I am implementing a FIFOSQLiteQueue from the module persistqueue in python.
I have two threads: one gathering data and adding it to the queue and one dequeing and processing the data in the queue.
However I want my dequeue function to only remove the item from the queue after processing instead of before processing to prevent data loss if the progress crashes during processing the data.
Can anyone help me with this?
this is the code I currently use for creating the queue, enqueuing, and dequeueing
from persistqueue import FIFOSQLiteQueue
from sys import path
max_queue_size = 3
current_working_dir = f"{path[0]}"
queue_path = f"{current_working_dir}/queue"
queue = FIFOSQLiteQueue(queue_path, multithreading=True, auto_commit=False)
def dequeue():
if queue.qsize() == 0:
return(503, f"Queue is empty")
params = queue.get()
queue.task_done()
return(200, params)
def enqueue(params):
if queue.qsize() >= max_queue_size:
return(503, f"Queue is full")
queue.put(params)
queue.task_done()
return(200, "Enqueued successfully")