Azure WorkerRole trigger on queue like WebJob

Question

I'm used to use webjob on Azure triggering an Azure queue. It works like a charm.

Azure tutorial webjob + queue

static void Main(string[] args)
{
    JobHost host = new JobHost();
    host.RunAndBlock();
}

public static void ProcessQueueMessage([QueueTrigger("logqueue")] string logMessage, TextWriter logger)
{
    logger.WriteLine(logMessage);
}

What's really good with queueTrigger is until the process triggered by a message isnot done, the message is keep invisible (not delete). So If you turn off the webjob (for webjob update for example) The message will be visible (after a little timeout) in the queue to be process by the updated webjob (perfect).

Now I wanna do the same thing but on a worker role. Today I do like this.

while (true)
{
     var cloudMessage = await sourceImportationQueue.GetMessageAsync();
     if (cloudMessage != null)
           sourceImportationQueue.DeleteMessage(cloudMessage);
      // process my job (few hours)
     else
           await Task.Delay(1000 * 5);
}

But if I stop the worker during is job, I lost the message. So how can I do like webJob triggering ?

Answer 1

Finally I find a simple solution. Before running my job of several hours, I launch a task KeepHiddenMessageAsync that update the message with the timeout. Before the end of the timeout a new update of the message is done. If an problem occur then the timeout of the message will be reached and the message will become visible.

        private bool jobIsComplete;

        private void Run()
        {
             while (true)
            {
                 jobIsComplete = false;
                 //get the message
                 var cloudMessage = await queue.GetMessageAsync();

                 if (cloudMessage != null)
                        //run the task to keep the message until end of the job and worker role stopping for an update for example 
                       var keepHiddenMessageTask = KeepHiddenMessageAsync(cloudMessage);

                        //
                        // process my job (few hours)
                        //

                      jobIsComplete = true;
                      await keepHiddenMessageTask;
                      await _queue.DeleteMessageAsync(cloudMessage);
                 else
                       await Task.Delay(1000 * 5);
            }
        }

        private async Task KeepHiddenMessageAsync(CloudQueueMessage iCloudQueueMessage)
        {
            while (true)
            {
                //Update message and hidding during 5 new minutes
                await _queue.UpdateMessageAsync(iCloudQueueMessage, TimeSpan.FromMinutes(5), MessageUpdateFields.Visibility);

                //Wait 4 minutes
                for (int i = 0; i < 60 * 4; i++)
                {
                    if (JobIsComplete)
                        return;
                    else
                        await Task.Delay(1000);
                }
            }
        }

Answer 2

By default once a queue message has been retrieved it becomes invisible for 5 minutes. After this delay, if the message has not been deleted from the queue it will become visible again so that it can be processed once again.

In your code sample you are deleting the message as soon as you get it from the queue. If you want to make it safe the message should only be deleted at the end of your process. Have you tried to move sourceImportationQueue.DeleteMessage(cloudMessage); at the end of the processing job?

Answer 3

It may not be possible to solve this issue without using some kind of persistent storage to track your job progress. As already identified you are deleting the message before your job begins so if the the job fails for any reason, including stopping the role, the message is lost. The maximum lock period for a message is 5 minutes meaning that the message will re-appear again while your job is still running and if the delete operation was moved to the end it would fail due to having lost the lock.

If your long running job is comprised of multiple smaller steps none of which exceed the 5 minute period then you would be able to call RenewLock() periodically to retain the lock on the message and stop it from re-appearing on the queue. As long as the lock never expires a DeleteMessage at the end would succeed in this case. It's probably unlikely that this fits your scenario though.

A possible solution would be to write the job status to, for example, an Azure table and record state throughout the processing of your job. Your worker role loop would be to check the table for any jobs not yet completed and continue any that exist and if none found check the service bus for any new jobs. This solution may also give you the chance to pick up failed jobs from the point they reached rather than starting your 2 hour job from the beginning again.