I have a springboot application deployed in PCF. I have a scheduler that runs every hour and picks up eligible jobs from a database to run. I need to implement some locking mechanism so that different instances don't pick up the same job.
So I introduced a table in database so the status of the job could be changed to running and multiple instance don't run the same job. So if there are 10 jobs submitted in db and have 10 instances of my app, each instance can pick up 1 job mark it as running and so on with other instances. However, I believe if all instances run scheduler at midnight, they all will read the job as not started and all of them will start executing the same job.
In a previous implementation I have restricted job execution to only the first PCF instance to avoid multiple executions of same job. That will not be an effective approach for this use case as that is not a scalable approach.
Any solutions recommended or another approach anyone else followed in a similar scenario? This is not yet fully implemented so don't have an actual example of this happening.