[MehmedGIT]

Seems there are a few race conditions in different places due to the fact that the MongoDB is used as source of truth and there are multiple agents (processing server, processing workers, etc.) changing database entries asynchronously. Hence, the processing server needs its separate cache for locked pages and processing requests (coming from the run_workflow method). The cache removes some race conditions, but other race conditions based on processing/workflow job states that are modified by different agents are still there.

One of the many race conditions I faced is in the run_workflow:

1. The processing server is submitting processing requests to the processing workers without blocking them to start processing as the job submission continues.
2. The first time the processing server pushes a request to the RabbitMQ a Mets server is started for the workspace used in the workflow request.
3. While still submitting the jobs, if the processing workers finish the submitted jobs faster than the processing server can push, the processing server cache with requests is empty when the processing callbacks arrive from the processing workers. Since the cache is empty the Mets server is shut down.
4. Any other requests coming after the Mets server shut down are in-processable.

The example above is now fixed by simply adding extra flags to the workflow jobs to track whether their submission is finished or not. There are also other flags for the processing job database entries to prevent any edge cases for the locking/unlocking page_ids for the workspace.

The race conditions happen only when specific timing of the job submission and job processing are met. Which makes things even harder to detect without any locking mechanisms. Also keep in mind, that since the processing server also provides a way to chain processing jobs using the processing endpoint (and not run_workflow) without having a specific workflow complicates things further. There are a few places where both endpoints need to be considered when using the similar methods to push/cache/queue jobs.

I will first try to resolve as many potential race conditions as I can. That may include more caching on the processing server side, by also utilizing locking mechanisms here and there. However, using simply the database as a source of truth is not a good architectural solution overall. No matter what mechanisms I implement (and I have tried some more than what I committed here), there is always some kind of a race condition.

Another point is, even if the processing server is in a state to restart after a failure, there will be potential missing processing callbacks missed from the processing workers in the period when the processing server was trying to recover. On top, it is hard to track what requests are still inside the RabbitMQ after the processing server recovers. The processing server can keep track of what was pushed, and assume these records are in the RabbitMQ, but it is always uncertain how much was processed while the processing server was gone.

I am open to suggestions if you have any regarding the processing server and overall recovery. However, it seems nearly impossible to recover from a processing server failure. The better strategy would be to make sure the processing server is robust as it can be.

---

What does this PR add:

• A workflow callback mechanism flag to the ocrd network client workflow command
• An actual implementation of the workflow callback mechanism inside the Processing Server
• A test endpoint on the Processing Server: /user_workflow_callback_test
• New variables inside DBWorkflowJob for better tracking of job status and preventing duplicate actions:

submission_complete: bool 
mets_server_stopped: bool
callback_sent: bool

• New variables inside DBProcessorJob for better tracking of job status and preventing duplicate actions:

result_callback_processed: bool
result_callback_in_progress: bool
result_callback_counter_decremented: bool

• New database methods for DBProcessorJob to prevent race conditions as much as possible by using the newly introduced variables in the database model
• Locking mechanism when doing workspace related actions inside the Processing Server to prevent race conditions leading to:

1. premature shut down of the Mets server
2. the Mets server never being shut down under certain circumstance

• Much better logic when deciding when to shut down the mets server
• Some refactoring inside the Processing Server