Workflow endpoint of the Processing Server

[joschrew]

Endpoint to run a workflow with the processing-server

Usage described here

[kba]

We really need more specific exceptions in ocrd/ocrd/task_sequence.py...

[MehmedGIT]

There is a need of request revocation method to remove stuck requests from the internal queue of a workspace. This situation happens when a processing job fails, and there are other jobs depending on the failed job. A simple solution would be to cancel all jobs that depend on the failed job. Then further cancel all jobs that are depending on the cancelled jobs.

EDIT: This is implemented.

[MehmedGIT]

Things to be considered yet (maybe not in #1083 directly since WebAPI docs adaptations would be required too):

1. Ease the user for checking job statuses, i.e., currently the processing server supports GET /processor/{processor_name}/{job_id} for checking the status of the processing job. However, this is inconvenient for the general user. The user should be able to check the processing step job status without knowing the name of the processor. The job_id is a globally unique id.
   Suggestion: GET /processor/{processor_name}/{job_id} -> GET /processor/job_status/{job_id}
2. Since there is no way to check the workflow job status, a separate endpoint is needed. In a similar manner to checking the processing job status. There is already a DB and Response model for the Workflow.
   Suggestion: provide an endpoint GET /workflow/job_status/{job_id}
3. Find an optimal way to update the job status of a workflow based on the job statuses of the building processing steps.

[MehmedGIT]

For the page-wise processing, I was also considering a way to reduce the amount of total requests. With a workflow that utilizes 9 processors and a workspace of 100 pages that makes 900 requests per workflow... We could also provide something like page_step. Instead of creating a one-page request, the page_step amount of pages per request will be created. This adds more complexity for testing but is also more flexible and convenient for the user. For now, I would just keep it as an idea - till we have everything tested and working.

EDIT: Feature postponed since it will bring more complexity and further increase the required timeout value for the METS Server.

Adding it as an extra note so it does not disappear somewhere above.

[bertsky]

Things to be considered yet (maybe not in #1083 directly since WebAPI docs adaptations would be required too):

1. Ease the user for checking job statuses, i.e., currently the processing server supports GET /processor/{processor_name}/{job_id} for checking the status of the processing job. However, this is inconvenient for the general user. The user should be able to check the processing step job status without knowing the name of the processor. The job_id is a globally unique id.
   Suggestion: GET /processor/{processor_name}/{job_id} -> GET /processor/job_status/{job_id}

Good idea.

Note: the current spec still says we should use GET /processor/job/{job_id} for this. So either core or spec needs to be updated anyway.

2. Since there is no way to check the workflow job status, a separate endpoint is needed. In a similar manner to checking the processing job status. There is already a DB and Response model for the Workflow.
   Suggestion: provide an endpoint GET /workflow/job_status/{job_id}

Also 👍

But @joschrew already implemented e6ce39e as GET /workflow/{job_id} now.

IMO this is ambiguous: as of the spec, this looks like querying the workflow definition.

The current spec still says we should use GET /workflow/job/{job_id} for this.

Both processing step and workflow status endpoints are also explained in the human-readable readme in these terms.

3. Find an optimal way to update the job status of a workflow based on the job statuses of the building processing steps.

IIUC @joschrew already found a way of doing that in the aforementioned commit. That solves it, right?

[bertsky]

We still need a network client CLI implementation for the new workflow endpoint. Should we track this in a new issue?

[MehmedGIT]

Anything you see in the spec should be up-to-date and matching with core v2.67.2. If that is not the case, let me know.

Regarding the network client CLI - that idea was left behind at some point and needs to be implemented. A separate issue makes sense.

[bertsky]

Anything you see in the spec should be up-to-date and matching with core v2.67.2. If that is not the case, let me know.

oops, sry, I did not look at master – everything is in order between spec and core, as you stated.

Regarding the network client CLI - that idea was left behind at some point and needs to be implemented. A separate issue makes sense.

#1264