-
Notifications
You must be signed in to change notification settings - Fork 358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WX-927 GCP Batch: LogsPolicy is now configurable #7491
Conversation
- When the "logs-policy" config entry is missing, "CLOUD_LOGGING" is set which is the default policy from Batch. - When the "logs-policy" is set to "PATH", a "task.log" file is stored within the file system under the task files, this is later pushed to Google Cloud Storage. This is an exampel where "task.log" can be found: - gs://project-id/cromwell-execution-root/workflow-name/workflow-id/call-myTask/task.log (workflow-id would be a UUID).
Do the task logs really only get copied to the storage bucket after the job completes? Cloud Life Sciences asynchronously copies task logs to GCS periodically while the job is running. This is the main utility of the task logs in that they allow users to see how jobs are progressing before they terminate. |
Unfortunately, yes. In theory, Google can push these logs directly to the Cloud Storage but we were told that the docs are wrong and that feature does not work (see https://cloud.google.com/php/docs/reference/cloud-batch/latest/V1.LogsPolicy).
|
Logs are seen as job progresses with the |
Thank you @dspeck1, I was able to see cloud task logs for my "Hello world!" workflow in realtime when filtering for In production many Cromwell users scatter thousands of jobs simultaneously per project. Would it be possible to have these cloud logs labeled with workflow id / root workflow id / task / shard / attempt so users can search for logs specific to a Cromwell job? I see there are some GCP Batch job ID-based labels but I don't know how to associate these to jobs in Cromwell / WDL terms. |
We add the workflow ID as a label. Are task and shared generated by Cromwell and would they be available from the parameters or somewhere else? |
Interesting, I'm running a local build from current In For the proposed additional labels, with respect to
|
Oh those labels are not propagated from Batch to Logging. Less than ideal, but you can filter with gcloud. Example below.
Then do a describe on the job. I will ask Google if there is something to propagate additional labels to logs. |
Thank you! This would be a big help in making the GCP Batch backend user-friendly at scale. |
Description
LogsPolicy is now configurable
This is an exampel where "task.log" can be found:
gs://project-id/cromwell-execution-root/workflow-name/workflow-id/call-myTask/task.log
(workflow-id would be a UUID).Release Notes Confirmation
CHANGELOG.md
CHANGELOG.md
in this PRCHANGELOG.md
because it doesn't impact community usersTerra Release Notes