Monitor a scan's progress

[cco3]

Is there any way to do this with a large image?

[tdruez]

At the moment, you could run a Pipeline directly, bypassing the task system, to get the progress log displayed in the terminal:

$ cd scancode.io
$ source bin/activate
$ scanpipe create-project PROJECT_NAME --input docker_image.tar
$ DJANGO_SETTINGS_MODULE='scancodeio.settings' bin/python scanpipe/pipelines/docker.py run --project PROJECT_NAME

This will display Pipeline steps live execution:

2020-12-09 04:49:01.279 Workflow starting (run-id 1607489341272759):
2020-12-09 04:49:01.284 [1607489341272759/start/1 (pid 19637)] Task is starting.
2020-12-09 04:49:02.569 [1607489341272759/start/1 (pid 19637)] Task finished successfully.
2020-12-09 04:49:02.574 [1607489341272759/extract_images/2 (pid 19644)] Task is starting.
2020-12-09 04:49:03.799 [1607489341272759/extract_images/2 (pid 19644)] Task finished successfully.
2020-12-09 04:49:03.805 [1607489341272759/extract_layers/3 (pid 19649)] Task is starting.
2020-12-09 04:49:05.016 [1607489341272759/extract_layers/3 (pid 19649)] Task finished successfully.
[...]

I will improve the task system to provide easy access to progress information about Pipeline execution.

[cco3]

Thanks!

[tdruez]

@cco3 a new status management command is also available to display a high level status of a Project.
It can be useful to monitor the database objects addition:

$ scanpipe status --project PROJECT_NAME

[tdruez]

The monitoring of Pipeline execution was improved and is available in the command line, the UI, and the REST API.

Command line

$ scanpipe status --project PROJECT_NAME

Project: MyProject
Create date: Feb 16 2021 06:08
Work directory: /var/projects/MyProject-40599791

Database:
 - CodebaseResource: 403
   - (no status): 398
   - ignored-not-interesting: 4
   - scanned: 1
 - DiscoveredPackage: 0
 - ProjectError: 1

Pipelines:
 [RUNNING] docker
   2021-02-16 06:08:06.95 Pipeline [docker] starting
   2021-02-16 06:08:06.95 Step [extract_images] starting
   2021-02-16 06:08:06.97 Step [extract_images] completed in 0.02 seconds
   2021-02-16 06:08:06.97 Step [extract_layers] starting
   2021-02-16 06:08:07.13 Step [extract_layers] completed in 0.16 seconds
   2021-02-16 06:08:07.14 Step [find_images_linux_distro] starting
   2021-02-16 06:08:07.14 Step [find_images_linux_distro] completed in 0.00 seconds
   2021-02-16 06:08:07.14 Step [collect_images_information] starting
[...]

UI

From the Project details view, click on the "Running" pipeline entry in the Pipeline section:

This will open a modal with the current progress:

REST API

GET /api/projects/<PROJECT_UUID>/
The progress values are available in the runs field.