Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run with --jobs=2 automatically on CI #924

Merged
merged 1 commit into from
Oct 11, 2023
Merged

Conversation

soutaro
Copy link
Owner

@soutaro soutaro commented Oct 4, 2023

Steep invokes worker processes to type check files as many as the processors. But, computers running CI jobs typically have large number of processors, while the resources each CI job can use is limited.

Assume the CI computer has 32 cores, but only two of them are available for CI job. Steep spawns ~32 worker processes based on the number of cores. The problem is it often hits the memory limit by starting too many workers.

This PR is to fix the problem by setting the default --jobs number to 2 (or # of cores if it's smaller) when:

  1. CI environment variable is defined, and
  2. --jobs option is not set

This can slow the type checking job during CI if more than two cores are available. In that case, specify --jobs option explicitly based on the number of the available cores.

Another possible implementation

This is kind of a workaround and calculating actually available number of processors would be a better solution for this problem. This is left for future work. 😃

@soutaro soutaro added this to the Steep 1.6 milestone Oct 4, 2023
@soutaro soutaro force-pushed the detect_ci_and_use_lower_jobs branch from 284cb22 to 064521e Compare October 4, 2023 11:56
@soutaro soutaro force-pushed the detect_ci_and_use_lower_jobs branch from 064521e to 4e9ca99 Compare October 4, 2023 11:57
@soutaro soutaro merged commit ab8cc3a into master Oct 11, 2023
22 checks passed
@soutaro soutaro deleted the detect_ci_and_use_lower_jobs branch October 11, 2023 05:44
@soutaro soutaro added the Released The PR is already included in a published release label Oct 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Released The PR is already included in a published release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant