-
Notifications
You must be signed in to change notification settings - Fork 358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Do not ever want to use an "e" machine on GCP #7474
Comments
This is with the GCP Batch backend correct? Machine Type is a parameter in Batch, but not a parameter in Cromwell. If a machine type is not defined Batch selects the machine type based on the CPU and Memory request. Setting |
Assuming that putting that in the runtime {cpuPlatform: "Intel Cascade Lake"} setting in the Cromwell conf file counts as "in the configuration", then yes. If we have to hand edit every single one of our WDL task files to put that in a runtime block, not I haven't tried that |
$ gustil cat gs://bucket/path/to/run.stdout | grep standard | grep machine | sort | uniq -c No happiness there |
Thanks. Setting |
"The E2 machine series also contains shared-core machine types that use context- switching to time-share a physical core between vCPUs for multitasking" Essentially, they are garbage machines that give you 1/2 the CPUs you ask for, and have horrid I/O. IMO, no one should ever be given one, unless they've explicitly asked for it. Especially in a bioinformatics environment, where you're going to be reading and writing large files on a regular basis. Where in the code is the E2 default set? That's the part I was unable to figure out. If I could have that, I can fix it, put in a PR, and make our own version that doesn't require us to rewrite all our task files. Thank you |
From the GCP docs, it seemed like Cascade Lake wasn't among the CPU platforms used for E2s. So I would think the
https://cloud.google.com/compute/docs/machine-resource#machine_type_comparison |
@GregoryDougherty can you try a different CPU platform that similarly excludes E2 based on the matrix linked above? |
I did, no change. I put "Intel Cascade Lake" in cpuPlatform in the .wdl code's runtime block. So it gave me an e2-highcpu-16 with an AMD EPYC chip. So, my question remains: where in the code is teh machine-type actually chosen for GCP Batch? There has to be something that is saying "yes, use e2". Where is that? What is the code that makes e2 a possible option, rather than just starting at n2 and going up from there? because our other choice is to always demand at least 40 CPUs, since that gets us out of e territory. But that's a really sub-optimal solution |
Fixes for this will be available in the next Cromwell release, no ETA yet. If you need the fixes immediately and are comfortable building from the |
Thank you |
We are running a wld pipeline on GCP. Requested 16 CPSs. we have
runtime {
cpuPlatform: "Intel Cascade Lake"
}
set in the .conf file
Nevertheless, the task was given an e2-standard-16 machine, and took 9 hours to turn a bam into a cram, and another hour to upload the 51 gb ram file
I ran the code again, only change was I used an n2-standard-16
The entire task, including the upload, finished in less than 30 minutes
How do we make it so we never, ever, get an "e" machine? I downloaded the cromwell code and looked through it, trying to figure out where it is you pick the machine type on GCP, but never found anything that looked like it would pick an e machine. What am I missing?
The cromwell that assigned the e2-standard-16 was cromwell-87.jar, if that matters
Thank you
The text was updated successfully, but these errors were encountered: