Do not ever want to use an "e" machine on GCP

[GregoryDougherty]

We are running a wld pipeline on GCP. Requested 16 CPSs. we have
runtime {
cpuPlatform: "Intel Cascade Lake"
}
set in the .conf file

Nevertheless, the task was given an e2-standard-16 machine, and took 9 hours to turn a bam into a cram, and another hour to upload the 51 gb ram file

I ran the code again, only change was I used an n2-standard-16
The entire task, including the upload, finished in less than 30 minutes

How do we make it so we never, ever, get an "e" machine? I downloaded the cromwell code and looked through it, trying to figure out where it is you pick the machine type on GCP, but never found anything that looked like it would pick an e machine. What am I missing?

The cromwell that assigned the e2-standard-16 was cromwell-87.jar, if that matters

Thank you

[dspeck1]

This is with the GCP Batch backend correct? Machine Type is a parameter in Batch, but not a parameter in Cromwell. If a machine type is not defined Batch selects the machine type based on the CPU and Memory request. Setting cpuPlatform is the way to not get an e series machine. Did you get an e series machine with "Intel Cascade Lake" in the configuration?

[GregoryDougherty]

Assuming that putting that in the runtime {cpuPlatform: "Intel Cascade Lake"} setting in the Cromwell conf file counts as "in the configuration", then yes.

If we have to hand edit every single one of our WDL task files to put that in a runtime block, not I haven't tried that

[GregoryDougherty]

$ gustil cat gs://bucket/path/to/run.stdout | grep standard | grep machine | sort | uniq -c
1601 machine_type: "e2-standard-2"

No happiness there

[dspeck1]

Thanks. Setting cpuPlatform in the runtime attributes is the only way to avoid scheduling to an e series machines through Cromwell. GCP Batch is limited to setting cpuPlatform or instance type. There is no preferred machine family type setting in GCP Batch. With that said the e series machine should not be that slow. Performance is supposed to be comparable to N1. If it repeatable open a support case with GCP.

[GregoryDougherty]

"The E2 machine series also contains shared-core machine types that use context- switching to time-share a physical core between vCPUs for multitasking"

Essentially, they are garbage machines that give you 1/2 the CPUs you ask for, and have horrid I/O. IMO, no one should ever be given one, unless they've explicitly asked for it. Especially in a bioinformatics environment, where you're going to be reading and writing large files on a regular basis.

Where in the code is the E2 default set? That's the part I was unable to figure out. If I could have that, I can fix it, put in a PR, and make our own version that doesn't require us to rewrite all our task files.

Thank you

[aednichols]

From the GCP docs, it seemed like Cascade Lake wasn't among the CPU platforms used for E2s. So I would think the cpuPlatform should accomplish the goal if it works as intended.

Intel Skylake, Broadwell, and Haswell, AMD EPYC Rome and EPYC Milan

https://cloud.google.com/compute/docs/machine-resource#machine_type_comparison

[aednichols]

@GregoryDougherty can you try a different CPU platform that similarly excludes E2 based on the matrix linked above?

[GregoryDougherty]

I did, no change.

I put "Intel Cascade Lake" in cpuPlatform in the .wdl code's runtime block. So it gave me an e2-highcpu-16 with an AMD EPYC chip.
Which was moderately faster, but still not nearly as good as an n2-standard-16

So, my question remains: where in the code is teh machine-type actually chosen for GCP Batch?

There has to be something that is saying "yes, use e2". Where is that? What is the code that makes e2 a possible option, rather than just starting at n2 and going up from there?

because our other choice is to always demand at least 40 CPUs, since that gets us out of e territory. But that's a really sub-optimal solution

[mcovarr]

Fixes for this will be available in the next Cromwell release, no ETA yet. If you need the fixes immediately and are comfortable building from the develop branch, that is also an option.

[GregoryDougherty]

Thank you