Retry with double memory [BA-5933] #5180

salonishah11 · 2019-09-16T22:56:59Z

This PR adds a config option for user defined retries. With memory-retry the user can specify an array of strings which when encountered in the stderr file by Cromwell, allows the task to be retried with a factor also mentioned in the config.

JIRA issue link.

kshakir

Excellent PoC. I'm happy to 👍 if the PoC needs to go out asap, but in general I think the concept of "DoubleMemory" could be replaced with a more generic "MoreMemory".

CHANGELOG.md

kshakir · 2019-09-16T23:28:35Z

...scala/cromwell/engine/workflow/lifecycle/execution/job/preparation/JobPreparationActor.scala

+
+    jobKey.memoryDoubleMultiplier match {
+      case 1 => runtimeAttributes
+      case multiplier: Int => doubleRuntimeMemory(multiplier)


ToL: I haven't been following the ticket too closely, so this is mostly thinking for future PRs. A hardcoded factor of 2.0 is fine for now, but I could see someone wanting eventually wanting a multiplier of 1.1 or even 10.0.

kshakir · 2019-09-16T23:30:16Z

backend/src/main/scala/cromwell/backend/backend.scala

@@ -26,7 +26,7 @@ import scala.util.Try
 /**
  * For uniquely identifying a job which has been or will be sent to the backend.
  */
-case class BackendJobDescriptorKey(call: CommandCallNode, index: Option[Int], attempt: Int) extends CallKey {
+case class BackendJobDescriptorKey(call: CommandCallNode, index: Option[Int], attempt: Int, memoryDoubleMultiplier: Int = 1) extends CallKey {


Perhaps call the field memoryMultiplier? While here, I'm also a fan of Double multipliers vs. Int.

kshakir · 2019-09-16T23:31:40Z

backend/src/main/scala/cromwell/backend/io/JobPaths.scala

@@ -32,6 +32,7 @@ trait JobPaths {

  def workflowPaths: WorkflowPaths
  def returnCodeFilename: String = "rc"
+  def doubleMemoryRetryRCFilename: String = "double_memory_retry_rc"


Assuming we remove the hardcoding of the multiplier, how about val memoryRetryRcFilename = "memory_retry_rc"?

Building on that idea, memory_retry_rc_filename would make it more clear it's a filename we're looking at and not a code.

kshakir · 2019-09-16T23:32:28Z

backend/src/main/scala/cromwell/backend/async/AsyncBackendJobExecutionActor.scala

@@ -99,8 +99,8 @@ trait AsyncBackendJobExecutionActor { this: Actor with ActorLogging with SlowJob
    case Finish(FailedNonRetryableExecutionHandle(throwable, returnCode)) =>
      completionPromise.success(JobFailedNonRetryableResponse(jobDescriptor.key, throwable, returnCode))
      context.stop(self)
-    case Finish(FailedRetryableExecutionHandle(throwable, returnCode)) =>
-      completionPromise.success(JobFailedRetryableResponse(jobDescriptor.key, throwable, returnCode))
+    case Finish(FailedRetryableExecutionHandle(throwable, returnCode, retryWithDoubleMemory)) =>


retryWithMoreMemory?

Perhaps another producty question for @ruchim. The JIRA ticket title says "more" but the description says "double".

kshakir · 2019-09-16T23:34:29Z

engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution/WorkflowExecutionActor.scala

    pushFailedCallMetadata(jobKey, returnCode, reason, retryableFailure = true)

-    val newJobKey = jobKey.copy(attempt = jobKey.attempt + 1)
+    val currentMemoryMultiplier = jobKey.memoryDoubleMultiplier
+    val newMemoryMultiplier = if (retryWithDoubleMemory) currentMemoryMultiplier * 2 else currentMemoryMultiplier


This magic number 2.0 could be another ~~knob~~ config/option.

...scala/cromwell/engine/workflow/lifecycle/execution/job/preparation/JobPreparationActor.scala

kshakir · 2019-09-16T23:48:12Z

src/ci/resources/papi_v2_application.conf

+        include "papi_provider_config.inc.conf"
+        genomics.compute-service-account = "centaur@broad-dsde-cromwell-dev.iam.gserviceaccount.com"
+        filesystems.http {}
+        retry-with-double-memory = ["OutOfMemoryError"]


ToL: Future PR..?

retry-memory { error-messages = ["OutOfMemoryError"] multiplier = 2.0 }

kshakir · 2019-09-16T23:50:39Z

...ala/cromwell/backend/google/pipelines/common/PipelinesApiAsyncBackendJobExecutionActor.scala

@@ -447,7 +447,8 @@ class PipelinesApiAsyncBackendJobExecutionActor(override val standardParams: Sta
          privateDockerKeyAndEncryptedToken = dockerKeyAndToken,
          womOutputRuntimeExtractor = jobDescriptor.workflowDescriptor.outputRuntimeExtractor,
          adjustedSizeDisks = adjustedSizeDisks,
-          virtualPrivateCloudConfiguration = jesAttributes.virtualPrivateCloudConfiguration
+          virtualPrivateCloudConfiguration = jesAttributes.virtualPrivateCloudConfiguration,
+          retryWithDoubleMemoryKeys = jesAttributes.retryWithDoubleMemoryKeys


Pro Tip: If you leave a trailing comma here and on the lines below, you can pay-it-forward so future devs won't get extra git blame for just adding a , to each list.

https://docs.scala-lang.org/sips/completed/trailing-commas.html

kshakir · 2019-09-16T23:55:53Z

.../v2alpha1/src/main/scala/cromwell/backend/google/pipelines/v2alpha1/api/ActionCommands.scala

@@ -141,6 +141,11 @@ object ActionCommands {
       |fi""".stripMargin
  }

+  def checkIfStderrContainsRetryKeys(retryLookupKeys: List[String]): String = {
+    val lookupKeysAsString = retryLookupKeys.mkString("|")
+    s"grep -E -q '$lookupKeysAsString' /cromwell_root/stderr ; echo $$? > /cromwell_root/double_memory_retry_rc"


ToL: Nice... nothing Papi here except rapid-prototyping.

salonishah11 · 2019-09-17T14:26:43Z

@kshakir I had thought of making the multiplier 2 as a config option. But since this ticket had evolved to be a PoC, I kept it constant at 2. If we decide to not rush this PR, than I am all in for 'MoreMemory' as compared to 'DoubleMemory'.

mcovarr

comments for now with a couple of Ruchi tags for clarification 😉

mcovarr · 2019-09-17T15:29:39Z

backend/src/main/scala/cromwell/backend/io/JobPaths.scala

@@ -32,6 +32,7 @@ trait JobPaths {

  def workflowPaths: WorkflowPaths
  def returnCodeFilename: String = "rc"
+  def doubleMemoryRetryRCFilename: String = "double_memory_retry_rc"


Building on that idea, memory_retry_rc_filename would make it more clear it's a filename we're looking at and not a code.

mcovarr · 2019-09-17T15:30:27Z

backend/src/main/scala/cromwell/backend/io/JobPaths.scala

@@ -72,6 +73,7 @@ trait JobPaths {
  lazy val script = callExecutionRoot.resolve(scriptFilename)
  lazy val dockerCid = callExecutionRoot.resolve(dockerCidFilename)
  lazy val returnCode = callExecutionRoot.resolve(returnCodeFilename)
+  lazy val doubleMemoryRetryRC = callExecutionRoot.resolve(doubleMemoryRetryRCFilename)


I assume if Khalid's comments above regarding "more" vs "double" are acted upon then this should change too.

mcovarr · 2019-09-17T15:55:51Z

...alpha1/src/test/scala/cromwell/backend/google/pipelines/v2alpha1/api/ActionBuilderSpec.scala

@@ -51,4 +54,42 @@ class ActionBuilderSpec extends FlatSpec with Matchers with TableDrivenPropertyC
    }
  }

+  def grepForRetryKeysCommand(lookupString: String) =
+    s"grep -E -q '$lookupString' /cromwell_root/stderr ; echo $$? > /cromwell_root/double_memory_retry_rc"


maybe this should be in one place in the production code with visibility such that the test code can see it

CHANGELOG.md

mcovarr · 2019-09-17T16:08:09Z

backend/src/main/scala/cromwell/backend/async/AsyncBackendJobExecutionActor.scala

@@ -99,8 +99,8 @@ trait AsyncBackendJobExecutionActor { this: Actor with ActorLogging with SlowJob
    case Finish(FailedNonRetryableExecutionHandle(throwable, returnCode)) =>
      completionPromise.success(JobFailedNonRetryableResponse(jobDescriptor.key, throwable, returnCode))
      context.stop(self)
-    case Finish(FailedRetryableExecutionHandle(throwable, returnCode)) =>
-      completionPromise.success(JobFailedRetryableResponse(jobDescriptor.key, throwable, returnCode))
+    case Finish(FailedRetryableExecutionHandle(throwable, returnCode, retryWithDoubleMemory)) =>


Perhaps another producty question for @ruchim. The JIRA ticket title says "more" but the description says "double".

mcovarr · 2019-09-17T16:13:27Z

backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala

+  /*
+    `CheckingForRetryWithDoubleMemory` action in Papiv2 exits with code 0 if the stderr file contains keys
+    mentioned in `retry-with-double-memory` config.
+   */


This is generic code, not specific to Papiv2.

mcovarr · 2019-09-17T16:28:06Z

.../cromwell/engine/workflow/lifecycle/execution/job/preparation/JobPreparationTestHelper.scala

+  mockJobKeyWithMemoryMultiplier4.node returns call
+  mockJobKeyWithMemoryMultiplier4.index returns None
+  mockJobKeyWithMemoryMultiplier4.attempt returns 3
+  mockJobKeyWithMemoryMultiplier4.memoryDoubleMultiplier returns 4


hmm a "double multiplier" with different values is a bit confusing

salonishah11 · 2019-09-19T18:49:31Z

@kshakir @mcovarr I have changed the multiplier factor to be configurable and made things more generic instead of ...doubleMemory.... It's now ready for re-review!

kshakir · 2019-09-19T21:41:42Z

...ain/scala/cromwell/backend/google/pipelines/common/PipelinesApiConfigurationAttributes.scala

@@ -49,13 +50,16 @@ object PipelinesApiConfigurationAttributes {

  final case class VirtualPrivateCloudConfiguration(name: String, subnetwork: Option[String], auth: GoogleAuthMode)
  final case class BatchRequestTimeoutConfiguration(readTimeoutMillis: Option[Int Refined Positive], connectTimeoutMillis: Option[Int Refined Positive])
+  final case class MemoryRetryConfiguration(errorKeys: List[String], multiplier: Double)


Should this be propagated as multiplier: Double Refined Positive, such that the multiplier type is always positive throughout the code?

Oh I somehow missed this comment. I will take a look. Thanks!

So the multiplier at this point, which is when Cromwell is started and Papi backend is initialized, is validated that it is a positive double, but not used. It is used in the StandardAsyncExecutionActor when a task has failed and the retryWithMoreMemory condition is true. I am not revalidating again because if it was not a valid double, Cromwell would have thrown an error when it was started and validated in the Papi backend. Does that make sense? I can remove the multiplier field from MemoryRetryConfiguration altogether to make it easy.

I can also revalidate that the multiplier is a positive double again at StandardAsyncExecutionActor, but if at this point it is not a valid double, we can switch the multiplier value to default (which is 2) or capture the error and fail the task without retrying.

After talking and coding f2f, the code has been updated to use Double Refined Positive type for multiplier all the way through.

kshakir

Sorry if I missed the reply to the open question above? If you point me at the reply and hit ♻️ and I'll re-review.

backend/src/main/scala/cromwell/backend/async/KnownJobFailureException.scala

docs/backends/Google.md

mcovarr · 2019-09-23T20:55:31Z

...ala/cromwell/backend/google/pipelines/common/PipelinesApiAsyncBackendJobExecutionActor.scala

@@ -141,6 +143,8 @@ class PipelinesApiAsyncBackendJobExecutionActor(override val standardParams: Sta
    case _ => false
  }

+  override lazy val memoryRetryFactor: Option[Double Refined Positive] = initializationData.papiConfiguration.papiAttributes.memoryRetryConfiguration.map(_.multiplier)


TOL hmm really this should be > 1.0

In strict sense, yes. But I am not sure if there is any harm to have a memory factor > 0 but less than < 1. Something to discuss during tech talk?

I'm fine with a memoryRetryFactor being GreaterEqual[] vs. Positive.

Also ToL for tech talk: I'm curious if this would be easier to follow, though it's about as verbose when used.

type MemoryRetryFactor = GreaterEqual[W.`1.0`.T]

Or is convention to alias the refined type?

type MemoryRetryFactor = Refined[Double, GreaterEqual[W.`1.0`.T]]

Or some mix of the two?

...nes/v1alpha2/src/main/scala/cromwell/backend/google/pipelines/v1alpha2/GenomicsFactory.scala

dinvlad · 2019-09-25T15:04:05Z

docs/backends/Google.md

+  }
+}
+```  
+this tells Cromwell to retry the task with 1.1x memory when it sees either `OutOfMemoryError` or `Killed` in the `stderr` file. If the task has 


One thing to be careful about is Killed might also mean preempted. So it may be better to check explicitly that the call wasn't preempted, and increase memory only when it wasn't.

That is certainly a good point. Since this PR has been developed as a POC, we are trying to keep it simple. But I will note this down for when we start adding more features to it. Thank you!

dinvlad · 2019-11-10T20:23:44Z

Unfortunately, this does not yet work in practice. I opened https://broadworkbench.atlassian.net/browse/BA-6112 re the cause.

kshakir reviewed Sep 16, 2019

View reviewed changes

gemmalam requested a review from mcovarr September 17, 2019 14:34

mcovarr reviewed Sep 17, 2019

View reviewed changes

salonishah11 requested review from kshakir and mcovarr September 19, 2019 18:49

kshakir reviewed Sep 19, 2019

View reviewed changes

gemmalam requested a review from kshakir September 20, 2019 14:34

kshakir reviewed Sep 20, 2019

View reviewed changes

salonishah11 requested a review from kshakir September 20, 2019 19:30

mcovarr approved these changes Sep 23, 2019

View reviewed changes

salonishah11 force-pushed the ss_double_memory branch from f7c3172 to 15dfb22 Compare September 23, 2019 21:52

salonishah11 requested a review from mcovarr September 24, 2019 19:18

mcovarr approved these changes Sep 24, 2019

View reviewed changes

kshakir approved these changes Sep 25, 2019

View reviewed changes

salonishah11 force-pushed the ss_double_memory branch from cba7284 to 86d3028 Compare September 25, 2019 15:34

dinvlad reviewed Sep 25, 2019

View reviewed changes

salonishah11 added 12 commits September 26, 2019 21:17

wip

6b26ee7

this works

95ce0de

with retryWithDoubleMemory as runtime attribute

36a410d

revert oops

8c9ce4f

progress here and there

6a3ca56

[force ci]

482ea9a

some tests

6ad254a

try this test [force ci]

f350c96

docs

e2abb49

oops [force ci]

bacdab6

changelog

62de623

clean up

832e840

salonishah11 added 12 commits September 26, 2019 21:17

oops 2

7841122

and 1 more

019e9b8

fixing tests

aacfd8b

fixing tests 2

f5a04af

PR feedback

f6cf77a

argh

d212fd6

refined positive all the way

fe3a484

refined double

c8f4f27

greater than 1

4513de5

doc

2d424c5

oops

d18084f

fix tests

fdaf311

salonishah11 force-pushed the ss_double_memory branch from 86d3028 to fdaf311 Compare September 27, 2019 01:19

gemmalam mentioned this pull request Sep 27, 2019

User-defined retries #1991

Open

salonishah11 merged commit e956df7 into develop Sep 27, 2019

salonishah11 deleted the ss_double_memory branch September 27, 2019 14:07

salonishah11 added a commit that referenced this pull request Sep 27, 2019

Retry with more memory [BA-5933] (#5180)

ef8d5bd

ParkvilleData mentioned this pull request Nov 4, 2020

Cromwell call caching and scatter #5966

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retry with double memory [BA-5933] #5180

Retry with double memory [BA-5933] #5180

salonishah11 commented Sep 16, 2019 •

edited

Loading

kshakir left a comment

kshakir Sep 16, 2019

kshakir Sep 16, 2019

kshakir Sep 16, 2019

mcovarr Sep 17, 2019

kshakir Sep 16, 2019

mcovarr Sep 17, 2019

kshakir Sep 16, 2019

kshakir Sep 16, 2019

kshakir Sep 16, 2019

kshakir Sep 16, 2019

salonishah11 commented Sep 17, 2019

mcovarr left a comment

mcovarr Sep 17, 2019

mcovarr Sep 17, 2019

mcovarr Sep 17, 2019

mcovarr Sep 17, 2019

mcovarr Sep 17, 2019

mcovarr Sep 17, 2019

salonishah11 commented Sep 19, 2019

kshakir Sep 19, 2019

salonishah11 Sep 20, 2019 •

edited

Loading

salonishah11 Sep 20, 2019

salonishah11 Sep 20, 2019

salonishah11 Sep 23, 2019

kshakir left a comment

mcovarr Sep 23, 2019

salonishah11 Sep 23, 2019

kshakir Sep 24, 2019

dinvlad Sep 25, 2019

salonishah11 Sep 27, 2019

dinvlad commented Nov 10, 2019

Retry with double memory [BA-5933] #5180

Retry with double memory [BA-5933] #5180

Conversation

salonishah11 commented Sep 16, 2019 • edited Loading

kshakir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salonishah11 commented Sep 17, 2019

mcovarr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salonishah11 commented Sep 19, 2019

Choose a reason for hiding this comment

salonishah11 Sep 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kshakir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dinvlad commented Nov 10, 2019

salonishah11 commented Sep 16, 2019 •

edited

Loading

salonishah11 Sep 20, 2019 •

edited

Loading