-
Notifications
You must be signed in to change notification settings - Fork 358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WX-1675] Record cloud quota delay to GroupMetrics table #7501
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
New actor whose purpose is to receive messages:
- record which group has run into quota exhaustion
- get message asking about if this group is quota exhausted or not (this will be implemented in follow up PR)
* Checks if the job has run into any cloud quota exhaustion and records it to GroupMetrics table | ||
* @param runStatus The run status | ||
*/ | ||
def checkAndRecordQuotaExhaustion(runStatus: StandardAsyncRunState): Unit = () |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Backend agnostic method and each backend can implement how to detect quota exhaustion. Currently only implemented in PipelinesApiAsyncBackendJobExecutionActor.scala
lazy val groupMetricsActor: ActorRef = | ||
context.actorOf(GroupMetricsActor.props(EngineServicesStore.engineDatabaseInterface)) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is where the singleton GroupMetricsActor is created and wired through other actors to make its way through to backend actor.
getm-manifest.json
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure how this was created, will delete it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this branch would be a good mob-time demo. I'm curious to see it in action, but I also don't want to deal with migrating/resetting my local Cromwell DB.
Have you thought about where you'd like to place the conversion from "quota timestamp is recent" to Boolean
"quota is exhausted"?
I was thinking of quotaExhaustionForGroupId
in the component, but maybe that's too low level and it should wait for the next PR.
case 0 => dataAccess.groupMetricsEntryIdsAutoInc += groupMetricsEntry | ||
case _ => assertUpdateCount("recordGroupMetricsEntry", updateCount, 1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hadn't seen this pattern before but I see that it's common in our DB. TIL.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you had a different pattern in mind to achieve this upsert let me know and I am happy to look into it!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That was not a leading question, it seems reasonable and I don't know of anything better.
@aednichols I was thinking probably in the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently this is only a simple test file that checks the actor receives messages when sent and calls appropriate database method. But in the follow up PR where GroupMetricsActor
will receive another message related to figuring out whether group is quota exhausted or not, I am expecting that this test file will get more useful tests then.
|
||
override def receive: Receive = { | ||
case RecordGroupQuotaExhaustion(group) => | ||
log.info(s"Recording quota exhaustion for group '$group'.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From discussion - remove this log line
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM once that log line is removed. Thanks for the walkthrough!
…dinstitute/cromwell into sps_record_quota_exhaustion merge origin to local
Jira: https://broadworkbench.atlassian.net/browse/WX-1675
Description
GroupMetrics
table when a job runs intoAwaitingCloudQuota
stateGroupMetricsActor
which takes in currentlyRecordGroupQuotaExhaustion
message to record the group that ran into quota exhaustion into the new tableExample screenshot:
![Screenshot 2024-08-20 at 4 27 49 PM](https://private-user-images.githubusercontent.com/16748522/359636857-fe403e65-d7f2-41b3-8279-38a05b3cbb6b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNTAxNTUsIm5iZiI6MTcyNjE0OTg1NSwicGF0aCI6Ii8xNjc0ODUyMi8zNTk2MzY4NTctZmU0MDNlNjUtZDdmMi00MWIzLTgyNzktMzhhMDViM2NiYjZiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDE0MDQxNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM5OGE5ODUxY2Q4MzA3YTYxMzVmMDgyY2JkYjFmOGI1MmM4OGVkM2I2OTZlNjQ5MDczMDQ2NmVjM2JhNTVkMWYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.N396M_EWBYANJtAVnL47kKx1ookNcCATrDKLMjnXlGY)
Release Notes Confirmation
CHANGELOG.md
CHANGELOG.md
in this PRCHANGELOG.md
because it doesn't impact community usersTerra Release Notes