Add additional info to alerts | Batch DB writes #489

aquarat · 2023-10-18T15:06:51Z

This PR is currently being tested live.

This PR does two main things:

Batches DB writes to improve performance/reduce DB load
Includes extra info in OpsGenie alerts related to dead gateways (how many out of how many gateways are dead and whether those gateways are AWS/GCP). This will make it easier to figure out if a dead gateway is a big deal or not.
Updates the hard-coded chains :(

Closes https://github.com/api3dao/tasks/issues/219
Closes https://github.com/api3dao/tasks/issues/199

This reverts commit f2354f2.

aquarat · 2023-10-19T12:12:20Z

src/alerting.ts

 export let prismaActive = prisma;

 export const setPrisma = (newPrisma: any) => {
  prismaActive = newPrisma;
 };

+type DbRecord = {
+  model: 'deviationValue' | 'rPCFailures' | 'compoundValues' | 'dataFeedApiValue' | 'gatewayFailures';


I don't know if there's a better way of typing this or handling this.

I can't think of anything that would be more elegant than this either (reminds me of the old telemetry 😄)

aquarat · 2023-10-24T13:26:46Z

src/alerting.ts

+
+const addRecord = (record: DbRecord) => {
+  // eslint-disable-next-line functional/immutable-data
+  recordsToInsert.push(record);


We batch pending DB inserts now, which speeds things up and reduces DB load.

aquarat · 2023-10-24T13:27:02Z

src/alerting.ts

+    await Promise.allSettled(
+      Object.entries(bufferedRecords).map(([model, data]) =>
+        // @ts-ignore
+        prismaActive[model].createMany({ data: data.map((item) => item.record) })


Not sure if this can be done better...

I like this solution 😄, anything else would be a much longer utility function

the createMany's are quite fast so this could even be made sequential if we find DB load to bee still too much later on 🤷

aquarat · 2023-10-24T13:27:34Z

src/alerting.ts

+  ).filter((result) => result.status === 'rejected');
+
+  if (results.length > 0) {
+    logger.error(`DB Writer error: ${results}`);


If we drop a sample it's not a big deal and if we drop everything, well, bigger issue that will be obvious.

aquarat · 2023-10-24T13:28:36Z

src/alerting.ts

  if (!nodaryResponse.success) {
-    await limitedSendToOpsGenieLowLevel(
+    limitedSendToOpsGenieLowLevel(


Sometimes we are limited by OG's rate limiting, in those case we don't want it to slow down the entire app. This was causing samples to be dropped.

aquarat · 2023-10-24T13:29:25Z

src/alerting.ts

@@ -189,6 +231,19 @@ export const recordRpcProviderResponseSuccess = async (contract: Api3ServerV1, s
  }
 };

+export const getBaseUrl = (fullUrl: string) => {


We use the base URL for (1) finding gateways and (2) hashing the gateway URL so it can be looked up later by Mertcan as part of his fault tracing.

aquarat · 2023-10-24T13:30:53Z

src/alerting.ts

      {
        message: `Dead gateway for Airnode Address ${airnodeAddress}`,
        priority: 'P3',
-        alias: `dead-gateway-${airnodeAddress}${generateOpsGenieAlias(gatewayUrl)}`,
+        alias: `dead-gateway-${airnodeAddress}${generateOpsGenieAlias(baseUrl)}`,


We're only interested in the base URL, not the URL+template ID (which is what we used before).

aquarat · 2023-10-24T13:32:11Z

src/constants.ts

@@ -46,5 +46,5 @@ export const DIRECT_GATEWAY_MIN_TIME_DEFAULT_MS = 20;
 export const DIRECT_GATEWAY_MAX_CONCURRENCY_DEFAULT = 10;

 // TODO: load these 2 from env var instead
-export const DATAFEED_READ_BATCH_SIZE = 100;
-export const DATAFEED_UPDATE_BATCH_SIZE = 10;
+export const DATAFEED_READ_BATCH_SIZE = 1;


Reading more than 1 has been unreliable on Kava - and I'm tired of troubleshooting this app - if you can see why it's failing, LMK.

wow! This would be a problem for the main airseeker as well, right?

aquarat · 2023-10-24T13:32:46Z

src/fetch-beacon-data.ts

@@ -92,7 +78,7 @@ export const fetchBeaconData = async (beaconId: string) => {
  }

  const goRes = await go(fetchFn, {
-    retries: 2,
+    retries: 0,


We don't need/want it to retry because if it fails we'll try the whole thing again very soon.

aquarat · 2023-10-24T13:33:10Z

src/main.ts

@@ -16,6 +16,10 @@ export const handleStopSignal = (signal: string) => {
  expireLimiterJobs();
  updateState((state) => ({ ...state, stopSignalReceived: true }));
  heartbeatReporter(getState().config);
+


Very annoying when the app doesn't quit quickly in response to ctrl+c. This fixes that.

vponline

LGTM, I didn't see any issues 🚀

vponline · 2023-10-25T09:35:55Z

src/alerting.ts

+    await Promise.allSettled(
+      Object.entries(bufferedRecords).map(([model, data]) =>
+        // @ts-ignore
+        prismaActive[model].createMany({ data: data.map((item) => item.record) })


I like this solution 😄, anything else would be a much longer utility function

vponline · 2023-10-25T09:37:26Z

src/alerting.ts

+    await Promise.allSettled(
+      Object.entries(bufferedRecords).map(([model, data]) =>
+        // @ts-ignore
+        prismaActive[model].createMany({ data: data.map((item) => item.record) })


the createMany's are quite fast so this could even be made sequential if we find DB load to bee still too much later on 🤷

vponline · 2023-10-25T09:41:46Z

src/alerting.ts

 export let prismaActive = prisma;

 export const setPrisma = (newPrisma: any) => {
  prismaActive = newPrisma;
 };

+type DbRecord = {
+  model: 'deviationValue' | 'rPCFailures' | 'compoundValues' | 'dataFeedApiValue' | 'gatewayFailures';


I can't think of anything that would be more elegant than this either (reminds me of the old telemetry 😄)

vponline · 2023-10-25T09:44:19Z

src/constants.ts

@@ -46,5 +46,5 @@ export const DIRECT_GATEWAY_MIN_TIME_DEFAULT_MS = 20;
 export const DIRECT_GATEWAY_MAX_CONCURRENCY_DEFAULT = 10;

 // TODO: load these 2 from env var instead
-export const DATAFEED_READ_BATCH_SIZE = 100;
-export const DATAFEED_UPDATE_BATCH_SIZE = 10;
+export const DATAFEED_READ_BATCH_SIZE = 1;


wow! This would be a problem for the main airseeker as well, right?

aquarat added 3 commits October 16, 2023 14:40

Only hash the base portion of the gateway URL

cca3652

Apply base url change to alias

15d8ce8

Add more detail to alerts description for dead gateways

64b7a7a

aquarat changed the base branch from main to airseeker-monitoring-tuning October 18, 2023 15:07

aquarat changed the base branch from airseeker-monitoring-tuning to airseeker-telemetry October 18, 2023 15:07

Add in batched DB writes

72a6467

aquarat changed the title ~~Add additional info to alerts~~ Add additional info to alerts | Batch DB writes Oct 18, 2023

aquarat added 5 commits October 19, 2023 11:10

Simplify data fetch loop

f2354f2

Revert "Simplify data fetch loop"

108e739

This reverts commit f2354f2.

Update chains

d5e6ccd

Reduce verbosity

04da36a

Refactor

6ed982c

aquarat requested a review from vponline October 19, 2023 12:11

aquarat commented Oct 19, 2023

View reviewed changes

aquarat marked this pull request as ready for review October 19, 2023 12:13

aquarat marked this pull request as draft October 19, 2023 13:13

aquarat added 4 commits October 19, 2023 14:21

Improve exit, improve call scheduling

b848b62

Fix tests

ce53148

Refactor scheduling for data retrieval loop

cfcea89

Don't retry in go by default, increase batch sizing for DB

1071693

aquarat marked this pull request as ready for review October 20, 2023 10:10

aquarat added 2 commits October 24, 2023 09:45

Fix Nodary retrieval by not waiting for alerts reporting to complete

6f50986

Failsafe batch read size down to one

a8b75fc

aquarat commented Oct 24, 2023

View reviewed changes

vponline approved these changes Oct 25, 2023

View reviewed changes

aquarat merged commit 20fc849 into airseeker-telemetry Oct 31, 2023

aquarat deleted the add-additional-info-to-alerts branch October 31, 2023 13:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add additional info to alerts | Batch DB writes #489

Add additional info to alerts | Batch DB writes #489

aquarat commented Oct 18, 2023 •

edited

Loading

aquarat Oct 19, 2023

vponline Oct 25, 2023

aquarat Oct 24, 2023

aquarat Oct 24, 2023

vponline Oct 25, 2023

vponline Oct 25, 2023

aquarat Oct 24, 2023

aquarat Oct 24, 2023

aquarat Oct 24, 2023

aquarat Oct 24, 2023

aquarat Oct 24, 2023

vponline Oct 25, 2023

aquarat Oct 24, 2023

aquarat Oct 24, 2023

vponline left a comment

vponline Oct 25, 2023

vponline Oct 25, 2023

vponline Oct 25, 2023

vponline Oct 25, 2023

Add additional info to alerts | Batch DB writes #489

Add additional info to alerts | Batch DB writes #489

Conversation

aquarat commented Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vponline left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aquarat commented Oct 18, 2023 •

edited

Loading