Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retries to test & emit warnings when connections fail #12528

Merged
merged 7 commits into from
Jul 20, 2022

Conversation

hithwen
Copy link
Contributor

@hithwen hithwen commented Jul 18, 2022

HANA connection some times flakes. In those cases either the client itself or the integration handle it gracefully but some or all metrics can be missing for a check run.

Example https://dev.azure.com/datadoghq/integrations-core/_build/results?buildId=105007&view=logs&j=b0ed5827-5770-56e3-9f82-70059854e6d2&t=879ef9f2-95c3-5e11-5b66-404511e0691a&l=663

This change introduces the following changes:

  • A warning service check is emitted when the connection flakes or gets lost (In the case of lost connection a critical service check is emitted if the re-connection fails)
  • Integration and e2e tests include a retry in case this happens.

@hithwen hithwen requested a review from a team as a code owner July 18, 2022 12:15
@ghost ghost added the integration/sap_hana label Jul 18, 2022
@hithwen hithwen changed the title Add attempts to test environment Add attempts to test environment & emit warnings when connections fail Jul 18, 2022
@codecov
Copy link

codecov bot commented Jul 18, 2022

Codecov Report

Merging #12528 (4519079) into master (296ef7e) will increase coverage by 0.01%.
The diff coverage is 92.85%.

Flag Coverage Δ
sap_hana 92.37% <92.85%> (+1.30%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@hithwen hithwen changed the title Add attempts to test environment & emit warnings when connections fail Add retries to test & emit warnings when connections fail Jul 19, 2022
@hithwen hithwen force-pushed the js/add-attempts-to-sap-hanna branch from 9b1c2a0 to 3bec9bc Compare July 20, 2022 08:52
FlorentClarret
FlorentClarret previously approved these changes Jul 20, 2022
aggregator = dd_agent_check(instance, rate=True)
while attempts and connection_flaked(aggregator):
aggregator = dd_agent_check(instance, rate=True)
attempts -= 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit - in this case we have 4 attempts, not 3

@FlorentClarret FlorentClarret self-requested a review July 20, 2022 09:03
@hithwen hithwen merged commit f3db5a5 into master Jul 20, 2022
@hithwen hithwen deleted the js/add-attempts-to-sap-hanna branch July 20, 2022 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants