Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(data-warehouse): Adjusted temporal and sql settings for long running tasks #25196

Merged
merged 1 commit into from
Sep 25, 2024

Conversation

Gilbert09
Copy link
Member

Problem

  • We have a long-running snowflake job that keeps getting activity timeouts on the import data step

Changes

  • Updates the activity heartbeat time out from 2mins to 5mins
    • Kept the heartbeater beating every 10 secs (I think something may be stopping it from beating?!)
  • Increased the start_to_close_timeout from 5 hours to 12 hours
  • Increased the SQL chunk size from 1,000 to 5,000 rows
  • Moved the import_data heartbeater to the root of the activity and not just wrapping the DLT pipeline run

Copy link
Member

@EDsCODE EDsCODE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assuming the tab indent changes are correct

@Gilbert09 Gilbert09 merged commit de87079 into master Sep 25, 2024
87 checks passed
@Gilbert09 Gilbert09 deleted the tom/long-running-activites branch September 25, 2024 16:03
Copy link

sentry-io bot commented Sep 25, 2024

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ DeltaError: Delta protocol violation: Generic action error: filed to get tombstones posthog.temporal.data_imports.pipelines.pipelin... View Issue
  • ‼️ PipelineStepFailed: Pipeline execution failed at stage normalize when processing package 1727291771.1948323 with exce... posthog.temporal.data_imports.pipelines.pipelin... View Issue
  • ‼️ PipelineStepFailed: Pipeline execution failed at stage load when processing package 1727291608.795646 with exception: posthog.temporal.data_imports.pipelines.pipelin... View Issue
  • ‼️ PipelineStepFailed: Pipeline execution failed at stage load when processing package 1727288871.7664406 with exception: posthog.temporal.data_imports.pipelines.pipelin... View Issue
  • ‼️ Exception: DLT type 'binary' is not a supported column type posthog.warehouse.data_load.validate_schema in ... View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants