Merge pull request #1397 from FrankTub/feature/teams-integration

Introduce the possibility to send elementary alerts to Microsoft Teams using an incoming webhook in Teams

dev-requirements.txt

@@ -16,3 +16,5 @@ types-pytz
 types-jsonschema
 types-PyYAML
 types-setuptools
+pandas-stubs
+types-retry

docs/_snippets/setup-teams-integration.mdx

@@ -0,0 +1,108 @@
+First create a Microsoft Teams team:
+
+<Accordion title="Create a new Team">
+
+## Create a new Team
+
+Go to the Microsoft Teams desktop app and create a new team.
+
+<img
+  src="https://res.cloudinary.com/dgxyrldax/image/upload/v1707202577/ghoizqzywicyipr7nbrm.png"
+  alt="Microsoft Teams team"
+/>
+
+Create a team from from a template and use the `From Scratch` template.
+
+<img
+  src="https://res.cloudinary.com/dgxyrldax/image/upload/v1707202897/dhftnz4zc0jsuay6eh2c.png"
+  alt="Microsoft Teams template"
+/>
+
+Choose `Public` as the kind of a team.
+
+<img
+  src="https://res.cloudinary.com/dgxyrldax/image/upload/v1707203017/s2buhk5yncmortzlvge5.png"
+  alt="Microsoft Teams public team"
+/>
+
+Call it `Elementary` (or whatever you prefer) and connect it to the workspace of your choice.
+
+<img
+  src="https://res.cloudinary.com/dgxyrldax/image/upload/v1707203137/jw2wgscz9cruarapmzpj.png"
+  alt="Microsoft Teams Elementary team"
+/>
+
+</Accordion>
+
+Now it is time to setup the webhook for this channel.
+
+<Accordion title="Create Teams Webhook">
+
+## Create a webhook
+
+Go to a channel in your Team and choose `Manage channel`
+
+<img
+  src="https://res.cloudinary.com/dgxyrldax/image/upload/v1707203620/npn3p0tsmdvk723etyxn.png"
+  alt="Teams manage channel"
+/>
+
+Choose `Edit` connectors.
+
+<img
+  src="https://res.cloudinary.com/dgxyrldax/image/upload/v1707203932/utnld7rzvgiwfgumzhtv.png"
+  alt="Teams edit connectors"
+/>
+
+Search for `Incoming webhook` and choose `Add`.
+
+<img
+  src="https://res.cloudinary.com/dgxyrldax/image/upload/v1707204047/esvfhescsxgttanzv3kx.png"
+  alt="Teams add incoming webhook"
+/>
+
+Choose `Add` again and add name your webhook `ElementaryWebhook` (or whatever you prefer). And `Create` the webhook.
+
+<img
+  src="https://res.cloudinary.com/dgxyrldax/image/upload/v1707204465/mcncjpvsnptd0gcsbb21.png"
+  alt="Teams create webhook"
+/>
+
+Copy the URL of the webhook.
+
+<img
+  src="https://res.cloudinary.com/dgxyrldax/image/upload/v1707204718/gkt2uhz2qaow1lm1frnp.png"
+  alt="Teams copy URL webhook"
+/>
+
+</Accordion>
+
+Lastly, pass the webhook to the CLI as a param or in the `config.yml` file:
+
+<Accordion title="Teams config as CLI params">
+
+## Webhook:
+
+Use the webhook URL when you execute edr monitor using the option `-tw, --teams-webhook`:
+
+```shell
+edr monitor --teams-webhook <your_teams_webhook_url>
+```
+
+</Accordion>
+
+<Accordion title="Teams config as in config.yml">
+
+The CLI reads the Teams integration from a file, copy it into a file called config.yml.
+Create it here: `HOME_DIR/.edr/config.yml`
+
+Here is the format in the yml itself:
+
+## Webhook:
+
+```yml config.yml
+teams:
+  teams_webhook: <your_teams_webhook_url>
+```
+
+</Accordion>

docs/oss/deployment-and-configuration/teams.mdx

@@ -0,0 +1,22 @@
+---
+title: "Teams setup for Elementary CLI"
+sidebarTitle: "Teams"
+---
+
+Elementary Teams integration includes sending [Teams alerts](/oss/guides/send-teams-alerts) on failures in dbt tests and models.
+
+## Integration options
+
+There is one integration option for Microsoft Teams: a Webhook. This method let you receive alerts from Elementary, but lacks
+some support that is available in the Slack integration solution.
+Below is features support comparison table (with Slack), to help you select the integration method.
+
+| Integration   | Elementary alerts | Elementary report | Multiple channels | Slack workflows |
+| ------------- | ----------------- | ----------------- | ----------------- | --------------- |
+| Teams Webhook | ✅                | ❌                | ❌                | ❌              |
+| Slack Token   | ✅                | ✅                | ✅                | ❌              |
+| Slack Webhook | ✅                | ❌                | ❌                | ✅              |
+
+## Teams integration setup
+
+<Snippet file="setup-teams-integration.mdx" />

docs/oss/guides/send-teams-alerts.mdx

@@ -0,0 +1,50 @@
+---
+title: "Setup Teams alerts"
+---
+
+<Snippet file="alerts/alerts-introduction.mdx" />
+
+## Setup Teams Integration
+
+<Info>
+
+**Before you start**
+
+Before you can start using the alerts, make sure to [install the dbt package](/oss/quickstart/quickstart-cli-package), [configure a profile and install the CLI](/oss/quickstart/quickstart-cli).
+This is **required for the alerts to work.**
+
+<br />
+</Info>
+
+<Snippet file="setup-teams-integration.mdx" />
+
+## Execute the CLI
+
+Make sure to run the following command after your dbt runs and tests:
+
+```
+edr monitor --teams-webhook <your_teams_webhook> --group-by [table | alert]
+```
+
+Or just `edr monitor` if you used `config.yml`.
+
+---
+
+## Alert on source freshness failures
+
+_Not supported in dbt cloud_
+
+To alert on source freshness, you will need to run `edr run-operation upload-source-freshness` right after each execution of `dbt source freshness`.
+This operation will upload the results to a table, and the execution of `edr monitor` will send the actual alert.
+
+- Note that `dbt source freshness` and `upload-source-freshness` needs to run from the same machine.
+- Note that `upload-source-freshness` requires passing `--project-dir` argument.
+
+## Continuous alerting
+
+In order to monitor continuously, use your orchestrator to execute it regularly (we recommend running it right after
+your dbt job ends to monitor the latest data updates).
+
+Read more about how to deploy [Elementary in production](/oss/deployment-and-configuration/elementary-in-production).
+If you need help or wish to consult on this, reach out to us
+on [Slack](https://elementary-data.com/community).

elementary/clients/teams/client.py

@@ -0,0 +1,90 @@
+from abc import ABC, abstractmethod
+from typing import Optional
+
+from pymsteams import cardsection, connectorcard, potentialaction  # type: ignore
+from retry import retry
+
+from elementary.config.config import Config
+from elementary.tracking.tracking_interface import Tracking
+from elementary.utils.log import get_logger
+
+logger = get_logger(__name__)
+
+OK_STATUS_CODE = 200
+
+
+class TeamsClient(ABC):
+    def __init__(
+        self,
+        webhook: str,
+        tracking: Optional[Tracking] = None,
+    ):
+        self.webhook = webhook
+        self.tracking = tracking
+        self.client = self._initial_client()
+
+    @staticmethod
+    def create_client(
+        config: Config, tracking: Optional[Tracking] = None
+    ) -> Optional["TeamsClient"]:
+        if not config.has_teams:
+            return None
+        if config.teams_webhook:
+            return TeamsWebhookClient(webhook=config.teams_webhook, tracking=tracking)
+        return None
+
+    @abstractmethod
+    def _initial_client(self):
+        raise NotImplementedError
+
+    @abstractmethod
+    def send_message(self, **kwargs):
+        raise NotImplementedError
+
+    @abstractmethod
+    def title(self, title: str):
+        raise NotImplementedError
+
+    @abstractmethod
+    def text(self, text: str):
+        raise NotImplementedError
+
+    @abstractmethod
+    def addSection(self, section: cardsection):
+        raise NotImplementedError
+
+    @abstractmethod
+    def addPotentialAction(self, action: potentialaction):
+        raise NotImplementedError
+
+
+class TeamsWebhookClient(TeamsClient):
+    def _initial_client(self):
+        return connectorcard(self.webhook)
+
+    @retry(tries=3, delay=1, backoff=2, max_delay=5)
+    def send_message(self, **kwargs) -> bool:
+        self.client.send()
+        response = self.client.last_http_response
+
+        if response.status_code == OK_STATUS_CODE:
+            return True
+        else:
+            logger.error(
+                "Could not post message to teams via webhook - %s. Error: %s",
+                {self.webhook},
+                {response.body},
+            )
+        return False
+
+    def title(self, title: str):
+        self.client.title(title)
+
+    def text(self, text: str):
+        self.client.text(text)
+
+    def addSection(self, section: cardsection):
+        self.client.addSection(section)
+
+    def addPotentialAction(self, action: potentialaction):
+        self.client.addPotentialAction(action)

elementary/config/config.py

@@ -16,6 +16,7 @@ class Config:
     _AWS = "aws"
     _GOOGLE = "google"
     _AZURE = "azure"
+    _TEAMS = "teams"
     _CONFIG_FILE_NAME = "config.yml"
 
     # Quoting env vars
@@ -63,6 +64,7 @@ def __init__(
         azure_connection_string: Optional[str] = None,
         azure_container_name: Optional[str] = None,
         report_url: Optional[str] = None,
+        teams_webhook: Optional[str] = None,
         env: str = "dev",
         run_dbt_deps_if_needed: Optional[bool] = None,
     ):
@@ -121,6 +123,12 @@ def __init__(
             GroupingType.BY_ALERT.value,
         )
 
+        teams_config = config.get(self._TEAMS, {})
+        self.teams_webhook = self._first_not_none(
+            teams_webhook,
+            teams_config.get("teams_webhook"),
+        )
+
         aws_config = config.get(self._AWS, {})
         self.aws_profile_name = self._first_not_none(
             aws_profile_name,
@@ -201,6 +209,10 @@ def has_send_report_platform(self):
     def has_slack(self) -> bool:
         return self.slack_webhook or (self.slack_token and self.slack_channel_name)
 
+    @property
+    def has_teams(self) -> bool:
+        return self.teams_webhook
+
     @property
     def has_s3(self):
         return self.s3_bucket_name
@@ -224,10 +236,20 @@ def has_gcs(self):
         return self.gcs_bucket_name and self.has_gcloud
 
     def validate_monitor(self):
+        provided_integrations = list(
+            filter(
+                lambda provided_integration: provided_integration,
+                [self.has_slack, self.has_teams],
+            )
+        )
         self._validate_timezone()
-        if not self.has_slack:
+        if not provided_integrations:
+            raise InvalidArgumentsError(
+                "Either a Slack token and a channel, a Slack webhook or a Microsoft Teams webhook is required."
+            )
+        if len(provided_integrations) > 1:
             raise InvalidArgumentsError(
-                "Either a Slack token and a channel or a Slack webhook is required."
+                "You provided both a Slack and Teams integration. Please provide only one so we know where to send the alerts."
             )
 
     def validate_send_report(self):

elementary/monitor/cli.py

@@ -261,6 +261,13 @@ def get_cli_properties() -> dict:
     help="Filter the alerts by tags:<tags separated by commas> / owners:<owners separated by commas> / models:<models separated by commas> / "
     "statuses:<warn/fail/error/skipped> / resource_types:<model/test>.",
 )
+@click.option(
+    "--teams-webhook",
+    "-tw",
+    type=str,
+    default=None,
+    help="A Microsoft Teams webhook URL for sending alerts to a specific channel in Teams.",
+)
 @click.pass_context
 def monitor(
     ctx,
@@ -289,6 +296,7 @@ def monitor(
     override_dbt_project_config,
     report_url,
     filters,
+    teams_webhook,
 ):
     """
     Get alerts on failures in dbt jobs.
@@ -318,6 +326,7 @@ def monitor(
         env=env,
         slack_group_alerts_by=group_by,
         report_url=report_url,
+        teams_webhook=teams_webhook,
     )
     anonymous_tracking = AnonymousCommandLineTracking(config)
     anonymous_tracking.set_env("use_select", bool(select))

elementary/monitor/data_monitoring/alerts/integrations/integrations.py

@@ -8,6 +8,9 @@
 from elementary.monitor.data_monitoring.alerts.integrations.slack.slack import (
     SlackIntegration,
 )
+from elementary.monitor.data_monitoring.alerts.integrations.teams.teams import (
+    TeamsIntegration,
+)
 from elementary.tracking.tracking_interface import Tracking
 
 
@@ -33,5 +36,11 @@ def get_integration(
                 tracking=tracking,
                 override_config_defaults=override_config_defaults,
             )
+        elif config.has_teams:
+            return TeamsIntegration(
+                config=config,
+                tracking=tracking,
+                override_config_defaults=override_config_defaults,
+            )
         else:
             raise UnsupportedAlertIntegrationError