DataDog integration
This document describes how to integrate Data Dog with our MWAA application environments.
Logging
In order to send logs to Datadog, we are using a Datadog forwarder Lambda stack. This setup is created by the former PE Team with the AWS account itself. But the default setup does not log any Airflow logs. Our Airflow Installations and logs are stored in Cloud Watch log groups.
In order to find our airflow logs in Datadog, we attach the Cloud Watch log groups we want to see in Datadog to the forwarder Lambda as trigger manually. That way we do not spam the Datadog system with everything.
For example for dev we attach:
- airflow-dev-managed-airflow-DAGProcessing
- airflow-dev-managed-airflow-Scheduler
- airflow-dev-managed-airflow-Task
- airflow-dev-managed-airflow-WebServer
- airflow-dev-managed-airflow-Worker
Additionally, we define our own Datadog index to keep the search in datadog more efficient. This is done by enhancing the Passat Terraform scripts https://github.com/PIT-PASSAT-Community/datadog/tree/main/tf-datadog-log-indexes.
our indexes are:
- data-cd-dev
- data-cd-stage
- data-cd-prod
Keep in mind that it is possible to allow automatic log indexing inside datadog via the forwarder, but this way we would include all cloud watch logs from our account and not only the MWAA logs.
Metrics
We are using the Datadog MWAA integration. This integration is activated in datadog and automatically collects some core metrics from our MWAA environments.
check https://app.datadoghq.eu/integrations?search=MWA&integrationId=amazon-mwaa for full details.
Service entry in service catalog
In order to add an entry to the service catalog, a meta description is added to the root folder of the repo named entity.datadog.yaml.
It is pulled from Datadog automatically since Datadog has read permission to the repo.
But be aware, that these permissions are bound to a specific person, who granted that access, this integration will fail, if the person leaves the company.
Monitors
Monitors are managed outside CDK via python in the infrastructure repository.