Decouple Customer Data Platform from Statista Platform

Context

Our current customer data platform (CDP), Twilio Segment, manages all user-related data, such as tracking and marketing events. However, Segment lacks built-in monitoring and logging capabilities, which creates several challenges:

• We are unable to detect or troubleshoot issues and bugs related to event handling in real-time.

• Events are sometimes missing in Segment despite being successfully sent, but we lack insight into which events are affected or why. This is also due to CDP API nature, which returns always 200 , even when events are not valid, or there are other underlying issues

• We do not currently have a mechanism in place to detect when a form on Statista platform stops sending events to Segment.

• The third party system (CDP) is tighlty coupled to all the statista pages. Which can slow down the platform, and/or cause huge effort to make any change in the API contract.

• Due to tight coupling and lack of schema definition, the Martech needs to self inject the CDP API in other team Repos/Projects which is less than ideal.

In order to address these gaps and improve the reliability of our event data, we have identified the need for an intermediary service that will allow us to validate, monitor and manage tracking events before they are sent to Segment , Furthermore completely decouple the CDP from the monolith and make it compatible with the single page application approach in Statista 4.0. This will enable us to ensure data consistency , react proactively to potential issues, and introduce new fields and features more easily in the future.

---
title: The current connection between Statista Platform and the CDP is directly via REST
---
graph LR 
    id1[Statista 3.5] --> CDP
    id2[Statista 4] --> CDP

Decision

We will introduce an intermediary service between Statista Platform and the CDP to intercept tracking events, providing validation, enhanced monitoring and error logging. The service will consist of:

• An API Gateway that receives event data from Statista Platform.
• Schema validation within the API Gateway to ensure all events conform to the correct structure.
• Logging of invalid requests in CloudWatch and DataDog for better monitoring and alerting.
• A FIFO-configured SQS queue that will store the validated events, using the user_id as the group_id to avoid race conditions.
• A Lambda function that processes the events from the SQS queue and sends them to Segment.

In the next iteration, we will also store events in a database for comparison purposes, allowing us to verify that events are properly received and stored in the CDP.

This decision ensures that we can actively monitor event traffic, detect failures in real-time, and identify missing events to improve data reliability and troubleshooting.

Proposed architecture to decouple the CDP from Statista Platform and provide monitoring and logging is visualized on this miro board: Monitor All Purchases

Consequences

Positive Impacts:

• Enhanced Visibility: By introducing monitoring capabilities, we will be able to detect and respond to issues in real-time.
• Early Schema Validation: by introducing Schema validation early on, we ensure that product teams are sending the correct and complete data and reduce the cost of manual work and fixing bug cycle
• Decoupled System from CDP: Eliminate the direct dependency between the platform and third party application, which has several high values like speeding up pages performance , higher flexibility, and enhanced maintenance.
• Improved Pages performance: this is related to decoupling from the CDP and as a result sending events which can improve the page performance especially checkouts
• Data Integrity Assurance: Storing events in a database will allow for easy comparison between events sent and those recorded in the CDP, helping us ensure that no events are lost.
• Prevention of Race Conditions: FIFO queues will prevent issues related to event ordering, ensuring that events are processed in the correct sequence for each user.

Negative Impacts:

• Increased System Complexity: Adding the intermediary service introduces additional components, such as the API Gateway, SQS queue, and Lambda function, which will require management and maintenance.
• Latency: The added steps in the process could introduce some latency in event processing to the CDP
• Operational Overhead: The new architecture will necessitate monitoring and management of the intermediary service components, increasing the operational workload.