Data Architect at TrackingplanPosted on: 05/11/2022
Full-remote, between UTC-1 and UTC+5 Trackingplan monitors in real time the customer data flowing between Web and iOS/Android Apps to all third-party integrations. Several of our customers have more than 10M MAU. That's at least 10GB daily per customer with changing daily and weekly traffic trends and peaks (e.g. black Friday). Our real-time data ingestion pipeline has to handle this kind of traffic coming from all the users of each of our customers. This challenges us to design, maintain and continuously improve an architecture that keeps the two counterparts latency and costs as low as possible and this requires careful design and execution. We are also planning to migrate from our current data preprocessing pipeline and DynamoDB database to an Analytics Database that allows us to compute a wider range of queries on the data in real time from within the user application. This position spans from maintaining and improving the current system implemented in Python on AWS Lambda to proposing and designing completely new architectures to implement with the rest of the team. We are looking for someone with experience dealing with large real-time data throughputs and modern NoSQL Databases such as ClickHouse, Druid or similar and Cloud/PaaS services on AWS/GCP/Azure.