Data Engineer

Recruitment Consultant: Data & AI, South Africa CONTRACT OPPORTUNITY - SOUTH AFRICA - IMMEDIATELY AVAILABLE PREFERRED Location: Remote – Quarterly travel to Joburg office Works closely with: AI Engineer, Data Scientist, BI Ops Team, Product & R&D About the Role You will work deeply with high-volume telematics data (≈9,000 messages per second), build scalable ingestion and processing pipelines, and help modernise a fast-maturing environment that is moving toward Microsoft Fabric and OneLake — while still requiring robust AWS, Spark, and Cassandra-based solutions for telematics workloads. This position requires someone who truly understands data: large-scale ingestion, real-time streaming, distributed systems, and the architectural thinking needed in an environment where data architecture capability is currently lacking. You will also work alongside AI Engineers and Data Scientists to enable internal AI-driven operational improvements and external customer-facing data products. Key Responsibilities Build, maintain, and optimise large-scale ingestion pipelines handling ~9,000 events/messages per second. Work extensively with telematics datasets Develop distributed processing workflows using Cassandra , Spark , Databricks , and related tooling. Ensure efficient extraction logic in Databricks while managing cost trade-offs vs Fabric/Synapse. Modern Data Platform Development Engineer pipelines across mixed environments: Microsoft Fabric + OneLake for BI, and AWS for telematics workloads. Help integrate telematics data into existing BI structures while ensuring reliability, performance, and cost efficiency. Implement transformations in ADX and push processed data into downstream operational systems. Step into a space where no formal data architecture capability currently exists — influence patterns, standards, and design. Own end-to-end structuring, modelling, and flow of telematics and ERP data into analytics-ready formats. Support internal R&D work: enabling new customer-facing data products and exploring AI-driven operational improvement opportunities. Enabling AI, Analytics & BI Teams Work alongside the AI Engineer/Data Scientist to ensure telematics and ERP data is accessible and usable. Collaborate with BI operational leads to support data insight generation and data monetisation efforts. Build the data foundation required to commercialise the dataset. Operationalising and Automating Data Design and implement resilient data pipelines using Synapse, Databricks, and AWS services. Build new data solutions and internal tools for the operations and R&D teams. Improve processing efficiency to reduce cloud compute and storage costs. Tech Stack & Environment Core Technologies Spark Databricks Synapse Microsoft Fabric + OneLake ADX (for fast transformations & unit health insights) Nice-to-Have AI/ML exposure (working with AI Engineers, understanding model data requirements) Experience in monetising data products or customer-facing data solutions Understanding of ERP data structures What Makes This Role Exciting You are stepping into the most business-critical hire for the team. Influence architecture and strategy in a place where the capability is still developing. Work with very high-scale telematics data and modern tech. Hybrid environment: Fabric/OneLake for BI, AWS + Spark/Cassandra for telematics. Strong partnership with AI Engineers — ideal for someone who enjoys ML-adjacent data engineering. Ideal Candidate Profile 5–8+ years of data engineering experience. Strong experience with big data, streaming data, and distributed systems . Capable of designing robust end-to-end ingestion pipelines. Hands-on with Spark, Databricks, Cassandra, or similar high-throughput systems. Comfortable structuring and modelling complex, messy operational datasets. Able to balance business needs, cost considerations, and technical best practice. Thrives in an evolving environment with expanding capabilities and new data products. Seniority level Mid-Senior level Employment type Contract Job function Information Technology Motor Vehicle Manufacturing Referrals increase your chances of interviewing at Tenth Revolution Group by 2x Get notified about new Data Engineer jobs in South Africa. #J-18808-Ljbffr