Senior Data Engineer | Data Integration Developer

13 hours ago
Pretoria, South Africa Active Recruitment Full-time

Introduction

Centurion  |  Gauteng  | Permanent

7+ years experience required - Senior

Hands on Engineer  | Data Integration Developer

This role is for someone that is not afraid to get their hands dirty.

 

Hands-on role that owns the full data engineering lifecycle:

  • from ingestion and transformation through to platform architecture and production deployment – designing, building and running complex data pipelines on AWS.

 

Technical Skills Required:

  • Python (advanced) – data pipeline development, automation and scripting.
  • SQL (advanced) – complex query writing, performance optimisation and stored procedures.
  • AWS (mid–advanced) – S3, Glue, Lambda, Redshift, RDS, IAM, CloudWatch, Athena.
  • Data warehousing and data modelling – star/snowflake schemas, dimensional and analytics-oriented design.
  • ETL/ELT pipeline development – batch, incremental and near-real-time patterns.
  • Data lakehouse architecture on cloud-native platforms.
  • CI/CD and Git – version control, automated testing and deployment pipelines.
  • Power BI – supporting BI teams, data model integration and semantic layer understanding.
  • Experience ingesting data from relational databases (e.g. SQL Server, Oracle), cloud storage, APIs, SFTP, MongoDB and file-based sources.

 

Highly Valued (Differentiating Skills):

  • Apache Spark / PySpark for large-scale distributed processing.
  • dbt for analytics engineering and transformation layer management.
  • Apache Airflow or AWS Step Functions for orchestration.
  • Terraform or CloudFormation (infrastructure-as-code).
  • Real-time streaming – Kafka, Kinesis or similar.
  • Machine learning workflow support – feature engineering and model data pipelines.
  • Microsoft Azure or GCP as secondary cloud exposure.

 

Senior Data Engineer, AWS, Python, SQL, Data Engineering, ETL/ELT, Data Lakehouse, Data Warehousing, Power BI, CI/CD, Agile, AI-enabled engineering, Financial Services, Centurion, Hybrid

Duties & Responsibilities

Delivery Ownership within Agile Execution

  • Contribute to data engineering outcomes aligned to sprint goals and programme-level commitments.
  • Take part in Agile ceremonies – planning, estimation, prioritisation and continuous improvement.
  • Decompose data features into implementable tasks with reliable effort estimates.
  • Ensure outputs meet agreed functional, performance and data quality expectations.

Data Engineering & Pipeline Development

  • Design and implement scalable ingestion and transformation pipelines across multiple systems and data domains, supporting batch and incremental processing.
  • Build in error handling, monitoring and alerting to keep pipelines robust.
  • Implement data validation and reconciliation mechanisms to maintain confidence in data assets.

Platform Design & Architecture (AWS)

  • Design solutions aligned to the data platform strategy and target architecture, and contribute to the evolution of the cloud-based data lakehouse environment.
  • Assess the impact of design choices on security, performance, cost and supportability.
  • Identify integration points, upstream/downstream dependencies and risks early in the delivery lifecycle.

Technical Leadership, Coaching & Standards

  • Provide guidance and technical oversight to less experienced data engineers.
  • Support analytics, BI and data science teams with clarity on data structures, availability and pipeline behaviour.
  • Review code and configuration, and help define shared standards, patterns and best practices.
  • Optimise and simplify existing pipelines and data models.

AI-Enabled Engineering (Governance-Led)

  • Promote responsible use of AI to improve engineering productivity and solution quality, within group and business-unit governance frameworks.
  • Use only enterprise-approved AI tooling, and ensure no sensitive, proprietary or client information is exposed to public or unapproved AI platforms.
  • Validate all AI-assisted outputs and retain accountability for correctness, compliance and production readiness.
  • Apply AI to understand data flows and lineage, improve pipeline logic, support query formulation and test scenario creation, and accelerate documentation.

DevOps, Operational Support & Incident Response

  • Apply version control, automated testing, CI/CD and structured releases.
  • Contribute to operational readiness, deployment pipelines and monitoring.
  • Diagnose and resolve data incidents, perform root cause analysis and implement corrective actions.

Security, Risk, Compliance & Governance

  • Build solutions with privacy, access control and regulatory considerations embedded by design.
  • Identify and mitigate data risks, and support remediation of audit findings and compliance gaps.
  • Maintain accurate documentation of pipelines, data models and operational considerations, and keep work tracking up to date in Jira.
  • Adhere to delivery governance, change management and release processes.

Stakeholder Engagement

  • Communicate technical progress, constraints and decisions clearly to business and technical stakeholders.
  • Collaborate across technology and analytics teams and contribute positively to team culture.

Desired Experience & Qualification

Skills & Experience:

  • 7+ years’ professional experience in data engineering or related roles.
  • Demonstrated delivery of production-ready, end-to-end data pipelines and cloud data platforms.
  • Strong exposure to AWS-based data architectures.
  • Experience across the full stack – ingestion, transformation, modelling and serving – not just a single layer.
  • Hands-on experience working within Agile or SAFe delivery environments.
  • Experience in financial services, insurance, investment management or other regulated data environments (preferred).
  • Practical experience applying AI tooling in engineering workflows – including using AI responsibly and validating its outputs – to support pipeline understanding, optimisation, testing or documentation.
  • Experience coaching or guiding other engineers.

Interested?

  • Senior Data Engineer

    2 days ago

    Pretoria East, South Africa Network IT Full-time

    Reference: ITA006289-Mfund-2Are you a passionate Data Engineer looking to work with modern cloud technologies, big data platforms, and scalable data architectures? Join a forward-thinking organisation where you'll play a pivotal role in building robust data pipelines, enabling business intelligence, and driving data-driven decision-making across the...

  • Data Engineer

    13 hours ago

    Pretoria, South Africa Boardroom Appointments Full-time

    REQUIRED MINIMUM EDUCATION/TRAINING: BCom or Bachelors degree in Information Technology / Computer Science / Engineering (IT) or similarData Engineering tools (Azure Data Engineer) certification will be an added advantageITIL or COBIT certification will be an added advantage REQUIRED MINIMUM WORK EXPERIENCE: 2 - 4 years of working experience in an IT...

  • Senior Data Engineer

    2 days ago

    Pretoria, South Africa Network Recruitment Full-time

    An innovative and data-driven organisation is seeking a talented Senior Data Engineer to join its growing team. This opportunity is ideal for a technically strong professional who enjoys solving complex data challenges and building scalable solutions that support analytics, reporting, and operational systems. In this role, you will be responsible for...

  • Data Scientist

    2 days ago

    Pretoria, Gauteng, South Africa King Price Group Full-time

    We’ve got a position available for an experienced end-to-end SQL developer to join our dynamic team. We call these Data Scientist, but we are really looking for a solid dev. Company: Porcupine UnionPurpose of position:This role involves the full BI lifecycle, from gathering business requirements to designing and implementing ETL processes, developing...

  • Remote Senior Data Engineer: Azure Data Platform

    2 days ago

    Pretoria, Gauteng, South Africa Hire Resolve Remote Full-time

    A recruitment agency is seeking a talented Senior Data Engineer/Developer to enhance their data infrastructure for real-time analytics and fraud detection. The successful applicant will optimize Azure-based data pipelines, ensuring data quality and performance. This position requires 7+ years of experience, proficiency in T-SQL, and familiarity with various...

  • Senior Data Engineer | AWS | AI

    6 days ago

    Pretoria, South Africa Active Recruitment Full-time

    IntroductionCenturion  |  Gauteng  | Permanent 7+ years experience required - Senior Hands on Engineer This role is for someone that is not afraid to get their hands dirty.   Hands-on role that owns the full data engineering lifecycle: from ingestion and transformation through to platform architecture and production deployment – designing, building...

  • Senior Analytics Engineer

    2 days ago

    Pretoria, Gauteng, South Africa Jobgether SRL Remote Full-time

    This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Senior Analytics Engineer – Marketing Data based in South Africa. Join a data-driven environment where analytics engineering plays a key role in transforming marketing data into actionable business insights.In this role, you...

  • Senior Cloud Data Engineer

    17 hours ago

    Pretoria, Gauteng, South Africa Hire Resolve Full-time

    Hire Resolve's client is looking for a Senior Cloud Engineer (Remote) to join their dynamic and growing Engineering team. The ideal candidate will have strong expertise in Databricks, Spark, Lakehouse architectures, Delta Lake tables, and other modern data engineering technologies. They should be proficient in T‑SQL and Python, with advanced coding...

  • Junior Data Engineer

    2 days ago

    Pretoria, South Africa Creative Leadership Solutions Full-time

    REQUIREMENTS Minimum Education (essential) - Bachelor's degree in Computer Science, Information Systems, Data Engineering or a related field. - Relevant Microsoft Fabric and/or Azure data certification. Minimum applicable experience (years): - 0-1 years of practical data engineering experience, including strong recent hands-on experience with Microsoft...

  • Intermediate Data Engineer

    2 days ago

    Pretoria, South Africa Creative Leadership Solutions Full-time

    REQUIREMENTS Minimum Education (essential) - Bachelor's degree in Computer Science, Information Systems, Data Engineering or a related field. - Relevant Microsoft Fabric and/or Azure data certification. Minimum applicable experience (years): - 3+ years of practical data engineering experience, including strong recent hands-on experience with Microsoft...

Similar jobs in Pretoria

Top cities