Senior Data Engineer

2 weeks ago

Cape Town, Western Cape, South Africa PBT Group Full time
Senior DataOps Engineer Role Summary

We are seeking a talented Senior DataOps Engineer to lead the development of DataOps practices within our organisation. As a seasoned professional with a strong DevOps background and software and data experience, you will be responsible for setting the direction for DataOps within the team, building scalable data pipelines, and ensuring efficient data operations in a fully cloud-based environment (AWS).

Key Responsibilities
  • Define and implement DataOps practices, guiding the team towards a more automated and efficient data pipeline management system.
  • Collaborate with data engineers, data scientists, and other stakeholders to establish best practices for data workflows and infrastructure.
  • Design, deploy, and manage scalable and reliable data pipelines to enable efficient data flow and processing across cloud environments.
  • Automate and optimize ETL/ELT processes, ensuring seamless data integration from various sources.
  • Apply DevOps principles to data workflows, focusing on continuous integration, continuous delivery (CI/CD), and infrastructure-as-code (IaC) for data operations.
  • Implement version control and automation for data-related processes, reducing errors and improving data quality.
  • Manage and optimize AWS cloud infrastructure to support data workflows, including S3, Redshift, Lambda, and RDS.
  • Monitor, maintain, and scale AWS resources for efficient data storage, processing, and analysis.
  • Ensure data security and compliance with relevant industry standards and regulations, implementing appropriate security protocols and monitoring.
  • Collaborate with security teams to ensure secure handling of sensitive data and manage access controls effectively.
  • Set up monitoring, logging, and alerting mechanisms for data pipelines to ensure high availability and performance.
  • Identify bottlenecks and inefficiencies in data processes, proposing and implementing optimisations.
  • Provide technical leadership and mentorship to junior team members, helping them to grow in their roles and expand their knowledge of DataOps practices.
  • Work closely with data engineering, software development, and IT teams to drive cross-functional initiatives.
Requirements and Qualifications
  • Proven experience as a DevOps Engineer, with significant exposure to data-focused environments.
  • Strong understanding of data concepts, including data pipelines, ETL/ELT processes, and data storage solutions.
  • Expertise in building, deploying, and managing data pipelines in cloud environments, especially AWS.
  • Familiarity with AWS services such as S3, Redshift, Lambda, RDS, and other relevant cloud-based data tools.
  • Experience with CI/CD pipelines, infrastructure-as-code, and automation tools (e.g., Jenkins, Terraform, Ansible).
  • Strong knowledge of version control systems (e.g., Git) and containerisation technologies (e.g., Docker, Kubernetes).
Preferred Skills
  • Experience with DataOps principles and practices, including data versioning, pipeline automation, and data observability.
  • Strong programming skills in languages such as Python, Java, or Scala, with experience in building data processing solutions.
  • Familiarity with data processing frameworks like Apache Spark, Kafka, or similar.
  • Exposure to security best practices for data storage and processing in cloud environments.
What We Offer

As a Senior DataOps Engineer at PBT Group, you will have the opportunity to work with a talented team of professionals, develop your skills and expertise, and contribute to the success of our organization.

We offer a competitive salary, comprehensive benefits package, and a collaborative and dynamic work environment.

  • Senior Data Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Goldman Tech Resourcing Full time

    Senior Data EngineerWe are seeking an experienced Senior Data Engineer to join our team at Goldman Tech Resourcing. The ideal candidate will have a strong background in designing and implementing data pipelines, with expertise in managing and optimizing Elasticsearch clusters. Additionally, they should have strong proficiency with data extraction,...

  • Senior Data Center Electrical Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Amazon Data Services South Africa (Pty) Ltd Full time

    About the RoleAs a Senior Data Center Electrical Engineer at Amazon Data Services South Africa (Pty) Ltd, you will play a critical role in ensuring the design, construction, and operation of our data centers meet the highest standards of safety, security, and efficiency. You will be responsible for providing technical expertise and guidance to our teams,...

  • Senior Network Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Dimension Data Full time

    Job Title: Senior Network EngineerWe are seeking a highly skilled Senior Network Engineer to join our team at Dimension Data. As a key member of our network engineering team, you will be responsible for designing, implementing, and maintaining complex network infrastructures for our clients.Key Responsibilities:Design and implement network architectures to...

  • Senior Data Center Electrical Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Amazon Data Services South Africa (Pty) Ltd Full time

    About the Role:As a Senior Data Center Electrical Engineer at Amazon Data Services South Africa (Pty) Ltd, you will play a critical role in ensuring the high availability and reliability of our data centers. Your expertise will be instrumental in designing, implementing, and maintaining electrical systems that meet the stringent requirements of our...

  • Senior Data Engineer

    3 weeks ago

    Cape Town, Western Cape, South Africa Network Recruitment Full time

    Job Title: Senior Data Engineer - Data ArchitectJob Summary:We are seeking a highly skilled Senior Data Engineer - Data Architect to join our team at Network Recruitment. As a Senior Data Engineer - Data Architect, you will be responsible for designing and implementing efficient data systems, ensuring data quality, and maintaining data infrastructure...

  • Senior Data Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Datonomy Solutions Full time

    We are seeking a highly skilled Senior Data Engineer to join our team at Datonomy Solutions. In this role, you will be responsible for designing and implementing data solutions using industry best practices. You should have a strong background in data and business intelligence, with experience in SQL technology, reporting technology, and cloud technology.Key...

  • Senior Data Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Network Recruitment Full time

    Senior Data Engineer - Data ArchitectFully remote opportunity for a Senior Data Engineer - Data Architect to join our team at Network Recruitment.Key Responsibilities:Design and implement data architectures to support business growth and scalabilityDevelop and maintain large-scale data pipelines and systemsCollaborate with cross-functional teams to ensure...

  • Senior Data Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Datonomy Solutions Full time

    We are looking for a Senior Data Engineer at Datonomy Solutions to support and manage BI analytics solutions. In this role, you should have a background in data and business intelligence. You should be analytical; an excellent communicator and a problem-solving aptitude is needed in this role. Ultimately, you will enhance our business intelligence systems to...

  • Senior Data Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Aculocity Full time

    Job Title: Senior Data EngineerJob Summary:Aculocity is seeking a highly skilled Senior Data Engineer to lead the design, development, and optimization of complex data infrastructure and pipelines. As a key member of our agile Data Engineering team, you will play a pivotal role in delivering high-impact reporting, analytics, and machine learning solutions...

  • Senior Data Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa E-Merge Full time

    A Fintech department, delivering financial information to the company, seeks a Senior Data Engineer to design and develop robust data frameworks and create data services that deliver actionable insights to their global customer base.E-Merge is a globally recognised brand that rewards technical prowess.Responsibilities:Improve processes and...

  • Senior Avaya Telephony Engineer

    2 days ago

    Cape Town, Western Cape, South Africa Dimension Data Full time

    Job OverviewWe are seeking an experienced Senior Avaya Telephony Engineer to join our team at Dimension Data. This role will be responsible for providing technical support and solutions to clients, ensuring that their IT infrastructure and systems remain operational.The ideal candidate will have a strong background in Avaya telephony administration, with...

  • Senior Data Center Electrical Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Amazon Data Services South Africa (Pty) Ltd Full time

    About the Job:We are seeking a highly skilled Senior Data Center Electrical Engineer to join our team at Amazon Data Services South Africa (Pty) Ltd. As a key member of our team, you will be responsible for providing full life-cycle support to our data centers from design inception through site improvement and maintenance.Key Responsibilities:• Perform...

  • Senior Data Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Aculocity Full time

    About the Role:The Senior Data Engineer will play a pivotal role in our agile Data Engineering team, leading the design, development, and optimization of complex data infrastructure and pipelines.Key Responsibilities:Develops and maintains scalable data pipelines and builds out new API integrations to support continuing increases in data volume and...

  • Senior Network Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Dimension Data Full time

    Job SummaryWe are seeking a highly skilled Senior Network Engineer to join our team at Dimension Data. As a key member of our network engineering team, you will be responsible for designing, implementing, and supporting new IT projects in the areas of modern networks, security solutions, and cloud technologies.You will work on a variety of complex...

  • Senior .NET Software Engineer

    2 days ago

    Cape Town, Western Cape, South Africa Dimension Data Full time

    **Job Description for Senior .NET Software Engineer at Dimension Data**We are seeking a highly skilled and experienced .NET Developer to join our team as a Senior .NET Software Engineer.The successful candidate will be responsible for designing, developing, testing, and implementing viable IT solutions using the latest .NET technologies.Main...

  • Senior Data Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Network Recruitment Full time

    About the RoleWe are seeking a skilled Data Engineer to join our team at Network Recruitment. As a Senior Data Engineer - Data Pipeline Architect, you will play a key role in designing and implementing efficient data pipelines for our clients in the financial services sector.Key ResponsibilitiesCollaborate with data scientists and analysts to develop...

  • Senior Data Engineering Specialist

    2 weeks ago

    Cape Town, Western Cape, South Africa OfferZen Ltd Full time

    At OfferZen Ltd, we're looking for a seasoned Senior Data Engineering Specialist to join our team. As a key member of our engineering team, you'll play a crucial role in designing, building, and maintaining our large-scale data systems.Key Responsibilities:Design, develop, and deploy large-scale data processing systemsCollaborate with cross-functional teams...

  • Senior Data Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Progressive IT Resourcing Full time

    Job Title: Senior Data Engineer - Cloud ExpertJob Summary: We are seeking a highly skilled Senior Data Engineer - Cloud Expert to join our team at Progressive IT Resourcing. As a key member of our engineering organization, you will be responsible for designing, building, and maintaining scalable data platforms on cloud technologies.Key...

  • Senior Data Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa LexisNexis South Africa Shared Services (Pty) Ltd Company Full time

    About the RoleWe're seeking a highly skilled Senior Data Engineer to join our team at LexisNexis. As a key member of our data engineering team, you will play a critical role in designing, building, and maintaining our data infrastructure.ResponsibilitiesDesign and implement scalable data pipelines to process large datasets in real-time.Collaborate with...

  • Senior Cloud Infrastructure Engineer

    2 days ago

    Cape Town, Western Cape, South Africa Dimension Data Full time

    Job SummaryWe are seeking a highly skilled Senior Cloud Infrastructure Engineer to join our team at Dimension Data. As a key member of our engineering team, you will be responsible for designing, deploying, and managing scalable cloud infrastructure on AWS.Key ResponsibilitiesDesign and implement efficient and reliable automated scripts to automate...