Senior Data Engineer

1 week ago

Johannesburg, Gauteng, South Africa Dariel Full time R500 000 - R1 200 000 per year

Data Engineer and / or Streaming Engineer

The Data Engineer's role entails building and supporting data pipelines that must be scalable, repeatable, and secure. This role functions as a core member of an agile team, whereby these professionals are responsible for the infrastructure that provides insights from raw data, handling and integrating diverse sources of data seamlessly. They enable solutions by handling large volumes of data in batch and real-time by leveraging emerging technologies from both the big data and cloud spaces. Additional responsibilities include developing proof of concepts and implementing complex big data solutions with a focus on collecting, parsing, managing, analysing, and visualising large datasets. They know how to apply technologies to solve the problems of working with large volumes of data in diverse formats to deliver innovative solutions. Data Engineering is a technical job that requires substantial expertise in a broad range of software development and programming fields. These professionals have a knowledge of data analysis, end-user requirements, and business requirements analysis to develop a clear understanding of the business need and to incorporate these needs into a technical solution. They have a solid understanding of physical database design and the systems development lifecycle.

Responsibilities

  • Architects' Data Analytics Framework
  • Translates complex functional and technical requirements into detailed architecture, design, and high-performing software
  • Leads Data and batch/real-time analytical solutions leveraging transformational technologies
  • Works on multiple projects as a technical lead, driving user story analysis and elaboration, design and development of software applications, testing, and building automation tools
  • Development and Operations
  • Database Development and Operations
  • Policies, Standards, and Procedures
  • Business Continuity & Disaster Recovery
  • Research and Evaluation
  • Creating data feeds from on-premise to AWS Cloud
  • Support data feeds in production on a break-fix basis
  • Creating data marts using Talend or a similar ETL development tool
  • Manipulating data using Python
  • Processing data using the Hadoop paradigm, particularly using EMR, AWS's distribution of Hadoop
  • Develop for Big Data and Business Intelligence, including automated testing and deployment

Requisite Experience, Education, Knowledge, and/ or Skills

  • Bachelor's Degree in Computer Science, Computer Engineering, or equivalent
  • AWS Certification
  • Extensive knowledge in different programming or scripting languages
  • Expert knowledge of data modelling and understanding of different data structures and their benefits and limitations under particular use cases
  • Capability to architect highly scalable distributed systems, using different open source tools
  • 5+ years of Data engineering or software engineering experience
  • 2+ years of Big Data experience
  • 2+ years' experience with Extract, Transform, and Load (ETL) processes
  • 2+ years of AWS experience
  • 5 years of demonstrated experience with object-oriented design, coding, and testing patterns, as well as experience in engineering (commercial or open source) software platforms and large-scale data infrastructures
  • Big Data batch and streaming tools
  • Talend
  • AWS: EMR, EC2, S3
  • Python
  • PySpark or Spark

  • Senior Data Management Security Engineer

    3 days ago

    Johannesburg, Gauteng, South Africa NTT DATA, Inc. Full time R800 000 - R1 200 000 per year

    Make an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.The Data Management Security Engineer...

  • Senior Microsoft Engineer

    2 weeks ago

    Johannesburg, Gauteng, South Africa NTT DATA Full time R900 000 - R1 200 000 per year

    Job Summary:The Senior Microsoft Engineer is responsible for designing, implementing, and maintaining Microsoft-based infrastructure solutions, both on-premises and in the cloud (primarily Azure). This role ensures the stability, integrity, and efficiency of IT systems and supports automation, security, and compliance...

  • Senior Data Engineer

    1 week ago

    Johannesburg, Gauteng, South Africa Harambee Youth Employment Accelerator Full time R120 000 - R240 000 per year

    We're Hiring: Senior Data Engineer – JohannesburgAre you ready to build technology that changes lives?Harambee Youth Employment Accelerator is looking for an exceptionalSenior Data Engineerto join our Software Development team. You'll work on collecting, storing, and processing large datasets to deliver optimal solutions—while partnering with our Head of...

  • Senior Data Engineer

    1 week ago

    Johannesburg, Gauteng, South Africa Harambee Youth Employment Accelerator Full time R600 000 - R1 200 000 per year

    Senior Data Engineer – JohannesburgHelp build technology that changes lives.About the RoleThe Harambee Software Development team is looking for an exceptional Senior Data Engineer to join our dynamic team based in Johannesburg.This role is focused on collecting, storing, and processing large sets of data to deliver optimal solutions—along with their...

  • Senior Data Engineer

    5 days ago

    Johannesburg, Gauteng, South Africa Harambee Full time R800 000 - R1 200 000 per year

    Senior Data Engineer - JohannesburgHelp build technology that changes lives.About the RoleThe Harambee Software Development team is looking for an exceptional Senior Data Engineer to join our dynamic team based in Johannesburg.This role is focused on collecting, storing, and processing large sets of data to deliver optimal solutions-along with their...

  • Senior Data Engineer

    3 days ago

    Johannesburg, Gauteng, South Africa Hire Resolve Full time R800 000 - R1 200 000 per year

    Hire Resolve's client is looking for a Senior Data Engineer to join their team in Johannesburg, GP. The Data Engineer will be responsible for designing, building, and maintaining scalable data pipelines to support telecommunications CDR processing, real-time data ingestion, and analytical workloads. This role requires expertise in data modeling, ETL...

  • Senior Data Integration Engineer

    5 days ago

    Johannesburg, Gauteng, South Africa Indsafri Full time R2 000 000 - R2 500 000 per year

    Job Description:Job Title: Senior Data Integration Engineer (Salesforce, Databricks & MuleSoft)Location: Johannesburg (Hybrid)Employment Type: ContractContract Tenure: 6 to 12 monthsJob SummaryWe are seeking a highly experienced and strategic Senior Data Integration Engineer to architect, build, and manage the data pipelines that power our customer...

  • Senior Data Engineer

    3 days ago

    Johannesburg, Gauteng, South Africa Hire Resolve Full time R1 200 000 - R2 400 000 per year

    Hire Resolve's client is seeking a highly skilled Senior Data Engineer to join their dynamic team within the telecommunications domain. The ideal candidate will be responsible for designing, developing, and maintaining scalable and reliable data pipelines to support the ingestion, processing, and analysis of Call Detail Records (CDRs) and real-time data...

  • Senior Data Engineer/Analyst

    3 days ago

    Johannesburg, Gauteng, South Africa Boardroom Appointments Full time R1 000 000 - R3 000 000 per year

    Senior Data Engineer/Analyst - 3 Year ContractQualifications & Experience:Must-Have:Bachelors or Masters degree in Computer Science, Data Science, Engineering, Mathematics, or a related field.5+ years of experience in data engineering, analytics, or BI development.Strong proficiency in SQL and Python for data manipulation and transformation.Experience with...

  • Senior Full Stack Data Engineer

    3 days ago

    Johannesburg, Gauteng, South Africa Hire Resolve Full time R120 000 - R260 000 per year

    Hire Resolve is currently seeking a driven and experienced Senior Full Stack Data Engineer to join our dynamic software company. As a Senior Full Stack Data Engineer, you will be responsible for designing, developing, and implementing data solutions to support our business needs. You will work with cross-functional teams to gather requirements, design data...