Senior Data Engineer

3 days ago

Durban, KwaZulu-Natal, South Africa Carlysle Full time R120 000 - R180 000 per year
Our client, based in Durban, is looking for a strategic and architectural level Engineering Specialist in this newly created role.

The main purpose of this role is to build from scratch their big data/bioinformatics solutions (including all hardware and software aspects). This will be a senior management role with decision-making responsibility. A key aspect of this role will be to define the strategy and then to design and roll out the entire infrastructure.

This role will suit someone from a Scientific institution or a university that has worked with scientific research solutions.

They have a implemented a hybrid working model with 2 days in the office and 3 days at home. You can also be based either in Durban OR in Somkhele (Richards Bay area).

You will be required to design, plan, setup, configure, support high performance data pipelines for scientific compute resources, working closely with the science team. The science team will have the science experience but not the technical experience and they are looking to bridge that gap. This person will sit within the IT team and provide assistance to the science departments.
 
Likely candidates will have past experience working at universities or other research and science related institutions, or some larger corporate type roles that do similar data heavy computation work.
 
They will be skilled in many niche technical aspects including databases, but this is not just a DBA only role. They must be skilled on the technical side of building the data pipelines which may involve Cloud and/or hardware and software skills and knowledge.

This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance. This is technical role (not just a DBA or Data management role) that needs a high degree of core IT technical background and skills
  • Design, develop, and maintain the end-to-end technical aspects (hardware - on prem and/or cloud) of a high performance compute and storage platform designed specifically for data pipelines in a research data intensive environment
  • Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers.
  • Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks
  • Optimize data performance, scalability, and security
  • Provide technical guidance and support to data analysts and research scientists.
  • Design data integrations and data quality frameworks
  • Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture
  • Document and effectively communicate data engineering processes and solutions.
  • Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments.
Minimum Qualifications:

  1. Bachelor's degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field
  2. Industry recognized IT related certification and technology qualification such as Databases and Data related certifications.
  3. This is a technical role so a strong focus needs to be on technical skills and experience
Minimum Experience:

  1. 7+ years' experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing
  2. Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics
  3. At least 5 years' advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems and preferably containerization technologies like Docker, Kubernetes etc
  4. Working knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc.
  5. Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM etc.
  6. Strong working experience with hardware compute platforms including high performance compute cluster hardware and related technologies.
Knowledge and Abilities:
  • Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc.
  • Experience of Big Data technologies such as Hadoop, Spark and Hive
  • Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow
  • Experience working with containerization is advantageous
  • Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous
  • Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous
  • Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous
  • Strong Experience designing end-to-end data pipelines including the compute hardware infrastructure.
  • Strong knowledge of data modeling, architecture, and governance principles
  • Strong Linux Administration skills
  • Programming skills in various languages advantageous
  • Strong data security and compliancy experience
  • Excellent communication, collaboration, and problem-solving skills
  • Ability to work independently and as part of a cross-functional team
  • Interest and enthusiasm for medical scientific research and its applications
SUMMARY:

There is large emphasis on the technical element in the role (having experience working with and designing hardware clusters from the ground up). They have a number of technical skills and experience already within the IT department but are looking for someone very strong, and with the past working experience, to help design these clusters and understand the technology in play for advanced scientific computational requirements. 

This goes beyond just the hardware element though and you will need strong Linux, Data Pipeline and coding/scripting skills. Although this is not a developer role, you should be comfortable with a certain level of coding/scripting and data analysis packages with strong virtualization skills like VMware, HyperV, OpenStack, KVM etc.

You will therefore be that technical link with the scientists and will help build platforms (could be hardware, software, cloud etc) and then also manage the Data and Data Pipelines on this system. This would include the compliancy, performance and security of the data too.

  • Senior Data Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa LexisNexis Full time R900 000 - R1 200 000 per year

    About the BusinessLexisNexis Legal & Professional provides legal, regulatory, and business information and analytics that help customers increase their productivity, improve decision-making, achieve better outcomes, and advance the rule of law around the world. As a digital pioneer, the company was the first to bring legal and business information online...

  • Senior BI and Data Engineer

    5 days ago

    Durban, KwaZulu-Natal, South Africa Pedros Full time R1 200 000 - R2 400 000 per year

    Job DescriptionThe Senior BI & Data Engineer will establish and lead the foundation of Pedros' Business Intelligence (BI) capability.This hands-on role is responsible for designing and implementing the data architecture, pipelines, and reporting systems that will enable accurate, scalable, and timely insights across the business.The incumbent will act as the...

  • Senior Data Engineer I

    3 days ago

    Durban, KwaZulu-Natal, South Africa LexisNexis Full time R1 000 000 - R2 500 000 per year

    About the BusinessLexisNexis Legal & Professional provides legal, regulatory, and business information and analytics that help customers increase their productivity, improve decision-making, achieve better outcomes, and advance the rule of law around the world. As a digital pioneer, the company was the first to bring legal and business information online...

  • Junior Data Engineer

    1 week ago

    Durban, KwaZulu-Natal, South Africa Old Mutual South Africa Full time R250 000 - R500 000 per year

    Let's Write Africa's Story TogetherOld Mutual is a firm believer in the African opportunity and our diverse talent reflects this.Job DescriptionAs a Junior Data Engineer, you will be responsible for supporting the design, development, and maintenance of our data infrastructure. You will work closely with the data engineering team to implement data pipelines,...

  • Junior Data Engineer

    1 week ago

    Durban, KwaZulu-Natal, South Africa Old Mutual Limited Full time R250 000 - R400 000 per year

    Let's Write Africa's Story TogetherOld Mutual is a firm believer in the African opportunity and our diverse talent reflects this. Job DescriptionAs a Junior Data Engineer, you will be responsible for supporting the design, development, and maintenance of our data infrastructure. You will work closely with the data engineering team to implement data...

  • Senior Engineer

    3 days ago

    Durban, KwaZulu-Natal, South Africa Hire Resolve Full time R900 000 - R1 200 000 per year

    Hire Resolves client is looking for a Senior Engineer to join their team in KZN. The compnay is committed to guiding your journey towards a sustainable future, using innovation and analytics to develop and implement advanced energy and emissions management solutions. We use technology as an enabler to help you use less energy and resources, and as a...

  • Senior Engineer

    1 week ago

    Durban, KwaZulu-Natal, South Africa AECOM Full time R900 000 - R1 200 000 per year

    Company DescriptionAECOM's Enterprise Capabilities team is looking for a Senior Engineer - Water to join our team in our South Africa offices.About Enterprise CapabilitiesFundamental to our Think and Act Globally strategy and digital adoption, Enterprise Capabilities (EC) is AECOM's primary vehicle for workshare. Operating across geographic boundaries and...

  • Senior Engineer

    1 week ago

    Durban, KwaZulu-Natal, South Africa AECOM Full time R500 000 - R1 200 000 per year

    Company DescriptionAECOM's Enterprise Capabilities team is looking for a Senior Engineer - Water to join our team in our South Africa offices.About Enterprise Capabilities Fundamental to our Think and Act Globally strategy and digital adoption, Enterprise Capabilities (EC) is AECOM's primary vehicle for workshare. Operating across geographic boundaries and...

  • Senior Engineer

    1 week ago

    Durban, KwaZulu-Natal, South Africa AECOM Full time R600 000 - R1 200 000 per year

    Company Description AECOM's Enterprise Capabilities team is looking for a Senior Engineer - Water to join our team in our South Africa offices.About Enterprise Capabilities Fundamental to our Think and Act Globally strategy and digital adoption, Enterprise Capabilities (EC) is AECOM's primary vehicle for workshare. Operating across geographic boundaries and...

  • Senior IT Systems Engineer

    3 days ago

    Durban, KwaZulu-Natal, South Africa HR Talent Partner Full time R900 000 - R1 200 000 per year

    Job OverviewApplication Deadline: 10 December 2025Job Location: Durban, Kwazulu NatalJob Title: Senior IT Systems EngineerEducation Level: Bachelors DegreeJob Level: SeniorMinimum Experience: 5- 7 YearsSenior IT Systems Engineer required in Durban.A well-known national Software Development and Media company is looking for a Senior IT Systems Engineer to join...