Senior Data Engineer

1 week ago

Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time

Our client, based in Durban, is looking for a Senior Data Engineer to join their team of highly proficient and experienced Engineers. They have implemented a hybrid working model with 2 days in the office and 3 days at home. You can also be based either in Durban OR in Somkhele (Richards Bay area).

You will be required to design, plan, setup, configure, and support high-performance data pipelines for scientific compute resources, working closely with the science team. The science team will have the science experience but not the technical experience, and we are looking to bridge that gap. This person will sit within the IT team and provide assistance to the science departments.

Likely candidates may have past experience working at universities or other research and science-related institutions, or some larger corporate type roles that do similar data-heavy computation work.

They will be skilled in many niche technical aspects including databases, but this is not just a DBA only role. They must be skilled on the technical side of building the data pipelines which may involve Cloud and/or hardware and software skills and knowledge.

POSITION INFO :

You will help build and maintain scalable data pipelines and related systems in a research-focused environment. You will be responsible for designing, developing, testing, and deploying data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT, and other stakeholders to ensure data quality, reliability, accessibility, security, and governance, as follows:

  • Design, develop, and maintain end-to-end technical aspects of all data pipelines required to support the research scientists and data managers.
  • Support ETL processes including data ingestion, transformation, validation, and integration processes using various tools and frameworks.
  • Optimize data performance, scalability, and security.
  • Provide technical guidance and support to data analysts and research scientists.
  • Design data integrations and data quality frameworks.
  • Work and collaborate with the rest of the IT department to help develop the strategy for long-term scientific Big Data platform architecture.
  • Document and effectively communicate data processes and solutions.
  • Make use of and help define the right cutting-edge technology, processes, and tools needed to drive technology within our science and research data management departments.

Minimum Qualifications :

  • Bachelor's degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field.
  • Industry-recognized IT related certification and technology qualification such as Databases and Data related certifications.
  • This is a technical role, so a strong focus needs to be on technical skills and experience.

Minimum Experience :

  • 7+ years' experience in Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing.
  • Strong experience with high-performance computing environments including Unix, Docker, Kubernetes, Hadoop, Kafka, Nifi or Spark, or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery, and Azure Synapse Analytics.
  • At least 5 years' advanced experience and very strong proficiency in UNIX, Linux, Windows.
  • Knowledge of various data-related programming, scripting, or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc.

Knowledge and Abilities :

  • Strong experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra, etc.
  • Experience of Big Data technologies such as Hadoop, Spark, and Hive.
  • Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow.
  • Experience working with containerization is advantageous.
  • Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous.
  • Experience working with Big Data Cloud-based (AWS, Azure, etc.) technologies is advantageous.
  • Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous.
  • Strong knowledge of data modeling, architecture, and governance principles.
  • Programming skills in various languages advantageous.
  • Strong data security and compliance experience.
  • Excellent communication, collaboration, and problem-solving skills.
  • Ability to work independently and as part of a cross-functional team.
  • Interest and enthusiasm for scientific research and its applications.
#J-18808-Ljbffr

  • Senior Process Engineer

    3 days ago

    Durban, KwaZulu-Natal, South Africa Gig Engineer Full time

    Gig Engineer is seeking an experienced Senior Process Engineer (Pr. Eng) for a 3-month gig on a part-time (freelance) basis with one of our clients based in Durban, South Africa. The role is ideally suited for an independent professional with a strong municipal water treatment plant design background who can provide an expert-level design review for external...

  • Engineering Project Manager

    1 week ago

    Durban, KwaZulu-Natal, South Africa Gig Engineer Full time

    The Engineering Manager is part of the Engineering team. The purpose of the Engineering Manager role is to effectively ensure that the facilities, layout and machinery used to produce new and existing materials and goods run to their maximum efficiency and output. This includes total preventative maintenance, managing breakdowns of mechanical, electrical and...

  • Senior Process Consultant

    2 days ago

    Durban, KwaZulu-Natal, South Africa Gig Engineer Full time

    Gig Engineer is seeking a Senior Process Consultant to conduct a design review of a municipal water treatment plant. The ideal candidate should be professionally registered and possess extensive experience in process design, control philosophies, and design verification.The selected candidate will assess process design and control philosophies, ensuring...

  • Senior Data Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time

    This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with...

  • Process Optimization Engineer

    2 days ago

    Durban, KwaZulu-Natal, South Africa The Chemical Engineer Full time

    About Our CompanyAt Unilever, we are committed to creating a better business and a better world. We strive to develop innovative solutions that make a difference in people's lives and believe in creating a diverse and inclusive workplace.Job SummaryThis role is perfect for individuals who are passionate about manufacturing excellence and have a strong...

  • Network Engineer

    5 days ago

    Durban, KwaZulu-Natal, South Africa NTT Data Full time

    NTT Data seeks an experienced Cybersecurity and Network Architect to lead our team of engineers in designing, implementing, and maintaining cutting-edge network security solutions.Responsibilities:Develop and maintain robust network security architectures.Liaise with cross-functional teams to identify and address security risks.Evaluate emerging technologies...

  • Production engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa The Chemical Engineer Full time

    ABOUT UNILEVERUnilever is the place where you can bring your purpose to life with the work that you do – creating a better business and a better world. You will work on brands that are loved and improve the lives of our consumers and the communities around us. You will lead innovations, big and small, that will make our business win and grow. You will...

  • Process Design Expert

    2 days ago

    Durban, KwaZulu-Natal, South Africa Gig Engineer Full time

    Gig Engineer is seeking an experienced Senior Process Engineer to review municipal water treatment plant designs for external verification purposes. The ideal candidate should be professionally registered and possess extensive experience in process design, control philosophies, and design verification.The selected candidate will conduct a detailed design...

  • Network Engineer

    5 days ago

    Durban, KwaZulu-Natal, South Africa NTT Data Full time

    The Network Engineer - Data Centre Specialist is responsible for designing, implementing, and maintaining the network infrastructure within data centres. This role ensures optimal performance, security, and reliability of all data centre network components.Responsibilities:Design and deploy data centre network infrastructure including routers, switches,...

  • Municipal Water Treatment Specialist

    2 days ago

    Durban, KwaZulu-Natal, South Africa Gig Engineer Full time

    We are looking for a highly skilled Senior Process Engineer to join our team as a Municipal Water Treatment Specialist. The successful candidate will have extensive experience in process design, control philosophies, and design verification.The role involves conducting a detailed design review of a municipal water treatment plant, providing external...

  • Senior Programs Coordinator

    1 week ago

    Durban, KwaZulu-Natal, South Africa Benchmark Senior Living Full time

    Senior Programs Coordinator - KwaZulu-Natal - Benchmark Senior LivingBenchmark Senior Living KwaZulu-Natal1 day agoDescriptionBenchmark Senior Living seeks a dedicated professional to coordinate senior programs that promote socialization, engagement, and overall well-being.Main Responsibilities:Create and implement activities that foster meaningful...

  • Senior Programs Coordinator

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa Benchmark Senior Living Full time

    Senior Programs Coordinator - Richards Bay - Benchmark Senior LivingBenchmark Senior Living Richards Bay2 days agoJob title: Senior Services AssociateDescriptionBenchmark Senior Living seeks a dedicated professional to coordinate senior programs that promote socialization, engagement, and overall well-being. As a Programming/Activity Assistant, you will be...

  • Cybersecurity and Network Engineer

    1 day ago

    Durban, KwaZulu-Natal, South Africa NTT Data Full time

    Network Engineering Expert">This Data Centre Specialist role is responsible for designing, implementing, and maintaining the network infrastructure within our data centres. As a key member of our IT team, you will be working closely with cross-functional teams to ensure the smooth operation of our data centre network infrastructure.Main...

  • Data Centre Specialist

    1 day ago

    Durban, KwaZulu-Natal, South Africa NTT Data Full time

    Data Centre Specialist is responsible for designing, implementing, and maintaining the network infrastructure within data centres. This role ensures optimal performance, security, and reliability of all data centre network components.Responsibilities:Design and deploy data centre network infrastructure including routers, switches, firewalls, and load...

  • Senior Data Strategist

    4 days ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    Senior Data Strategist Job Description: Goldman Tech Resourcing is a leading mobile and digital solutions company looking for a Senior Data Strategist to join our team.Responsibilities: The ideal candidate will have proven experience leading data teams or projects in open-source or AWS cloud environments, and expertise in data warehousing, ETL pipelines, and...

  • Senior Data Architect

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Carlysle Human Capital. In this role, you will be responsible for designing and developing scalable data pipelines and related systems in a research-focused environment.

  • Data Centre Infrastructure Specialist

    5 days ago

    Durban, KwaZulu-Natal, South Africa NTT Data Full time

    The IT Infrastructure Lead - Networking at NTT Data is responsible for designing, implementing, and maintaining the network infrastructure within data centres.Key Responsibilities:Design and deploy data centre network infrastructure including routers, switches, firewalls, and load balancers.Configure and manage network devices to ensure stable and secure...

  • Senior Services Associate

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa Benchmark Senior Living Full time

    Senior Services Associate - Richards Bay - Benchmark Senior LivingBenchmark Senior Living Richards Bay2 days agoDescriptionActivities AssistantFoster Meaningful Connections among Seniors at Benchmark Senior LivingWe are seeking a compassionate and skilled individual to join our team in KwaZulu-Natal.About the Role:Create and implement activities that promote...

  • Data Engineer

    6 days ago

    Durban, KwaZulu-Natal, South Africa Network Recruitment Full time

    FinTech | Data Engineer | South Africa (Remote/Hybrid) | Python, AWS & PySpark Experience a MustAn innovative FinTech firm is hiring a Data Engineer to drive data infrastructure initiatives and streamline data processes. If you thrive in a fast-paced, cloud-first environment and enjoy designing efficient data systems from scratch, this is your chance to...

  • Senior Data Scientist

    1 week ago

    Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time

    At Carlysle Human Capital, we are seeking a Senior Data Scientist to lead our team of highly skilled engineers in designing, planning, and implementing high-performance data pipelines for scientific compute resources. This role involves working closely with the science team to bridge the gap between technical and scientific expertise.This position requires...