Senior Data Engineer

2 months ago

Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time
Job Title: Head of Data Engineering

We are seeking a highly skilled and experienced Head of Data Engineering to join our team at Carlysle Human Capital. As a key member of our IT department, you will be responsible for designing, developing, and maintaining scalable data pipelines and related systems in a research-focused environment.

Key Responsibilities:
  • Design and develop end-to-end technical aspects of high-performance compute and storage platforms for data pipelines in a research data-intensive environment.
  • Design and develop end-to-end data pipelines, including software, databases, and processes, to support research scientists and data managers.
  • Support ETL processes, including data ingestion, transformation, validation, and integration, using various tools and frameworks.
  • Optimize data performance, scalability, and security.
  • Provide technical guidance and support to data analysts and research scientists.
  • Design data integrations and data quality frameworks.
  • Collaborate with the IT department to develop the strategy for long-term scientific Big Data platform architecture.
  • Document and communicate data engineering processes and solutions effectively.
Requirements:
  • Bachelor's degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field.
  • Industry-recognized IT-related certification and technology qualification, such as databases and data-related certifications.
  • 7+ years of experience in Data Engineering, High Performance Computing, Data Warehousing, or Big Data Processing.
  • Strong experience with technologies such as Hadoop, Kafka, Nifi, or Spark, or Cloud-based Big Data processing environments like Amazon Redshift, Google BigQuery, and Azure Synapse Analytics.
  • At least 5 years of advanced experience and strong proficiency in UNIX, Linux, Windows Operating Systems, and preferably containerization technologies like Docker, Kubernetes, etc.
  • Working knowledge of various data-related programming, scripting, or data engineering tools, such as Python, R, Julia, T-SQL, PowerShell, etc.
  • Strong working experience with software compute and virtualization platforms, such as VMware, Hyper-V, OpenStack, KVM, etc.
  • Strong working experience with hardware compute platforms, including high-performance compute cluster hardware and related technologies.
Preferred Qualifications:
  • Strong experience working with various relational database technologies, such as MS SQL, MySQL, PostgreSQL, as well as NoSQL databases, such as MongoDB, Cassandra, etc.
  • Experience with Big Data technologies, such as Hadoop, Spark, and Hive.
  • Experience with data pipeline tools, such as Airflow, Spark, Kafka, or Dataflow.
  • Experience working with containerization is advantageous.
  • Experience with data quality and testing tools, such as Great Expectations, dbt, or DataGrip, is advantageous.
  • Experience working with Big Data Cloud-based (AWS, Azure, etc.) technologies is advantageous.
  • Experience with data warehouse and data lake technologies, such as BigQuery, Redshift, or Snowflake, is advantageous.
  • Strong experience designing end-to-end data pipelines, including compute hardware infrastructure.
  • Strong knowledge of data modeling, architecture, and governance principles.
  • Strong Linux Administration skills.
  • Programming skills in various languages are advantageous.
  • Strong data security and compliance experience.
  • Excellent communication, collaboration, and problem-solving skills.
  • Ability to work independently and as part of a cross-functional team.
  • Interest and enthusiasm for medical scientific research and its applications.

  • Senior Data Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    Senior Data Engineer PositionWe are seeking a highly skilled Senior Data Engineer to join our team at Goldman Tech Resourcing. The ideal candidate will possess strong proficiency in data extraction, transformation, and loading (ETL) processes.Familiarity with data modeling and schema design for efficient data storage and retrieval is essential.Proficiency in...

  • Senior Mechanical Engineer

    2 months ago

    Durban, KwaZulu-Natal, South Africa Tampa Gardens Senior Living Full time

    Job DescriptionWe are seeking a highly skilled Senior Mechanical Engineer to join our team at Tampa Gardens Senior Living. As a key member of our engineering team, you will be responsible for the effective execution of mechanical engineering projects.Key ResponsibilitiesProject ExecutionDeliver engineered packages that meet project budget, schedule, and...

  • Senior Data Engineer

    4 weeks ago

    Durban, KwaZulu-Natal, South Africa LexisNexis (Proprietary) Limited Company Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at LexisNexis (Proprietary) Limited Company. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining analytical solutions using Amazon Redshift and Tableau.Key ResponsibilitiesDesign and develop scalable data models and analytical...

  • Senior Mechanical Engineer

    2 months ago

    Durban, KwaZulu-Natal, South Africa Tampa Gardens Senior Living Full time

    Senior Mechanical Engineer - Mining and Minerals ProjectsThe Mechanical Engineer will be responsible for the effective compilation of proposals and tenders, as well as the provision of cost-effective, compliant, and safe engineering, design, and construction support during project execution. This will be rendered in the Mining/Minerals and Water market...

  • Senior Data Engineer

    4 weeks ago

    Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time

    Job DescriptionThis role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research-focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals.Key...

  • Senior Data Engineer

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa OfferZen Ltd Full time

    About the RoleData Engineer at OfferZen Ltd: We are seeking a talented Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems. You will work closely with our data analysts and scientists to ensure that our data is accurate, reliable, and easily accessible.Key...

  • Senior Data Engineer I

    1 month ago

    Durban, KwaZulu-Natal, South Africa LexisNexis (Proprietary) Limited Company Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at LexisNexis (Proprietary) Limited Company. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining analytical solutions using Amazon Redshift and Tableau.Key ResponsibilitiesDesign and develop scalable data models and analytical...

  • Senior Data Architect

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    We are seeking a Senior Data Architect to lead our data engineering team. The successful candidate will have strong proficiency in data extraction, transformation, and loading processes, and will be responsible for designing efficient data storage and retrieval systems. Key Responsibilities:Design and implement data architecture solutionsDevelop data models...

  • Senior Data Engineer for Advanced Scientific Computing

    4 days ago

    Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Carlysle Human Capital. As a key member of our IT department, you will be responsible for designing and developing scalable data pipelines and related systems in a research-focused environment.Job DescriptionDesign and develop end-to-end technical aspects of...

  • Senior Data Engineering Expert

    1 week ago

    Durban, KwaZulu-Natal, South Africa LexisNexis (Proprietary) Limited Company Full time

    About the RoleAs a Senior Data Engineer with LexisNexis (Proprietary) Limited Company, you will be responsible for designing, developing, and maintaining analytical solutions using Amazon Redshift and Tableau. You will apply your expertise in data modeling, SQL querying, data analysis, and data presentation techniques to drive business insights and...

  • Data Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa LexisNexis (Proprietary) Limited Company Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at LexisNexis (Proprietary) Limited Company. This role will be responsible for designing, developing, and maintaining analytical solutions using Amazon Redshift and Tableau.Responsibilities• Design and develop scalable data models and analytical solutions using Amazon...

  • Senior Cloud Data Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa OfferZen Ltd Full time

    Job Description:Are you looking for a challenging role as a Data Engineer? OfferZen Ltd is seeking a talented professional with experience in handling large data sets and working with big data tools such as Hadoop, Spark, and Kafka. Key Responsibilities:Design and develop data pipelines and workflow management toolsWork with SQL and NoSQL...

  • Lead Data Engineering Specialist

    5 days ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    Job DescriptionWe are seeking a highly skilled Senior Data Engineer to join our team at Goldman Tech Resourcing.About the Role:The ideal candidate will have strong proficiency with data extraction, transformation, and loading (ETL) processes. They should be familiar with data modeling and schema design for efficient data storage and retrieval. Good...

  • MS Engineer

    1 month ago

    Durban, KwaZulu-Natal, South Africa NTT DATA Full time

    About the RoleWe are seeking a highly skilled MS Engineer to join our team at NTT DATA. As a key member of our team, you will be responsible for providing a managed service to clients to ensure that their IT infrastructure and systems remain operational.Key ResponsibilitiesMonitor client infrastructure and solutions to identify potential issues and...

  • Senior Geotechnical Engineer

    4 weeks ago

    Durban, KwaZulu-Natal, South Africa Hire Resolve Full time

    Job SummaryHire Resolve is seeking a highly skilled Senior Geotechnical Engineer to join a reputable construction company. As the Senior Geotechnical Engineer, you will oversee geotechnical engineering aspects of various construction projects while ensuring compliance with all safety and regulatory requirements.Key Responsibilities:Manage geotechnical...

  • Principal Data Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    We are seeking a highly skilled Principal Data Engineer to lead our data engineering team at Goldman Tech Resourcing. The ideal candidate will have a strong background in data extraction, transformation, and loading (ETL) processes, as well as familiarity with data modeling and schema design for efficient data storage and retrieval. Proficiency in...

  • Data Engineer

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa OfferZen Ltd Full time

    About the Role:We are seeking a skilled Data Engineer to join our team at OfferZen Ltd. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems.Key Responsibilities:Design and implement data pipelines using big data tools such as Hadoop, Spark, and Kafka.Develop and maintain SQL and NoSQL databases.Work...

  • Head of Data Engineering

    1 month ago

    Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time

    Job Title: Head of Data EngineeringWe are seeking a highly skilled and experienced Head of Data Engineering to join our team at Carlysle Human Capital. As a key member of our IT department, you will be responsible for designing, developing, and maintaining scalable data pipelines and related systems in a research-focused environment.Key...

  • Server Engineer

    4 weeks ago

    Durban, KwaZulu-Natal, South Africa NTT DATA Full time

    Job DescriptionMake an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.Your day at NTT...

  • Managed Services Engineer

    1 week ago

    Durban, KwaZulu-Natal, South Africa NTT DATA Full time

    About the RoleWe are seeking a talented Managed Services Engineer to join our team at NTT DATA.Job DescriptionThe Managed Services Cross Technology Engineer (L1) is an entry-level engineering role, responsible for providing a managed service to clients to ensure their IT infrastructure and systems remain operational.Your Key ResponsibilitiesMonitor client...