Senior Data Engineer

4 weeks ago

Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time
Job Description

This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research-focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals.

Key Responsibilities
  • Design, develop, and maintain the end-to-end technical aspects (hardware - on prem and/or cloud) of a high-performance compute and storage platform designed specifically for data pipelines in a research data-intensive environment
  • Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers.
  • Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks
  • Optimize data performance, scalability, and security
  • Provide technical guidance and support to data analysts and research scientists.
  • Design data integrations and data quality frameworks
  • Work and collaborate with the rest of the IT department to help develop the strategy for long-term scientific Big Data platform architecture
  • Document and effectively communicate data engineering processes and solutions.
  • Make use of and help define the right cutting-edge technology, processes, and tools needed to drive technology within our science and research data management departments.
Requirements
  • Bachelor's degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field
  • Industry-recognized IT-related certification and technology qualification such as Databases and Data-related certifications.
  • This is a technical role, so a strong focus needs to be on technical skills and experience
Experience
  • 7+ years of experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing
  • Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery, and Azure Synapse Analytics
  • At least 5 years of advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems, and preferably containerization technologies like Docker, Kubernetes, etc.
  • Working knowledge of various data-related programming, scripting, or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc.
  • Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM, etc.
  • Strong working experience with hardware compute platforms including high-performance compute cluster hardware and related technologies.
Knowledge and Abilities
  • Strong experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL, as well as NoSQL databases such as MongoDB, Cassandra, etc.
  • Experience of Big Data technologies such as Hadoop, Spark, and Hive
  • Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow
  • Experience working with containerization is advantageous
  • Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous
  • Experience working with Big Data Cloud-based (AWS, Azure, etc.) technologies is advantageous
  • Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake is advantageous
  • Strong experience designing end-to-end data pipelines including the compute hardware infrastructure.
  • Strong knowledge of data modeling, architecture, and governance principles
  • Strong Linux Administration skills
  • Programming skills in various languages are advantageous
  • Strong data security and compliance experience
  • Excellent communication, collaboration, and problem-solving skills
  • Ability to work independently and as part of a cross-functional team
  • Interest and enthusiasm for medical scientific research and its applications

  • Senior Data Engineer

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    Senior Data Engineer PositionWe are seeking a highly skilled Senior Data Engineer to join our team at Goldman Tech Resourcing. The ideal candidate will possess strong proficiency in data extraction, transformation, and loading (ETL) processes.Familiarity with data modeling and schema design for efficient data storage and retrieval is essential.Proficiency in...

  • Senior Data Engineer

    4 weeks ago

    Durban, KwaZulu-Natal, South Africa LexisNexis (Proprietary) Limited Company Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at LexisNexis (Proprietary) Limited Company. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining analytical solutions using Amazon Redshift and Tableau.Key ResponsibilitiesDesign and develop scalable data models and analytical...

  • Senior Data Engineer

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa OfferZen Ltd Full time

    About the RoleData Engineer at OfferZen Ltd: We are seeking a talented Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems. You will work closely with our data analysts and scientists to ensure that our data is accurate, reliable, and easily accessible.Key...

  • Senior Data Architect

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    We are seeking a Senior Data Architect to lead our data engineering team. The successful candidate will have strong proficiency in data extraction, transformation, and loading processes, and will be responsible for designing efficient data storage and retrieval systems. Key Responsibilities:Design and implement data architecture solutionsDevelop data models...

  • Senior Data Engineer for Advanced Scientific Computing

    7 days ago

    Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Carlysle Human Capital. As a key member of our IT department, you will be responsible for designing and developing scalable data pipelines and related systems in a research-focused environment.Job DescriptionDesign and develop end-to-end technical aspects of...

  • Senior Data Engineering Expert

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa LexisNexis (Proprietary) Limited Company Full time

    About the RoleAs a Senior Data Engineer with LexisNexis (Proprietary) Limited Company, you will be responsible for designing, developing, and maintaining analytical solutions using Amazon Redshift and Tableau. You will apply your expertise in data modeling, SQL querying, data analysis, and data presentation techniques to drive business insights and...

  • Data Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa LexisNexis (Proprietary) Limited Company Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at LexisNexis (Proprietary) Limited Company. This role will be responsible for designing, developing, and maintaining analytical solutions using Amazon Redshift and Tableau.Responsibilities• Design and develop scalable data models and analytical solutions using Amazon...

  • Senior Cloud Data Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa OfferZen Ltd Full time

    Job Description:Are you looking for a challenging role as a Data Engineer? OfferZen Ltd is seeking a talented professional with experience in handling large data sets and working with big data tools such as Hadoop, Spark, and Kafka. Key Responsibilities:Design and develop data pipelines and workflow management toolsWork with SQL and NoSQL...

  • Lead Data Engineering Specialist

    1 week ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    Job DescriptionWe are seeking a highly skilled Senior Data Engineer to join our team at Goldman Tech Resourcing.About the Role:The ideal candidate will have strong proficiency with data extraction, transformation, and loading (ETL) processes. They should be familiar with data modeling and schema design for efficient data storage and retrieval. Good...

  • Senior Geotechnical Engineer

    4 weeks ago

    Durban, KwaZulu-Natal, South Africa Hire Resolve Full time

    Job SummaryHire Resolve is seeking a highly skilled Senior Geotechnical Engineer to join a reputable construction company. As the Senior Geotechnical Engineer, you will oversee geotechnical engineering aspects of various construction projects while ensuring compliance with all safety and regulatory requirements.Key Responsibilities:Manage geotechnical...

  • Principal Data Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    We are seeking a highly skilled Principal Data Engineer to lead our data engineering team at Goldman Tech Resourcing. The ideal candidate will have a strong background in data extraction, transformation, and loading (ETL) processes, as well as familiarity with data modeling and schema design for efficient data storage and retrieval. Proficiency in...

  • Data Engineer

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa OfferZen Ltd Full time

    About the Role:We are seeking a skilled Data Engineer to join our team at OfferZen Ltd. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems.Key Responsibilities:Design and implement data pipelines using big data tools such as Hadoop, Spark, and Kafka.Develop and maintain SQL and NoSQL databases.Work...

  • Server Engineer

    4 weeks ago

    Durban, KwaZulu-Natal, South Africa NTT DATA Full time

    Job DescriptionMake an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.Your day at NTT...

  • Managed Services Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa NTT DATA Full time

    About the RoleWe are seeking a talented Managed Services Engineer to join our team at NTT DATA.Job DescriptionThe Managed Services Cross Technology Engineer (L1) is an entry-level engineering role, responsible for providing a managed service to clients to ensure their IT infrastructure and systems remain operational.Your Key ResponsibilitiesMonitor client...

  • Senior Data Architect

    1 week ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    We are seeking a highly skilled Data Engineer to join our team at Goldman Tech Resourcing. The ideal candidate will have strong proficiency with ETL processes, data modeling, and schema design for efficient data storage and retrieval.A strong background in Python, Scala, or Java programming languages is required. The candidate should be able to work with...

  • Cloud Infrastructure Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa NTT DATA Full time

    Job OverviewMake a difference with NTT DATANavigate a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.Your role at NTT...

  • MS Engineer

    4 weeks ago

    Durban, KwaZulu-Natal, South Africa NTT DATA Full time

    Job OverviewMake a difference with NTT DATAWe are a global leader in business and technology services, committed to helping clients innovate, optimize, and transform for long-term success. Our team is passionate about delivering exceptional results and making a positive impact on our clients and society.Your roleThe Managed Services Cross Technology Engineer...

  • Senior Structural Engineer

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa Tumaini Consulting Full time

    Job Title: Senior Structural EngineerJob Summary:We are seeking an experienced Senior Structural Engineer to join our team at Tumaini Consulting. As a Senior Structural Engineer, you will be responsible for leading the structural department and overseeing the design of reinforced concrete, steel, and timber structures.Responsibilities:Lead the structural...

  • Senior Project Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa Talented Recruitment Full time

    Job Title: Senior Project EngineerTalented Recruitment is seeking a highly skilled Senior Project Engineer to join our team. As a Senior Project Engineer, you will be responsible for leading engineering duties in planning, designing, and construction management projects.Key Responsibilities:Obtaining or maintaining professional status in accordance with...

  • Senior Business Analyst

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa Data Centrix Full time

    **Job Title:** Senior Business AnalystCritical RequirementsHold a Matric or Grade 12 qualificationPossess an IT qualification, such as a certification, diploma, or degreeHold a BA certification, preferably a FTI DiplomaExperienceMinimum 3-5 years of experience as a Business Analyst in projectsOpenText product experience is an advantageCompetenciesEffective...