Head of Data Engineering

3 months ago

Durban, South Africa Carlysle Human Capital Full time
This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance. This is technical role (not just a DBA or Data management role) that needs a high degree of core IT technical background and skills
  • Design, develop, and maintain the end-to-end technical aspects (hardware - on prem and/or cloud) of a high performance compute and storage platform designed specifically for data pipelines in a research data intensive environment
  • Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers.
  • Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks
  • Optimize data performance, scalability, and security
  • Provide technical guidance and support to data analysts and research scientists.
  • Design data integrations and data quality frameworks
  • Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture
  • Document and effectively communicate data engineering processes and solutions.
  • Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments.
Minimum Qualifications:
  1. Bachelor's degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field
  2. Industry recognized IT related certification and technology qualification such as Databases and Data related certifications.
  3. This is a technical role so a strong focus needs to be on technical skills and experience
Minimum Experience:
  1. 7+ years experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing
  2. Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics
  3. At least 5 years advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems and preferably containerization technologies like Docker, Kubernetes etc
  4. Working knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc.
  5. Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM etc.
  6. Strong working experience with hardware compute platforms including high performance compute cluster hardware and related technologies.
Knowledge and Abilities:
  • Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc.
  • Experience of Big Data technologies such as Hadoop, Spark and Hive
  • Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow
  • Experience working with containerization is advantageous
  • Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous
  • Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous
  • Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous
  • Strong Experience designing end-to-end data pipelines including the compute hardware infrastructure.
  • Strong knowledge of data modeling, architecture, and governance principles
  • Strong Linux Administration skills
  • Programming skills in various languages advantageous
  • Strong data security and compliancy experience
  • Excellent communication, collaboration, and problem-solving skills
  • Ability to work independently and as part of a cross-functional team
  • Interest and enthusiasm for medical scientific research and its applications
SUMMARY:

There is large emphasis on the technical element in the role (having experience working with and designing hardware clusters from the ground up). They have a number of technical skills and experience already within the IT department but are looking for someone very strong, and with the past working experience, to help design these clusters and understand the technology in play for advanced scientific computational requirements.

This goes beyond just the hardware element though and you will need strong Linux, Data Pipeline and coding/scripting skills. Although this is not a developer role, you should be comfortable with a certain level of coding/scripting and data analysis packages with strong virtualization skills like VMware, HyperV, OpenStack, KVM etc.

You will therefore be that technical link with the scientists and will help build platforms (could be hardware, software, cloud etc) and then also manage the Data and Data Pipelines on this system. This would include the compliancy, performance and security of the data too.

  • Site Management Specialist

    4 weeks ago

    Durban, KwaZulu-Natal, South Africa Hutech Head Office Full time

    Job OverviewHutech Head Office seeks an experienced Site Supervisor to lead our operations in Durban, KwaZulu-Natal. As a key member of our team, you will be responsible for overseeing daily activities, managing production data, and ensuring optimal fuel stock and boiler house metrics.Key Responsibilities:Capture and analyze production data to inform...

  • Head of Department It

    7 months ago

    Durban, South Africa Richfield Graduate Institute of Technology Full time

    The Head of Department is an academic position whose main objective is to provide academic leadership through leading, managing and developing the department to ensure it achieves the highest possible standards of academic excellence. The Head of Department reports to the Dean of Faculty of Information Technology **Minimum Requirements**: - A Masters or...

  • Head of CSO

    6 months ago

    Durban South, South Africa Dimension Data Full time

    FunctieomschrijvingDo you want to make a real difference in your career? Do you want to progress within one of the largest security companies with a strong focus on innovation and new technologies? Do you want to work for an employer that values top-notch talent? Apply for this Head of CSO role!Responsible for high-level relationship management with clients....

  • Senior Data Engineer

    1 week ago

    Durban, South Africa Old Mutual Full time

    Job title : Senior Data EngineerJob Location : KwaZulu-Natal, DurbanDeadline : December 28, 2024Quick Recommended Links Jobs by Location Job by industries Job Description Senior Data Engineer will lead the development and optimization of our data infrastructure, playing a critical role in shaping the future of our data architecture. They will be...

  • Senior Data Engineer

    7 days ago

    Durban, South Africa Old Mutual Full time

    Job title : Senior Data Engineer Job Location : KwaZulu-Natal, Durban Deadline : December 28, 2024 Quick Recommended Links Jobs by Location Job by industries Job Description Senior Data Engineer will lead the development and optimization of our data infrastructure, playing a critical role in shaping the future of our data architecture....

  • Data Centre Operations

    6 months ago

    Durban South, South Africa Dimension Data Full time

    FunctieomschrijvingData Centre Operations (KZN)LOCATIONDurban, South AfricaTYPEPermanentDo you want greater control over your own career? We believe this Data Centre Operations role was created for you! Please read through and apply!Data Centre Operations (DCO) person: Data Centre - manages the operational day to day tasks onsite and assists the Site...

  • Managed Services Engineer

    1 month ago

    Durban, KwaZulu-Natal, South Africa NTT DATA Full time

    About the RoleWe are seeking a talented Managed Services Engineer to join our team at NTT DATA.Job DescriptionThe Managed Services Cross Technology Engineer (L1) is an entry-level engineering role, responsible for providing a managed service to clients to ensure their IT infrastructure and systems remain operational.Your Key ResponsibilitiesMonitor client...

  • Network Engineer

    2 weeks ago

    Durban, KwaZulu-Natal, South Africa NTT Data Full time

    NTT Data: A Leader in Digital TransformationWe are seeking a highly skilled Network Engineer to join our team at NTT Data. This is an exceptional opportunity for a motivated and experienced professional to take their career to the next level.About the Role:This position is responsible for providing advanced infrastructure support for our network...

  • Data Analyst

    6 months ago

    Durban, South Africa Zeal HR Full time

    **Job Title**:Data Analyst **Introduction**: Objectives of the Role Ensure that DMA, RNG and GET tasks allocated for testing are assessed and reported on in accordance with the approved methodologies, working documents and templates and in line with regulatory, accreditation and contractual requirements. Primary Responsibilities - Conduct Game Engine...

  • Senior Data Analyst Manager

    6 months ago

    Durban, South Africa Dina Gates Recruitment on behalf of Mining Materials Company Full time

    We are recruiting for a full-time Senior Data Analyst position at one of our retail bank clients with head office in Gauteng. **KEY REQUIREMENTS**: - **A minimum of 8 years** of experience in **data analysis, database management, and data manipulation**: - Experience with large datasets and statistical and data modeling techniques - **In-depth knowledge...

  • Managed Services Engineer

    1 month ago

    Durban, South Africa NTT DATA Full time

    Job Description Make an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it’s a place where you can grow, belong and thrive.Your day at NTT...

  • Data Analyst

    6 months ago

    Durban, South Africa Zeal HR Full time

    **Job Title**:Data Analyst - Coding **Introduction**: Objectives of the Role - Ensure that DMA, RNG and GET tasks allocated for testing are assessed and reported on in accordance with the approved methodologies, working documents and templates and in line with regulatory, accreditation and contractual requirements. - Ensure data generation and conversion...

  • Head of Fresh Foods Operations

    4 days ago

    Durban, KwaZulu-Natal, South Africa Shoprite Group of Companies Full time

    Job Title: Head of Fresh Foods OperationsWe are seeking an experienced and skilled professional to fill the role of Head of Fresh Foods Operations at the Shoprite Group of Companies.About the Role:Leads and manages the daily operations of the fresh foods department, ensuring efficient and effective management of stock, logistics, and staff.Develops and...

  • Data Officer

    7 months ago

    Durban, South Africa Games Global Full time

    We are looking for a Data Officer who would be is responsible for providing assistance and support to the Head of Data Governance, focusing on records management and acting as the deputy data protection officer you will maintain and update all documents as required, undertake the drafting of reports, conduct research into privacy related matters and...

  • Senior Data Recovery Engineer

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa beBee Professionals Full time

    BeBee Professionals is a leading organization in the field of data recovery, with a presence in Durban. We are seeking a highly skilled Senior Data Recovery Engineer to join our team.Job OverviewThe successful candidate will be responsible for recovering data from damaged or corrupted storage media, including hard drives, flash drives, and other digital...

  • Data Engineering Expert for Innovative Solutions

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa Goldman Tech Resourcing Full time

    At Goldman Tech Resourcing, we are on the lookout for a skilled Senior Data Engineer to spearhead our data engineering efforts. As a key member of our team, you will be responsible for designing and implementing cutting-edge data pipelines that drive business growth and customer engagement.We are seeking an experienced professional with a strong background...

  • Data Engineer for Advanced Scientific Computing

    3 weeks ago

    Durban, KwaZulu-Natal, South Africa Carlysle Human Capital Full time

    We are Carlysle Human Capital, a leading provider of human capital solutions. We have an exciting opportunity for a highly skilled Data Engineer to join our team.About the RoleThis is a technical role that requires a strong focus on designing and developing scalable data pipelines and related systems in a research-focused environment. You will be responsible...

  • Head of Data Management

    5 months ago

    Durban, South Africa CAPRISA Careers Full time

    CAPRISA is an award-winning global research organisation located on the campus of the Nelson Mandela School of Medicine in Durban, South Africa and undertakes research in HIV, TB and Covid at four clinical research sites in KwaZulu-Natal. CAPRISA’s goal is to undertake globally relevant and locally responsive research that contributes to understanding HIV...

  • Network and Security Engineer

    6 months ago

    Durban South, South Africa Dimension Data Full time

    FunctieomschrijvingNetwork and Security EngineerLOCATIONDurban, South AfricaTYPEPermanentDo you want to learn and grow on your own? Do you have a passion for networking and security but you don't feel like being micro managed by management? Apply for this Network and Security Engineer role!MS Engineer (L2) is responsible for providing a managed service to...

  • Head of Social

    1 week ago

    Durban, South Africa Gorilla Full time

    “That is awesome!” If that short sentence is the response you like to hear when people see your work, then please read on - you may be the Head of Social our Jungle is looking for. Are you awesome at these head of social things? Social media strategy development Define and lead Gorilla’s overarching social media strategy, ensuring alignment with client...