Data Engineer

1 month ago

Midrand, South Africa Quarphix Full time
Experience: Minimum of 5 years of experience, preferably in a financial background as a Data Engineer.
Education: Bachelors degree in Information Technology, Computer Science, Software Development, Engineering, or a related field.
Job Description:

Key Accountabilities:

  • Source data from internal and external sources, engaging with technical subject matter experts.
  • Explore, analyze, and profile data from various internal and external sources, assisting data scientists in preparing data for analytical purposes.
  • Ensure delivered solutions meet Systems Integration and User Acceptance Testing criteria.
  • Productionalize solutions and ensure daily data refresh processes run successfully.
Data Architecture & Data Engineering:
  • Understand the technical landscape and bank-wide architecture connected to or dependent on the supported business area.
  • Translate data architecture direction and associated business requirements, leveraging analytical and creative problem-solving skills to synthesize data solution designs.
  • Participate in design thinking processes to deliver data solution blueprints.
  • Utilize state-of-the-art relational and No-SQL databases, integration, and streaming platforms to deliver sustainable business-specific data solutions.
  • Design data retrieval, storage, and distribution solutions, contributing to all phases of the development lifecycle.
  • Develop high-quality data processing, retrieval, storage, and distribution designs in a test-driven and domain-driven/cross-domain environment.
  • Build analytics tools utilizing the data pipeline, producing well-organized, optimized, and documented source code and algorithms.
  • Create and maintain sophisticated CI/CD pipelines, supporting deployment to multi-site environments.
  • Automate tasks through appropriate tools and scripting technologies (e.g., Ansible, Chef).
  • Debug existing source code and polish feature sets.
  • Assemble large, complex data sets meeting business requirements and manage the data pipeline.
  • Build infrastructure to automate extremely high volumes of data delivery.
  • Create data tools for analytics and data science teams, optimizing data sets for business benefit.
  • Ensure designs and solutions align with technical organization principles (self-service, repeatability, testability, scalability, and resilience).
  • Apply general design patterns and paradigms to deliver technical solutions.
  • Support the continuous optimization, improvement, and automation of data processing, retrieval, storage, and distribution processes.
  • Ensure quality assurance and testing of all data solutions align with QA Engineering and architectural guidelines.
  • Implement and align with Group Security standards to ensure the undisputable separation, security, and quality of the organization's data.
  • Contribute to and ensure solutions align with the design and direction of Group Architecture, adhering to data standards, principles, preferences, and practices.
  • Contribute to and ensure solutions align with the design and direction of Group Infrastructure standards and practices (e.g., OLAs, IAAS, PAAS, SAAS, Containerization).
  • Monitor the performance of data solution designs and ensure ongoing optimization of data solutions.
  • Stay updated on data processing, retrieval, storage, and distribution technologies and processes (global best practices and trends) to ensure best practices.
People:
  • Coach and mentor other engineers.
  • Conduct peer reviews, testing, and problem-solving within and across the broader team.
  • Build data science team capability in the use of data solutions.
Risk & Governance:
  • Identify technical risks and mitigate these (pre, during & post deployment).
  • Update/Design all application documentation aligned to organizational technical standards and risk/governance frameworks.
  • Create business cases and solution specifications for various governance processes (e.g., CTO approvals).
  • Participate in incident management and DR activity, applying critical thinking, problem-solving, and technical expertise to resolve major incidents.
  • Deliver projects on time and within budget.

  • F5 Engineer

    1 month ago

    Midrand, South Africa Dimension Data Full time

    FunctieomschrijvingOur client requires an F5 Engineer who will oversee, manage, and coordinate network policy. They assist in the configuration and troubleshooting of F5 load balancers. L2 or L3 The engineer must manage the load balancing network and application requirements. They collaborate and coordinate with internal support teams and give timely...

  • AWS Data Engineer

    9 hours ago

    Midrand, South Africa Liyema Consulting Full time

    ESSENTIAL SKILLS REQUIREMENTS: - Technical Skills / Technology (in order of importance): - Terraform - Python 3.x - SQL - Oracle/PostgreSQL - PySpark - Boto3 - ETL - Docker - Linux / Unix - Big Data - Powershell / Bash - Cloud Data Hub (CDH) - CDEC Blueprint - Any additional responsibilities assigned in the Agile Working Model (AWM)...

  • AWS Data Engineer

    13 hours ago

    Midrand, South Africa Liyema Consulting Full time

    ESSENTIAL SKILLS REQUIREMENTS:- Technical Skills / Technology (in order of importance): - Terraform - Python 3.x - SQL - Oracle/PostgreSQL - PySpark - Boto3 - ETL - Docker - Linux / Unix - Big Data - Powershell / Bash - Cloud Data Hub (CDH) - CDEC Blueprint - Any additional responsibilities assigned in the Agile Working Model (AWM)...

  • AWS Data Engineer

    2 hours ago

    Midrand, South Africa Adzuna ZA B C2 Full time

    ESSENTIAL SKILLS REQUIREMENTS:- Technical Skills / Technology (in order of importance): - Terraform - Python 3.x - SQL - Oracle/PostgreSQL - PySpark - Boto3 - ETL - Docker - Linux / Unix - Big Data - Powershell / Bash - Cloud Data Hub (CDH) - CDEC Blueprint - Any additional responsibilities assigned in the Agile Working Model (AWM) CharterADVANTAGEOUS SKILLS...

  • AWS Data Engineer EXP 0675

    3 weeks ago

    Midrand, South Africa Open Source (Pty) Ltd Full time

    ESSENTIAL SKILLS REQUIREMENTS: Above average experience/understanding (in order of importance):e.g. Technical Skills / TechnologyTerraform Python 3x SQL - Oracle/PostgreSQL Py Spark Boto3 ETL Docker Linux / Unix Big Data Powershell / Bash BMW Cloud Data Hub (CDH) BMW CDEC Blueprint · Experience in working with Enterprise Collaboration tools such...

  • AWS Data Engineer EXP 0675

    1 month ago

    Midrand, South Africa Open Source (Pty) Ltd Full time

    ESSENTIAL SKILLS REQUIREMENTS:Above average experience/understanding (in order of importance):e.g. Technical Skills / TechnologyTerraformPython 3xSQL - Oracle/PostgreSQLPy SparkBoto3ETLDockerLinux / UnixBig DataPowershell / BashBMW Cloud Data Hub (CDH)BMW CDEC Blueprint · Experience in working with Enterprise Collaboration tools such as Confluence,...

  • Senior Specialist Core Data Engineering

    1 month ago

    Midrand, South Africa Vodafone Full time

    **Role purpose**: To research, Engineer, Design, build, test, deploy, evolve, and form a technology strategy for packet core, including automation and product design. **Your responsibilities will include**: All key accountabilities below are with engagement with all CVDi departments. The role cannot operate individually Design PS Core technical policy and...

  • Data Architect/Data Lead

    3 weeks ago

    Midrand, South Africa LINKFIELDS INNOVATIONS (PTY) LTD Full time

    We are seeking a skilled Data Architect to join our team and take ownership of our organization's data architecture. The ideal candidate will have a strong background in designing, implementing, and managing complex data systems. They will work closely with our engineering, data science, and business teams to understand requirements, design solutions, and...

  • Data Architect/Data Lead

    3 weeks ago

    Midrand, South Africa LINKFIELDS INNOVATIONS (PTY) LTD Full time

    We are seeking a skilled Data Architect to join our team and take ownership of our organization's data architecture. The ideal candidate will have a strong background in designing, implementing, and managing complex data systems. They will work closely with our engineering, data science, and business teams to understand requirements, design solutions, and...

  • Data Architect/Data Lead

    3 weeks ago

    Midrand, South Africa LINKFIELDS INNOVATIONS (PTY) LTD Full time

    We are seeking a skilled Data Architect to join our team and take ownership of our organization's data architecture. The ideal candidate will have a strong background in designing, implementing, and managing complex data systems. They will work closely with our engineering, data science, and business teams to understand requirements, design solutions, and...

  • Data Architect/Data Lead

    2 weeks ago

    Midrand, South Africa Adzuna ZA B C2 Full time

    We are seeking a skilled Data Architect to join our team and take ownership of our organization's data architecture. The ideal candidate will have a strong background in designing, implementing, and managing complex data systems. They will work closely with our engineering, data science, and business teams to understand requirements, design solutions,...

  • Software Engineer

    1 month ago

    Midrand, South Africa Lim Personnel Full time

    Software Engineer / Developer (Cloud Data Platform) (Expert)ESSENTIAL SKILLS REQUIREMENTS:Languages:Python application development skillsTechnologies:Experience in working with AWS services (Lambda, EC2, IAM, S3, Step Functions, Fargate, Batch, Code Pipeline)Familiar with Terraform and Infrastructure-as-Code conceptsWorking Mode / Mindset:Agile Development...

  • Data Streaming Platform Engineer

    1 month ago

    Midrand, South Africa Liyema Consulting Full time

    Role Title: Data Streaming Platform Engineer (Advanced)Essential Skills Requirements:- Sound knowledge in Kotlin, Java, JavaScript, Python, or C#- Experience with Microservices, Cloud Architectures, and Container Architectures (e.g., Docker)- Proficiency in developing solutions on public cloud platforms like Microsoft Azure and AWS- Familiarity with...

  • Data Streaming Platform Engineer

    3 weeks ago

    Midrand, South Africa Liyema Consulting Full time

    Role Title: Data Streaming Platform Engineer (Advanced) Essential Skills Requirements: - Sound knowledge in Kotlin, Java, JavaScript, Python, or C#- Experience with Microservices, Cloud Architectures, and Container Architectures (e.g., Docker)- Proficiency in developing solutions on public cloud platforms like Microsoft Azure and AWS- Familiarity with...

  • Data Streaming Platform Engineer

    1 month ago

    Midrand, South Africa Adzuna ZA B C2 Full time

    Role Title: Data Streaming Platform Engineer (Advanced)Essential Skills Requirements:- Sound knowledge in Kotlin, Java, JavaScript, Python, or C#- Experience with Microservices, Cloud Architectures, and Container Architectures (e.g., Docker)- Proficiency in developing solutions on public cloud platforms like Microsoft Azure and AWS- Familiarity with...

  • AWS Data Engineers Required

    2 weeks ago

    Midrand, South Africa HR Genie Full time

    Job Title: AWS Data Engineers Required - Project Based - Contractual-to start asap-JHB - HybridLocation: Gauteng, Johannesburg Metro, MidrandJob Type: Contract, expected to work Full-Time hoursPrimary Industry: Banking and FinanceSecondary Industry: Banking and FinanceJob Duties:1. Design, develop, and implement AWS cloud solutions for data engineering...

  • AWS Data Engineers Required

    2 weeks ago

    Midrand, South Africa Africonology Full time

    Job Title: AWS Data Engineers Required - Project Based - Contractual-to start asap-JHB - HybridLocation: Gauteng, Johannesburg Metro, MidrandJob Type: Contract, expected to work Full-Time hoursPrimary Industry: Banking and FinanceSecondary Industry: Banking and FinanceJob Duties:1. Design, develop, and implement AWS cloud solutions for data engineering...

  • AWS Data Engineers Required

    2 weeks ago

    Midrand, South Africa Adzuna ZA B C2 Full time

    Job Title: AWS Data Engineers Required - Project Based - Contractual-to start asap-JHB - HybridLocation: Gauteng, Johannesburg Metro, MidrandJob Type: Contract, expected to work Full-Time hoursPrimary Industry: Banking and FinanceSecondary Industry: Banking and FinanceJob Duties:1. Design, develop, and implement AWS cloud solutions for data engineering...

  • Worskhop Data Capturer

    4 weeks ago

    Midrand, South Africa RJPersonnel Full time

    2years - Data capturing within the automotive industry - Data capturing, of motor vehicles, diesel, petrol, pumps etc. parts, and types - Capturing truck parts and types - Spare parts database listings - Stock analysis - Warehouse stock support **Qualifications and Experience**: - Matric / Grade 12 - Diploma in IT Engineering or IT packages or...

  • Data Culture Lead

    4 weeks ago

    Midrand, South Africa Liyema Consulting Full time

    **Key Responsibilities**: - Partner with client (product and marketing) teams to understand business problems and marketing strategies as well as engaging with IT to unearth trends in behavior. - Translate open-ended business problems into analytical hypotheses and test the hypotheses leveraging various data sources. - Work through the phases of a data...