Data Engineer

3 weeks ago

Pretoria, South Africa Mi-C3 International Ltd Full time

As a Data Integration Engineer, you will be responsible for designing, implementing, and maintaining data integration solutions to handle real‑time streaming data from various sources like IoT / IIoT protocols, third‑party APIs, or even raw files. Your main objective will be to process data in real‑time and provide valuable insights for our organization. You will work with a diverse range of Big Data tools and technologies. The successful candidate will have experience in embedded systems bring‑up requirements engineering management, systems integration, developing programs to drive HW and SW planning and articulating the big picture. Additionally, you will be involved in the development of a Data Streaming platform using Nifi. Responsibilities Data Integration Design: Collaborate with cross‑functional teams to understand data requirements, source systems, and data formats, and design efficient data integration pipelines for real‑time data streaming from multiple sources. Programming Languages: Develop custom data processing components and applications using Java and Python to meet specific business requirements. ETL Development: Implement Extract, Transform, Load (ETL) processes to ingest and transform data from various streaming sources into a format suitable for analysis and storage. Real‑time Data Processing: Develop and optimize data processing workflows to ensure timely handling of streaming data, maintaining low‑latency and high‑throughput capabilities. Big Data Tools: Utilize and maintain various Big Data tools such as Apache NiFi, Spark, Kafka, etc. to build scalable and robust data integration solutions. Message Broker Configuration: Set up and configure message brokers like RabbitMQ, AMQP, and Kafka to enable efficient data exchange between different systems and applications. IoT / IIoT Protocols Integration: Integrate and work with IoT / IIoT protocols such as MQTT, SNMP, CoAP, TCP, and WebSockets to capture data from edge devices and industrial systems. Data Quality and Validation: Implement data validation checks and data quality measures to ensure the accuracy and reliability of the integrated data. Performance Monitoring: Monitor the performance and health of data integration pipelines, making necessary adjustments to optimize data flow and resource utilization. Troubleshooting and Issue Resolution: Diagnose and resolve issues related to data integration, ensuring smooth and uninterrupted data streaming. Technical Requirements Strong experience in designing and implementing data integration solutions for real‑time streaming data. Proficiency in using Big Data technologies such as Apache NiFi, Apache Spark and Kafka. Familiarity with message brokers like RabbitMQ, AMQP, and Kafka for data exchange and event‑driven architectures. Hands‑on experience with IoT / IIoT protocols such as MQTT, SNMP, CoAP, TCP, and WebSockets. Proficiency in programming languages such as Java and Python for developing custom data processing components. Knowledge of data quality assurance and validation techniques to ensure reliable data. Ability to troubleshoot and resolve issues related to data integration and streaming processes. Strong analytical and problem‑solving skills, with a keen eye for detail. Excellent communication and teamwork skills to collaborate effectively with cross‑functional teams. Experience with cloud‑based platforms and distributed systems is advantageous. Have a never ending curious mindset to learn and work with new tools and technologies. #J-18808-Ljbffr

  • Data Engineer

    14 hours ago

    Pretoria, South Africa It Ridge Technologies Full time

    Role overview : We are looking for an experienced Data Engineer to design and manage a resilient data pipeline that effectively processes and integrates both structured and unstructured data into data lakes, warehouses, databases, and other data sources. In this role, you will thrive in an agile setting, working closely with ICT teams, data scientists, and...

  • Data Engineer

    2 weeks ago

    Pretoria, South Africa Kamo Placements Cc Full time

    Reference : JHB G-1Data Engineer Duties & Responsibilities JOB DESCRIPTION Data Engineering & Pipeline ManagementDesign, build, and optimize T-SQL stored procedures, functions, and scripts for high-volume data processing and ECM scoring.Develop, deploy, and monitor end-to-end ETL / ELT workflows (e.g., SQL Server Agent, SSIS, Azure Data Factory, or Airflow)...

  • Data Engineer

    2 weeks ago

    Pretoria, South Africa Bizmod Full time

    A leading financial institution is seeking a Senior Data Engineer / Analyst to join a multi‑year IT transformation programme aimed at renewing and modernising its financial surveillance systems. Key Responsibilities Define and optimise business processes and value chains relevant to the Financial Surveillance function. Identify, analyse, and improve...

  • Data Engineer

    1 week ago

    Pretoria, Gauteng, South Africa Interfront SOC Ltd. Full time R500 000 - R1 200 000 per year

    Key Responsibilities:Data Engineering & Pipeline ManagementDesign, build, and optimize T-SQL stored procedures, functions, and scripts for high-volume data processing and ECM scoring.Develop, deploy, and monitor end-to-end ETL/ELT workflows (e.g., SQL Server Agent, SSIS, Azure Data Factory, or Airflow) with checkpoint/rollback, job tracking, and recovery...

  • Data Engineer

    5 days ago

    Pretoria, Gauteng, South Africa Boardroom Appointments Full time R600 000 - R1 200 000 per year

    This roles responsibility is to design, develop and maintain data-based solutions including ensuring that the operationalization of data pipelines and data stores are high-performing, efficient, organized, and reliable, given a set of business requirements and constraints.The Data Engineer will build and maintain secure and compliant data processing...

  • Data Engineer

    1 week ago

    Pretoria, Gauteng, South Africa Interfront Full time R800 000 - R1 200 000 per year

    Job Reference Number: Recruit198-199Department: TaxBusiness Unit:Industry: Information TechnologyJob Type: ContractPositions Available: 2Salary: Market RelatedPerform data cleansing, preparation & analysis to support business intelligence & machine learning. Design, build & maintain robust data pipelines, feature stores & scheduled workflows that power the...

  • Data Engineer

    5 days ago

    Pretoria, Gauteng, South Africa Boardroom Appointments Full time R600 000 - R1 200 000 per year

    REQUIRED MINIMUM EDUCATION/TRAINING:BCom or Bachelors degree in Information Technology / Computer Science / Engineering (IT) or similarData Engineering tools (Azure Data Engineer) certification will be an added advantageITIL or COBIT certification will be an added advantageREQUIRED MINIMUM WORK EXPERIENCE:2 - 4 years of working experience in an IT...

  • Data Engineer

    2 weeks ago

    Pretoria, Gauteng, South Africa Mediro ICT Full time R900 000 - R1 200 000 per year

    Design, develop, and maintain scalable data pipelines to ingest, process, and store structured and unstructured data from multiple sources.Develop ETL/ELT processes to transform raw data into clean, analysable formats for reporting, analytics and dashboards.Work with cloud data platformsEnsure data security and complianceIntegrate and consolidate data from...

  • Data Process Engineer

    3 weeks ago

    Pretoria, South Africa Bicrm Full time

    Our Johannesburg Office is Hiring 24 months renewable contract At BICRM, we are at the forefront of innovation in the financial services industry. We are seeking a Data / Process Engineer to join our team and support the Financial Surveillance department at a leading bank. This role involves conducting business process modelling and engineering to enhance...

  • Senior Data Engineer

    3 weeks ago

    Pretoria, South Africa Moyo Full time

    WELCOME TO MOYO Senior Data Engineer We are searching for an experienced Senior Data Engineer to join us in driving significance together. Primary Duties and Responsibilities In this consulting role, you will play a key role in understanding our clients' business objectives and translating them into robust, scalable data solutions. You will lead the...