Data Engineering Lead(Big Data)

1 day ago


Johannesburg, South Africa Adcorp Holdings Full time

**Roles and Responsibilities** Internal Process**
- Responsible for the end-to-end technical solution. This means that key decisions, once discussed and debated, are appropriately understood, and implemented.
- Collaborate with the Product Development Lead and the Business Enablement Lead to ensure that all stakeholder expectations are met.
- Ensure that the company’s Edge platform can cater to all varieties of data, including batch and real-time, and is safe, secure, scalable, and stable.
- Manage the work items on behalf of the team, breaking down and assigning work items per sprint. Managing elements of sprint planning.
- DevOps CI/CD pipeline management, control, and oversight.
- Engage other platform owners and agree on optimal integration paths and methods.
- Outstanding communication and presentation skills, with the ability to convey complex concepts to non-technical stakeholders.

**Client Services**
- Ongoing relationship building with internal and external stakeholders to ensure smooth business processing.
- Ability to develop new improved processes and to facilitate discussions with external and internal stakeholders.
- Committed to Client experience.
- Enable business communications between business areas.
- Participate and contribute to a culture that builds rewarding relationships, facilitates feedback, and provides exceptional client service.
- Build trust and creditability with respect to Data in the financial market.

**People Management**
- Develop and maintain productive and collaborative working relationships with colleagues, peers, and stakeholders.
- Continuously develop own expertise in terms of professional, industry, and legislation knowledge.
- Contribute to continuous innovation through the development, sharing, and implementation of new ideas.
- Ability to work proactively and independently to deliver precise and accurate work.
- Take ownership of driving personal career development.
- Comfortable working with a strictly agile project methodology. This includes breaking down features and user stories for the team of data engineers into appropriately sized deliverables and when required, defining standards, performing code reviews, and validating for quality.
- Display an inquisitive, detailed-orientated nature, and be unafraid to ask difficult questions, or challenge the status quo when the need arises.
- Responsible for leveraging financial industry experience to build a modern, secure, scalable solution that is appropriate for the clients and partners that the company services.
- Makes use of, and develops cost-effective, right-sized solutions and services within Microsoft Azure’s Cloud Platform to meet the core platform needs.
- Displays consideration of the increasing risks faced as a society and as an organisation and suggests appropriate responses with regard to the solutions being built.

**Roles and Responsibilities (cont.)** Finance**
- Identify solutions to enhance cost -effectiveness and increase operational efficiency.
- Provide input into the risk identification processes and communicate recommendations in the appropriate forum.

**Job Requirements**
- Grade 12 or equivalent
- A bachelor’s degree in computer science or another closely related field
- 5-8 years of working experience in a related field.
- Proven work experience as a senior data engineer, or as a Data Engineering Lead
- Strong Data Bricks and Python skills are a must. SQL skills as an alternative to Python will also be considered
- Understand how big data solutions / NoSQL solutions differ from standard data solutions and explain why choices made might be different.
- Strong DevOps understanding, including CI/CD

Job Reference: TCN01680


  • Data Scientist

    2 days ago


    Johannesburg, South Africa Data Centrix Full time

    Inherent RequirementsDegree (Honours, Masters or PHD) in quantitative fields such as Actuarial Science , Statistics, Computer Science, Engineering, Mathematics and / or a combination of these.Professional certifications in data science and related technologies (e.g. Python, Azure, AWS, Spark, big data, machine learning, cloud infrastructure.A minimum of 5...


  • Johannesburg, Gauteng, South Africa E-Merge Full time

    E-Merge seeks a Big Data Engineer to collect, process, and analyse large datasets. Key responsibilities include:Main Duties:Developing data collection strategies and executing themPinpointing areas for optimisation and delivering recommendationsBuilding software applications for data analysis and automationThe ideal candidate should have a Bachelor's degree...


  • Johannesburg South, South Africa Network Recruitment Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team and contribute to the development of scalable data pipelines.Key Responsibilities and RequirementsDesign, develop, and maintain scalable data pipelines to process large volumes of structured and unstructured data.Collaborate with Data Scientists, Analysts, and other teams to...

  • Big Data Architect

    2 days ago


    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for a Big Data Architect. In this position you have the opportunity to help lead the change on operational strategy and your responsibilities will include to validate the reference architecture and provide details for the architecture of platforms, leading and working hands on towards implementation and delivery to production of...

  • Big Data Developer

    5 days ago


    Johannesburg, Gauteng, South Africa Kaelo Full time

    We are looking for a skilled Big Data Developer to join our team at Kaelo. As a Big Data Developer, you will be responsible for designing, developing, and deploying large-scale data processing systems using big data technologies.In this role, you will collaborate with data analysts, scientists, and other stakeholders to translate business requirements into...

  • Data Specialist

    5 days ago


    Johannesburg, South Africa Dimension Data Full time

    **Role Description** The IT Data Specialist applies data analysis, data modelling and quality assurance techniques based on a detailed understanding of business processes to establish, modify or maintain internal data structures and associated components (entity, relationship and attribute descriptions). They develop and maintain of a formal description of...

  • Big Data Engineer

    2 days ago


    Johannesburg, Gauteng, South Africa Psybergate Full time

    About the RoleWe are looking for an experienced Data Engineer to join our team at Psybergate. The ideal candidate will have extensive experience in designing, developing, and maintaining data infrastructure.Main ResponsibilitiesDesigning, building, and maintaining scalable data pipelines using Ab Initio.Developing ETL processes to extract, transform, and...


  • Johannesburg, Gauteng, South Africa Liyema Consulting Full time

    Job Summary:We are seeking an experienced Data Architecture Strategist to lead the design, implementation, and evolution of enterprise-level data architectures. This role will involve architecting big data platforms and systems for scalability, performance, and reliability, as well as enabling advanced analytics, statistical modeling, and machine learning to...


  • Johannesburg, South Africa Data Centrix Full time

    Requirements - Minimum 5 years in a data-centric environment working in either SQL Data-Warehousing, - Analytics or as a dedicated Data-Modeller. - Above average TSQL and SQL server skills. - Logical, and Physical Data-Modelling Expertise. - Experience with Forward and Reverse Engineering of Existing Databases. - Good understanding of Star Schemas, Snowflake...


  • Johannesburg, South Africa e-Merge IT Recruitment Full time

    This industry leading insurance company is urgently seeking a Data Governance Lead to join their ranks. The main goal of this position is to implement a Data Governance Operating Framework for the organisation, emphasizing the enhancement of data governance and data quality by adjusting organizational behaviour, policies, standards, principles, governance...

  • AWS Data Engineer

    11 hours ago


    Johannesburg, South Africa PBT Group Full time

    PBT Group is currently offering an opportunity for an AWS Data Engineer with 2 to 5 years of relevant experience.   The role of a Data Engineer involves constructing and maintaining data pipelines and datamarts, emphasizing scalability, repeatability, and security. Data Engineers play a pivotal role in facilitating the acquisition of data from diverse...


  • Johannesburg, South Africa Dimension Data Full time

    As a Financial Data Analyst, you'll analyse data and develop analytical models to provide financial reporting, forecasts and risk management. You will have experience in either an analyst or senior role within the finance department of a large organisation. You should also be fluent in SQL and familiar with R programming language. **Requirements - Acquire...


  • Johannesburg, Gauteng, South Africa Liyema Consulting Full time

    Company OverviewLiyema Consulting is a pioneering firm dedicated to building a Centre of Excellence in data architecture and analytics.Job SummaryWe are seeking an experienced Data Architect Lead to lead the design, implementation, and evolution of enterprise-level data architectures.Key ResponsibilitiesArchitect big data platforms and systems for...


  • Johannesburg, South Africa NTT DATA Full time

    **Make an impact with NTT DATA** Join a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion - it’s a place where you can grow, belong and thrive. **Your day at NTT DATA** This...

  • AWS Data Engineer

    14 hours ago


    Johannesburg, South Africa PBT Group Full time

    PBT Group is currently offering an opportunity for an AWS Data Engineer with 2 to 5 years of relevant experience. The role of a Data Engineer involves constructing and maintaining data pipelines and datamarts, emphasizing scalability, repeatability, and security. Data Engineers play a pivotal role in facilitating the acquisition of data from diverse...


  • Johannesburg, Gauteng, South Africa E-Merge Full time

    Job DescriptionE-Merge is a leading company in the health industry, seeking a highly skilled Hadoop Administrator to join our winning team. The successful candidate will have a strong background in Linux and Kubernetes experience.About the RoleThe ideal candidate will possess knowledge of various technical platforms, operating systems, and processes, with a...


  • Johannesburg South, South Africa Network Recruitment Full time

    We are seeking a highly skilled Data Engineer to join our team at Network Recruitment.About the RoleThis is an exciting opportunity for a motivated and experienced Data Engineer to work with our clients to build and maintain scalable data pipelines. The ideal candidate will have a strong background in designing, developing, and maintaining complex data...


  • Johannesburg, South Africa NTT DATA Full time

    **Make an impact with NTT DATA** Join a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion - it’s a place where you can grow, belong and thrive. **Your day at NTT DATA** The...


  • Johannesburg, Gauteng, South Africa E-Merge Full time

    E-Merge is a leading innovator in the tech industry, dedicated to providing cutting-edge solutions that drive business success globally.The current need is for a Senior Data Engineer with a passion and love for Azure Cloud. There is so much to explore here.As a key team member, you will work with top industry players, analysing and organising raw data;...

  • Data Engineer

    3 days ago


    Johannesburg, South Africa Jobted ZA C2 Full time

    What you will be doing - The ideal candidate will have extensive experience in data engineering, particularly with Ab Initio, and will be responsible for designing, developing, and maintaining the data infrastructure. - Design, build, and maintain scalable data pipelines using Ab Initio. - Develop ETL processes to extract, transform, and load data from...