Data Engineering Lead(Big Data)

4 weeks ago

Johannesburg, South Africa TalentCru Full time

The purpose of this role is to lead the technical delivery team, using experience to deliver right sized and well architected solutions in order to meet targets. The successful candidate will be expected to use their experience working in Azure to understand the available technology choices and selectively implement services safely and securely. As well as demonstrable experience working with the technologies and processes, such as Data Bricks, Python, Unity catalog, DevOps (CI/CD) and Plotly.

The company’s Edge platform has been built, from the ground up, to cater for advanced analytics and AI solutions. As such, the successful candidate will be able to work with leading data Scientists, within the company’s Data Services team or those that work for our strategic partners and clients.

Internal Process

  • Responsible for the end-to-end technical solution. This means that key decisions, once discussed and debated, are appropriately understood, and implemented.
  • Collaborate with the Product Development Lead and the Business Enablement Lead to ensure that all stakeholder expectations are met.
  • Ensure that the company’s Edge platform can cater to all varieties of data, including batch and real-time, and is safe, secure, scalable, and stable.
  • Manage the work items on behalf of the team, breaking down and assigning work items per sprint. Managing elements of sprint planning.
  • DevOps CI/CD pipeline management, control, and oversight.
  • Engage other platform owners and agree on optimal integration paths and methods.
  • Outstanding communication and presentation skills, with the ability to convey complex concepts to non-technical stakeholders.

Client Services

  • Ongoing relationship building with internal and external stakeholders to ensure smooth business processing.
  • Ability to develop new improved processes and to facilitate discussions with external and internal stakeholders.
  • Committed to Client experience.
  • Enable business communications between business areas.
  • Participate and contribute to a culture that builds rewarding relationships, facilitates feedback, and provides exceptional client service.
  • Build trust and creditability with respect to Data in the financial market.

People Management

  • Develop and maintain productive and collaborative working relationships with colleagues, peers, and stakeholders.
  • Continuously develop own expertise in terms of professional, industry, and legislation knowledge.
  • Contribute to continuous innovation through the development, sharing, and implementation of new ideas.
  • Ability to work proactively and independently to deliver precise and accurate work.
  • Take ownership of driving personal career development.
  • Comfortable working with a strictly agile project methodology. This includes breaking down features and user stories for the team of data engineers into appropriately sized deliverables and when required, defining standards, performing code reviews, and validating for quality.
  • Display an inquisitive, detailed-orientated nature, and be unafraid to ask difficult questions, or challenge the status quo when the need arises.
  • Responsible for leveraging financial industry experience to build a modern, secure, scalable solution that is appropriate for the clients and partners that the company services.
  • Makes use of, and develops cost-effective, right-sized solutions and services within Microsoft Azure’s Cloud Platform to meet the core platform needs.
  • Displays consideration of the increasing risks faced as a society and as an organisation and suggests appropriate responses with regard to the solutions being built.

 

Finance

  • Identify solutions to enhance cost -effectiveness and increase operational efficiency.
  • Provide input into the risk identification processes and communicate recommendations in the appropriate forum.
  • Grade 12 or equivalent
  • A bachelor’s degree in computer science or another closely related field
  • 5-8 years of working experience in a related field.
  • Proven work experience as a senior data engineer, or as a Data Engineering Lead
  • Strong Data Bricks and Python skills are a must. SQL skills as an alternative to Python will also be considered
  • Understand how big data solutions / NoSQL solutions differ from standard data solutions and explain why choices made might be different.
  • Mature experience working with cloud services: Azure, AWS, or GCP. This experience will allow the candidate to propose appropriate solutions and alternatives.
  • Strong DevOps understanding, including CI/CD

  • Big Data Product Manager Specialist

    4 days ago

    Johannesburg, South Africa Vodafone Full time

    **Role purpose**: As product Manager for Big Data, you will work with various senior stakeholder in the international business. You be responsible for keeping track of the Big Data and AI agenda for Vodacom International. This will mean Big Data requirements gathering and tracking of the success. **Your responsibilities will include**: Creation of the Big...

  • Big Data Product Owner Specialist

    3 days ago

    Johannesburg, South Africa Vodafone Full time

    **Role purpose**: As product Manager for Big Data, you will work with various senior stakeholder in the international business. You be responsible for keeping track of the Big Data and AI agenda for Vodacom International. This will mean Big Data requirements gathering and tracking of the success. **Your responsibilities will include**: Creation of the Big...

  • Data Engineering

    4 weeks ago

    Johannesburg, South Africa IT Ridge Technologies Full time

    As a Data Engineering & Analytics vertical market lead you will be responsible for providing advisory and thought leadership on the provision of analytics environments leveraging Cloud based platforms, big data technologies, including integration with existing data and analytics platforms and toolsYou will drive pre-sales (proactive, reactive pursuits). Come...

  • Data Engineer

    4 weeks ago

    Johannesburg, South Africa Puma Energy Full time

    Main Purpose: Collaborate with data scientists and business stakeholders to design, develop, and maintain efficient data pipelines feeding into the organization's data lake. Ensure the data lake contains accurate, up-to-date, and high-quality data, enabling data scientists to develop insightful analytics and business stakeholders to make well-informed...

  • Executive Head: Big Data, Loyalty and CVM Technology

    1 week ago

    Johannesburg, South Africa Vodafone Full time

    . When it comes to putting people first, we're number 1.  The number 1 Top Employer in South Africa.  Certified by the Top Employer Institute 2024. Role purpose: The overarching purpose of this role is to lead the Big Data, Business Intelligence, Data & Analytics, CVM, and Loyalty technology strategy, technical solution and functional...

  • Data Center Technician

    7 days ago

    Johannesburg, South Africa Dimension Data Full time

    **Requirements**: - The Data Center Technician (DCT) responsibility is the maintenance of all Data Centre White Space, ensuring a 24/7 service offering within Internet Solutions Data Centers.The DCT provides a service to clients to ensure that their IT infrastructure and systems remain operational through proactively identifying, investigating and resolving...

  • Data Engineer

    4 weeks ago

    Johannesburg, South Africa PRR Recruitment Services Full time

    This well known company is looking for a Data Engineer. will be based in Rivonia The ideal candidate will have strong experience in the following: Data cleaning SQL ExportBig data Sets BSc Degree in Maths and Stats will be required a min of 3 years experience. Stable track record is required. Send your Cv to marinda@prrrecruitment.co.za

  • Data Architect

    4 weeks ago

    Johannesburg, South Africa Dimension Data Full time

    FunctieomschrijvingAre you tired of being micro managed? Do you want to work for a company where the CEO acts like a consultant and has few final decisions to make? We are looking for an experienced Data Architect who can deliver complex data solutions with minimal oversight.As a Data Architect, you'll be responsible for designing and implementing the...

  • Data Culture Lead

    5 days ago

    Johannesburg, South Africa HelloKindred Full time

    Company Description **Who is HelloKindred?** HelloKindred are specialists in staffing marketing, creative and technology roles, offering a range of talent solutions that can be delivered on-site, remotely or hybrid. Our vision is to make work accessible and people’s lives better. We do this by disrupting traditional employment barriers - connecting...

  • Data Center Services Sales Specialist

    20 hours ago

    Johannesburg, South Africa Dimension Data Full time

    **Senior Data Center Services Sales Specialist** **Role description** Dimension data is seeking a senior data centre services sales specialist to join our sales team in Johannesburg. The primary focus of this role is to take our data centre services value proposition to market, retaining existing business and growing market share. As a senior data centre...

  • AWS Data Engineer with DataBricks

    2 weeks ago

    Johannesburg, South Africa Adzuna ZA B C2 Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    4 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    4 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    2 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience.   Duties: Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake. Analyze, re-architect and...

  • Senior Data Engineer

    4 weeks ago

    Johannesburg, South Africa The Individual Touch Full time

    You will take responsibility for managing our master data set, developing reports, and troubleshooting data issues. To do well in this role you need a very fine eye for detail, experience as a data analyst, and deep understanding of the popular data analysis tools and databases: Full job description available for short-listed candidates.Experience/Skills:5+...

  • Data Streaming Engineer

    4 weeks ago

    Johannesburg, South Africa Golden Rule Full time

    The Role An opportunity exists for a Senior Data Streaming Engineer who will Build, enhance, and maintain our real-time data pipeline. The role will work with various infrastructure and operations teams to maintain our data infrastructure. As a senior engineer on the Data Streaming Platform team, you will be responsible for developing and maintaining...

  • Data Engineer

    1 week ago

    Johannesburg, South Africa Adzuna ZA B C2 Full time

    Our client, a leading company within the Financial Services industry, is seeking a highly skilled Senior Data Engineer to join their dynamic team. In this role, you will play a critical part in designing, developing, and maintaining robust data pipelines, ensuring the availability, reliability, and accessibility of our data assets. Your expertise in data...

  • Data Engineer

    4 weeks ago

    Johannesburg, South Africa Network Recruitment Full time

    Our client, a leading company within the Financial Services industry, is seeking a highly skilled Senior Data Engineer to join their dynamic team. In this role, you will play a critical part in designing, developing, and maintaining robust data pipelines, ensuring the availability, reliability, and accessibility of our data assets. Your expertise in data...

  • Data Engineer

    2 weeks ago

    Johannesburg, South Africa Network Recruitment Full time

    Our client, a leading company within the Financial Services industry, is seeking a highly skilled Senior Data Engineer to join their dynamic team. In this role, you will play a critical part in designing, developing, and maintaining robust data pipelines, ensuring the availability, reliability, and accessibility of our data assets. Your expertise in data...

  • Head of Data Engineering

    4 weeks ago

    Johannesburg, South Africa PBT Group Full time

    Define and lead the execution of the data engineering strategy with the intent to significantly evolve the enterprise data warehouse, ensuring seamless data pipelines and maintain data for availability and usability by the insights division. Lead the transformation of the data human capital to de-risk the environment and ensure adaptability to the changing...