DevOps Data Engineer

6 months ago

Cape Town, South Africa PBT Group Full time

PBT Group has a requirement for an intermediate DevOps Data Engineer.

 

Duties:

  • Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties – Glue, Step-functions, Kafka CC, PySpark, DynamoDB, Delta.io, RedShift, Lambda, DeltaLake, Python,.
  • Analyze, re-architect and re-platform on-premise data warehouses to data platforms on AWS cloud using AWS or 3rd party services and Kafka CC.
  • Design and build production data pipelines from ingestion to consumption within a big data architecture, using Java, PySpark, Scala, Kafka CC.
  • Design and implement data engineering, ingestion and curation functions on AWS cloud using AWS native or custom programming.
  • Perform detail assessments of current state data platforms and create an appropriate transition path to AWS cloud.
  • Design, implement and support an analytical data infrastructure providing ad-hoc access to large datasets and computing power.
  • Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using SQL , AWS big data technologies and Kafka CC.
  • Creation and support of real-time data pipelines built on AWS technologies including Glue, Lambda, Step Functions, PySpark , Athena and Kafka CC
  • Continual research of the latest big data and visualization technologies to provide new capabilities and increase efficiency
  • Working closely with team members to drive real-time model implementations for monitoring and alerting of risk systems.
  • Collaborate with other tech teams to implement advanced analytics algorithms that exploit our rich datasets for statistical analysis, prediction, clustering and machine learning
  • Help continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers

 

Required Skills:

  • Advanced working “data engineering” knowledge and experience working with modern data practices, using Delta.io , CDC management and data load practices.
  • Experience building/operating highly available, distributed systems of data extraction, ingestion, and processing of large data sets
  • Experience working with distributed systems as it pertains to data storage and computing
  • Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
  • Strong analytic skills related to working with unstructured datasets.
  • Build processes supporting data transformation, data structures, meta data, dependency, and workload management.
  • A successful history of manipulating, processing and extracting value from large, disconnected data sets.
  • Working knowledge of message queuing, stream processing, and highly scalable Big Data, data stores.
  • Strong project management and organizational skills.
  • Experience supporting and working with cross-functional teams in a dynamic environment.
  • Experience in a Data Engineer or similar roles
  • Experience with big data tools is a must: Delta.io, PySpark, Kafka, etc.
  • Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
  • Experience with data pipeline and workflow management tools: Step functions , glue workflow etc.
  • Experience with AWS cloud services: EC2, EMR, RDS, Redshift

 

Required Qualifications / Training:

  • Relevant data warehouse and BI solution training is essential.
  • B.Sc. or related degree is advantageous.
  • 5+ years programming experience.

 

* In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

 

  • DevOps Engineer

    6 months ago

    Cape Town, South Africa RTT Full time

    The DevOps engineer is responsible for deploying product updates, identifying production issues, and implementing integrations that meet our customers’ needs. You will work with developers and engineers to ensure that software development follows established processes and works as intended. The DevOps engineer will combines release engineering,...

  • Intermediate DevOps Engineer

    3 weeks ago

    Cape Town, Western Cape, South Africa Black Swan Data Full time

    Job Summary:We are seeking an experienced DevOps Engineer with a strong background in Kubernetes, AWS, and Linux to join our team. As a Cloud Infrastructure Specialist, you will play a critical role in designing, implementing, and maintaining our cloud infrastructure, with a focus on Kubernetes-based container orchestration, AWS services, and Linux...

  • Intermediate DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa Black Swan Data Full time

    Job Summary:We are seeking a highly skilled DevOps Engineer to join our team at Black Swan Data. As a key member of our infrastructure team, you will be responsible for designing, implementing, and maintaining our cloud infrastructure, with a focus on Kubernetes-based container orchestration, AWS services, and Linux systems.Key Responsibilities:Design and...

  • Intermediate DevOps Engineer

    6 months ago

    Cape Town, South Africa Black Swan Data Full time

    Summary: We are seeking an experienced and passionate DevOps Engineer with a strong background in Kubernetes, AWS, and Linux to join our team. As an Intermediate DevOps Engineer, you will play a critical role in designing, implementing, and maintaining our cloud infrastructure, with a focus on Kubernetes-based container orchestration, AWS services, and...

  • Senior DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa Santam Full time

    Job Title: Senior DevOps EngineerSantam's Guidewire BI DataHub team is seeking a Senior DevOps Engineer with a solid Solution Architecture background to support and maintain the BI DataHub technical infrastructure, DevOps processes & procedures.The incumbent will work with the broader BI Team, specifically with Technical Architecture and Release & Deployment...

  • Senior DevOps Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Black Pen Recruitment Full time

    About the RoleBlack Pen Recruitment is seeking an experienced Senior DevOps Engineer to join our client's technology team. As a Senior DevOps Engineer, you will be responsible for designing, implementing, and managing scalable, secure infrastructure across cloud, on-premises, and government environments.ResponsibilitiesDesign and implement reliable,...

  • Senior DevOps Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Sanlam Full time

    Job Title: Senior DevOps EngineerJob Summary:We are seeking a highly skilled Senior DevOps Engineer to join our team at Santam. As a Senior DevOps Engineer, you will play a key role in ensuring the smooth operation of our BI DataHub technical infrastructure.Key Responsibilities:Provide technical support for BI DataHub technology, including hardware,...

  • DevOps Engineer

    2 months ago

    Cape Town, South Africa Endeavour Recruitment Solutions Full time

    Endeavour Recruitment has an opportunity for a DevOps Engineer to join our client’s fast-growing Information Technology and Services company, specializing in the SMME market. This is a permanent role, and we are open to all candidates who are willing to relocate to Cape Town, South Africa, as this work placement is onsite. This is an excellent...

  • DevOps Engineer

    2 months ago

    Cape Town, South Africa Tumaini Consulting Full time

    Our client, a forward-thinking technology company specializing in innovative software development and digital solutions, is currently seeking a DevOps Engineer to join the team. Requirements: · 3-4 years of experience as a DevOps Engineer · Strong knowledge of Agile processes, automated deployments, and source control · Proficiency in coding, scripting,...

  • Senior DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa Black Pen Recruitment Full time

    Job Title: Senior DevOps EngineerOur client, a US-based technology start-up, is seeking an experienced Senior DevOps Engineer to play a critical role in ensuring the seamless operation, scalability, and security of their technology platform. The successful candidate will design, implement, and manage scalable, secure infrastructure across cloud, on-premises,...

  • DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa RTT Full time

    Job DescriptionThe DevOps engineer at RTT is responsible for ensuring the smooth operation of our software development processes. This includes deploying product updates, identifying production issues, and implementing integrations that meet our customers' needs.Key ResponsibilitiesBuilding and implementing various development, testing, automation tools, and...

  • DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa CSG Recruit Full time

    Key ResponsibilitiesAs a DevOps Engineer at CSG Recruit, you will be responsible for managing and maintaining application platforms to ensure their stability and high performance. This includes investigating and documenting best practices, methodologies, and tools for deployment, DevOps, monitoring, release, and configuration management.Requirements3-5 years...

  • DevOps Engineer

    1 month ago

    Cape Town, South Africa zeroG - AI in Aviation Full time

    PayU, a leading payment and Fintech company in 50+ high-growth markets throughout Asia, Central and Eastern Europe, Latin America, the Middle East and Africa, part of Prosus group, one of the largest technology investors in the world is redefining the way people buy and sell online for our 300.000 + merchants and millions of consumers. As a leading online...

  • DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa BeingIT® Full time

    About the RoleWe are seeking a highly skilled DevOps Engineer to join our team at BeingIT®. As a key member of our development team, you will be responsible for managing and overseeing the release process for software applications, ensuring timely and efficient deployments.Key ResponsibilitiesCollaborate with development teams to integrate Continuous...

  • DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa Sci Tech Placements Full time

    Job Title: DevOps EngineerWe are seeking a highly skilled DevOps Engineer to join our team at Sci Tech Placements. As a key member of our infrastructure team, you will be responsible for designing and maintaining scalable, containerized environments for AI model deployment.Key Responsibilities:Design and implement containerized environments for AI model...

  • DevOps Engineer

    1 week ago

    Cape Town, Western Cape, South Africa Being IT Solutions Full time R40 000 - R70 000 per month

    About the RoleWe are seeking an experienced DevOps Engineer to join our team at Being IT Solutions. As a key member of our infrastructure group, you will be responsible for ensuring the smooth operation and maintenance of our software release lifecycle.The successful candidate will have a proven track record of successfully implementing Continuous...

  • DevOps Engineer

    1 week ago

    Cape Town, Western Cape, South Africa Being IT Solutions Full time R40 000 - R70 000 per month

    About the RoleWe are seeking an experienced DevOps Engineer to join our team at Being IT Solutions. As a key member of our infrastructure group, you will be responsible for ensuring the smooth operation and maintenance of our software release lifecycle.The successful candidate will have a proven track record of successfully implementing Continuous...

  • Senior DevOps Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Tumaini Consulting Full time

    DevOps Engineer Job DescriptionOur client, a forward-thinking digital agency, is seeking a DevOps Engineer to join their team. Key requirements include:Knowledge of Agile processesExperience with automated deployments & source controlCoding and scripting abilitiesStrong problem-solving & troubleshooting skillsExcellent documentation skillsFamiliarity with...

  • DevOps Engineer

    6 months ago

    Cape Town, South Africa PayU Full time

    About PayU PayU, a leading payment and Fintech company in 50+ high-growth markets throughout Asia, Central and Eastern Europe, Latin America, the Middle East and Africa, part of Prosus group, one of the largest technology investors in the world is redefining the way people buy and sell online for our + merchants and millions of consumers. As a leading...

  • DevOps Engineer

    2 months ago

    Western Cape, South Africa OfferZen Ltd Full time

    DevOps Engineer Are you a talented DevOps engineer looking for your Dream Developer Job? OfferZen, the developer jobs marketplace, reverses the job search process. We connect developers with a vast network of top tech companies in Africa and Europe that are eager to apply to interview you This could be the last job posting you ever have to click on Why waste...