DevOps Data Engineer

5 months ago

Cape Town, South Africa PBT Group Full time

PBT Group has a requirement for an intermediate DevOps Data Engineer.

 

Duties:

  • Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties – Glue, Step-functions, Kafka CC, PySpark, DynamoDB, Delta.io, RedShift, Lambda, DeltaLake, Python,.
  • Analyze, re-architect and re-platform on-premise data warehouses to data platforms on AWS cloud using AWS or 3rd party services and Kafka CC.
  • Design and build production data pipelines from ingestion to consumption within a big data architecture, using Java, PySpark, Scala, Kafka CC.
  • Design and implement data engineering, ingestion and curation functions on AWS cloud using AWS native or custom programming.
  • Perform detail assessments of current state data platforms and create an appropriate transition path to AWS cloud.
  • Design, implement and support an analytical data infrastructure providing ad-hoc access to large datasets and computing power.
  • Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using SQL , AWS big data technologies and Kafka CC.
  • Creation and support of real-time data pipelines built on AWS technologies including Glue, Lambda, Step Functions, PySpark , Athena and Kafka CC
  • Continual research of the latest big data and visualization technologies to provide new capabilities and increase efficiency
  • Working closely with team members to drive real-time model implementations for monitoring and alerting of risk systems.
  • Collaborate with other tech teams to implement advanced analytics algorithms that exploit our rich datasets for statistical analysis, prediction, clustering and machine learning
  • Help continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers

 

Required Skills:

  • Advanced working “data engineering” knowledge and experience working with modern data practices, using Delta.io , CDC management and data load practices.
  • Experience building/operating highly available, distributed systems of data extraction, ingestion, and processing of large data sets
  • Experience working with distributed systems as it pertains to data storage and computing
  • Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
  • Strong analytic skills related to working with unstructured datasets.
  • Build processes supporting data transformation, data structures, meta data, dependency, and workload management.
  • A successful history of manipulating, processing and extracting value from large, disconnected data sets.
  • Working knowledge of message queuing, stream processing, and highly scalable Big Data, data stores.
  • Strong project management and organizational skills.
  • Experience supporting and working with cross-functional teams in a dynamic environment.
  • Experience in a Data Engineer or similar roles
  • Experience with big data tools is a must: Delta.io, PySpark, Kafka, etc.
  • Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
  • Experience with data pipeline and workflow management tools: Step functions , glue workflow etc.
  • Experience with AWS cloud services: EC2, EMR, RDS, Redshift

 

Required Qualifications / Training:

  • Relevant data warehouse and BI solution training is essential.
  • B.Sc. or related degree is advantageous.
  • 5+ years programming experience.

 

* In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

 

  • DevOps Data Engineer

    1 month ago

    Cape Town, Western Cape, South Africa PBT Group Full time

    PBT Group: Intermediate DevOps Data EngineerWe are seeking an experienced DevOps Data Engineer to join our team at PBT Group. As a key member of our data engineering team, you will be responsible for designing, building, and operationalizing large-scale enterprise data solutions and applications using AWS data and analytics services.Key...

  • Intermediate DevOps Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Black Swan Data Full time

    Job Title: Intermediate DevOps EngineerWe are seeking a highly skilled and experienced DevOps Engineer to join our team at Black Swan Data. As an Intermediate DevOps Engineer, you will play a critical role in designing, implementing, and maintaining our cloud infrastructure, with a focus on Kubernetes, AWS, and Linux.Key Responsibilities:Design and build...

  • Intermediate DevOps Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Black Swan Data Full time

    Job Summary:We are seeking a highly skilled DevOps Engineer with expertise in Kubernetes, AWS, and Linux to join our team at Black Swan Data. As an Intermediate DevOps Engineer, you will play a critical role in designing, implementing, and maintaining our cloud infrastructure, with a focus on Kubernetes-based container orchestration, AWS services, and Linux...

  • DevOps Engineer

    5 months ago

    Cape Town, South Africa RTT Full time

    The DevOps engineer is responsible for deploying product updates, identifying production issues, and implementing integrations that meet our customers’ needs. You will work with developers and engineers to ensure that software development follows established processes and works as intended. The DevOps engineer will combines release engineering,...

  • Intermediate DevOps Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Black Swan Data Full time

    Job Summary:We are seeking a highly skilled DevOps Engineer to join our team at Black Swan Data. As a key member of our infrastructure team, you will be responsible for designing, implementing, and maintaining our cloud infrastructure, with a focus on Kubernetes-based container orchestration, AWS services, and Linux systems.Key Responsibilities:Design and...

  • Intermediate DevOps Engineer

    5 months ago

    Cape Town, South Africa Black Swan Data Full time

    Summary: We are seeking an experienced and passionate DevOps Engineer with a strong background in Kubernetes, AWS, and Linux to join our team. As an Intermediate DevOps Engineer, you will play a critical role in designing, implementing, and maintaining our cloud infrastructure, with a focus on Kubernetes-based container orchestration, AWS services, and...

  • Senior DevOps Engineer

    1 week ago

    Cape Town, Western Cape, South Africa Santam Full time

    Job Title: Senior DevOps EngineerSantam's Guidewire BI DataHub team is seeking a Senior DevOps Engineer with a solid Solution Architecture background to support and maintain the BI DataHub technical infrastructure, DevOps processes & procedures.The incumbent will work with the broader BI Team, specifically with Technical Architecture and Release & Deployment...

  • DevOps Engineer

    3 weeks ago

    Cape Town, Western Cape, South Africa Tumaini Consulting Full time

    DevOps EngineerWe are seeking a highly skilled DevOps Engineer to join our team at Tumaini Consulting. Our client, a forward-thinking technology company, specializes in innovative software development and digital solutions.Key Responsibilities:Design and implement automated deployment processes using Agile methodologies.Develop and maintain high-quality...

  • DevOps Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa Communicate Recruitment Full time

    Job Title: DevOps EngineerWe are seeking a highly skilled DevOps Engineer to join our team at Communicate Recruitment. As a key member of our IT team, you will be responsible for ensuring the smooth operation of our cloud-based infrastructure.Key Responsibilities:Design, implement, and maintain scalable and secure cloud-based systems using AWS and...

  • DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa 4cgroup Full time

    {"title": "DevOps Engineer", "description": "Job SummaryWe are seeking a highly skilled DevOps Engineer to join our team at 4cgroup. As a DevOps Engineer, you will be responsible for building, testing, and maintaining our infrastructure and tools to enable the speedy development and release of software.Main ResponsibilitiesDesign and implement scalable and...

  • Intermediate DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa Black Swan Data Full time

    Cloud Infrastructure EngineerWe are seeking a highly skilled Cloud Infrastructure Engineer to join our team at Black Swan Data. As a key member of our infrastructure team, you will be responsible for designing, implementing, and maintaining our cloud infrastructure, with a focus on Kubernetes-based container orchestration, AWS services, and Linux systems.Key...

  • DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa RTT Full time

    Job DescriptionThe DevOps engineer at RTT is responsible for ensuring the smooth deployment of product updates, identifying and resolving production issues, and implementing integrations that meet our customers' needs.Key ResponsibilitiesDesign and implement various development, testing, and automation tools, as well as IT infrastructure to support software...

  • DevOps Engineer

    1 month ago

    Cape Town, Western Cape, South Africa RTT Full time

    Job DescriptionThe DevOps engineer at RTT is responsible for ensuring the smooth deployment of product updates, identifying and resolving production issues, and implementing integrations that meet our customers' needs.Key ResponsibilitiesDesign and implement various development, testing, and automation tools, as well as IT infrastructure to support software...

  • DevOps Engineer

    3 weeks ago

    Cape Town, Western Cape, South Africa RareCruit Full time

    Job Title: Hybrid DevOps EngineerWe are seeking a highly skilled Hybrid DevOps Engineer to join our team at RareCruit. As a Hybrid DevOps Engineer, you will be responsible for designing, implementing, and maintaining our DevOps environment, ensuring seamless collaboration between development and operations teams.Responsibilities:Enhancements & Bug Fixes:...

  • DevOps Engineer

    3 weeks ago

    Cape Town, South Africa Endeavour Recruitment Solutions Full time

    Endeavour Recruitment has an opportunity for a DevOps Engineer to join our client’s fast-growing Information Technology and Services company, specializing in the SMME market. This is a permanent role, and we are open to all candidates who are willing to relocate to Cape Town, South Africa, as this work placement is onsite. This is an excellent...

  • DevOps Engineer

    1 month ago

    Cape Town, South Africa Tumaini Consulting Full time

    Our client, a forward-thinking technology company specializing in innovative software development and digital solutions, is currently seeking a DevOps Engineer to join the team. Requirements: · 3-4 years of experience as a DevOps Engineer · Strong knowledge of Agile processes, automated deployments, and source control · Proficiency in coding, scripting,...

  • Senior DevOps Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Black Pen Recruitment Full time

    Job Title: Senior DevOps EngineerOur client, a US-based technology start-up, is seeking an experienced Senior DevOps Engineer to play a critical role in ensuring the seamless operation, scalability, and security of their technology platform. The successful candidate will design, implement, and manage scalable, secure infrastructure across cloud, on-premises,...

  • Senior DevOps Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Santam Full time

    Job Title: Senior BI DevOps EngineerSantam's Guidewire BI DataHub team is seeking a highly skilled Senior DevOps Engineer to join their team. As a key member of the team, you will be responsible for supporting and maintaining the BI DataHub technical infrastructure, DevOps processes, and procedures.Key Responsibilities:Technical support of BI Datahub...

  • DevOps Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa RTT Full time

    Job DescriptionThe DevOps engineer at RTT is responsible for ensuring the smooth operation of our software development processes. This includes deploying product updates, identifying production issues, and implementing integrations that meet our customers' needs.Key ResponsibilitiesBuilding and implementing various development, testing, automation tools, and...

  • Senior DevOps Engineer

    3 weeks ago

    Cape Town, Western Cape, South Africa Black Pen Recruitment Full time

    Job Title: Senior DevOps EngineerOur client, a leading technology startup, is seeking an experienced Senior DevOps Engineer to join their team. As a key member of the infrastructure team, you will play a critical role in ensuring the seamless operation, scalability, and security of their technology platform.Key Responsibilities:Design, implement, and manage...