AWS Data Engineer

2 weeks ago

Cape Town, Western Cape, South Africa PBT Group Full time

PBT Group is currently offering an opportunity for an AWS Data Engineer with 4+ years of relevant experience.

The role of a Data Engineer involves constructing and maintaining data pipelines and datamarts, emphasizing scalability, repeatability, and security. Data Engineers play a pivotal role in facilitating the acquisition of data from diverse sources, ensuring its conformity to data quality standards, and enabling downstream users to access data promptly. This position is an integral part of an agile team.

These professionals are entrusted with the responsibility of establishing the infrastructure required to derive insights from raw data, integrating data from various sources seamlessly. They empower solutions by efficiently managing substantial volumes of data, both in batch and real-time, utilizing cutting-edge technologies from the realms of big data and cloud computing. Additional responsibilities encompass the development of proof-of-concepts and the implementation of intricate big data solutions, with a primary focus on collecting, parsing, managing, analyzing, and visualizing extensive datasets. They are adept at employing technologies to resolve challenges associated with handling vast amounts of data in diverse formats, thereby delivering innovative solutions.

Data Engineering is a technically demanding role that necessitates a broad spectrum of expertise in software development and programming. These professionals possess knowledge in data analysis, understanding end-user and business requirements, and have the ability to translate these needs into technical solutions. They exhibit a strong grasp of physical database design and the systems development lifecycle. Collaboration within a team environment is essential for success in this role.

Key Responsibilities:

  • Architecting Data analytics framework.
  • Translating complex functional and technical requirements into detailed architecture, design, and high-performance software.
  • Leading the development of data and batch/real-time analytical solutions by leveraging transformative technologies.
  • Engaging in multiple projects as a technical lead, overseeing user story analysis, design, software development, testing, and automation tool creation.

Duties: Primary Job Objectives:

  • Development and Operations
  • Database Development and Operations
  • Establishment and Adherence to Policies, Standards, and Procedures
  • Communication
  • Business Continuity and Disaster Recovery Planning
  • Research and Evaluation
  • Coaching and Mentoring

Required Skills, Knowledge, and Experience:

  • A minimum of 5 years of experience in Data Engineering or Software Engineering.
  • Demonstrated leadership experience, managing teams of engineers for 3-5 years.
  • A minimum of 2 years of experience in Big Data.
  • At least 5 years of experience with Extract, Transform, and Load (ETL) processes.
  • A minimum of 2 years of experience with AWS (Amazon Web Services).
  • Demonstrated experience with agile or other rapid application development methodologies for at least 2 years (e.g., Agile, Kanban, Scrum).
  • 5 years of proven expertise in object-oriented design, coding, testing patterns, and working with commercial or open-source software platforms and large-scale data infrastructures.
  • Proficiency in creating data feeds from on-premise to AWS Cloud (2 years).
  • Support experience for data feeds in production on a break-fix basis (2 years).
  • A minimum of 4 years of experience in creating data marts using Talend or similar ETL development tools.
  • Proficiency in data manipulation using Python and PySpark (2 years).
  • Experience in processing data using the Hadoop paradigm, particularly with EMR, AWS's distribution of Hadoop (2 years).
  • DevOps experience in Big Data and Business Intelligence, including automated testing and deployment (2 years).
  • Extensive knowledge of various programming or scripting languages.
  • Expertise in data modeling and an understanding of different data structures and their suitability for specific use cases.

Additional Technical Skills Required:

  • The ability to design highly scalable distributed systems using various open-source tools.
  • Proficiency in both batch and streaming Big Data tools.
  • Experience with Talend for at least 1 year.
  • Familiarity with AWS services such as EMR, EC2, and S3 for at least 1 year.
  • Proficiency in Python for at least 1 year.
  • Familiarity with PySpark or Spark (desirable for at least 1 year).
  • Experience in Business Intelligence data modeling for 3 years.
  • Proficiency in SQL for 3 years.

Qualifications/Certifications:

  • A Bachelor's degree in computer science, computer engineering, or equivalent work experience for a minimum of 4 years.
  • AWS Certification, at least at the associate level.

* In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

 

  • AWS Data Engineer

    2 weeks ago

    Cape Town, Western Cape, South Africa Network Recruitment Full time

    Minimum Experience: 5 years of experience in Python/C# development 3 years of experience in AWS Data Engineering Education Requirements: Bachelors degree in Computer Science, Information Systems, or a related field Advantageous: AWS Certified Machine Learning Specialty Certificate Responsibilities: Design and create data models that can extract information...

  • Data Pipeline Engineer AWS

    2 days ago

    Cape Town, Western Cape, South Africa Ampstek Full time

    AWS Data Engineer Position at Ampstek:About the RoleThis role involves designing, developing and maintaining scalable data solutions on AWS Cloud.The ideal candidate will have experience in data pipelines, ETL processes and data lake/warehouse solutions using AWS services.ResponsibilitiesImplement data security, governance and compliance best...

  • AWS Data Engineer

    6 days ago

    Cape Town, Western Cape, South Africa PBT Group Full time

    The Data Engineer will be responsible for designing, implementing, and optimising data pipelines and data solutions using AWS and Snowflake. The successful candidate will work closely with Data Analysts, Data Scientists, and other stakeholders to ensure efficient data storage, processing, and accessibility in a cloud environment.Key Responsibilities:Design,...

  • Data Engineer

    1 week ago

    Cape Town, Western Cape, South Africa Datonomy Solutions Full time

    Job Overview:We are seeking an experienced Data Engineer with strong Solution Architecture expertise to lead the data engineering efforts for a financial data migration project. The role focuses on designing, building, and optimizing data pipelines to support the transition from an on-premise Hyperion system to OneStream in the cloud. The ideal candidate...

  • Data Engineering

    3 weeks ago

    Cape Town, Western Cape, South Africa PBT Group Full time

    The Data Engineering & Streaming Lead will be responsible for building, enhancing, and maintaining real-time data pipelines. The role involves working closely with various infrastructure and operations teams to maintain data infrastructure. As a senior engineer, the candidate will develop and maintain data ingestion pipelines, ensuring the bank's continued...

  • Data Engineering

    3 weeks ago

    Cape Town, Western Cape, South Africa PBT Group Full time

    The Data Engineering & Streaming Lead will be responsible for building, enhancing, and maintaining real-time data pipelines. The role involves working closely with various infrastructure and operations teams to maintain data infrastructure. As a senior engineer, the candidate will develop and maintain data ingestion pipelines, ensuring the bank's continued...

  • Data Engineering

    4 days ago

    Cape Town, Western Cape, South Africa PBT Group Full time

    The Data Engineering & Streaming Lead will be responsible for building, enhancing, and maintaining real-time data pipelines. The role involves working closely with various infrastructure and operations teams to maintain data infrastructure. As a senior engineer, the candidate will develop and maintain data ingestion pipelines, ensuring the bank's continued...

  • AWS Cloud Solutions Architect

    2 days ago

    Cape Town, Western Cape, South Africa Ampstek Full time

    Ampstek is seeking a highly skilled AWS Data Engineer to design, develop and maintain scalable data solutions on AWS Cloud.Job DescriptionWe are looking for an experienced professional who can work with data pipelines, ETL processes and data lake/warehouse solutions using AWS services.Design and build data pipelines using AWS Glue, AWS Lambda and AWS Step...

  • Cloud Data Engineer

    2 days ago

    Cape Town, Western Cape, South Africa Ampstek Full time

    Ampstek is seeking a highly skilled Data Engineer to join our team in Cape Town.About the RoleThis role involves designing, developing and maintaining scalable data solutions on AWS Cloud.The ideal candidate will have experience in data pipelines, ETL processes and data lake/warehouse solutions using AWS services.ResponsibilitiesDesign and build data...

  • Security Engineer, Aws Trust

    1 day ago

    Cape Town, Western Cape, South Africa Ananzi Full time

    AWS Trust and Safety (T&S) Risk & Response (R&R) is seeking a motivated Security Engineer with a strong background in incident response, threat investigation, and developing solutions to security issues.As a Security Engineer in R&R, you will employ your technical skills to develop solutions to complex and ambiguous security-related events originating from...

  • Software Development Engineer, AWS Alameda

    4 weeks ago

    Cape Town, Western Cape, South Africa Amazon Full time

    Software Development Engineer, AWS AlamedaAWS Alameda is shaping the future of how Control Planes for AWS Services will be offered. The Alameda team builds innovative and secure technologies on a massive scale that manage the control planes for AWS services and keep them secure and scalable for their customers. As a software development engineer on this...

  • Data Engineer

    4 weeks ago

    Cape Town, Western Cape, South Africa PBT Group Full time

    ? Join PBT Group as a Senior Data Engineer ? Are you a dynamic data engineer passionate about optimising large-scale data ecosystems? PBT Group is looking for a Data Engineer to design and implement robust, scalable processes that support our growing data engineering capability. This role is pivotal in advancing our cloud engineering and data management...

  • Data Engineer

    3 weeks ago

    Cape Town, Western Cape, South Africa PBT Group Full time

    ? Join PBT Group as a Senior Data Engineer ?Are you a dynamic data engineer passionate about optimising large-scale data ecosystems? PBT Group is looking for a Data Engineer to design and implement robust, scalable processes that support our growing data engineering capability. This role is pivotal in advancing our cloud engineering and data management...

  • AWS Security Engineer Lead

    7 days ago

    Cape Town, Western Cape, South Africa Orionlabs Full time

    Cyber Security Engineer PositionOrion Labs is seeking a skilled Cyber Security Engineer to ensure the security of our clients' cloud environments and applications.Key Responsibilities:Audit and secure cloud environments using AWS.Assess application security to identify potential weaknesses.Conduct penetration testing to identify vulnerabilities.Implement...

  • Principal Lead: Data Engineer

    15 hours ago

    Cape Town, Western Cape, South Africa BigTalent Full time

    BigTalent City of Cape Town, Western Cape, South AfricaPrincipal Lead: Data EngineerOur financial services client is looking for a Principal Lead Data Engineer to drive the Technical Vision, Strategy and Execution of their data engineering capability.The role will lead the design and implementation of scalable, high-performance data pipelines, data...

  • Senior Data Engineer Amazon Web Services

    2 days ago

    Cape Town, Western Cape, South Africa Ampstek Full time

    Ampstek is seeking a talented Data Engineer to join our team in Cape Town.About the JobThis role involves designing, developing and maintaining scalable data solutions on AWS Cloud.The ideal candidate will have experience in data pipelines, ETL processes and data lake/warehouse solutions using AWS services.Candidates should be able to design and build data...

  • Senior Data Engineer

    1 week ago

    Cape Town, Western Cape, South Africa Communicate Recruitment Full time

    The successful candidate will be responsible for building, optimizing, and maintaining complex data pipelines and databases. They will collaborate with data scientists and analysts to ensure seamless data accessibility and quality while developing ETL processes and scalable storage solutions. The role involves implementing best practices for data security,...

  • Data Streaming Engineer

    3 weeks ago

    Cape Town, Western Cape, South Africa PBT Group Full time

    Are you a skilled Data Streaming Engineer passionate about real-time data processing and cutting-edge technology? Join our PBT team and play a pivotal role in shaping data streaming architecture. This position offers the opportunity to work with industry-leading tools like Kafka and Flink, collaborating with talented development and data engineering teams to...

  • Data Engineering Lead

    12 hours ago

    Cape Town, Western Cape, South Africa BigTalent Full time

    BigTalent, a leading organization in the financial services sector, is seeking an exceptional Data Engineering Lead to drive their technical vision and strategy.Job OverviewThe successful candidate will lead the design and implementation of scalable data pipelines, platforms, and infrastructure to support business objectives.The ideal candidate will be a...

  • Software Dev Engineer, AWS Firmware Services

    4 weeks ago

    Cape Town, Western Cape, South Africa Amazon Full time

    Software Dev Engineer, AWS Firmware ServicesJob ID: 2871969 | Amazon Data Services, Inc.AWS Infrastructure Services owns the design, planning, delivery, and operation of all AWS global infrastructure. In other words, we're the people who keep the cloud running. We support all AWS data centers and all of the servers, storage, networking, power, and cooling...