AWS Data Engineer/JHB

4 weeks ago

Johannesburg, South Africa Rooftop Recruitment Full time

AWS Data Engineer/JHB

Contract Type: 12 months

Our client is seeking a highly motivated and experienced AWS Data Engineer to join their Global

Markets division. The role is responsible for closely working the traders, quantitative analysts, and

Data Scientists to design, develop, and maintain data infrastructure and pipelines for our global markets business.

The ideal candidate should have a strong background in C# or Python programming and building data pipelines on AWS, especially with AWS Glue Jobs using PySpark or AWS GlueSpark. This role offers an exciting opportunity to collaborate with leading financial institutions, contributing to the design and implementation of data pipelines to support global markets business. It will be advantageous if the candidate has the AWS Certified Machine Learning

Specialty Certificate or can develop machine learning (ML) models and automate ML pipelines.

Minimum Experience:

5 Years Python/C# Development

3 Years AWS Data Engineering

Education Requirements:

Bachelor's degree in Computer Science, Information Systems, or related field.

Advantageous: AWS Certified Machine Learning Specialty Certificate

Responsibilities differ across client engagements but may include:

Creating data models that can be used to extract information from various sources and store it in a usable format.

Lead the design, implementation, and successful delivery of large-scale, critical, or difficult data solutions involving a significant amount of data.

Utilize expertise in SQL and have a strong understanding of ETL and data modelling.

Ability to ingest data into AWS S3, perform ETL into RDS or Redshift.

Use AWS Lambda (C# or Python) for event-driven data transformations.

COMPETENCIES

Designing and implementing security measures to protect data from unauthorized access or misuse.

Maintaining the integrity of data by designing backup and recovery procedures.

Work on automating the migration process in AWS from development to production.

You will deliver digestible, contemporary, and immediate data content to support and drive business decisions.

You will be involved in all aspects of data engineering from delivery planning, estimating and analysis, all the way through to data architecture and pipeline design, delivery, and production implementation.

From day one, you will be involved in the design and implementation of complex data solutions ranging from batch to streaming and event-driven architectures, across cloud, on-premise and hybrid client technology landscapes.

Optimize cloud workloads for cost, scalability, availability, governance, compliance, etc.

Must have experience with AWS Glue Jobs using PySpark or AWS Glue Spark.

Real-time ingestion using KAFKA is an added advantage.

Strong SQL and C# or Python programming knowledge.

Objective oriented principles in C# or Python: classes and inheritance.

Expert knowledge of data engineering packages and libraries and related functions in

C# or Python.

AWS technical certifications (Developer Associate or Solutions Architect).

Experience with development and delivery of microservices using serverless AWS services (S3, RDS, Aurora, DynamoDB, Lambda, SNS, SQS, Kinesis, IAM).

Ability to understand and articulate requirements to technical and non-technical audiences.

Have experience working with RDBMS databases, such as Postgres, SQL Server and MySQL.

Apply knowledge of scripting and automation using tools like PowerShell, Python,

Bash, Ruby, Perl, etc.

Stakeholder management and communication skills, including prioritising, problem solving and interpersonal relationship building.

Effectively and efficiently troubleshoot data issues and errors.

Strong experience in SDLC delivery, including waterfall, hybrid, and Agile methodologies.

Experience delivering in an agile environment.

Experience in implementing and delivering data solutions and pipelines on AWS Cloud Platform.

A strong understanding of data modelling, data structures, databases, and ETL processes.

An in-depth understanding of large-scale data sets, including both structured and unstructured data.

Knowledge and experience in delivering CI/CD and DevOps capabilities in a data environment.

Ability to clearly communicate complex technical ideas.

Experience in the financial industry is a plus.

An AWS Certified Machine Learning Specialty Certificate is an advantage.

  • AWS Data Engineer with DataBricks

    4 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    4 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    3 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience.   Duties: Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake. Analyze, re-architect and...

  • AWS Cloud Data Engineer

    4 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Cloud Data Engineer. Duties:Coordinate/Prioritise the work that the Developers and Product OwnerAssist with cross-team projects that involve development.AWS practices are followed with sound security and governance.Coordinate with the cloud Center of Excellence team for any AWS account level security and...

  • AWS Engineer

    4 weeks ago

    Johannesburg, South Africa IOCO Full time

    We're looking for an AWS Engineer to join our team! As part of our dynamic and growing organization, you'll have the opportunity to work on cutting-edge projects and collaborate with a talented team.What you’ll do: Research and suggest new technologiesProactively identify gaps and continuously work to improve the customer experienceCollaborate with...

  • AWS Engineer

    3 weeks ago

    Johannesburg, South Africa IOCO Full time

    We're looking for an AWS Engineer to join our team! As part of our dynamic and growing organization, you'll have the opportunity to work on cutting-edge projects and collaborate with a talented team. What you’ll do: Research and suggest new technologies Proactively identify gaps and continuously work to improve the customer experience Collaborate...

  • AWS DevOps Engineer

    4 weeks ago

    Johannesburg, South Africa Edge Executives Full time

    We have an opportunity for an AWS DevOps Engineer with one of our international clients. Requirements:Must be AWS certified  Must have a tertiary qualification 5+ years of experience as a DevOps Engineer  In-depth knowledge of AWS services - EC2, S3, RDS, Lambda Experience with infrastructure as code tool - Terraform or CloudFormation Docker and...

  • AWS DevOps Engineer

    4 weeks ago

    Johannesburg, South Africa Edge Executives Full time

    We have an opportunity for an AWS DevOps Engineer with one of our international clients. Requirements:Must be AWS certified Must have a tertiary qualification5+ years of experience as a DevOps Engineer In-depth knowledge of AWS services - EC2, S3, RDS, LambdaExperience with infrastructure as code tool - Terraform or CloudFormationDocker and...

  • AWS DevOps Engineer

    4 weeks ago

    Johannesburg, South Africa Adzuna ZA B C2 Full time

    We have an opportunity for an AWS DevOps Engineer with one of our international clients. Requirements:Must be AWS certified Must have a tertiary qualification5+ years of experience as a DevOps Engineer In-depth knowledge of AWS services - EC2, S3, RDS, LambdaExperience with infrastructure as code tool - Terraform or CloudFormationDocker and...

  • Data Streaming Engineer

    4 weeks ago

    Johannesburg, South Africa Golden Rule Full time

    The Role An opportunity exists for a Senior Data Streaming Engineer who will Build, enhance, and maintain our real-time data pipeline. The role will work with various infrastructure and operations teams to maintain our data infrastructure. As a senior engineer on the Data Streaming Platform team, you will be responsible for developing and maintaining...

  • Data Engineer

    3 weeks ago

    Johannesburg, South Africa EQ Plus Full time

    Duties: Responsible for solution design and development of Various Functionality in AWS for the project flow Develop and maintain automated ETL pipelines (with monitoring) using scripting languages such as Python, SQL and AWS services such as S3, Glue, Lambda, SNS, Redshift, SQS, KMS Develop Glue Jobs for batch data processing and created Glue Catalog for...

  • Data Engineer

    4 weeks ago

    Johannesburg, South Africa EQ Plus Full time

    Duties:Responsible for solution design and development of Various Functionality in AWS for the project flowDevelop and maintain automated ETL pipelines (with monitoring) using scripting languages such as Python, SQL and AWS services such as S3, Glue, Lambda, SNS, Redshift, SQS, KMSDevelop Glue Jobs for batch data processing and created Glue Catalog for...

  • Data Engineer

    3 weeks ago

    Johannesburg, South Africa Adzuna ZA B C2 Full time

    Duties:Responsible for solution design and development of Various Functionality in AWS for the project flowDevelop and maintain automated ETL pipelines (with monitoring) using scripting languages such as Python, SQL and AWS services such as S3, Glue, Lambda, SNS, Redshift, SQS, KMSDevelop Glue Jobs for batch data processing and created Glue Catalog for...

  • Senior Aws Administrator

    7 days ago

    Johannesburg, South Africa TotalEnergies Full time

    Information Systems - JOHANNESBURG-BIERMANN AV(ZAF) - South Africa - As a Data Engineer you would be responsible for the following: - ** Functional Expertise**: - Provide functional expertise and guidance around leveraging the capabilities of data engineering platforms to business stakeholders. - Act as liaison between business key users and the Digital &...

  • Data Analyst

    4 weeks ago

    Johannesburg, South Africa Mediro Application Consulting Full time

    Permanent - Only South African citizens my apply - Full-time at office in Sandton (Not hybrid or remote work) - 3-5 years experience in Data analytics - Must have at least 2 to 3 years AWS work experience - Clean ITC/credit record and no criminal record **Technical skills required (compulsory) 3 5 years min**: - SAS - AWS (2 to 3 years required) -...

  • Jnr Data Engineer

    4 weeks ago

    Johannesburg, South Africa Salt Employee Benefits Full time

    Data is one of the most valuable resources a company owns, and savvy data engineers are required to build an infrastructure to keep it organized. This role requires a significant set of technical skills, including a deep knowledge of SQL and multiple programming languages.  The incumbent should have stellar communication skills as you will work across...

  • Senior Data Engineer

    4 weeks ago

    Johannesburg, South Africa Salt Employee Benefits Full time

    Salt Employee Benefits is looking for a savvy Snr Data Engineer to join our IT team. This role requires the incumbent to have  a significant set of technical skills, including a deep knowledge of SQL database design and multiple programming languages. The successful incumbent should have stellar communication skills as she/he will work across departments...

  • Senior Software Engineer

    4 weeks ago

    Johannesburg, South Africa Dimension Data Full time

    FunctieomschrijvingAre you tired of being micro-managed? Are you looking for a company where no-one tells you how to do your job? Do you want to work in a team, but not be told what to do and when to do it? If so, we believe this Senior Software Engineer role is perfect for you!As a Senior Software Engineer, you'll be responsible for building and maintaining...

  • Data Engineering Technical Lead

    4 weeks ago

    Johannesburg, South Africa Megan Frg Consulting Full time

    **Location**: Sandton, Johannesburg **Salary**: Highly Negotiable **About the Role**: **Requirements**: 5+ years data engineering technical experience **Technical skills to include as many of the following**: Data Modelling (e.g. Kimball/Data Vault) Scripting/coding (e.g Python, T-SQL, SQL, Spark, PySpark,) ETL/ELT (Eg. Alteryx, SSIS, dbt, AWS Glue) SQL...

  • Data Engineering Technical Lead

    4 weeks ago

    Johannesburg, South Africa Megan Förg Consulting Full time

    **About the Role**: **Requirements**: 5+ years data engineering technical experience Technical skills to include as many of the following: Data Modelling (e.g. Kimball/Data Vault) Scripting/coding (e.g Python, T-SQL, SQL, Spark, PySpark,) ETL/ELT (Eg. Alteryx, SSIS, dbt, AWS Glue) SQL (Snowflake, SQL Server, Oracle) Visualisation (Power Bi, Tableau) Data...