AWS Data Engineer/JHB

4 weeks ago

Johannesburg, South Africa Rooftop Recruitment Full time

AWS Data Engineer/JHB

Contract Type: 12 months

Our client is seeking a highly motivated and experienced AWS Data Engineer to join their Global

Markets division. The role is responsible for closely working the traders, quantitative analysts, and

Data Scientists to design, develop, and maintain data infrastructure and pipelines for our global markets business.

The ideal candidate should have a strong background in C# or Python programming and building data pipelines on AWS, especially with AWS Glue Jobs using PySpark or AWS GlueSpark. This role offers an exciting opportunity to collaborate with leading financial institutions, contributing to the design and implementation of data pipelines to support global markets business. It will be advantageous if the candidate has the AWS Certified Machine Learning

Specialty Certificate or can develop machine learning (ML) models and automate ML pipelines.

Minimum Experience:

5 Years Python/C# Development

3 Years AWS Data Engineering

Education Requirements:

Bachelor's degree in Computer Science, Information Systems, or related field.

Advantageous: AWS Certified Machine Learning Specialty Certificate

Responsibilities differ across client engagements but may include:

Creating data models that can be used to extract information from various sources and store it in a usable format.

Lead the design, implementation, and successful delivery of large-scale, critical, or difficult data solutions involving a significant amount of data.

Utilize expertise in SQL and have a strong understanding of ETL and data modelling.

Ability to ingest data into AWS S3, perform ETL into RDS or Redshift.

Use AWS Lambda (C# or Python) for event-driven data transformations.

COMPETENCIES

Designing and implementing security measures to protect data from unauthorized access or misuse.

Maintaining the integrity of data by designing backup and recovery procedures.

Work on automating the migration process in AWS from development to production.

You will deliver digestible, contemporary, and immediate data content to support and drive business decisions.

You will be involved in all aspects of data engineering from delivery planning, estimating and analysis, all the way through to data architecture and pipeline design, delivery, and production implementation.

From day one, you will be involved in the design and implementation of complex data solutions ranging from batch to streaming and event-driven architectures, across cloud, on-premise and hybrid client technology landscapes.

Optimize cloud workloads for cost, scalability, availability, governance, compliance, etc.

Must have experience with AWS Glue Jobs using PySpark or AWS Glue Spark.

Real-time ingestion using KAFKA is an added advantage.

Strong SQL and C# or Python programming knowledge.

Objective oriented principles in C# or Python: classes and inheritance.

Expert knowledge of data engineering packages and libraries and related functions in

C# or Python.

AWS technical certifications (Developer Associate or Solutions Architect).

Experience with development and delivery of microservices using serverless AWS services (S3, RDS, Aurora, DynamoDB, Lambda, SNS, SQS, Kinesis, IAM).

Ability to understand and articulate requirements to technical and non-technical audiences.

Have experience working with RDBMS databases, such as Postgres, SQL Server and MySQL.

Apply knowledge of scripting and automation using tools like PowerShell, Python,

Bash, Ruby, Perl, etc.

Stakeholder management and communication skills, including prioritising, problem solving and interpersonal relationship building.

Effectively and efficiently troubleshoot data issues and errors.

Strong experience in SDLC delivery, including waterfall, hybrid, and Agile methodologies.

Experience delivering in an agile environment.

Experience in implementing and delivering data solutions and pipelines on AWS Cloud Platform.

A strong understanding of data modelling, data structures, databases, and ETL processes.

An in-depth understanding of large-scale data sets, including both structured and unstructured data.

Knowledge and experience in delivering CI/CD and DevOps capabilities in a data environment.

Ability to clearly communicate complex technical ideas.

Experience in the financial industry is a plus.

An AWS Certified Machine Learning Specialty Certificate is an advantage.

  • Aws Data Engineer

    2 weeks ago

    Johannesburg, Gauteng, South Africa E-Merge Full time

    This technology company is enthusiastic about resolving business challenges for their clients and is actively seeking an AWS Data Engineer to join their team. By employing advanced data analytics, actuarial services, software development, and automation, this remarkable business serves both large corporations listed on the JSE and non-listed clients in the...

  • Cloud Data Engineer

    2 weeks ago

    Johannesburg, Gauteng, South Africa Dimension Data Full time

    The Cloud Data Engineer, AWS Data Pipeline, AWS Storage Gateway and Digital Assets will work on the Amazon Web Services (AWS) infrastructure team. The engineer's primary focus is to provide data services that enhance our clients' businesses. You'll be a key member of the data pipeline team helping us deliver these solutions quickly and efficiently while...

  • AWS Data Engineer

    1 month ago

    Johannesburg, South Africa ACB Consulting Full time

    “As part of our next project, we need to migrate existing cloud infrastructure from a set of source AWS accounts to a new set of target accounts. We need to make some functional changes and enhancements along the way so it won’t be simple copy-and-paste. Additionally, the target infrastructure needs to be managed by Terraform (which is currently not used...

  • Aws Data Engineer

    2 weeks ago

    Johannesburg, Gauteng, South Africa E-Merge Full time

    A dynamic global group with a focus on innovation topics as well as comprehensive IT solutions and Global Team collaboration driving experiences with a deep insight into the compliance and IT Methodology processes is recruiting for an AWS Data Engineer to join an environment with cutting edge global IT system landscape and processesRequirements:Demonstrate...

  • Cloud Data Engineer

    4 weeks ago

    Johannesburg, South Africa Dimension Data Full time

    The Cloud Data Engineer, AWS Data Pipeline, AWS Storage Gateway and Digital Assets will work on the Amazon Web Services (AWS) infrastructure team. The engineer's primary focus is to provide data services that enhance our clients' businesses. You'll be a key member of the data pipeline team helping us deliver these solutions quickly and efficiently while...

  • AWS Data Engineer

    2 weeks ago

    Johannesburg, Gauteng, South Africa ACB Consulting Full time

    "As part of our next project, we need to migrate existing cloud infrastructure from a set of source AWS accounts to a new set of target accounts. We need to make some functional changes and enhancements along the way so it won't be simple copy-and-paste. Additionally, the target infrastructure needs to be managed by Terraform (which is currently not used in...

  • AWS Data Engineer

    1 month ago

    Johannesburg, South Africa ACB Consulting Full time

    “As part of our next project, we need to migrate existing cloud infrastructure from a set of source AWS accounts to a new set of target accounts. We need to make some functional changes and enhancements along the way so it won’t be simple copy-and-paste. Additionally, the target infrastructure needs to be managed by Terraform (which is currently not used...

  • AWS Data Engineer with DataBricks

    3 months ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    3 months ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    2 months ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience.   Duties: Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake. Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    4 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience.   Duties: Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake. Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    4 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.Analyze, re-architect and...

  • AWS Data Engineer with DataBricks

    4 weeks ago

    Johannesburg, South Africa PBT Group Full time

    PBT Group has an opportunity for an AWS Data Engineer, with DataBricks experience. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.Analyze, re-architect and...

  • AWS Data Cloud Engineer

    2 weeks ago

    Johannesburg, Gauteng, South Africa Planet10 Full time

    A leader in the insurance space is on the lookout for a AWS Data Cloud Engineer (big Data Engineer) Knowledge /Qualifications High knowledge in AWS – security groups, permissions, VPC, roles, services. Experience developing engineering applications in Corporate companies Demonstrated project development and leadership skills Master's degree in Computer...

  • Aws Data Cloud Engineer

    2 weeks ago

    Johannesburg, Gauteng, South Africa Planet10 Full time

    Job Description:A leader in the insurance space is on the lookout for a AWS Data Cloud Engineer (Big Data Engineer)Knowledge/Qualifications:High knowledge in AWS - security groups, permissions, VPC, roles, services.Experience developing engineering applications in Corporate companies.Demonstrated project development and leadership skills.Master's degree in...

  • AWS Data Engineer

    2 weeks ago

    Johannesburg, Gauteng, South Africa PM Connection Full time

    Job Title: Cloud Infrastructure Migration SpecialistCompany: Innovative Solutions Inc.As part of our exciting upcoming project, we are looking for a skilled professional to lead the migration of our cloud infrastructure from existing AWS accounts to new target accounts. This task involves more than just a simple transfer; it requires strategic functional...

  • AWS Data Cloud Engineer

    1 month ago

    Johannesburg, South Africa Planet10 Full time

    A leader in the insurance space is on the lookout for a AWS Data Cloud Engineer (big Data Engineer) Knowledge /Qualifications  High knowledge in AWS – security groups, permissions, VPC, roles, services.Experience developing engineering applications in Corporate companiesDemonstrated project development and leadership skillsMaster's degree in...

  • AWS Data Cloud Engineer

    1 month ago

    Johannesburg, South Africa Planet10 Full time

    A leader in the insurance space is on the lookout for a AWS Data Cloud Engineer (big Data Engineer)  Knowledge /Qualifications   High knowledge in AWS – security groups, permissions, VPC, roles, services. Experience developing engineering applications in Corporate companies Demonstrated project development and leadership skills Master's degree...

  • AWS Data Cloud Engineer

    4 weeks ago

    Johannesburg, South Africa Planet10 Full time

    A leader in the insurance space is on the lookout for a AWS Data Cloud Engineer (big Data Engineer)  Knowledge /Qualifications   High knowledge in AWS – security groups, permissions, VPC, roles, services. Experience developing engineering applications in Corporate companies Demonstrated project development and leadership skills Master's degree...

  • AWS Data Cloud Engineer

    4 weeks ago

    Johannesburg, South Africa Planet10 Full time

    A leader in the insurance space is on the lookout for a AWS Data Cloud Engineer (big Data Engineer) Knowledge /Qualifications  High knowledge in AWS – security groups, permissions, VPC, roles, services.Experience developing engineering applications in Corporate companiesDemonstrated project development and leadership skillsMaster's degree in...