AWS Data Engineer

3 weeks ago

Cape Town, South Africa PBT Group Full time

PBT Group has an opportunity for an AWS Data Engineer, preferably with Hadoop experience.

 

Duties:

  • Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake.
  • Analyze, re-architect and re-platform on-premise data warehouses to data platforms on AWS cloud using AWS or 3rd party services.
  • Design and build production data pipelines from ingestion to consumption within a big data architecture, using Java, Python, Scala.
  • Design and implement data engineering, ingestion and curation functions on AWS cloud using AWS native or custom programming.
  • Perform detail assessments of current state data platforms and create an appropriate transition path to AWS cloud.
  • Design, implement and support an analytical data infrastructure providing ad-hoc access to large datasets and computing power.
  • Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using SQL and AWS big data technologies.
  • Creation and support of real-time data pipelines built on AWS technologies including Glue, Redshift/Spectrum, Kinesis, EMR and Athena
  • Continual research of the latest big data and visualization technologies to provide new capabilities and increase efficiency
  • Working closely with team members to drive real-time model implementations for monitoring and alerting of risk systems.
  • Collaborate with other tech teams to implement advanced analytics algorithms that exploit our rich datasets for statistical analysis, prediction, clustering and machine learning
  • Help continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers

 

Experience:

  • Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
  • Experience building/operating highly available, distributed systems of data extraction, ingestion, and processing of large data sets
  • Experience working with distributed systems as it pertains to data storage and computing
  • Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
  • Strong analytic skills related to working with unstructured datasets.
  • Build processes supporting data transformation, data structures, meta data, dependency and workload management.
  • A successful history of manipulating, processing and extracting value from large disconnected data sets.
  • Working knowledge of message queuing, stream processing, and highly scalable Big Data, data stores.
  • Strong project management and organizational skills.
  • Experience supporting and working with cross-functional teams in a dynamic environment.
  • Experience in a Data Engineer or similar roles
  • Experience with big data tools is a must: Hadoop, Spark, Kafka, etc.
  • Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
  • Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
  • Experience with AWS cloud services: EC2, EMR, RDS, Redshift
  • Experience with stream-processing systems: Storm, Spark-Streaming, etc.
  • Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc

 

Qualifications/ Certification:

  • Bachelor's Degree in Computer Science, Information Technology or other relevant fields
  • Has experience in any of the following AWS Athena and Glue Pyspark, EMR, DynamoDB, Redshift, Kinesis, Lambda, Snowflake
  • Proficient in AWS Redshift, S3, Glue, Athena, DynamoDB, EMR
  • Knowledge of software engineering best practices across the development lifecycle, including agile methodologies, coding standards, code reviews, source management, build processes, testing, and operations

 

* In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

 

  • AWS Data Engineer

    4 weeks ago

    Cape Town, South Africa PBT Group Full time

    PBT Group is currently offering an opportunity for an AWS Data Engineer with 2 to 5 years of relevant experience.The role of a Data Engineer involves constructing and maintaining data pipelines and datamarts, emphasizing scalability, repeatability, and security. Data Engineers play a pivotal role in facilitating the acquisition of data from diverse sources,...

  • Junior AWS Engineer

    4 weeks ago

    Cape Town, South Africa Dimension Data Full time

    FunctieomschrijvingResponsibilities Develop reusable and reliable automated scripts and make them part of continuous integration / build systems Deploy, scale and automate AWS services across multiple regions. Participate in the research and development of the automated and replicable "infrastructure as a code" approaches. Tune system parameters and...

  • AWS Engineer

    4 weeks ago

    Cape Town, South Africa IOCO Full time

    We are seeking a skilled AWS Mid/Senior Level Engineer to join our dynamic team. The AWS Engineer will play a key role in designing, implementing, and managing AWS cloud solutions, ensuring high performance, scalability, and security.6 Month contract - renewableHybrid roleMatric / Tertiary qualification preferable.AWS CertificationMin 5 years...

  • AWS Engineer

    3 weeks ago

    Cape Town, South Africa IOCO Full time

    We are seeking a skilled AWS Mid/Senior Level Engineer to join our dynamic team. The AWS Engineer will play a key role in designing, implementing, and managing AWS cloud solutions, ensuring high performance, scalability, and security. 6 Month contract - renewable Hybrid role Matric / Tertiary qualification preferable. AWS Certification Min 5 years...

  • AWS Architect

    4 weeks ago

    Cape Town, South Africa Dimension Data Full time

    FunctieomschrijvingAre you tired of the corporate culture? Are you passionate about technology and want to work on cutting edge infrastructure? We are looking forward to hearing from you!As an AWS Architect, you'll be responsible for designing and architecting the services that will power our cloud based applications. You'll also be responsible for ensuring...

  • AWS Cloud Engineer

    4 weeks ago

    Cape Town, South Africa Parvana Full time

    Client Details: Our client works with multiple organisations to find a better way to transact. They bring together committed people dedicated to delivering innovative enterprise solutions that help their customers contribute to economic growth. As a vibrant and innovative SaaS company, they deliver industry-leading expertise and technology to solve real...

  • DevOps Data Engineer

    4 weeks ago

    Cape Town, South Africa PBT Group Full time

    PBT Group has a requirement for an intermediate DevOps Data Engineer. Duties:Strong experience with AWS, Terraform, CI/CD, GitDesign, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties – Glue, Step-functions, Kafka CC, PySpark, DynamoDB,...

  • DevOps Data Engineer

    3 weeks ago

    Cape Town, South Africa PBT Group Full time

    PBT Group has a requirement for an intermediate DevOps Data Engineer.   Duties: Strong experience with AWS, Terraform, CI/CD, Git Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties – Glue, Step-functions, Kafka CC, PySpark, DynamoDB,...

  • DevOps Data Engineer

    4 weeks ago

    Cape Town, South Africa PBT Group Full time

    PBT Group has a requirement for an intermediate DevOps Data Engineer. Duties:Strong experience with AWS, Terraform, CI/CD, GitDesign, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties – Glue, Step-functions, Kafka CC, PySpark, DynamoDB,...

  • DevOps Data Engineer

    3 weeks ago

    Cape Town, South Africa PBT Group Full time

    PBT Group has a requirement for an intermediate DevOps Data Engineer.   Duties: Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties – Glue, Step-functions, Kafka CC, PySpark, DynamoDB, Delta.io, RedShift, Lambda, DeltaLake, Python,....

  • DevOps Data Engineer

    4 weeks ago

    Cape Town, South Africa PBT Group Full time

    PBT Group has a requirement for an intermediate DevOps Data Engineer. Duties:Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties – Glue, Step-functions, Kafka CC, PySpark, DynamoDB, Delta.io, RedShift, Lambda, DeltaLake,...

  • Data Engineer

    4 weeks ago

    Cape Town, South Africa Dimension Data Full time

    As a Data Engineer, you'll design and build the data architecture of our products. You will use your technical knowledge to help us develop new products by taking an idea from concept to reality. You'll work on projects across multiple teams in order to achieve company goals and objectives. Your role will involve: - Analyzing business requirements; -...

  • Aws Administrator Senior

    3 weeks ago

    Cape Town, South Africa Network Contracting Full time

    **Required**: A numerate honours degree (e.g. Computer Science, Mathematics, Engineering). Ability to develop technical solutions and functional specifications based on business requirements. Knowledge of leading technologies including AWS certifications such as AWS Certified Solutions Architect, AWS Certified DevOps Engineer, or AWS Certified SysOps...

  • API Cloud Engineer

    4 weeks ago

    Cape Town, South Africa PBT Group Full time

    PBT Group is seeking an AWS-experienced API Cloud Engineer to fill a promising opportunity. Skills and experience required include building Cloud-based Solutions (AWS, Azure, GCP), expertise in C#/.NET 6.0 and REST APIs, communication skills, and the ability to interpret architectural designs and produce solutions that will aid data migration and topic...

  • Intermediate DevOps Engineer

    4 weeks ago

    Cape Town, South Africa Black Swan Data Full time

    Summary: We are seeking an experienced and passionate DevOps Engineer with a strong background in Kubernetes, AWS, and Linux to join our team. As an Intermediate DevOps Engineer, you will play a critical role in designing, implementing, and maintaining our cloud infrastructure, with a focus on Kubernetes-based container orchestration, AWS services, and...

  • Data Streaming Engineer

    4 weeks ago

    Cape Town, South Africa PBT Group Full time

    PBT Group has an opportunity for a Data Streaming Engineer to build, enhance, and maintain real-time data pipeline. Work with various infrastructure and operations teams to maintain data infrastructure. As a senior engineer on the Data Streaming Platform team, you will be responsible for developing and maintaining data ingestion pipelines vital to the...

  • Senior Cloud Engineer

    4 weeks ago

    Cape Town, South Africa Progressive Edge Full time

    Senior Cloud Engineer (AWS)Location: Winelands / Cape TownA flourishing fintech and payments enterprise is rapidly expanding through innovative initiatives and state-of-the-art solutions in the payment processing sector. These innovations aim to enhance payment solutions with automation and cutting-edge technologies, spanning across e-commerce to advanced...

  • Senior Cloud Engineer

    4 weeks ago

    Cape Town, South Africa Progressive Edge Full time

    Senior Cloud Engineer (AWS)Location: Winelands / Cape TownA flourishing fintech and payments enterprise is rapidly expanding through innovative initiatives and state-of-the-art solutions in the payment processing sector. These innovations aim to enhance payment solutions with automation and cutting-edge technologies, spanning across e-commerce to advanced...

  • AWS Solutions Architect

    3 weeks ago

    Cape Town, South Africa PikUniq Full time

    Attention: Please note that the hiring team responsible for this position will be using the PikUniq platform for candidate screening and conducting one-way interviews. Each application received will be carefully evaluated and screened. Further information on how to get started will be provided. We are looking forward to receiving your...

  • Network Infrastructure Manager

    19 hours ago

    Cape Town, South Africa Amazon Data Services South Africa (Pty) Ltd Full time

    An engineering degree or equivalent with extensive engineering or operations experience. - Previous people management experience, building & managing technical teams. - Experience working on high scale networks; both the logical and physical connectivity of the networking and how it works. - Broad understanding of cabling infrastructure, infrastructure...