Data Engineer

Data Engineer

Role Purpose

We are seeking an experienced and passionate Data Engineer to design and build robust data infrastructure that facilitates large data volumes.

The role requires a blend of data engineering skills (taking data from various datasets and collections to clean, store, transform and analyse) and database administration skills (working with and manage relational databases).

You will work closely with business analysts and software engineers to ensure efficient data pipelines, quality data sources, and scalable solutions that support key business decisions and innovation.

Key Responsibilities

• Assembling large, complex sets of data that meet non-functional and functional business requirements.
• Design, develop, monitor and maintain scalable data ingestion pipelines and ETL processes. Building required infrastructure for optimal extraction, transformation and loading of data from various data sources using Integration and SQL technologies, often these are Cloud based.
• Identifying, designing and implementing internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes.
• Building analytical tools to utilize the data pipeline, providing actionable insight into key business performance metrics.
• Ensure data quality, consistency, integrity, and security across all systems. And drive continuous improvement of data engineering best practices and tooling.

Required Skills and Experience

• Bachelor's degree in Computer Science, Engineering, Mathematics, or related field.
• 5-7 years of experience in a database management, data engineering or similar role.
• Proficiency in programming languages such as Python or Scala.
• Strong proficiency in SQL and experience with relational databases (e.g. MSSQL, PostgreSQL, MySQL).
• Hands-on experience with No-SQL database technologies.
• Experience with Solr embeddings
• Experience with Event-driven architectures
• Experience in database optimization and performance tuning required.
• Good understanding of Data Integration patterns.
• Experience setting up MS SQL Replication and Data Archiving strategies will be beneficial.
• Experience with cloud platforms (e.g., AWS, GCP, Azure) and services like EC2, S3, Lambda, CloudWatch, EventBridge, Redshift, BigQuery, or Snowflake.
• Familiarity with big data technologies like Apache Spark, Data Bricks, and Hive.
• Familiarity with data modelling, warehousing concepts, and data governance practices.
• Exposure to Data cleansing and de-duplication techniques will be beneficial.

Desirable Skills

• Experience with stream processing tools (e.g., Kafka, Spark Streaming, Flink).
• Knowledge of containerization (Docker) and orchestration tools (Kubernetes).
• Familiarity with Jenkins and CloudFormation
• Exposure to BI tooling such as Power BI or Yellowfin is advantageous.
• Understanding of CI/CD principles and infrastructure-as-code.
• Exposure to machine learning workflows and MLOps.

Key Attributes

• Analytical mindset with a strong attention to detail.
• Proactive and solution-oriented with a continuous improvement mindset.
• Strong communication and collaboration skills.
• Able to work independently and as part of a cross-functional team.

Why Join Us?

• Work on exciting data-driven projects with real-world impact.
• Collaborate with a high-performing, inclusive, and passionate team.
• Access to learning opportunities, mentorship, and growth.
• Flexible hybrid work arrangements

Salary

Salary offered will be commensurate with level of skill and experience.

Location - Remote/Hybrid

Kaleidocode (Pty) Ltd is based in Durban, KZN, South Africa. Remote applications are welcome, but visits to our Durban or Centurion offices or client sites may be required.​

Responses

If you have not received a response within 4 weeks please consider your application to have been unsuccessful.

Data Privacy

By applying for this position, you consent to providing Kaleidocode Pivot Pty Ltd and Kaleidocode Pty Ltd with your personal information. This information will be used for the purposes of your application for a position within our company, and will not be made available to any other parties without your prior consent. Should your application be successful, we may require further personal or special information which will be obtained with your consent.

---