Specialist: Data Engineering

**Core Description**:
Support data driven solutions by leveraging your **software and data engineering** experience across a wide set of use cases within the Telkom Group and throughout its customer base. In this role, you will become a member of Telkom Strategic Insights (TSI), a new team within Telkom Group’s Strategic Division; focused on maturing Telkom’s data landscape by building data driven solutions for large-scale problems. To succeed, you should be highly skilled in **batch and real-time data ingestion, and data transformation with the ability to design, build and scale data pipelines for efficient data integration and processing. **TSI is one of the biggest data orientated teams in the country, with over 50 data professionals spread over 4 smaller teams: Data Management, Data Insights, AI/ML, and Engineering. This team structure enables us to achieve a holistic balance between start-up and corporate culture.

**Competencies**:

- Extensive knowledge and experience with data engineering and integration concepts and terms.
- Strong relational and non-relational database foundational knowledge.
- Able to create and maintain robust, automated data pipelines and DAG workflows.
- Experienced with analysis and improvement of business and software systems and processes.
- Deep knowledge and understanding of best practices, standards and deployment options enabling data transfer in batches or real time, on-prem and in the cloud.
- Cloud platform management (GCP, Azure, AWS, etc.)
- Experience with Linux, Python 3, Java, Scala, SQL, JavaScript, and tools in our stack including: PySpark, Hadoop, Cloudera, Kafka, Airflow, Argo, Prefect, Dagster, Redis, RabbitMQ, Beam, Arrow, Arrow Flight, Iceberg, Dremio, Ab Initio, Tableau, Alteryx, PowerBI, GitLab, Terraform, GCP, PowerPoint, Word & Excel.
- Communication (written and verbal)
- Stakeholder management
- Skilled planner
- Takes initiative
- Growth mindset
- Can do attitude
- Systems orientated
- Innovative thinker
- Works well in teams
- Works well under pressure
- Relishes dynamic/changing environment
- Hard working and conscientious
- Highly skilled in multitasking and context switching
- Excellent time management
- Can solve difficult problems
- Strives to achieve high quality output

**Responsibilities**:

- Data Engineering
- Architect, develop and maintain a declarative data integration solution with the ability to move data between databases, file systems and other locations while tracking data quality according to measures set by data owner.
- Enhance our big data and real-time data capabilities by using both open-source and internally developed tools, designed to efficiently move data while ensuring integrity.
- Oversee data support for specialized data centric services we provide, including ETL/ELT, data modelling, database design, data quality assessments, data format conversion, data transformation and data tool selection.
- Maintain up-to-date knowledge of data engineering tools, related technologies, data security and data privacy laws.
- Guide standards and best practices across the data stack.
- Processes, Automation and DevOps
- Contribute to the development and maintenance of our toolsets used to automate various jobs and scheduled tasks.
- Identify systemic inefficiencies, conceptualise possible solutions and drive their development.
- Ensure consistent documentation of all implemented tools, systems, and processes.
- Business and Leadership
- Use data engineering techniques to innovate and solve problems, translating business requirements into system designs.
- Engage with stakeholders to support the design and delivery of data science projects and solutions.
- Lead and develop a team of junior data engineers.
- Contribute to our agile way of work and innovation culture.

**Required Certification**:
Any data or cloud platform (GCP, Azure, AWS) certification is required, other relevant certifications will be highly advantageous.

**Qualifications**:
A formal qualification of at least NQF level 6 in Computer Science, Mathematics, Statistics, Software and/or Data Engineering or a related field. Any relevant specialised certifications or a post-graduate degree will be especially advantageous.

**Experience**:
3-5 years relevant experience, of which at least 2 years must have been in a data engineering environment.

**Special Requirements**:

- Experience with PySpark and other big data technologies
- Conscious of the various laws and privacy concerns surrounding personal and corporate data.
- Experience with task and workflow automation, DAGs, and relevant tooling.
- Experience with OOP and automated testing in Python.
- Experience with Google Cloud Platform and its products.
- Comfortable with mono-repo architecture and the relevant tooling.
- Experience with git workflows and interfaces.
- Experience with semantic versioning.

**Comments**: