Data Engineer

REQUIREMENTS Minimum education (essential): BSc in Computer Science, Engineering or relevant field Minimum applicable experience (years): 2-4 years Required nature of experience: Experience with SQL Server and Azure Synapse Analytics/Microsoft Fabric for query writing, indexing, performance tuning and schema design. Hands-on experience developing ETL pipelines, including data extraction from REST/SOAP APIs, databases and flat files. Proficiency in data transformation using Python and Azure-native tools. Experience with data warehousing. Background in data modelling, including dimensional modelling, schema evolution and versioning. Practical knowledge of cloud-based data storage and processing using Azure Blob Storage. Familiarity with pipeline optimisation, fault tolerance, monitoring and security best practices. Experience developing web applications using C# and the .NET platform. Experience with front-end development using Blazor, React.js, JavaScript/TypeScript, HTML, CSS/SCSS. Skills and Knowledge (essential): SQL Server, Azure Synapse Analytics, Azure Blob Storage, Microsoft Fabric Python REST/SOAP APIs, Data Extraction, Transformation, Loading (ETL) Azure Data Factory, Pipeline Orchestration Dimensional Modelling, Schema Evolution, Data Warehousing Power BI Performance Optimisation, Indexing, Query Tuning Cloud Data Processing, Backups C#, .NET, Blazor JavaScript/TypeScript, HTML, CSS/SCS Other: Proficient in Afrikaans and English Own transport and license KEY PERFORMANCE AREAS AND OBJECTIVES ETL and Pipeline Development Design, build, and orchestrate efficient ETL pipelines using Azure Synapse for both batch and near-real-time data ingestion. Extract data from a variety of structured and unstructured sources including REST APIs, SOAP APIs, databases, and flat files. Apply robust data transformation logic using Python and native Azure Synapse transformation tools. Optimise data flows for performance, scalability, and cost-effectiveness. Implement retry mechanisms, logging and monitoring within pipelines to ensure data integrity and fault tolerance. Data Architecture and Management Design and manage scalable and efficient data architectures using Microsoft SQL Server and Azure services, including Synapse Analytics/Microsoft Fabric and Blob Storage. Develop robust schema designs, indexes and query strategies to support analytical and operational workloads. Support schema evolution and version control, ensuring long-term maintainability and consistency across datasets. Implement and maintain metadata repositories and data dictionaries for improved data governance and transparency. Define and maintain role-based access control to ensure data security and compliance. Data Warehousing and BI Integration Architect and manage enterprise data warehouses using Azure Synapse Analytics. Apply best practices for data loading, partitioning strategies, and storage optimisation. Integrate warehousing solutions with Power BI and other analytics platforms for seamless business intelligence consumption. Data Modelling & Standards Develop and maintain conceptual, logical and physical data models. Implement dimensional modelling techniques (e.g., star/snowflake schemas) to support advanced analytics and reporting. Apply normalisation standards and relational modelling techniques to support OLTP and OLAP workloads. Ensure consistency of data models across systems and support schema versioning and evolution. Reporting and Communication Provide clear, timely updates on task status and progress to senior developers / management. Contribute to reports, manuals, and other documentation related to software status, operation, and maintenance. Collaborate effectively with team members and stakeholders using the appropriate communication channels. Maintain system and product change logs and release notes according to company standards. Automation, Monitoring and Optimisation Automate recurring data engineering tasks and deploy solutions with CI/CD best practices. Implement monitoring and alerting mechanisms to detect data quality issues and pipeline failures. Analyse and optimise query performance across platforms (SQL Server, Azure Synapse). Support scalability planning and cost control by monitoring pipeline execution and resource usage Security and Best Practices Enforce security best practices for data access, including encryption and secure authentication. Ensure compliance with data governance policies and applicable regulatory standards. Document processes, architectural decisions and technical implementations in alignment with organisational standards Contribution to The Team Collaborate with developers, data analysts, data scientists and business teams to understand data requirements and deliver scalable solutions. Work with the team to integrate pipelines with source control and deployment workflows Quality Management and Compliance Document data processes, transformations and architectural decisions. Maintain high standards of software quality within the team by adhering to good processes, practices and habits. Ensure compliance to the established processes and standards for the development lifecycle, including but not limited to data archival. Safeguard confidential information and data.