Data Engineer – Azure Databricks

PBT Group is seeking an experienced Data Engineer to join a project focused on implementing a modern Lakehouse architecture using the Medallion framework.

The successful candidate will be responsible for developing and maintaining data ingestion pipelines, landing data in the Bronze layer, and supporting data transformation and semantic modelling through the Silver and Gold layers. The objective is to deliver governed, reusable, production-ready and AI-ready data products that support advanced analytics, Power BI reporting and data science use cases.

Working closely with an in-house Data Scientist and other technical stakeholders, the Data Engineer will contribute to the development of reliable data products and business views, with a strong emphasis on production deployment, data governance, performance optimisation and operational excellence.

Key Responsibilities

Data Engineering and Lakehouse Architecture

• Design, develop and maintain scalable data pipelines using Azure Databricks and Azure Data Factory (ADF).

• Implement and support Medallion architecture, including Bronze, Silver and Gold data layers.

• Ingest data from multiple sources and ensure reliable landing, transformation and processing.

• Develop and maintain data integration and ingestion patterns that support batch and real-time data processing.

• Contribute to the design and implementation of robust, scalable and maintainable Lakehouse solutions.

Data Transformation and Modelling

• Develop data pipelines and transformation logic using Python, PySpark and SQL.

• Support semantic data modelling across Silver and Gold layers to deliver high-quality, reusable and governed data products.

• Develop data models and business views that support Power BI reporting, analytics and data science requirements.

• Ensure datasets are accurate, consistent, documented and suitable for downstream analytical and AI use cases.

• Apply data warehousing principles and appropriate modelling techniques to support business requirements.

Azure Data Services and Integration

• Work with Azure data services and technologies to develop, integrate and maintain data solutions.

• Implement data ingestion and integration pipelines using Azure Data Factory, Azure Databricks and other relevant Azure services.

• Apply API integration and data ingestion patterns to connect systems and source platforms.

• Where required by the solution, work with Azure Functions, Apache Kafka and Apache Spark to support integration, processing and streaming requirements.

Data Governance and Quality

• Apply data governance, access control and catalogue management practices using Databricks Unity Catalog and relevant gateway capabilities.

• Ensure that data products comply with established governance, security, quality and access requirements.

• Support data lineage, documentation, discoverability and reusability across the data environment.

• Ensure that data products are well governed and structured to support analytics and AI-driven use cases.

Production Deployment and DevOps

• Develop, deploy and maintain production-ready data pipelines and data products.

• Apply Git version control, release management and established DevOps practices.

• Contribute to CI/CD pipeline development, automation and controlled deployment processes.

• Monitor and troubleshoot pipeline performance, data quality issues and production incidents.

• Perform performance tuning and optimisation to improve pipeline efficiency, reliability and scalability.

• Maintain clear technical documentation covering architecture, design decisions, pipelines, transformations and data models.

Collaboration and Continuous Improvement

• Collaborate with Data Scientists and other stakeholders to translate analytical and business requirements into production-ready data products.

• Develop datasets and business views that support Power BI reporting and AI use cases.

• Work within an Agile delivery framework and contribute to planning, reviews and continuous improvement.

• Support the use of Databricks Genie where relevant to the solution.

• Promote engineering best practices, reusable components and operational excellence throughout the data delivery lifecycle.

Essential Skills and Experience

Candidates should demonstrate hands-on experience in the following areas:

<