Data Engineer

Job description

Implement best practices for data governance, security, and compliance across Azure Data Lake and related technologies.

Monitor, troubleshoot, and optimize the performance of data pipelines, ensuring high availability and reliability.

Drive the adoption of DevOps principles in data engineering to automate deployment and ensure continuous integration continuous delivery CI CD practices.

Stay updated with the latest trends and advancements in Azure data services, machine learning, and analytics.

Qualification

Required Skills and Qualifications

Bachelors degree in computer science, Information Technology, Data Science, or related field.

3 plus years of experience in data engineering and visualization, with at least 2 years of hands on experience working with Power BI Azure Synapse Analytics and Azure Data Lake

Proficient in Python SQL for building data transformation and ETL pipelines

Strong knowledge of Power BI including creating and publishing interactive dashboards reports and data models

Extensive knowledge of Azure Synapse Analytics for data warehousing querying, and large scale analytics

Hands on experience with Apache Spark for big data processing transformation and machine learning workflows

Understanding of data governance best practices in a cloud based environment including data security privacy and compliance

Familiarity with version control systems e.g Git and DevOps practices in data engineering

Strong problem solving and troubleshooting skills for identifying and resolving data pipeline and processing issues

Ability to work in an agile environment and manage multiple priorities efficiently

Power BI

Azure Synapse

ADLS

SQL Data database

Microsoft Fabric

Azure data services

Pipeline orchestration

Python

PySpark