Data Engineer

About the role

We are seeking a Data Engineer to design, build, and maintain data pipelines and Lakehouse structures within the Microsoft Fabric ecosystem. Reporting to the solution architect, you will translate architectural designs into high‑performance, scalable data solutions that support mission‑critical EU programmes.

Key responsibilities

• Implement Delta Lake storage layers (Bronze, Silver, Gold) with proper partitioning and Z‑Ordering.
• Develop robust ETL/ELT workflows using Fabric Data Factory, Spark, and custom code to ingest data from diverse sources into OneLake.
• Write and maintain high‑performance data processing modules or custom connectors in Java and/or Python.
• Integrate Microsoft Fabric with external platforms such as AWS, GCP, or on‑premises systems via Shortcuts and APIs.
• Create complex data transformations in Spark Notebooks, applying cleansing and business logic.
• Conduct unit testing, data profiling, and performance benchmarking to ensure pipeline stability and data integrity.

Required profile

• Expert proficiency in SQL and Python (PySpark); strong working knowledge of Java.
• Solid understanding of software engineering principles applied to data (DRY code, modularity, error handling, logging).
• Experience with structured, semi‑structured (JSON/XML) and unstructured data formats.
• Practical experience with Git, branching strategies, and CI/CD pipelines using Azure DevOps or GitHub Actions.
• Deep knowledge of Apache Spark internals, memory management, and execution plans.
• Good written and spoken English; French is a plus.

Required skills

• SQL
• Python / PySpark
• Java
• Microsoft Fabric
• Delta Lake
• Fabric Data Factory
• OneLake
• Apache Spark
• Azure DevOps
• Git / GitHub Actions
• AWS
• GCP
• APIs and Shortcuts