Data engineering on Azure: where to start

Data engineering on Azure comes up constantly in job postings and technology roadmaps, but where do you actually start if you’re new to the field? This guide covers the fundamentals, the main tools, and the current certification path, which has changed recently.

What is data engineering, in one sentence?

Data engineering is the process of integrating, transforming, and consolidating data from structured, unstructured, and streaming sources into formats that can support analytics solutions.

In practical terms, it is the work that makes data usable before an analyst or a Power BI report can do anything with it.

The main responsibilities of a data engineer

• Design and implement pipelines that move and transform data between systems.
• Select and implement the right storage architecture based on business needs, whether that means a modern data warehouse, a data lake, or a hybrid approach.
• Ensure that pipelines and data stores remain performant, reliable, and well governed.
• Secure data both at rest and in transit, while documenting lineage for compliance and traceability.

The most commonly used Azure tools

• Azure Data Factory: Data orchestration and integration for building ETL pipelines without having to write all of the low-level code.
• Azure Databricks: An Apache Spark-based platform for large-scale data processing and advanced machine learning.
• Azure Data Lake Storage: A foundational storage layer for raw and semi-structured data at scale.
• Azure Synapse Analytics and, increasingly, Microsoft Fabric: Warehousing and analytics layers that consume the data prepared by your pipelines.

An important change: the certification path has evolved

Microsoft retired the DP-203 exam, Data Engineering on Microsoft Azure, which had been the reference certification in this area for years.

Its official replacement is DP-700, Fabric Data Engineer Associate. It covers many of the same foundational concepts, but within the unified Microsoft Fabric environment rather than across separate Azure services such as Synapse and the classic version of Data Factory.

If you are planning a certification today, DP-700 is the current path to follow, not DP-203.

What changes with Fabric in practice?

For someone already familiar with traditional Azure services, moving to Fabric is more of a shift in how the tools are organized than a complete reset.

OneLake takes on the role that Azure Data Lake Storage Gen2 traditionally played as the storage foundation. Fabric Data Factory follows the same orchestration logic as Azure Data Factory, but is built directly into Fabric. Fabric lakehouses and warehouses take the place of many of the dedicated or serverless SQL scenarios previously handled in Synapse.

The underlying concepts, such as medallion architecture and ingestion pipelines, remain largely the same.

Where to start

1. Strengthen your SQL skills and, ideally, learn Python or Scala, two of the most common languages used for data processing.
2. Learn the fundamentals of pipelines and orchestration through Azure Data Factory or, if you are starting today, directly through Fabric Data Factory.
3. Practise transforming data with Apache Spark, either in Azure Databricks or with Fabric’s native notebooks.
4. Understand medallion architecture, using bronze, silver, and gold layers as a reference model for organizing data in a data lake or lakehouse.

Should you focus on traditional Azure or Microsoft Fabric?

For an organization starting a new data initiative today, Microsoft is clearly shifting toward Fabric as its reference platform, and the retirement of DP-203 reflects that direction.

That said, the core skills of data engineering, including pipeline design, dimensional modelling, and governance, remain transferable from one environment to another.

The strongest long-term approach is still to learn the concepts first, then choose the specific tools based on your organization’s context.

In summary

Data engineering on Azure remains a fundamental discipline, but its centre of gravity is moving toward Microsoft Fabric.

Our data engineering team supports organizations that are building or modernizing their data pipelines, whether they are still using traditional Azure services or already moving toward Fabric.

Evaluating your data engineering strategy? Talk to one of our experts.

‍