Disclaimer: This is work in progress intended to consolidate information from various sources for learning purposes. For the latest information please consult the documentation (see the links below)!
Last updated: 11-Apr-2025
[Microsoft Fabric] Copy job in Data Factory- {def}
- {benefit} simplifies data ingestion with built-in patterns for batch and incremental copy, eliminating the need for pipeline creation [1]
- across cloud data stores [1]
- from on-premises data stores behind a firewall [1]
- within a virtual network via a gateway [1]
- elevates the data ingestion experience to a more streamlined and user-friendly process from any source to any destination [1]
- {benefit} provides seamless data integration
- through over 100 built-in connectors [3]
- provides essential tools for data operations [3]
- {benefit} provides intuitive experience
- easy configuration and monitoring [1]
- {benefit} efficiency
- enable incremental copying effortlessly, reducing manual intervention [1]
- {benefit} less resource utilization and faster copy durations
- flexibility to control data movement [1]
- choose which tables and columns to copy
- map the data
- define read/write behavior
- set schedules that fit requirements whether [1]
- applies for a one-time or recurring jobs [1]
- {benefit} robust performance
- the serverless setup enables data transfer with large-scale parallelism
- maximizes data movement throughput [1]
- fully utilizes network bandwidth and data store IOPS for optimal performance [3]
- {feature} monitoring
- once a job executed, users can monitor its progress and metrics through either [1]
- the Copy job panel
- shows data from the most recent runs [1]
- reports several metrics
- status
- row read
- row written
- throughput
- the Monitoring hub
- acts as a centralized portal for reviewing runs across various items [4]
- {mode} full copy
- copies all data from the source to the destination at once
- {mode|preview} incremental copy
- the initial job run copies all data, and subsequent job runs only copy changes since the last run [1]
- an incremental column must be selected for each table to identify changes [1]
- used as a watermark
- allows comparing its value with the same from last run in order to copy the new or updated data only [1]
- the incremental column can be a timestamp or an increasing INT [1]
- {scenario} copying from a database
- new or updated rows will be captured and moved to the destination [1]
- {scenario} copying from a storage store
- new or updated files identified by their LastModifiedTime are captured and moved to the destination [1]
- {scenario} copy data to storage store
- new rows from the tables or files are copied to new files in the destination [1]
- files with the same name are overwritten [1]
- {scenario} copy data to database
- new rows from the tables or files are appended to destination tables [1]
- the update method to merge or overwrite [1]
- {default} appends data to the destination [1]
- the update method can be adjusted to
- {operation} merge
- a key column must be provided
- {default} the primary key is used, if available [1]
- {operation} overwrite
- availability
- the same regional availability as the pipeline [1]
- billing meter
- Data Movement, with an identical consumption rate [1]
- {feature} robust Public API
- {benefit} allows to automate and manage Copy Job efficiently [2]
- {feature} Git Integration
- {benefit} allows to leverage Git repositories in Azure DevOps or GitHub [2]
- {benefit} allows to seamlessly deploy Copy Job with Fabric’s built-in CI/CD workflows [2]
- {feature|preview} VNET gateway support
- enables secure connections to data sources within virtual network or behind firewalls
- Copy Job can be executed directly on the VNet data gateway, ensuring seamless and secure data movement [2]
- {feature} upsert to Azure SQL Database
- {feature} overwrite to Fabric Lakehouse
- {feature} [Jul-2025] native CDC
- enables efficient and automated replication of changed data including inserted, updated and deleted records from a source to a destination [5]
- ensures destination data stays up to date without manual effort
- improves efficiency in data integration while reducing the load on source systems [5]
- see Data Movement - Incremental Copy meter
- consumption rate of 3 CU
- {benefit} zero manual intervention
- automatically captures incremental changes directly from the source [5]
- {benefit} automatic replication
- keeps destination data continuously synchronized with source changes [5]
- {benefit} optimized performance
- processes only changed data
- reduces processing time and minimizing load on the source [5]
- smarter incremental copy
- automatically detects CDC-enabled source tables and allows to select either CDC-based or watermark-based incremental copy for each table [5]
- applies to
- CDC-enabled tables
- CDC automatically captures and replicates actions on data
- non-CDC-enabled tables
- Copy Job detects changes by comparing an incremental column against the last run [5]
- then merges or appends the changed data to the destination based on configuration [5]
- supported connectors
- ⇐ applies to sources and destinations
- Azure SQL DB [5]
- On-premises SQL Server [5]
- Azure SQL Managed Instance [5]
- {enhancement} column mapping for simple data modification to storage as destination store [2]
- {enhancement} data preview to help select the right incremental column [2]
- {enhancement} search functionality to quickly find tables or columns [2]
- {enhancement} real-time monitoring with an in-progress view of running Copy Jobs [2]
- {enhancement} customizable update methods & schedules before job creation [2]
References:
[1] Microsoft Learn (2025) Fabric: What is the Copy job in Data Factory for
Microsoft Fabric? [link]
[2] Microsoft Fabric Updates Blog (2025) Recap of Data Factory Announcements
at Fabric Conference US 2025 [link]
[3] Microsoft Fabric Updates Blog (2025) Fabric: Announcing Public
Preview: Copy Job in Microsoft Fabric [link]
[4] Microsoft Learn (2025) Fabric: Learn how to monitor a Copy job in
Data Factory for Microsoft Fabric [link]
[5] Microsoft Fabric Updates Blog (2025) Fabric: Simplifying Data Ingestion with Copy job – Introducing Change Data Capture (CDC) Support (Preview) [link]
[6] Microsoft Learn (2025) Fabric: Change data capture (CDC) in Copy Job (Preview) [link]
[5] Microsoft Fabric Updates Blog (2025) Fabric: Simplifying Data Ingestion with Copy job – Introducing Change Data Capture (CDC) Support (Preview) [link]
[6] Microsoft Learn (2025) Fabric: Change data capture (CDC) in Copy Job (Preview) [link]
Resources:
[R1] Microsoft Learn (2025) Fabric: Learn how to create a Copy job in Data Factory for Microsoft Fabric [link]
[R1] Microsoft Learn (2025) Fabric: Learn how to create a Copy job in Data Factory for Microsoft Fabric [link]
Acronyms:
API - Application Programming Interface
CDC - Change Data Capture
CI/CD - Continuous Integration and Continuous Deployment
CDC - Change Data Capture
CI/CD - Continuous Integration and Continuous Deployment
CU - Capacity Unit
DevOps - Development & Operations
DF - Data Factory
IOPS - Input/Output Operations Per Second
IOPS - Input/Output Operations Per Second
VNet - Virtual Network
