Before DataSync, moving large volumes of data into AWS meant either shipping physical storage devices, or running general-purpose tools like rsync over the open internet — tools that were never designed for the scale, reliability, or security requirements of enterprise data migration. A single rsync process is fundamentally single-threaded in its file-comparison logic, doesn’t natively encrypt data in flight unless wrapped in additional tooling, and offers no built-in mechanism to verify that a multi-terabyte transfer completed with byte-for-byte integrity rather than silently corrupting a handful of files along the way. At petabyte scale, over links that may span thousands of miles and pass through unreliable networks, these gaps stop being theoretical and start being the actual cause of failed migrations.

Production example: media and entertainment companies migrating petabyte-scale video archives to AWS have cited DataSync’s parallel, multi-threaded transfer architecture as the difference between a migration measured in weeks versus one measured in months, because moving that volume of large media files with single-threaded tools simply cannot saturate available network bandwidth the way a purpose-built parallel transfer engine can.

This section assumes you already know DataSync “transfers data into and out of AWS.” It focuses on the architecture and vocabulary that matter once you’re actually configuring and operating transfer tasks.

Agents, Locations, and Tasks — the Three Building Blocks

Every DataSync transfer is built from three distinct objects. An Agent is the software component (deployed as a VM image or, for AWS-to-AWS transfers, not needed at all) that performs the actual data movement on the source or destination side when that side isn’t natively reachable by the DataSync service directly — for example, an on-premises NFS share needs an agent deployed in that data center to read from it. A Location represents a specific storage endpoint — an S3 bucket, an EFS file system, an on-premises NFS or SMB share, or a supported third-party or other-cloud object store — configured with the connection details DataSync needs to read or write there. A Task ties a source Location to a destination Location, along with the transfer configuration: what to include or exclude, how to handle deletions, and how verification should run.

Agentless vs. Agent-Required Transfers

A subtlety that trips up teams designing their architecture: transfers between AWS-native locations that DataSync can reach directly (S3 to EFS, for instance) don’t require an agent at all — DataSync’s managed service infrastructure handles the movement. Transfers involving on-premises storage, or self-managed storage in another cloud, require deploying at least one DataSync Agent inside that environment, because the agent is what actually establishes the connection to the storage protocol (NFS, SMB, or a self-managed object storage API) that DataSync’s managed service has no direct network path to reach.

Task Execution and Incremental Transfers

A Task can be run once or on a recurring schedule. Critically, DataSync doesn’t re-copy everything on every run — after the initial full transfer, subsequent executions perform an incremental sync by comparing file metadata (size, modification time, and optionally content checksums) between source and destination, transferring only what’s new or changed. This is what makes DataSync practical for ongoing replication scenarios, not just one-time migrations — a nightly task can keep a destination continuously synchronized with a constantly changing source without re-transferring unchanged data every night.

Filters, Deletion Handling, and Verification Modes

Tasks support include/exclude filters based on file path patterns, letting you scope a transfer to a specific subdirectory tree or exclude temporary files without needing a separate staging step. Deletion handling is configurable: by default a task does not delete files at the destination that were removed at the source, but this can be explicitly enabled for scenarios where the destination should mirror the source exactly, including deletions. Verification runs at one of several levels — from a lightweight metadata-only check up to a full data verification that recomputes and compares checksums for every transferred file — and the deeper verification level you choose trades transfer speed for stronger integrity guarantees.

The architecture cleanly separates control plane from data plane. The DataSync control plane — task scheduling, configuration, reporting — runs entirely as an AWS-managed service, meaning you never patch, scale, or manage the orchestration layer yourself. The data plane, where the actual bytes move, runs either entirely within AWS-managed infrastructure (for AWS-to-AWS transfers) or partially through a customer-deployed Agent (for any transfer touching on-premises or self-managed storage), which is the piece of the architecture you’re actually responsible for provisioning and keeping healthy.

graph LR
    subgraph OnPrem["On-Premises Data Center"]
        NFS["NFS / SMB
File Share"]
        AGENT["DataSync Agent
(VM Appliance)"]
    end

    subgraph AWSCloud["AWS Cloud"]
        CP["DataSync Control Plane
(Managed Service)"]
        S3["Amazon S3"]
        EFS["Amazon EFS"]
        FSX["Amazon FSx"]
    end

    NFS --> AGENT
    AGENT -->|"Encrypted
Data Transfer"| CP
    CP --> S3
    CP --> EFS
    CP --> FSX
    CP -.->|"Schedule, Filters,
Verification Config"| AGENT
        

This separation is what allows DataSync to scale a single task’s throughput by deploying multiple agents against the same source and letting the control plane parallelize work across them, without customers needing to build that orchestration logic themselves — the control plane already knows how to fan work out across however many agents are available and healthy.

When a task executes, DataSync’s control plane first performs a listing and comparison pass — enumerating files at both source and destination and identifying, based on metadata (and optionally content hashes), exactly which files need to be transferred, updated, or deleted. This comparison pass is what enables the incremental sync behavior described earlier: on a re-run against an unchanged dataset, DataSync can determine that little or nothing needs to move without re-transferring any actual file content.

For files that do need transferring, DataSync breaks large files into chunks and moves them in parallel across multiple concurrent streams, rather than transferring one file start-to-finish before beginning the next. This parallelism, combined with a proprietary network transfer protocol optimized for high-throughput links (rather than relying on standard TCP file-copy semantics the way a basic rsync-over-SSH transfer would), is the primary mechanism behind DataSync’s throughput advantage over generic copy tools — it’s specifically engineered to fill available bandwidth on high-latency, high-bandwidth links where a naive single-stream copy would leave most of that bandwidth unused due to TCP window-size limitations over long round-trip times.

After transfer, if verification is enabled, DataSync recomputes checksums on the destination copy and compares them against source checksums computed during the read phase, flagging any mismatch as a failed file in the task’s execution report rather than silently completing with a corrupted destination copy.

Trace a typical enterprise migration scenario. An organization deploys a DataSync Agent as a VM inside their data center, pointed at an existing NFS file share containing years of accumulated engineering documents. They create a source Location referencing that NFS share through the agent, and a destination Location referencing an S3 bucket in their target AWS account. A Task links the two, configured to preserve file timestamps and run an initial full verification pass given the one-time, high-stakes nature of this migration.

The first task execution transfers the full dataset — potentially terabytes — over days, parallelized across as many concurrent streams as the agent and network allow, throttled to a configured bandwidth cap so it doesn’t compete with production traffic during business hours. Once the bulk transfer completes and verification confirms integrity, the team schedules the same task to run nightly for the following two weeks, transferring only newly modified files each night, so that by the time they’re ready to fully cut over, the S3 destination is only minutes behind the live source rather than days stale. On cutover night, they run the task once more to capture any final changes, then redirect applications to read from S3 instead of the original NFS share.

DataSync’s throughput scales along two independent axes: the number of concurrent streams a single agent can drive against the source storage system, and the number of agents deployed in parallel against the same source when a single agent’s network interface becomes the bottleneck. For very large migrations, deploying multiple agents against different subsets of the source dataset — each handling its own task against the same destination — is the standard pattern for pushing total throughput beyond what any single agent’s network capacity allows.

On the destination side, writing to S3 benefits from S3’s own effectively unlimited horizontal write scalability, meaning the destination is rarely the bottleneck in an on-premises-to-S3 migration — the constraint is almost always the source storage system’s read throughput and the network path’s available bandwidth, not anything on the AWS side.

The DataSync control plane is a fully managed, multi-AZ AWS service, meaning task scheduling and orchestration itself is resilient to the loss of a single Availability Zone without customer intervention. The reliability consideration that actually falls to the customer is the agent — since the agent is a customer-deployed VM, a single agent represents a single point of failure for any task that depends on it, and enterprises running critical, time-sensitive migrations commonly deploy redundant agents specifically so a task can continue, or be quickly redirected, if one agent instance becomes unavailable mid-transfer.

Data is encrypted in transit by default between the agent and AWS, and data at rest inherits the encryption configuration of the destination storage service (S3 server-side encryption, EFS encryption, and so on). IAM policies control which principals can create, modify, or execute tasks, and separately, resource-based policies on the destination storage (like an S3 bucket policy) control what the DataSync service role itself is permitted to write, giving two independent layers of access control over who can configure a transfer and what that transfer is actually allowed to touch.

Agent deployment is typically automated through the organization’s existing VM provisioning pipeline — the agent ships as a standard VM image importable into VMware, Hyper-V, or KVM environments, or as an EC2 AMI for transfers originating from another cloud’s compute. Task definitions, like most production AWS resources, are commonly managed as infrastructure-as-code so that recurring migration or replication tasks are version-controlled and reproducible rather than clicked together manually and undocumented.

Media companies migrating large video and post-production archives have described using DataSync’s parallel, multi-agent transfer capability to move multi-petabyte on-premises storage libraries into S3 within timeframes that would have been impractical with generic file-copy tooling, given the sheer volume of large binary media files involved. Life sciences and genomics organizations, dealing with large sequencing datasets generated continuously by lab instruments, have used scheduled DataSync tasks to keep an S3-based analysis pipeline continuously synchronized with on-premises instrument storage, rather than manually staging and uploading new data batches. Enterprises undergoing data-center consolidation projects have used DataSync as the core transfer mechanism for their broader migration programs, relying on its built-in verification specifically because the cost of silently corrupting even a small percentage of files across a multi-petabyte transfer would be far more expensive to discover and remediate after the fact than to prevent during transfer.