Before meeting the tools, let us understand the problem they solve. Copying a file feels trivial. Copying ten million files across the internet is a different sport entirely.

The simple idea and the hard reality

Data movement means taking information that lives in one place and making an exact copy of it in another place. The two places could be your laptop and the cloud, two different cloud accounts, or even two regions of the world. On the surface this is nothing more than “copy and paste.” In practice, four things make it hard: the amount of data, the distance it travels, the number of separate files, and the fact that something will always go wrong along the way.

When you copy one small document, none of these matter. When you copy a hundred terabytes made of forty million files, every one of them matters. A single dropped connection can waste a day. A single wrong permission can stop a job at midnight. A single tiny file multiplied by millions can be slower than one giant file of the same total size.

Three goals every transfer must meet

Whatever tool you use, a good transfer achieves three goals at once. The first is speed: finishing in a reasonable time by using all the network capacity you have. The second is safety: the data arrives complete and unchanged, and nobody who should not see it can see it. The third is repeatability: you can run the same transfer again tomorrow, or resume it after a failure, without starting over or creating duplicates.

Most beginner mistakes come from chasing only one goal. People who chase speed forget about safety and leave a wide-open access link lying around. People who chase safety copy files one at a time by hand and never finish. People who ignore repeatability end up with a half-copied mess and no record of what happened.

Doing the math before you start

The most valuable habit in data movement is to estimate the time before you press start. The rule is simple. A network link measured in megabits per second (Mbps) or gigabits per second (Gbps) tells you how many bits cross the wire each second. Divide by eight to get bytes. A 1 Gbps link carries at best 125 megabytes per second, which is about 10.8 terabytes per day if it ran flat out all day long.

Real links never run at their full rated speed. Overhead, other users on the same network, and busy servers usually leave you with something like 60 to 80 percent of the headline number. The table below shows what that means for moving 100 terabytes.

An illustrative example

Picture a hypothetical online retailer with 40 terabytes of product photos and order documents on servers in its own office. It has a 500 Mbps internet connection. At 70 percent efficiency, that link delivers about 44 megabytes per second, or roughly 3.8 terabytes per day. The first full copy would take around eleven days. After that, only the new and changed files need to be copied each night, which might be a few gigabytes. A good tool handles both jobs: the big first load and the small daily top-up. That pattern, big load first and small updates after, shows up again and again in this guide.

A tool is only as useful as your understanding of the place it delivers to. This chapter introduces the vocabulary of Azure Storage in plain words.

The storage account: your warehouse

Everything in Azure Storage lives inside a storage account. A storage account is a named container for your data that sits in one Azure region, such as East US or West Europe. It has its own web address, its own security settings, and its own bill. Inside the account you can keep several different kinds of data, each handled by a separate service.

The services inside the account

Containers, blobs, and the folder illusion

Inside the blob service, data is organised into containers. A container is the top-level bucket that holds blobs. Containers cannot be nested inside each other. A blob is simply one object with a name, such as invoices/2025/march/inv-001.pdf.

Here is a surprise that confuses almost every beginner. In a normal blob container, there are no real folders. The slashes in a blob name are just characters. Tools such as Storage Explorer draw them as folders to make life easier, but underneath, the container is a flat list of names. This matters for data movement because “renaming a folder” actually means copying and deleting every blob whose name starts with that prefix.

When you turn on the hierarchical namespace option, the account becomes a Data Lake Storage Gen2 account. Now folders are real objects, and renaming a folder is a single quick operation. Both AzCopy and Storage Explorer understand this difference and behave sensibly in either mode.

Three types of blobs

Blobs come in three flavours. Block blobs are built from blocks that can be uploaded in any order and then committed together. They are the default and the best fit for files, media, and backups. Append blobs only allow adding data to the end, which suits log files. Page blobs store data in fixed-size pages and are used mainly for virtual machine disks. AzCopy lets you choose the blob type when uploading, and for most work you will use block blobs.

Access tiers: hot, cool, cold, and archive

Not all data is read equally often. Azure lets you place each blob in an access tier that trades storage price against access price. The Hot tier costs more to store and less to read. The Cool and Cold tiers cost less to store but charge more to read and expect you to keep the data for a minimum time, roughly 30 days for Cool and 90 days for Cold. The Archive tier is the cheapest to store, expects about 180 days of retention, and keeps data offline, so you must wait to “rehydrate” a blob before reading it.

Why does this matter for moving data? Because the tier you choose at upload time affects your bill from day one. AzCopy can set the tier while it uploads, which is better than uploading everything as Hot and moving it later, because a later move can trigger extra charges.

Redundancy: how many copies Azure keeps

Azure keeps several copies of your data to protect against hardware failure. Locally redundant storage (LRS) keeps three copies inside one data centre. Zone-redundant storage (ZRS) spreads them across three separate zones in a region. Geo-redundant storage (GRS) adds a second region far away. You pick this setting on the storage account, not in the tool, but it shapes your plan: moving data into a geo-redundant account means Azure also copies it to the paired region in the background, which does not slow your upload but is worth knowing about.

Both tools are free. Both talk to the same storage accounts. They simply have different personalities and different best uses.

What is AzCopy?

AzCopy is a command-line utility published by Microsoft for copying data to, from, and between Azure Storage accounts. “Command-line” means you type instructions into a text window instead of clicking buttons. It is a single small program with no installer: you download it, put it somewhere on your computer, and run it. It works on Windows, Linux, and macOS, and it is also available inside Azure Cloud Shell, the browser-based terminal in the Azure portal.

AzCopy is built for scale and automation. It can move many files at the same time, it breaks big files into pieces and sends the pieces in parallel, it retries what fails, and it remembers the state of each job so you can resume after an interruption. Because it is just a program, you can place it inside a script, schedule it to run every night, or call it from an automated build pipeline.

The current major version is called AzCopy v10. Older versions such as v8 and v7 are retired, so if you find an old blog post that mentions them, treat the details with caution.

What is Azure Storage Explorer?

Azure Storage Explorer is a free desktop application for Windows, macOS, and Linux. It shows your storage accounts as a tree on the left and the contents of whatever you select on the right, much like the file manager on your own computer. You can upload and download by dragging, create and delete containers, edit properties, generate access links, and look inside blobs, file shares, queues, and tables. Azure also offers a similar browser-based experience called Storage browser inside the Azure portal, which is handy when you cannot install software.

Storage Explorer is built for people. It is the best way to look around, to check that a transfer worked, to fix one file, or to share a single blob with a colleague. A key fact to remember is that Storage Explorer uses AzCopy under the hood for its blob and file transfers. When you drag a folder into Storage Explorer, the heavy lifting is done by AzCopy, with the graphical app acting as a friendly front desk.

Side-by-side comparison

Related helpers you may meet

You will see two other names in documentation. The Azure CLI is a general command-line tool for managing Azure, and its storage copy command actually calls AzCopy for the heavy work. Azure PowerShell offers similar commands for people who prefer that language. If you already live in one of these, you can often stay there and still get AzCopy’s speed.

Even though AzCopy is a single file, inside it works like a small factory with several departments. Knowing the departments makes errors and settings much easier to understand.

flowchart LR
  CLI["AzCopy command line front end"] --> JM["Job Manager"]
  AUTH["Credential handler"] --> JM
  JM --> ENUM["Enumerator and scanner"]
  ENUM --> PLAN["Job plan files on local disk"]
  PLAN --> SCHED["Transfer scheduler"]
  SCHED --> POOL["Worker pool and chunk queue"]
  POOL --> SRC["Source location"]
  POOL --> DST["Destination location"]
  JM --> LOG["Log writer"]
  POOL --> LOG

The figure shows a job moving from left to right. Let us walk through each part in turn.

The command-line front end

This is the part you talk to. It reads the words you type, such as which action you want, where the source is, where the destination is, and which options you chose. It checks that your request makes sense and then hands a clean job description to the rest of the program. If you mistype an option, this is the department that complains.

The credential handler

Before anything moves, AzCopy must prove to Azure that you are allowed to do this. The credential handler figures out which method you chose: a signed-in Microsoft Entra ID identity, a managed identity, a service principal, or a shared access signature (SAS) token attached to the web address. It attaches the right proof to every request. Chapter 6 covers these methods in detail.

The Job Manager

Every copy, sync, or remove request becomes a job with a unique identifier. The Job Manager owns the life of that job. It creates it, tracks progress, counts successes and failures, and answers when you ask “how is job number so-and-so doing?” Because jobs have identifiers, you can list them, inspect them, and resume them later.

The enumerator and scanner

To copy a folder of files, AzCopy must first know what is in the folder. The enumerator walks through the source, whether that is a local disk, a container, or another cloud, and produces a list of items to transfer. The clever part is that it does this at the same time as transferring. It does not wait to finish listing a million files before starting to send the first one. This overlap keeps the network busy from the first second.

Job plan files

As the scanner finds items, AzCopy writes a plan describing each transfer into small files on your local disk. These plan files live in a hidden folder named .azcopy inside your user home directory. The plan records which files are pending, in progress, completed, or failed. This is the memory that makes resuming possible. If the program crashes or you close the laptop, the plan still knows where things stood.

The transfer scheduler and worker pool

The scheduler decides which pieces of work run next. Each file may be broken into chunks, and each chunk becomes a unit of work placed on a queue. A worker pool is a set of tiny parallel workers, each of which picks up a chunk, fetches it from the source, and writes it to the destination. The number of workers is the concurrency, and it is the single biggest dial for speed.

The log writer

Every important event is written to a log file named after the job identifier, in the same .azcopy folder. The log records what was attempted, what failed, and why. When something goes wrong at three in the morning, this log is your detective’s notebook.

This chapter follows a large file on its journey so you can see exactly why AzCopy is fast and why it can survive interruptions.

Why big files are cut into blocks

Uploading a 50 gigabyte file as one long stream would be slow and fragile. If the connection died at 49 gigabytes, you would start from zero. Azure block blobs solve this by letting a file arrive as many separate blocks. AzCopy slices the file into blocks, uploads many blocks at the same time over separate connections, and finally sends a “commit” message that tells Azure to assemble the blocks into one blob in the right order.

The service allows up to 50,000 blocks per blob and each block can be as large as 4,000 mebibytes. Multiply those numbers and you get the roughly 190 tebibyte ceiling for a single blob mentioned earlier. AzCopy chooses a block size automatically based on the file size, and you can override it if you have a reason.

sequenceDiagram
  participant Disk as Local disk
  participant AZ as AzCopy workers
  participant Blob as Azure Blob service
  Disk->>AZ: Read block 1, 2, 3 and more in parallel
  AZ->>Blob: Put Block 1
  AZ->>Blob: Put Block 2
  AZ->>Blob: Put Block 3
  Blob-->>AZ: Acknowledge each block
  AZ->>AZ: Retry any block that failed
  AZ->>Blob: Put Block List to commit
  Blob-->>AZ: Blob is now complete

Many small files are a different challenge

A single huge file is limited by bandwidth: how many bits per second the wire can carry. A million tiny files are limited by something else, the number of requests per second. Each small file needs its own set of conversations with the server, and the time spent talking can exceed the time spent carrying data. For this reason, a folder of ten million 10-kilobyte files can take far longer than one 100-gigabyte file, even though the totals are similar. AzCopy handles this by running many files in parallel, but the request rate that Azure Storage allows still sets a ceiling.

Retries and the idea of idempotency

Networks are messy. Packets get lost, servers say “I am busy, try again,” and wireless connections hiccup. AzCopy treats these as normal events. When a chunk fails with a temporary error, it waits a short time and tries again, and it lengthens the wait after each failure. This pattern is called exponential backoff, and it prevents a flood of angry retries from making an overloaded server even slower.

Retrying is safe because most storage operations are idempotent, a long word meaning that doing the same thing twice leaves the same result as doing it once. Uploading block number 7 twice just replaces block 7 with identical data. This property is why a robust tool can be so relentless about retrying.

Integrity checking with hashes

How do you know the copy is identical to the original? The answer is a hash, a short fingerprint calculated from a file’s contents. If even one bit changes, the fingerprint changes. AzCopy can calculate an MD5 fingerprint of each local file during upload and store it with the blob. On download, it can recompute the fingerprint and compare. If they differ, the file is flagged as failed rather than silently accepted. Turning these checks on costs a little time but gives strong confidence in important data.

The life of a job, step by step

Resuming a job after an interruption

Because the plan files remember every item’s state, AzCopy can resume a job. You list your jobs, find the identifier, and ask AzCopy to resume it. Only the unfinished items are retried. You can also ask it to resume while skipping or retrying only failed items, which is useful when you have fixed the cause of the failure.

Server-to-server copies: the data never visits you

When both the source and the destination are in Azure Storage, or when the source is a supported cloud such as Amazon S3, AzCopy can ask the destination service to pull the data directly from the source. The bytes travel across the provider networks, not through your laptop. Your machine only sends small instructions. This is why a laptop on a slow hotel connection can still orchestrate a massive account-to-account copy: the laptop is the conductor, not the orchestra.

Before Azure lets a single byte in or out, it asks two questions. Who are you, and are you allowed to do this? This chapter explains how AzCopy and Storage Explorer answer both.

Authentication versus authorization

Authentication is proving your identity, like showing a passport at a border. Authorization is the list of things that identity is allowed to do, like the visa stamped inside the passport. A person can be perfectly authenticated and still be refused because their visa does not cover the action they attempt. Many beginner errors that show as “403 Forbidden” are authorization problems, not sign-in problems.

Method one: Microsoft Entra ID sign-in

Microsoft Entra ID (formerly known as Azure Active Directory) is the identity system that holds your work or school account. With AzCopy, you sign in once with a login command, and AzCopy then uses short-lived tokens issued to you. No secret ever sits inside a file or a script. This is the method Microsoft recommends for people, because access can be controlled centrally, revoked instantly, and audited.

Signing in is only half the story. You also need a role assignment that grants data permissions. The most common roles for blobs are Storage Blob Data Reader (read and list), Storage Blob Data Contributor (read, write, and delete), and Storage Blob Data Owner (everything, including setting access rules on Data Lake folders). A role can be assigned at the level of the storage account, a single container, or even a resource group or subscription above them.

Method two: shared access signatures (SAS)

A shared access signature is a long web address with a special token attached. The token states what is allowed, for how long, and from which network addresses. Anyone who holds the link can do exactly what the token permits, without signing in. SAS tokens are handy for giving a contractor temporary access or for automated jobs that cannot sign in interactively.

There are three kinds. A user delegation SAS is signed using Entra ID credentials, so it inherits the safety of your identity system and is the preferred kind. A service SAS grants access to one service, such as one container, and is signed with the account key. An account SAS can cover several services and is also signed with the account key. The key-signed kinds are powerful, so their lifetime should be short and their permissions narrow.

Method three: managed identities and service principals

Automated jobs need to sign in without a human. Two options exist. A managed identity is an identity Azure itself creates and attaches to a resource such as a virtual machine. The machine can sign in as that identity without any password stored anywhere. A service principal is an application identity you create yourself, authenticated by a secret or a certificate. Both can be given the same data roles as a person. When your scripts run inside Azure, a managed identity is the safest choice because there is no secret to leak.

What about the account key?

Every storage account has two powerful access keys, which act like master passwords for the whole account. AzCopy intentionally does not accept account keys. This is a deliberate safety choice that nudges you toward Entra ID or SAS. Storage Explorer, by contrast, can use an account key when you attach an account by name and key, and it can also fetch a key on your behalf if your Azure role is allowed to list them. Many security teams turn off key-based access on storage accounts entirely, in which case only identity-based access and user delegation SAS tokens work.

Choosing the right method

One more detail worth knowing: Azure Files has historically leaned more on SAS in AzCopy than Blob storage does. Newer releases add identity-based options for file shares, so check the current official documentation before you plan a file share transfer.

AzCopy has a small set of verbs. Learn these few and you can express almost any data movement task.

The main commands, in plain words

The four directions of copy

Think of copy as having four directions, and each one has slightly different behaviour.

Copy versus sync: a crucial difference

Copy transfers everything you point it at, and by default it overwrites files that already exist at the destination. You can change that by telling it to skip existing files, to overwrite only when the source is newer, or to ask you each time. Sync first compares the two sides and then transfers only the differences. By default it compares the last-modified time, and a hash comparison mode is available for stricter checking.

Sync can also remove files at the destination that no longer exist at the source, but only if you ask for that with a delete option. This is an important safety valve. Without it, sync only ever adds and updates. With it, sync makes the destination an exact mirror, which means a mistake in the source, such as an accidentally emptied folder, can propagate as deletions.

Useful options you will reach for

• Recursive: include everything inside subfolders, not just the top level.
• Include and exclude patterns: filter by file name pattern, such as only images or everything except temporary files.
• Dry run: show what would happen without doing it. This is the single best habit for safe operations.
• Block blob tier: choose Hot, Cool, Cold, or Archive at upload time.
• Put MD5 and check MD5: store hashes on upload and verify them on download.
• Cap on megabits per second: limit bandwidth so you do not starve other users of the network.
• Log level: choose how detailed the log should be, from quiet to chatty.
• List of files: supply a text file naming exactly which items to move, useful when you already know the set.
• Follow symbolic links: by default, shortcut-like links are skipped, and this option tells AzCopy to follow them.

Special content: permissions and metadata

Files carry more than their bytes. They have names, timestamps, metadata labels, and, on file shares, permissions. AzCopy preserves metadata and, for Azure Files, can preserve SMB attributes, timestamps, and security descriptors if you ask. For Data Lake Storage Gen2, newer versions can carry access control information between accounts. These options matter during migrations, where users expect their folder permissions to still make sense on the other side.

A hypothetical scenario

A design agency keeps 6 terabytes of finished project files on an office server. It signs in with Entra ID, creates a container, and runs a first copy at night with a bandwidth cap so staff can still use the internet during the day. After the first week, the nightly job switches to sync so only changed files move. When a laptop is lost, the agency downloads a few projects from Azure. The same tool serves backup, restore, and migration.

Speed comes from understanding where the bottleneck is. This chapter teaches you to find it, and then to loosen it.

The chain of limits

A transfer is a chain of steps, and the whole chain moves only as fast as its slowest link. The chain includes the source disk, your computer’s processor and memory, your network card, your office network, your internet connection, the Azure network, and finally the storage account itself. Making one link ten times faster does nothing if another link is the real limit.

Know your workload: big files or many files

The dials you can turn

Concurrency controls how many transfers run at the same moment. AzCopy picks a sensible default that scales with the number of processor cores and tunes itself while running. You can raise or lower it through an environment setting. Higher values help on fast links with many files. Too high a value can overwhelm a small computer or trigger throttling at the server.

Block size controls how large each block is. Larger blocks mean fewer requests, which suits large files on fast links. Smaller blocks suit unreliable links, because a failed block costs less to retry. The automatic choice is good for most work.

Bandwidth cap deliberately limits speed. People sometimes forget that going slower can be the right call: capping at 70 percent of a shared office link keeps video calls working during a daytime migration.

Buffer memory is the amount of computer memory AzCopy may use to hold blocks in flight. If the machine is short of memory, reduce it. If you have plenty, increasing it can help throughput on very fast links.

Throttling: when Azure says “slow down”

Azure Storage publishes scalability targets, which are limits on how much traffic one account and one partition can handle. Storage divides blob names into ranges, and a heavy burst against one narrow range can hit the ceiling. When that happens, the service answers with “server busy” messages (HTTP status 503) or timeouts. AzCopy reads these, backs off, and retries, so the job slows but does not fail. If you see large amounts of throttling in the log, spread the load, reduce concurrency, or request higher limits from Azure support when your business case calls for it.

Measure before you tune

AzCopy includes a benchmark mode that uploads and downloads generated data to a target container so you can see the speed your setup can reach, without involving real files. Run it from the machine that will do the real transfer. If the benchmark is fast but the real job is slow, your source disk or your file mix is the cause. If the benchmark is slow too, look at the network or the computer itself.

Placing the machine near the data

Distance adds delay. If you are copying between accounts, run the command from anywhere, because the data does not pass through you. If you must upload from on-premises, consider the nearest Azure region and, for sustained heavy needs, a dedicated private connection such as ExpressRoute, which avoids the shared public internet. Another approach is to run AzCopy on a virtual machine inside Azure in the same region as the storage account, especially when the source is another cloud, because the pulled data then crosses fewer networks.

Now we turn to the friendly side. Storage Explorer packs a surprising amount of power into a window that looks like an ordinary file manager.

flowchart TB
  USER["Person at the keyboard"] --> UI["Storage Explorer desktop app"]
  UI --> CONN["Sign in and connection manager"]
  UI --> TREE["Resource tree and detail panes"]
  UI --> ACT["Activities queue"]
  CONN --> ENTRA["Microsoft Entra ID"]
  TREE --> API["Storage service calls"]
  ACT --> AZ["Bundled AzCopy engine"]
  API --> STORE["Azure Storage account"]
  AZ --> STORE

The pieces of the application

The connection manager handles how you reach your storage. You can sign in with your Azure account, and Storage Explorer then lists every subscription and storage account your identity can see. You can also attach a single resource using a connection string, an account name with a key, a SAS link, or a link to a public container. A local emulator called Azurite is supported too, which lets developers practise with a pretend storage account on their own computer.

The resource tree on the left shows accounts, then services such as blob containers, file shares, queues, and tables. Selecting any node shows its content on the right. Because the tree reads from the storage service directly, what you see is always the live state, not a cached picture.

The Activities pane tracks transfers and long operations. Uploads, downloads, and copies appear there with progress, speed, and counts of successes and failures. Failed items can be retried from this pane, and you can often open a detailed log. Behind the scenes, this queue hands the heavy work to the bundled AzCopy engine.

What you can do with it

Two ways to authorize inside Storage Explorer

When you sign in with your Azure account, Storage Explorer can reach data in two ways. It can use your identity and data roles directly, which is the safest option. Or, if your account has permission to list account keys, it may fall back to using a key behind the scenes. The practical lesson is that “I can see the storage account in the tree” does not mean “I can open its containers.” If you only hold a management role, you may see the account but get an error opening a container until someone grants you a data role.

The browser-based sibling

The Azure portal includes Storage browser, a lighter web version of the same idea. It needs no installation, which makes it ideal on locked-down machines. It is less capable for very large transfers because the browser sits in the middle, so treat it as a viewing and light editing tool.

A good tool used carelessly still causes damage. These workflows and habits keep Storage Explorer helpful rather than hazardous.

Getting connected the right way

For everyday use, sign in with your organisation account and let the tool discover your subscriptions. If you only need one account, you can filter the subscription list so your tree stays tidy. Use a connection string or account key only when no identity option exists, and treat those secrets like passwords. Storage Explorer stores connection details in the operating system’s secure credential store, but anyone who can use your logged-in computer can use the connections, so lock your screen.

Four common daily tasks

Be careful with delete and overwrite

Graphical tools make destructive actions feel casual. A right-click and a confirm dialog can delete a container holding millions of blobs. Before deleting, ask three questions. Is soft delete or versioning enabled so I can recover? Am I connected to the account I think I am? Is this the production environment or the test one? Many painful incidents begin with a person deleting the right folder in the wrong account.

Large jobs inside a graphical tool

Storage Explorer can handle big transfers, since AzCopy powers them. But there are practical limits. The app must stay open and the computer must stay awake. A crash of the application interrupts the transfer, though the Activities pane can usually retry unfinished items. There is no schedule, no script, and no way to run it on a server without a screen. If a job will run for many hours or must be repeated, graduate to AzCopy.

Settings worth knowing

The settings let you adjust transfer concurrency, configure a proxy if your company network requires one, and tune how the app signs in. If uploads are overwhelming a shared office connection, lowering the concurrency setting is a gentle fix. If your company inspects traffic with a proxy that re-signs certificates, you may need to import that certificate into the application so connections are trusted.

A hypothetical day

A support engineer at a software company gets a ticket: a customer cannot download a report. The engineer opens Storage Explorer, finds the container for that customer, sees that the blob’s content type is wrong, fixes it in properties, then generates a one-hour read-only link and pastes it into the ticket. The whole task takes four minutes with no commands. This is exactly where a graphical tool shines.

AzCopy and Storage Explorer are two tools in a larger toolbox. Knowing when to use something else is a mark of real experience.

flowchart TD
  A["You need to move data"] --> B{"How much and how often"}
  B -->|"A few files, one time, by hand"| C["Storage Explorer"]
  B -->|"Many files, large, or repeated"| D["AzCopy"]
  B -->|"Tens of TB and a slow network"| E["Offline Data Box family"]
  B -->|"Transforms, schedules, many sources"| F["Azure Data Factory pipelines"]
  B -->|"Managed on-premises file migration"| G["Azure Storage Mover"]

The other tools, explained

A practical comparison

The time rule of thumb for offline devices

Go back to the math in chapter 1. If the estimated network transfer time is measured in weeks or months, and a shipped device could arrive, be filled, and return in about the same or less, then offline wins. The ship-and-load approach has a fixed schedule that does not depend on your internet speed. Many teams combine approaches: ship a device for the first giant bulk load, then use AzCopy over the network to sync the changes made while the device was in transit.

Large migrations succeed because of preparation, not because of speed. This chapter walks through the lifecycle from first idea to final sign-off.

flowchart LR
  A["Assess"] --> B["Plan"]
  B --> C["Prepare Azure"]
  C --> D["Pilot"]
  D --> E["Bulk load"]
  E --> F["Delta sync"]
  F --> G["Verify"]
  G --> H["Cutover"]
  H --> I["Decommission"]
  F --> F

Stage 1: Assess the data you actually have

Count before you move. How many files? How large in total? What is the size of the biggest file? How are the files spread across folders? What file names exist that might cause trouble, such as names with unusual characters, very long paths, or names that differ only by capital letters? Blob names are case sensitive, while Windows file systems usually are not, so two files that Windows treats as the same name could collide or surprise you. Also ask which data is old and rarely read, because that data may belong in a cooler tier from day one.

Stage 2: Plan the destination

Decide on the storage account design. How many accounts? Which region? Which redundancy level? Will you turn on the hierarchical namespace? Which container layout will make sense in five years? Choosing a clean naming scheme now saves enormous pain later, because renaming in a flat blob namespace means copying everything again. Decide on lifecycle rules, such as moving data to Cool after 30 days and to Archive after a year.

Stage 3: Prepare Azure

Create the account and containers. Assign data roles to the identity that will run the transfer. Configure the firewall and private network access so that the machine doing the transfer can reach the account. Enable soft delete and versioning so a mistake can be undone. Decide how you will authenticate the scripts. Make sure logging and monitoring are on before the first byte flows.

Stage 4: Run a pilot

Transfer a small but representative slice, perhaps one percent of the data including the largest files and the strangest file names. Measure speed. Check results by comparing counts, sizes, and hashes. Fix surprises while the stakes are low. A pilot also lets you give the business a realistic completion date instead of an optimistic guess.

Stage 5: The bulk load

Run the main transfer, often across nights and weekends and with a bandwidth cap to protect daytime users. Break the job into logical slices, such as one top-level folder per job, so that a problem affects only one slice and so that you can run slices in parallel on different machines. Keep every job identifier and log file.

Stage 6: Delta sync until cutover

While the bulk load ran, users kept working, so the source has changed. Run sync jobs repeatedly. Each pass moves only what changed since the last one, so each pass is shorter. Eventually a pass is so short that you can freeze the source briefly for a final sync. This repeating loop is shown by the arrow pointing back to the same box in the figure.

Stage 7: Verify

Compare counts and total sizes between source and destination. Sample files and compare hashes. Check that metadata and content types look right. Run a trial restore of a few items. Verification is not the same as “the tool said it finished.” A tool can report success on what it was asked to copy, but it cannot tell you that you forgot to ask for a folder.

Stage 8: Cutover and decommission

Switch applications and users to the new location during an agreed window. Keep the old data untouched for a safety period, such as 30 to 90 days, then retire it. Document what you did, because the next person, possibly you in a year, will thank you.

What flows where during a transfer

It helps to picture the data path clearly. For an upload, bytes flow from the source disk, through AzCopy’s memory buffers, over your network and the internet, to the Azure front end, and into storage, where they are written to several copies according to your redundancy setting. Control messages, such as authentication and block commits, flow alongside. For an account-to-account copy, the control messages come from your machine, but the bytes flow only inside the cloud. Understanding this tells you what to fix when something is slow: for uploads, look at your side; for account-to-account copies, look at the service limits.

Data movement briefly creates risk, because information leaves its normal protected home and travels. Good habits shrink that risk to almost nothing.

Encryption in transit and at rest

AzCopy and Storage Explorer talk to Azure over secure HTTPS connections, which means the data is encrypted while it travels. Once it lands, Azure Storage encrypts all data at rest automatically, using Microsoft-managed keys by default. Organisations with stricter rules can use customer-managed keys stored in Azure Key Vault. You do not need to encrypt files yourself before uploading, though some teams do so for extra protection of highly sensitive material.

Least privilege: only what is needed

Least privilege means giving each identity the minimum access required for its job and no more. A nightly upload job needs write access to one container, not owner rights on the whole account. A reporting job that only downloads needs read access. Assign roles at the narrowest sensible scope, and review them regularly. Roles granted for a one-off migration should be removed when the migration ends.

Treat SAS tokens like cash

• Prefer user delegation SAS tokens over account-key-signed ones.
• Set the shortest practical expiry. Hours, not months.
• Grant only the permissions needed, such as read and list but not write or delete.
• Restrict by network address when the client has a stable address.
• Never paste SAS links into chat channels, tickets, or public repositories. Search engines and bots scan these places.
• For service SAS tokens, use a stored access policy so you can revoke a token without rotating the account key.

Network controls

A storage account has a built-in firewall that can restrict access to chosen networks and addresses. A private endpoint gives the account a private address inside your virtual network, so traffic never crosses the public internet. When these controls are on, your transfer machine must be inside an allowed network, or its address must be on the allow list. A very common surprise is a perfectly valid sign-in that still fails with an authorization error, because the storage firewall blocked the machine’s network address. Always check the firewall when “it worked from my other computer.”

Turning off shared keys

The strongest posture is to disable shared key authorization on the storage account entirely, so only Entra ID and user delegation SAS work. This removes the master-password risk. Do this after confirming that all tools and applications that need access have identity-based options. Storage Explorer then relies on your signed-in identity rather than keys.

Protecting against mistakes and attacks

Security also means surviving mistakes and malicious deletion. Turn on soft delete for blobs and containers so deleted items can be recovered for a retention period. Turn on blob versioning so overwrites keep previous versions. Consider immutable storage, which makes data write-once, for records that must not change. Use resource locks to prevent accidental deletion of the whole storage account. Together these form a safety net that matters especially when sync with deletion is in use.

Auditing who did what

Enable diagnostic logging on the storage account so that read, write, and delete operations are recorded, with the identity or token used. When something strange happens, such as a sudden spike in deletes, those logs tell you who did it and from where. Send them to a central log store with alerts, so someone is told within minutes rather than discovering the problem next quarter.

Things will go wrong. The difference between a bad night and a quick fix is knowing where to look.

Three places to look

Reading common errors

Most failures fall into a few families, and learning to recognise them makes you fast.

A systematic way to debug

Tracking progress without watching the screen

For long jobs, you do not want to stare at a console. Run AzCopy through a scheduler or pipeline that captures its output, and parse the final summary for failure counts. Many teams alert if the number of failed transfers is above zero, or if a nightly sync has not completed by a certain hour. Pair this with storage metrics alerts, for example when the error rate or the throttling percentage rises sharply.

Clock drift and other sneaky causes

Signed tokens depend on time. If the clock on your machine is several minutes wrong, tokens may appear to be not yet valid or already expired, and authorization fails mysteriously. Other sneaky causes include very long file paths that exceed operating system limits, files locked by another program, antivirus software scanning every file AzCopy touches, and laptops going to sleep in the middle of a job. Check these when the usual suspects are innocent.

A design pattern is a proven solution to a recurring problem. An anti-pattern is a tempting approach that keeps causing trouble. Here are the ones every data mover should know.

Patterns worth adopting

Anti-patterns to avoid

More anti-patterns in brief

• Using a graphical tool for a week-long job. The app and the laptop become single points of failure.
• Skipping verification. “No errors shown” is not proof that everything arrived.
• Running with maximum permissions “just for now.” Temporary broad access has a habit of becoming permanent.
• Hardcoding the account name and container in many scripts. Use a single configuration source, so changes happen in one place.
• Ignoring the tier at upload time. Uploading everything as Hot and fixing later can create avoidable charges.
• Forgetting the firewall. A locked-down account refuses even valid credentials from an unlisted network.

This chapter gathers the habits that separate smooth migrations from stressful ones, and shows where the money goes so there are no surprises on the bill.

Where the cost comes from

Moving data touches several kinds of charges. Understanding them helps you plan, and none of them requires you to memorise price lists, since prices change by region and over time.

Data entering Azure is generally not charged for bandwidth, which is good news for uploads. But operations are charged, so a migration of a hundred million small files costs more in operations than a migration of a hundred large files with the same total size. This is another reason to bundle small files when you can.

Habits that pay off

Common mistakes, in plain words

• Trusting a default overwrite. A copy can replace newer files at the destination with older ones. Check overwrite behaviour before running.
• Mixing up source and destination. The order matters. Read the dry-run output out loud.
• Forgetting trailing path details. Copying a folder and copying the contents of a folder differ, and the result can be one extra level of nesting. Test on a small sample.
• Not planning for file name problems. Characters that are legal on one system may be awkward on another, and very long paths can fail. Scan names in the assessment stage.
• Letting the machine sleep. A laptop that hibernates at 2 a.m. stops your job. Change power settings or use a server.
• Forgetting that sync is one-directional. Changes made only at the destination are not brought back, and with deletion enabled they may be removed.
• Ignoring time zones and clocks. Sync decisions based on modified time depend on clocks being sensible.
• Believing speed numbers from marketing. Measure on your own network with your own data.

Keeping the tools up to date

AzCopy is a single file, so updating is as simple as replacing it with the latest download. New versions fix bugs, improve speed, and add support for new storage features. Storage Explorer prompts you when an update is available. In scripts that run unattended, pin a specific version and test upgrades deliberately rather than letting a surprise change appear on a Saturday night.

The following scenarios are illustrative composites that show how the ideas combine. The names are invented, and the numbers are chosen to teach, not to describe any real company.

What these scenarios share

In every case, the winning move is choosing a tool to match the job rather than forcing one tool to do everything. The graphical tool handles looking, fixing, and sharing. The command-line tool handles volume, repetition, and automation. An offline device handles sheer size when the network is the limit. And in every case, the safety nets of identity-based access, soft delete, verification, and logging are on before the work begins.

Short answers to the questions beginners ask most often.

You began with the problem that moving data is harder than it looks, and you now have a complete map: the storage you are moving into, two complementary tools, how they work inside, how to make them fast and safe, and how to choose among all the options.

Think of the whole subject as a small logistics company. AzCopy is the freight team that works at scale, on a schedule, with a clipboard that survives power cuts. Storage Explorer is the front counter where people look, fix, and share. Data Box is the truck for loads too big for the road. Identity, least privilege, soft delete, and logging are the locks, insurance, and cameras that make the whole operation trustworthy.