Azure Data Lake Storage Gen2, often shortened to ADLS Gen2, is a storage service in Azure designed to hold very large amounts of data of any kind and to let analytics tools read it quickly, with real folders and fine-grained permissions.

Start with the idea of storage

Every application needs somewhere to keep files. Cloud providers offer a simple kind of storage called object storage, where each file is stored as a separate object with a name. Object storage is cheap, huge, and very durable. But it has a weakness: it has no real folders. A name such as sales/2026/october/data.csv is just one long label. Everything is in one flat space, and the slashes are only decoration.

For small jobs this does not matter. For big data work, it matters a great deal. Analytics tools often need to rename a folder, move a folder, or delete a folder with a million files. On a flat space, each of those jobs means touching a million separate objects, one at a time. It is slow, and if it stops halfway, it leaves a mess.

What Gen2 adds

ADLS Gen2 is not a separate product with its own boxes. It is a set of extra abilities that you switch on in an ordinary Azure storage account. The key switch is called the hierarchical namespace, which turns the flat space into a true tree of folders and files. With it, renaming or deleting a folder is a single quick action. On top of this, Gen2 adds file-and-folder permissions in the style used by Linux, and an access method designed for analytics engines.

• Scale. It holds from a few files to many petabytes, with no servers for you to manage.
• Any data. Tables, logs, images, audio, documents, and streams all fit, in any format.
• Real folders. A hierarchy makes organizing, renaming, and securing data natural and fast.
• Fine-grained access. You can set permissions on individual folders and files, not just on a whole container.
• Analytics-ready. Tools such as Spark, Synapse, Databricks, and Data Factory read it directly and in parallel.
• Low cost. It uses the same low-priced object storage underneath, with tiers that make cold data very cheap.

To see why this storage exists, look at how companies handled data before data lakes, and what went wrong.

The old way: the data warehouse only

For decades, companies stored analytical data in a data warehouse: a database with carefully designed tables. Before data could go in, it had to be cleaned and shaped to fit. That works well for tidy business numbers. But it forces you to decide in advance what questions you will ask, and it handles messy data, such as logs, images, or sensor streams, badly. It is also expensive per terabyte.

The problem of many kinds of data

Modern companies collect data of every shape: click streams from websites, readings from machines, text from emails, pictures from cameras, files from partners. Much of it has no neat table structure. Throwing it away wastes value that may be needed later, for example to train a machine learning model on last year’s behavior.

The data lake idea

A data lake is a central store where you keep data in its original form first, and decide how to use it later. You save everything cheaply, then shape the part you need when you need it. This is sometimes called “schema on read”: the structure is applied when you read the data, not when you write it.

Why the first generation of lakes struggled

Early lakes on flat object storage had real troubles. Renaming a folder was slow and unsafe. Permissions were coarse: you could allow someone into a whole container or none of it. Analytics engines had to work around these gaps. Gen2 was created to keep the low price and huge scale of object storage while fixing these weaknesses.

What it solves in practice

When it is not the right tool

A data lake is not a database. It does not offer fast single-row lookups, transactions across many records, or indexes by itself. It holds files. Tools on top of it, or databases beside it, add those abilities. If your need is a fast application database, use one, and use the lake for the analytical copy.

Data Lake Storage Gen2 has a small vocabulary. Learn these words once, and the documentation will read easily.

Two ways to see the same data

One of the neat features of Gen2 is that the same data can be reached in two ways. Through the blob endpoint, it looks like blob storage, and tools built for blobs work. Through the dfs endpoint, it looks like a file system with directories, and tools built for analytics work. You do not copy the data between them. It is one set of files with two doors.

Naming matters

Storage account names must be unique across Azure, use only lowercase letters and numbers, and stay within a short length limit. Choose them with care, because the name becomes part of every address, and you cannot rename the account later. A clear pattern, including the environment and purpose, saves confusion.

A data lake is not only storage. Around the storage sit the tools that fill it, shape it, read it, and protect it.

flowchart LR
    subgraph IN["Data sources"]
        A1["Databases"]
        A2["Applications and logs"]
        A3["Devices and streams"]
        A4["Partner files"]
    end
    subgraph ING["Ingestion tools"]
        B1["Data Factory pipelines"]
        B2["Event streaming services"]
        B3["Direct upload tools"]
    end
    subgraph LAKE["Data Lake Storage Gen2 account"]
        C1["Container: raw zone"]
        C2["Container: cleaned zone"]
        C3["Container: curated zone"]
    end
    subgraph USE["Consumers"]
        D1["Spark and Databricks"]
        D2["Synapse and SQL engines"]
        D3["Machine learning"]
        D4["Reports and dashboards"]
    end
    subgraph GOV["Control and care"]
        E1["Entra ID, RBAC, and ACLs"]
        E2["Network rules and private endpoints"]
        E3["Monitoring and lifecycle rules"]
    end
    A1 --> B1
    A2 --> B3
    A3 --> B2
    A4 --> B1
    B1 --> C1
    B2 --> C1
    B3 --> C1
    C1 --> C2
    C2 --> C3
    C2 --> D1
    C3 --> D2
    C3 --> D3
    C3 --> D4
    E1 -.-> LAKE
    E2 -.-> LAKE
    E3 -.-> LAKE
    

Read the diagram from left to right. Data starts in sources such as databases, applications, devices, and partner files. Ingestion tools bring it into the lake, where it first lands in a raw zone. Further steps clean it into a second zone and shape it into a curated zone ready for use. Consumers such as analytics engines, machine learning tools, and reports read from the zones that suit them. Along the bottom, the dotted lines show the controls that surround the whole lake: identity and permissions, network rules, monitoring, and lifecycle rules.

The storage layer

At the core is the storage account with the hierarchical namespace switched on. Inside it are containers, and inside those are directories and files. Azure keeps multiple copies according to your redundancy choice, and charges by the amount stored and by the operations performed.

The ingestion layer

Tools such as Azure Data Factory, event streaming services, and command line utilities push data into the lake. Some copy in batches on a schedule. Others write continuously as events arrive. Chapter 8 covers these routes.

The processing layer

Engines such as Spark, Synapse, and Databricks read files from the lake, transform them, and write results back. They run on their own compute, separate from storage. This separation of storage and compute is a defining feature: you can switch off the compute and keep the data, and you can run many different engines on the same data at the same time.

The consumption layer

Reports, dashboards, and models read the curated data. Often a SQL engine sits in front, letting analysts query the files with familiar commands.

The governance layer

Identity, permissions, network rules, encryption, auditing, and lifecycle rules protect and tidy the lake. They are not extras. In a lake that holds data from the whole business, they are as important as the storage itself.

This single setting is what turns an ordinary storage account into a Data Lake Storage Gen2 account. It is worth understanding deeply, because it explains most of Gen2’s advantages.

Flat versus hierarchical

In a flat namespace, the store keeps a list of object names. A name such as logs/2026/10/30/app.log is a single string. The slashes mean nothing to the store. If you ask to rename the “folder” logs/2026/10, the store must find every object whose name starts with that text, copy each to a new name, and delete the old one. With a million objects, that is a million copies, and failure partway leaves some renamed and some not.

In a hierarchical namespace, folders are real objects in a tree. Renaming a directory changes one entry in the tree, and everything beneath it comes along. The change is quick, and it happens as a single, all-or-nothing action. This property is called atomic: it either fully happens or does not happen at all.

What the hierarchy gives you

Why analytics tools love it

Big data engines write results into a temporary folder and then rename it to the final name, so that readers never see a half-finished result. On a flat store, that rename is a slow copy. On Gen2, it is nearly instant. Jobs finish sooner and fail less often at the very last step.

Turning it on

The hierarchical namespace is chosen when you create the storage account. Microsoft has also offered a way to upgrade an existing blob account, with conditions and a period in which the account is restricted, and with some features that do not work together with the hierarchical namespace. The details, including the list of unsupported features, have changed over time, so check current documentation before you plan. Test with a copy of your data and workloads first.

Features that behave differently

• Some blob features are not available, or are limited, on accounts with the hierarchical namespace. Examples that have been listed as limited in the past include certain versioning and snapshot features. Confirm the current list.
• Some network file protocols and SFTP access are available on these accounts with specific requirements and extra charges. Check current availability.
• Tools written only for flat blob storage may treat directories oddly. Test them before relying on them.

A lake without a plan becomes a swamp. The most valuable work you will do is deciding, before the first file lands, how data will be organized.

The zone idea

Most well-run lakes divide data into zones that reflect how refined it is. A popular naming is the medallion pattern, with three levels:

Other names exist, such as raw, enriched, and workspace, or landing, standardized, and curated. The names matter less than agreeing on them. Many lakes also keep a sandbox area where individuals can experiment without affecting shared data, and a landing area where outside partners drop files before checks are made.

Containers or folders for zones?

You can make each zone its own container, or each zone a top-level folder within one container. Separate containers make it simpler to set different permissions and to apply lifecycle rules by zone. Folders are simpler to manage when the number of zones is small. Many teams use one container per zone for the main layers.

Designing folder paths

A good folder path tells you what a file is without opening it. A common pattern uses the subject first, then the source, then the date:

• Zone (as a container or top folder), such as raw or curated.
• Business area or subject, such as sales or finance.
• Source system or data set, such as orders or customers.
• Date parts, such as year, month, and day, so that tools can read just the days they need.

Putting the date at the end, in year, month, day order, keeps related files together and makes it easy to read one day or one month. It also lets lifecycle rules and permissions work on natural boundaries.

Rules for good paths

Think about security when drawing folders

Permissions work most easily on folder boundaries. If the HR data lives in one folder tree and the sales data in another, access can be granted by folder. If they are mixed, you must set permissions file by file. Draw your folders so that each tree has one audience.

Data in the lake is useful only if tools can reach it. This chapter shows the doors into the lake and what happens when a request arrives.

sequenceDiagram
    participant C as Client or Analytics Tool
    participant E as Microsoft Entra ID
    participant S as Storage Service
    participant N as Hierarchical Namespace
    participant D as Data Files
    C->>E: Ask for a token for the storage service
    E-->>C: Return a signed token
    C->>S: Request to read a file with the token attached
    S->>S: Check network rules and the token
    S->>S: Check Azure role assignments
    S->>N: Check the access control lists along the path
    N-->>S: Permission granted or denied
    S->>D: Read the file contents
    D-->>S: Return the data
    S-->>C: Send the data back
    

The sequence follows one read request. The client first obtains a token from Microsoft Entra ID, proving who it is. It sends the request with the token. The storage service checks the network rules and the token, then checks role assignments, then checks the access control lists on the folders along the path. If everything allows it, the data is read and returned. Any check can stop the request.

The two endpoints

The same account name appears in both addresses, followed by a different part. Which one a tool uses is usually chosen for you, but it matters for network rules: if you lock down the account with private endpoints, you may need to create a private endpoint for each of the two doors you use.

The ABFS driver

Analytics engines use a connector called the Azure Blob File System driver. It speaks to the dfs endpoint, understands directories, and is built for big parallel reads and writes. Paths for it begin with a short scheme name and include the container and the account, followed by the folder path. You will see these addresses in notebooks and pipeline settings. The secure version of the scheme uses encrypted connections and should be your default.

Tools people use

Mounting and drives

Some platforms let you mount a lake folder so that it appears as a normal drive or folder in a notebook or virtual machine. This is convenient for exploration, but it hides the access rules behind a shared identity, so use it with care. Prefer direct access using each user’s own identity where you can.

A lake is only as useful as the data inside it. There are several routes for data to arrive, and each suits a different kind of source.

Batch loads

In a batch load, data is moved in chunks at set times, such as every night. Azure Data Factory or Synapse pipelines are the common tools. They read from databases, files, and applications, and write to the lake, often converting formats on the way. Batch is simple, predictable, and good for data that does not need to be fresh to the second.

Streaming loads

In a streaming load, data arrives continuously as events happen, such as clicks, sensor readings, or transactions. Event streaming services collect them, and a capture feature or a processing job writes them to the lake in small files, often grouped by time. Streaming gives fresher data but needs more care with file sizes and ordering.

File drops

Partners often send files. They may use a secure file transfer, a web upload, or a shared link. A common design is a separate landing area in the lake, with narrow permissions, where outside files arrive. A pipeline then checks them, for size, format, and safety, and moves good ones into the raw zone.

Bulk migration

When moving existing data into the lake for the first time, large copies can take days. Command line copy tools, high-speed pipelines, and offline transfer devices can all help. Plan for restarts, checks of file counts and sizes, and the bandwidth you will use.

The small files problem

Analytics engines are efficient at reading a few large files, and slow at reading millions of tiny ones. Each file adds overhead, and each request costs a small fee. Streaming and frequent loads tend to create tiny files. The fix is to compact them: periodically merge small files into larger ones, aiming for sizes in the range of hundreds of megabytes to a gigabyte, depending on the engine and the format. Plan compaction from the start.

Keep the raw data raw

Write incoming data to the raw zone exactly as it arrived, adding only the folder structure and perhaps a record of when it arrived. Do not clean or reshape it there. If a later step is wrong, you can reprocess from the untouched originals.

Data you cannot lose must be kept in more than one place. Redundancy options decide how many copies Azure keeps, and how far apart they are.

Why copies matter

Hardware fails. Disks break, racks lose power, buildings flood. Azure protects your data by keeping several copies, so a single failure never destroys it. You choose how many copies and how widely they are spread. More spread means more protection and a higher price.

The redundancy choices

How the geo copy works

The copy in the second region is made asynchronously, which means a short time after the write, not at the same moment. If the main region is lost suddenly, the very latest writes may not have reached the second region yet. The gap is usually small, but it is not zero. This is called the recovery point. If your business cannot lose any data, plan for it explicitly.

Failover

With geo redundancy, you can start a failover to make the secondary region the new main one. Failover has consequences: some recent data may be lost, the account becomes locally redundant in the new region until it is changed back, and some features may have limits on accounts with the hierarchical namespace. Treat failover as a decision for serious outages, rehearse it, and check the current conditions in Microsoft’s documentation.

Choosing a level

• Re-creatable data, such as a working copy of data that can be rebuilt, may be fine with local redundancy.
• Important production data usually deserves zone redundancy as a minimum, so a data center failure does not stop the lake.
• Business-critical or regulated data often needs geo or geo-zone redundancy, and a documented recovery plan.
• Data residency rules may limit which regions can hold copies. Check the paired region for your chosen one.

Not all data is used equally. Last week’s data is read daily; data from five years ago may be read once a year. Tiers let you pay less for data you touch rarely.

The tiers

Cooler tiers have minimum storage periods. If you delete or move data sooner, you pay an early deletion charge. Archive data is offline: before you can read it, you must rehydrate it, which means moving it back to a hotter tier, and this can take hours. The exact minimum periods, prices, and support on accounts with the hierarchical namespace should be checked in current documentation.

Setting the tier

The account has a default access tier, usually hot or cool, which applies to new data. You can also set the tier on individual files. Moving a file between tiers is a small operation, but with many files it adds up in cost, so automate it.

Lifecycle management rules

A lifecycle management policy is a set of rules that Azure applies automatically. Each rule selects data by folder prefix, file type, or tags, and says what to do as the data ages. Typical actions include:

• Move to cool after thirty days without being changed.
• Move to cold or archive after ninety days or a year.
• Delete after the required retention period, such as seven years.
• Delete old temporary or staging files after a week.

Match rules to zones

Watch for hidden costs

Do not move small files to cool or archive without thinking. Each move is a charged operation, and small files may cost more to move than they save. Cooler tiers also charge more to read, so moving data that jobs still read often can raise the bill instead of lowering it.

Data does not just sit in a lake. It arrives, is refined, is used, ages, and eventually leaves. Planning each stage keeps the lake healthy.

flowchart TD
    A["Source data is created"] --> B["Ingested into the raw zone"]
    B --> C["Validated and cleaned into the cleaned zone"]
    C --> D["Shaped into the curated zone"]
    D --> E["Read by reports, models, and analysts"]
    E --> F["Data ages and is read less often"]
    F --> G["Lifecycle rule moves it to a cooler tier"]
    G --> H["Retention period ends"]
    H --> I["Data is deleted or archived under policy"]
    B --> J["Raw copy kept for reprocessing"]
    J --> C
    

The diagram shows the main path from top to bottom. Data is created at the source, lands raw, is cleaned and shaped, and is used. As it ages, it is read less, moves to cooler tiers, and finally leaves the lake when its retention period ends. A side loop shows the raw copy, which is kept so that cleaning can be redone whenever rules change.

Stage one: arrival

Data lands in the raw zone as received. Record where it came from, when it arrived, and how many records it holds. These small notes make later problems far easier to trace.

Stage two: cleaning

Pipelines check structure, remove duplicates, fix types, and set aside bad records in a separate folder for review. Keep a log of what was changed, so analysts can trust the cleaned data and explain it.

Stage three: shaping

Cleaned data is combined and reshaped for particular uses, such as sales summaries or customer profiles. These are the datasets people actually read, so they deserve names, owners, descriptions, and quality checks.

Stage four: use

Reports, dashboards, and models read the curated data. Usage tells you which data is valuable, and which can be retired.

Stage five: aging

Over time, data is read less often. Lifecycle rules shift it to cheaper tiers. Old versions of curated tables can be trimmed.

Stage six: retention and deletion

Every dataset should have a retention period, driven by law, contracts, and business need. When it ends, delete the data or move it to a long-term archive. Keeping data forever has a cost, and for personal data, it can also be a legal risk. Rules about erasure requests also apply, and designing for them early is far easier than adding them later.

How you store data inside files makes a huge difference to speed and cost. The right format and layout can make queries ten times faster and storage several times smaller.

Row formats and column formats

Some formats store data row by row, like a list of records. Others store it column by column. Analytics usually asks questions about a few columns across many rows, such as “total sales by region”. A column format lets the engine read only the columns it needs and skip the rest, which saves huge amounts of reading.

Why Delta and similar table formats matter

Plain files are hard to update safely. If two jobs write at the same time, or a job fails halfway, readers can see half-finished data. Table formats such as Delta keep a log of changes and make each write all-or-nothing. They add features that databases take for granted: updates, deletes, merges, time travel to earlier versions, and consistent reads. Many modern lakes keep their cleaned and curated zones in such a format.

Partitioning

Partitioning means splitting a dataset into folders by the value of one or two columns, usually the date. When a query asks for one month, the engine reads only the folders for that month and skips the rest. This can cut reading from terabytes to gigabytes.

• Choose columns that queries filter on, most often date.
• Avoid too many partitions. Partitioning by a column with millions of values creates millions of tiny folders and files.
• Keep partitions big enough. A partition should hold files of a healthy size, not a few kilobytes.
• Do not partition small tables. The overhead is not worth it.

File size

Aim for files that are large enough to read efficiently and small enough to process in parallel. A common guideline is somewhere between a hundred megabytes and a gigabyte per file, though the best size depends on the engine. Both thousands of tiny files and a few gigantic ones cause trouble.

Compression

Compressing files saves storage and reading time at the cost of a little processing. Modern column formats compress internally by default. For text formats, compression is worth enabling, with a type that the reading tools can split for parallel work.

A data lake collects data from the whole business, so it becomes a very attractive target. Security starts with knowing who is asking, and what they are allowed to do.

Four ways to prove who you are

Why the account key is dangerous

An account key gives full access to the entire account and bypasses roles and ACLs. Anyone holding it can read, change, or delete everything, and the logs cannot tell who they were. It is also hard to rotate, because many tools may depend on it. Mature setups disable key-based access so that only Entra identities work. Do this carefully, checking that no tool still depends on keys.

Azure roles for the data

Role-based access control gives broad permissions at the level of the account, a container, or higher. The roles most used with data lakes are:

A key point about management and data

Azure separates the management plane, which creates and configures accounts, from the data plane, which reads and writes files. A person who is Owner of the account can change its settings, but does not automatically get permission to read the files unless they also hold a data role or use the account key. This separation is helpful: platform administrators can run the account without seeing the data.

Assigning roles sensibly

• Use groups, not individuals, so access follows team membership.
• Assign at the narrowest scope that works: a container rather than the whole account when possible.
• Prefer reader roles for most analysts, and give write roles only to pipelines and engineers.
• Review regularly, and remove access when people change roles or leave.
• Use conditional access and strong sign-in for people with broad rights.

Roles are broad. Access control lists let you be precise: this team may read this folder, that team may write that one, and nobody else may see either.

What an ACL is

An access control list is a list of entries attached to a directory or file. Each entry names an identity, which can be a user, a group, or a service, and gives it some combination of three rights: read, write, and execute. This follows the permission model long used on Linux and other Unix systems.

Walking the path

To reach a file deep in a tree, a person needs the execute right on every directory above it, starting at the container root. This surprises beginners: granting read on a leaf folder is not enough if the parent folders refuse entry. Always check the whole path.

Access ACLs and default ACLs

• Access ACLs control access to the item they are attached to.
• Default ACLs are set on a directory and act as a template. New files and subdirectories created below it inherit them. They do not change items that already exist.

This second point causes a lot of trouble. If you add a default ACL to a folder that already has a million files, those files do not get it. You must apply the change recursively to existing items. Tools can do this in bulk, but it takes time on large trees. Set default ACLs when you create the folder, before data arrives.

Use groups, not individuals

ACLs have a limit on the number of entries per item, so listing individual users does not scale. Grant rights to Entra groups instead, and manage membership in the group. Then one ACL entry covers a whole team, and people moving between teams change group membership, not folder permissions.

Typical permission design

Roles and ACLs together

A common approach is to give broad roles only to administrators and pipelines, and use ACLs on groups for everyone else. Because roles are checked first and can override ACLs, avoid giving ordinary users a data role at the account level when you want ACLs to control them.

Even with good identities and permissions, it is wise to limit where requests may come from, and to protect data against anyone who gets at the storage itself.

Network controls

By default, a storage account is reachable over the public internet by anyone who can authenticate. You can narrow this in layers:

Private endpoints and the two doors

For a Gen2 account, you may need a private endpoint for the dfs endpoint, which analytics tools use, and another for the blob endpoint, which some tools still use. Create the ones your tools need. You also need private DNS set up so that the account name resolves to the private address from inside your network. A missing DNS zone is the most common reason a private endpoint appears not to work.

Encryption at rest

All data in a storage account is encrypted at rest automatically, using strong encryption and keys managed by Microsoft. If your rules require control of the keys, you can use customer-managed keys stored in Azure Key Vault. This gives you the power to rotate or revoke the key, at the price of extra responsibility: if the key is lost or access to it is removed, the data becomes unreadable. Some organizations also use an extra layer of encryption at the infrastructure level.

Encryption in transit

Require secure transfer so that all connections use encrypted channels, and set a minimum version of the transport security protocol. Modern tools support this by default. Disable old, weak settings.

Protection against deletion and change

• Soft delete keeps deleted data for a set number of days so that it can be restored.
• Immutable storage makes data write-once and unchangeable for a time, for regulatory records.
• Resource locks stop someone from deleting the whole account by accident.
• Backups and copies to another account or region protect against serious mistakes or attacks. Check which of these features work on accounts with the hierarchical namespace, since support has varied.

Auditing and threat detection

Turn on logging of read, write, and delete requests, and send the logs to a central workspace. Microsoft also offers threat protection for storage that can warn about unusual access, such as a flood of reads from a strange location. Alerts only help if someone responds, so connect them to a person or a process.

A lake that nobody watches will grow, drift, and sometimes leak. Monitoring shows health and cost. Governance shows what you have and whether it follows the rules.

What to monitor

Tools

Alerts worth setting

• Sudden growth in used capacity or transactions.
• High rates of throttled or failed requests.
• Large downloads or egress at unusual times.
• Any use of the account key, if keys should be off.
• Changes to firewall rules, public access, or role assignments.

Governance basics

Governance means knowing what data you hold, who owns it, how sensitive it is, and where it came from. A catalog scans folders and records datasets, with labels for sensitive content such as personal information. Lineage shows how one dataset was built from others, which is vital when a source is wrong and you must find everything it affected. Azure Policy can enforce rules, such as requiring private endpoints, encryption settings, or tags on every account.

Storing data is cheap per gigabyte, but a large lake has many meters. Understanding them helps you keep the bill steady as the lake grows.

Where the money goes

The biggest levers

Reserved capacity

For large, steady amounts of data, Azure offers reserved capacity: you commit to a quantity for one or three years in return for a discount. This suits predictable lakes. It is a poor choice if you expect to shrink or move the data soon. Check current terms and rates.

Finding waste

Use the inventory report to list files by size, age, and tier. Look for large old files in hot, huge counts of tiny files, folders that nobody has read in a year, and duplicate copies of the same dataset. A quarterly cleanup often pays for itself many times.

Data Lake Storage Gen2 can serve very high throughput, but only if the data and the jobs are arranged to take advantage of it.

What limits speed

A storage account has limits on how many requests per second it can handle and how much data per second it can move. These limits are high, but they exist, and the numbers differ by region and account type. When a workload pushes beyond them, the service replies with a message asking the client to slow down and try again. This is called throttling. Well-built tools retry automatically with growing pauses, but heavy throttling makes jobs slow.

Habits that help

• Use large files. Fewer, bigger files mean fewer requests and faster scans.
• Read in parallel. Analytics engines split the work over many machines; give them enough files and partitions to spread the load.
• Use column formats and partition pruning, so jobs read only what they need.
• Co-locate compute and storage in the same region, to avoid long network trips.
• Avoid hot spots. Many clients hammering one file or one folder at the same instant can slow down. Spread them out.
• Avoid unnecessary listing. Listing a folder with millions of entries is slow and costly. Use partitioned paths and metadata where possible.
• Prefer the dfs endpoint and the ABFS driver for analytics, which is built for parallel work.
• Use premium accounts if you need very low latency and very high transaction rates; they cost more per gigabyte and fit particular workloads.

Standard and premium

Multiple accounts

If one account cannot meet the limits, you can spread the load across several accounts, for example by zone, by business unit, or by region. This also separates cost and permissions. The price is more to manage, so do it for a reason, not by habit.

Test before you trust

Run a realistic test with the file sizes, counts, and access patterns you expect. Measure throughput and look for throttling in the metrics. Surprises are cheaper to find in a test than on the day of a big launch.

Experienced architects write down major choices as short decision notes. Here are common decisions for data lake designs, written in that style so you can see how the reasoning goes.

Patterns that work

Anti-patterns to avoid

This chapter gathers lessons that many teams learn the hard way. You can learn them the easy way, by reading them first.

Best practices

• Enable the hierarchical namespace at creation for any analytics account.
• Write a folder standard before loading data, with zones, subjects, and date partitions.
• Keep raw data raw, and rebuild other zones from it when rules change.
• Use Entra identities, managed identities for services, and disable account key access when possible.
• Grant access through groups, with roles at the narrowest scope and ACLs for folder detail.
• Set default ACLs before data arrives, and remember that they do not change existing items.
• Use private endpoints and disable public access for sensitive data.
• Convert to Parquet or Delta from the cleaned zone onward, and partition by the columns queries filter on.
• Compact small files and aim for healthy file sizes.
• Automate lifecycle rules, aligned with zone and retention.
• Choose redundancy by data value, and add soft delete and a backup plan.
• Log and alert on usage, errors, and configuration changes.
• Catalog datasets, give each an owner, and review them regularly.
• Tag resources for cost and governance.

Common mistakes

Here are typical ways organizations use Data Lake Storage Gen2. These are common patterns described in general terms, not accounts of any named company.

A typical retailer building a single view of sales

A retail company keeps store systems, an online shop, and a loyalty program, each with its own database. Every night, pipelines copy new records into a raw zone organized by source and date. A cleaning step removes duplicates and fixes product codes, writing Parquet files into the cleaned zone. A curated zone holds daily sales by store and product. Analysts query it with SQL, and data scientists use the cleaned zone to build demand forecasts. Old raw data moves to cooler tiers after ninety days.

A typical bank with strict controls

A financial company stores transaction history for risk models. The lake lives behind private endpoints with public access disabled. Account keys are turned off, so every request comes from an Entra identity and appears in the logs. Pipelines have write rights only in raw and cleaned zones. Analysts read curated data through group-based ACLs. Immutable storage protects records that regulations require to be kept unaltered. A catalog tracks the sensitivity of each dataset.

A typical manufacturer with sensor data

A manufacturer collects readings from thousands of machines. An event streaming service captures them into the lake as small files by the minute. A nightly job compacts them into larger Parquet files partitioned by day and machine type. Engineers use the data to predict failures and plan maintenance. Because the raw readings are huge and rarely needed after a few weeks, lifecycle rules move them to the archive tier after sixty days, while the summarized data stays hot.

A typical media company storing large files

A media business stores video and audio files for editing and for machine learning that tags content. Files are large, so throughput matters more than request counts. The company keeps recent projects in hot storage and moves finished ones to cool and then archive. A metadata table in the curated zone records where each file is and what it contains, so searching does not require scanning folders.

A typical software company sharing data across teams

A software firm gives each product team a folder tree in the lake and an Entra group per tree. A platform team owns the account, the network rules, and the policies, but cannot read team data because it holds only management roles. Teams share chosen datasets by adding another group to the read ACL of a specific curated folder. The catalog shows who owns each dataset and who may read it.

Services that work with the lake

When something fails in the lake, an ordered approach finds the cause quickly. Most problems come from permissions, networks, or file layout.

A method that works

Common symptoms and likely causes

Quick answers to questions beginners ask most often.

Azure Data Lake Storage Gen2 gives you cheap, huge, well-organized storage for analytics. It rewards planning of folders, identities, and lifecycle, and it punishes a casual start.

In this guide you met Data Lake Storage Gen2 as blob storage with a real folder tree. You learned the vocabulary and architecture, the hierarchical namespace, zone and folder design, and the ways data gets in and out. You saw how redundancy, tiers, and lifecycle rules protect and price your data, how file formats and partitioning make analytics fast, and how authentication, roles, ACLs, networks, and encryption keep it safe. You also covered monitoring, cost, performance, patterns, common mistakes, and a method for troubleshooting.