Before settings and diagrams, we need a clear picture of the idea of a data lake and of where this service fits.

The short definition

A data lake is a central place that holds large amounts of data in its original form, whether it is neat tables, loose text files, images, or streams of events. Unlike a traditional database, a lake does not force you to decide the exact shape of the data before you store it. You pour data in first, and different tools shape it later according to the question being asked. Azure Data Lake Storage Gen2 is Microsoft’s service for building such a lake on Azure.

The most important fact about it is how it is built. Data Lake Storage Gen2 is not a separate product with its own separate machinery. It is a set of capabilities that you switch on inside an Azure Storage account, on top of Azure Blob Storage, by enabling a feature called the hierarchical namespace. That feature adds real folders, fast folder operations, and fine-grained permissions to the very cheap and very scalable storage underneath. You get the economics of object storage and the behaviour of a file system in one place.

Why it exists

Analytics tools such as Apache Spark and Hadoop were designed for file systems with true directories. They often write results to a temporary folder and then rename the folder to its final name when the job succeeds. On plain object storage, a rename means copying every object and then deleting the old ones, which is slow for big jobs and can leave a mess if the job stops halfway. Data Lake Storage Gen2 was created so that these tools could use cheap, huge, durable cloud storage without paying that penalty.

A note about the first generation

There was an earlier product called Data Lake Storage Gen1, a separate service with its own design. Microsoft has retired Gen1 and recommends Gen2 for all new and migrated work. If you meet old documents, diagrams, or code that mention Gen1, treat them as history. Everything in this guide is about Gen2, which is the living, supported service.

A handful of words appear again and again. Learn them once and the rest of the guide will read smoothly.

More words you will meet

Three more ideas deserve a plain explanation. The first is the data plane versus the control plane. The control plane is where you create and configure the storage account, usually through the Azure portal or automation. The data plane is where tools read and write your files. Permissions for the two are separate, so a person may manage an account without being able to read its data, which is a good thing for security.

The second is the difference between schema on write and schema on read. A database applies a fixed structure when data is written. A lake lets you store data first and apply structure when you read it. This gives great flexibility but needs discipline, because data without any structure or ownership slowly turns into what people call a data swamp.

The third is the idea of a file format. Plain text formats such as comma-separated values are easy to read but slow and large. Columnar formats such as Parquet store data column by column, which compresses well and lets tools read only the columns they need. Lakehouse formats such as Delta add a transaction log on top of Parquet files. The choice of format often matters more for performance than any storage setting.

A few words people often confuse

A container is not a folder, because it is the top-level boundary where settings such as immutability policies apply. A directory is not a prefix once the hierarchical namespace is on, because it exists as a real object with its own permissions. A storage account is not a lake, because one lake may span several accounts, and one account may hold several unrelated lakes. Keeping these terms straight prevents design meetings from drifting into arguments about words instead of choices.

Now we look at the building blocks of the service and how a request travels through them.

The big picture

The service has a simple outer shape. You create a storage account and enable the hierarchical namespace. Inside the account you create containers. Inside each container you create directories and files. Tools reach the data through the network using one of two endpoints. Behind the scenes, Azure spreads your data across many machines and keeps several copies for durability, according to the redundancy option you choose.

flowchart TD
    A[Storage account with hierarchical namespace] --> B[Container named raw]
    A --> C[Container named curated]
    B --> D[Directory sales]
    D --> E[Directory 2026]
    E --> F[File orders part 1]
    E --> G[File orders part 2]
    A --> H[Blob endpoint]
    A --> I[Data lake endpoint]
        

Two endpoints for one set of data

A storage account with the hierarchical namespace has two addresses for reaching the same data. The blob endpoint speaks the standard Blob Storage interface, which many applications already understand. The data lake endpoint speaks an interface designed for file-system operations such as create directory, rename, and set permissions. Analytics tools use a special driver called the Azure Blob File System driver, often shortened to ABFS, which talks to the data lake endpoint and presents the storage to tools as if it were a Hadoop-style file system. Its address scheme starts with abfss, where the final s means secure, encrypted transport.

The storage layer

Underneath, your data lives in the same distributed storage system that powers Azure Blob Storage. It is built to be extremely durable by keeping multiple copies. When the hierarchical namespace is on, an extra metadata layer keeps track of the directory tree, ownership, and permissions. This layer is what makes folder renames fast and permissions precise.

The identity and permission layer

Every request is checked. The service first identifies who is asking, usually through Microsoft Entra ID, then checks role assignments and, for directories and files, access control lists. We will study the exact order of these checks later, because it is one of the most important topics for anyone designing a lake.

The management layer

The control plane handles creation of accounts, settings, lifecycle rules, networking rules, and encryption choices. It does not touch your files. Changing the control plane does not move your data, though some settings affect how the data is billed or protected.

Who talks to whom

Every request passes through the same simple path: the client finds the endpoint by name, proves its identity, and then asks for an operation on a path. The front-end layer checks permissions and passes work to the partition layer, which finds the right data, and the stream layer finally reads or writes the replicated bytes. Because each layer scales on its own, a busy account can spread load across many servers without any action from you, as long as the client sends many requests in parallel.

This single feature is the heart of the service. Understanding it explains the speed, the permissions model, and several limits you will meet.

Flat storage and its problem

In ordinary object storage, there are no real directories. An object has a name, and that name may contain slash characters, so tools can display it as if it lived in folders. But the folders are an illusion made from the names. If you want to rename a folder, the system must find every object whose name starts with the old folder name and change each one individually, which means copying and deleting. For a folder with millions of objects, that can take a long time and cannot be done as one all-or-nothing step.

A true tree of directories

With the hierarchical namespace turned on, the service keeps a real tree. Each directory is an object in its own right, with a list of children, an owner, and permissions. Renaming a directory changes one entry in the tree, no matter how many files lie beneath it. The same is true for deleting a directory. Both operations are atomic, meaning they either finish completely or not at all. For analytics jobs that write to a temporary location and then rename it on success, this atomic step is the difference between a safe commit and a messy half-finished result.

flowchart LR
    subgraph FLAT[Flat namespace]
        F1[Rename folder] --> F2[Copy every object]
        F2 --> F3[Delete every old object]
    end
    subgraph TREE[Hierarchical namespace]
        T1[Rename folder] --> T2[Update one directory entry]
    end
        

Why this speeds up analytics

Big data tools do many directory-style operations. They list directories to find files, they create temporary folders for each task, and they rename folders to publish results. With real directories, listing a folder reads one directory’s children instead of scanning a huge flat list for matching names. Renames and deletes are instant. The net effect is that jobs finish faster and are less likely to leave partial output behind.

Permissions at every level

Because every directory and file is a real object in the tree, each can carry its own owner and access control list. This is the foundation of the second permission system in the lake, which we will study in a later chapter. Without the hierarchical namespace, you cannot have these lists, and you must rely on roles and tokens alone.

What the hierarchical namespace changes and what it limits

Switching it on changes more than speed. Some features of ordinary blob storage behave differently or are not available when it is enabled. According to Microsoft’s feature support table, blob versioning, point-in-time restore, change feed, and object replication are not supported on accounts with the hierarchical namespace. Soft delete for blobs and containers, lifecycle management, access tiers on standard accounts, static websites, and event notifications are supported, and blob snapshots are in preview. Because feature support changes over time, always check the current table before you rely on a particular protection.

Several choices made when creating an account shape its cost, resilience, and speed for years. Make them with care.

Account types

Data Lake Storage Gen2 can be enabled on a standard general-purpose version two account, which is the usual choice for most lakes, and on a premium block blob account, which uses faster solid-state storage for workloads that need very low delay and very high operation rates. Standard accounts support access tiers and tiering rules. Premium block blob accounts do not support access tiers, so every byte is paid for at the premium rate. Choose premium only when measured needs demand it, such as interactive analytics on small hot datasets or machine learning training with many small reads.

Redundancy: how many copies, and where

Redundancy decides what happens if hardware, a building, or a whole region fails. Locally redundant storage keeps three copies inside one data centre, which is the cheapest option but cannot survive the loss of that building. Zone-redundant storage spreads copies across separate data centres in one region. Geo-redundant storage adds an asynchronous copy in a paired region far away. Geo-zone-redundant storage combines zone spreading in the primary region with a copy in the paired region. Read-access variants let applications read from the secondary region.

Access tiers

Data in a lake is rarely equal. Fresh data is read constantly, while data from last year is touched only for audits. Access tiers let you pay less to store data that is read less. The hot tier has the highest storage price and the lowest access cost. The cool and cold tiers lower the storage price but charge more for reading and impose minimum storage periods. The archive tier is the cheapest to store but data must be brought back, which can take hours, before it can be read. Lifecycle rules can move data between tiers automatically based on age or last access.

Scale and limits

A single account can hold a very large amount of data and sustain very high rates of reads and writes, but there are documented scalability targets for capacity, requests per second, and bandwidth. A well-designed lake spreads its load across enough files and folders, and large estates sometimes use several accounts to stay within targets and to separate domains. Check the current scalability targets before sizing a platform, because they are updated from time to time and can sometimes be raised by request.

A lake without a design becomes a swamp. These choices decide whether people can find, trust, and quickly read your data.

Zones: the journey from raw to ready

Most successful lakes organise data into zones that reflect how far it has been cleaned and shaped. A popular pattern is called the medallion design, with three layers often named bronze, silver, and gold. Bronze holds raw data exactly as it arrived, which preserves the original evidence and lets you reprocess it later. Silver holds cleaned and combined data, with errors fixed, duplicates removed, and types made consistent. Gold holds data shaped for a particular use, such as tables for reports, features for machine learning, or summaries for business teams.

flowchart LR
    SRC[Source systems] --> BR[Bronze raw zone]
    BR --> SI[Silver cleaned zone]
    SI --> GO[Gold curated zone]
    GO --> BI[Reports and dashboards]
    GO --> ML[Machine learning]
    SI --> DS[Data science exploration]
        

Folder structure

Inside each zone, a clear folder path tells people and tools what the data is and where it came from. A common pattern is to include the zone, the source or business domain, the dataset name, and then time-based folders such as year, month, and day. Time-based folders help tools skip data they do not need, a trick called partition pruning. They also make it easy to apply different permissions or lifecycle rules to old data and to delete a whole month with one atomic folder operation.

Folder depth and the limits of permissions

Folder design affects security. Because permissions are stored on each directory and new items inherit from their parent only at creation time, it is wise to grant access at higher levels, to groups rather than to individuals, and to avoid designs that force you to set different permissions on thousands of tiny folders. Keep the hierarchy deep enough to be meaningful and shallow enough to manage.

File formats

File sizes matter

Analytics engines read files in parallel, but each file has overhead, such as opening it and reading its footer. Millions of tiny files overwhelm this machinery, because the time spent on bookkeeping dwarfs the time spent reading data. Very small files also increase the number of storage requests, which increases cost. The opposite extreme, a few gigantic files, limits how much work can be done in parallel. A good target is files of a few hundred megabytes up to a few gigabytes, which keeps overhead low while allowing plenty of parallel reads. Many platforms offer a compaction step that merges small files into larger ones.

Let us follow a single piece of data from the moment it is created to the day it is deleted, and then follow the life of the lake itself.

The journey of one dataset

Suppose a shop records every sale in a database. At night, an ingestion tool copies the day’s new sales into the raw zone of the lake as files, without changing them. A processing job then reads those files, removes duplicates, fixes dates, joins product details, and writes the result to the cleaned zone in a columnar format. A second job summarises sales by store and product and writes the result to the curated zone. A reporting tool reads the curated files each morning, and managers see the dashboard. Months later, a lifecycle rule moves old raw files to a cooler tier, and years later another rule deletes them according to policy.

sequenceDiagram
    participant DB as Sales database
    participant ING as Ingestion tool
    participant RAW as Raw zone
    participant ENG as Processing engine
    participant CUR as Curated zone
    participant BI as Reporting tool
    DB->>ING: New sales for the day
    ING->>RAW: Write files unchanged
    ENG->>RAW: Read new files
    ENG->>ENG: Clean and combine
    ENG->>CUR: Write tables
    BI->>CUR: Read tables
    BI-->>BI: Show dashboard
        

The write path inside the service

When a tool writes a file, it sends data in blocks. The service stores the blocks, and when the tool signals completion, the file becomes visible with its full contents. The namespace metadata is updated so the directory tree shows the new file. The data is copied to the number of locations required by your redundancy setting before success is reported for the highest durability levels. For analytics jobs, a common safe pattern is to write into a temporary directory and then rename it to the final name, relying on the atomic rename to publish the whole result at once.

The read path

For a read, the tool asks the service for a range of bytes within a file. Modern engines ask for several ranges at the same time, from many files, which is why large lakes can serve enormous throughput. Access control is checked at the start, and tokens are refreshed as needed. Reading from a tool in the same region as the account gives the lowest delay and avoids data transfer charges.

The life of the lake itself

Lifecycle management rules

Lifecycle management lets you write simple rules that act on data automatically. A rule might move files not modified for thirty days to the cool tier, move files not modified for a year to the archive tier, and delete files after seven years. Rules can be limited to certain folders by matching on the path prefix, which is why a clean folder design pays off. Because tiering and deletion save money and reduce risk, every lake should have such rules, reviewed by data owners.

A lake is only useful when data flows in and insight flows out. Here are the common ways of doing both.

Bringing data in

Reading data out

Many services read from the lake. Spark-based platforms such as Azure Databricks, Synapse Spark, and Fabric process data at scale. Serverless query engines can run structured queries directly over files without loading them into a database. Data warehouses can load or query lake data. Machine learning services read training data from the lake. Business intelligence tools such as Power BI can connect to curated data. Microsoft Fabric can expose existing lake folders through shortcuts, so that data stays where it is while being used inside the Fabric workspace.

flowchart LR
    DBS[Databases] --> PIPE[Pipelines]
    APPS[Applications] --> PIPE
    EVT[Event streams] --> CAP[Event capture]
    FILES[Partner files] --> SFTP[File transfer]
    PIPE --> LAKE[Data lake]
    CAP --> LAKE
    SFTP --> LAKE
    LAKE --> SPK[Spark platforms]
    LAKE --> SQL[Serverless query]
    LAKE --> ML[Machine learning]
    LAKE --> BIT[Business intelligence]
        

Batch and streaming together

Some data arrives in large periodic batches, such as a nightly export. Other data arrives continuously, such as sensor readings. A good lake handles both. Streaming data is often first landed as small files and then compacted into larger ones, or written directly into a table format that handles the small-file problem. The key is to decide how fresh each consumer needs the data to be, because freshness costs money and complexity.

Choosing the right access path

Planning for failure during ingestion

Pipelines fail, networks drop, and sources send bad files. Build ingestion that can safely run twice without creating duplicates, by writing to a temporary folder and renaming on success, or by recording which files have already been processed. Keep a rejected area for files that fail checks, so that nothing is silently lost and owners can fix the cause. Alert on pipelines that did not run, not only on pipelines that failed, because silence is also a sign of trouble.

The lake often holds the most valuable data a company owns. Access control is the first line of defence, and it has two cooperating systems.

Who is asking: authentication

Authentication answers “who are you?”. The preferred method is Microsoft Entra ID, which can identify people, applications, and Azure services, including through managed identities that need no stored password. Other methods exist: a shared account key, which gives full power to anyone who holds it, and shared access signatures, which are limited tokens that can be handed to others. Because the shared key is so powerful, many organisations disable it and rely on Entra identities, which can be audited and removed centrally.

System one: role-based access control

Azure roles are assigned at the subscription, resource group, account, or container level. For data, the key built-in roles are Storage Blob Data Owner, Storage Blob Data Contributor, and Storage Blob Data Reader. A reader can read data, a contributor can read, write, and delete, and an owner can do all of that and also manage permissions. Role assignments are inherited from higher levels, so a role at the account level covers every container within it. Roles are broad, simple, and easy to audit, but they cannot grant access to one folder and not to its neighbour.

System two: access control lists

Access control lists, available when the hierarchical namespace is on, apply to individual directories and files. Each list holds entries that pair an identity with permissions to read, write, and execute. For a file, read and write have their obvious meanings and execute means nothing. For a directory, read lets you list its contents, write lets you create or delete children, and execute lets you pass through the directory to reach what lies below. To reach a file with ACLs alone, an identity needs execute permission on every directory along the path, starting from the root of the container.

How a request is judged

When both systems exist, Azure checks roles first. If a role assignment grants the needed permission, the request is allowed and the access control lists are not even consulted. If no role grants it, the service then evaluates the access control lists. Inside the list evaluation, the service checks identities in a fixed order and stops at the first match: a superuser, the owning user, a named user or application, the owning group or a named group, and finally everyone else. A mask can limit the effective permissions of named users and groups. Understanding this order explains many surprises, such as a person who seems to have permission through a group but is blocked because an earlier entry matched.

flowchart TD
    R[Request arrives] --> A{Role grants the permission}
    A -->|Yes| OK[Allowed]
    A -->|No| B{Access control list check}
    B --> C[Superuser then owner then named user then groups then other]
    C --> D{Permission found along the whole path}
    D -->|Yes| OK
    D -->|No| NO[Denied]
        

Important rules and limits

A group-based design

The sustainable way to manage access is through security groups. Create groups that describe roles, such as readers of the sales curated data and writers of the sales raw data. Put the groups, not individuals, in the lists. When a person joins or leaves a team, you change group membership once, and no list needs to be rewritten across a large folder tree. Keep group sizes reasonable, since very large groups can cause performance problems in how identity tokens are built.

Identity decides who may enter. The network and encryption layers decide where they can enter from and what an intruder would find.

Layer 1: network isolation

By default a storage account has a public endpoint reachable from the internet, protected only by authentication. For sensitive lakes, close that door. The account firewall can limit access to chosen networks and addresses. Private endpoints give the account a private address inside your virtual network, so traffic never crosses the public internet. A detail that surprises many teams: the blob and data lake interfaces are separate sub-resources of the account, so a private endpoint is needed for each interface that your tools use. If a tool uses the data lake interface and you created an endpoint only for the blob interface, the connection fails.

flowchart TD
    APP[Analytics services in virtual network] --> PE1[Private endpoint for data lake interface]
    APP --> PE2[Private endpoint for blob interface]
    PE1 --> ACC[Storage account]
    PE2 --> ACC
    OFF[On-premises network] -->|VPN or private circuit| APP
    NET[Public internet] -.->|blocked by firewall| ACC
        

Layer 2: encrypted connections

All traffic should be encrypted in transit. Require secure transfer so that only encrypted connections are accepted, and set a minimum protocol version so that old, weak versions are refused. The driver scheme that ends in s, the secure variant, makes sure analytics tools use encrypted connections.

Layer 3: encryption at rest

Data is always encrypted on disk. By default the keys are managed by Microsoft. If your rules require control over keys, you can use customer-managed keys held in a key vault, which lets you rotate or revoke them. Some organisations also enable an extra layer of encryption at the infrastructure level for stricter requirements. As always, taking control of keys brings responsibility: losing access to a key can make data unreadable.

Layer 4: removing the weakest keys

Disable anonymous access so that nothing can be read without credentials. Consider disabling shared key authorisation so that only Entra identities and carefully issued tokens work. If you must use shared access signatures, prefer user delegation signatures, which are backed by an identity instead of the account key, keep their lifetimes short, and limit their permissions to the minimum.

Layer 5: protection against deletion and tampering

Layer 6: logging and review

Turn on diagnostic logs for read, write, and delete operations and send them to a place your security team watches. Review who has access regularly. Set alerts on unusual events such as a sudden burst of deletions, access from an unexpected location, or a large download of sensitive data.

Governance is the set of habits that keep a lake trustworthy as it grows. Without it, even a perfectly secured lake becomes confusing and risky.

Ownership and stewardship

Every dataset needs a named owner who decides who may use it and what quality it must have. A steward handles the daily care: documenting meaning, watching quality, and answering questions. When no one owns a dataset, nobody fixes it when it breaks and nobody deletes it when it is no longer needed. Record the owner in a catalogue, not only in someone’s memory.

Cataloguing and lineage

A data catalogue lists datasets, their descriptions, their schemas, and their owners. Microsoft Purview can scan storage accounts, discover what is inside, classify sensitive columns such as personal identifiers, and show lineage, which is the map of where data came from and which reports depend on it. Lineage answers urgent questions: if this source is wrong, which dashboards are affected? If a person asks for their data to be erased, where does it live?

Naming and tagging standards

Names should tell a story. Accounts, containers, and folders should follow a pattern that includes the environment, the domain, and the purpose. Azure tags on the account record the owner, cost centre, data classification, and expiry date. Consistent names and tags make cost reports readable and automation possible, because scripts can find all accounts with a certain tag.

Regulation and residency

Laws about privacy and data location may limit where data can live and how long it can be kept. Choose the region for each account with those laws in mind, and remember that geo-redundant options copy data to a paired region, which may sit in a different country. Azure Policy can enforce rules across a company, such as requiring private endpoints, denying accounts without encryption settings, or limiting regions. Use it to turn written policy into automatic guardrails.

Protecting personal data

Where personal data must be stored, apply the principle of least data: store only what you need, mask or pseudonymise identifiers early in the pipeline, and keep the key to re-identify in a separate protected place. Keep sensitive data in its own container or account, with tighter access and longer log retention. Plan how you will honour deletion requests, since data copied across many zones and derived tables is hard to find without lineage.

Data loss comes in several forms: a failed disk, a failed data centre, a failed region, a mistaken deletion, or a malicious act. Each needs a different defence.

Redundancy: protection against hardware and site failure

Every account keeps multiple copies of data. Locally redundant storage keeps three copies inside one data centre. Zone-redundant storage spreads copies across availability zones in one region, surviving the loss of a whole data centre. Geo-redundant storage adds an asynchronous copy in a paired region far away, and geo-zone-redundant storage combines zone spreading in the primary region with a copy in the paired region. The read-access variants allow reads from the secondary copy.

Failover

With geo-redundancy, if the primary region suffers a long outage, you can start a customer-managed failover, which promotes the secondary region to primary. Because geo-replication is asynchronous, the most recent writes may not have arrived, a gap measured by the last sync time. After failover, the account becomes locally redundant in its new region until you reconfigure geo-redundancy. Some features and account configurations have restrictions on failover, and the rules for accounts with the hierarchical namespace have changed over time, so check the current documentation before relying on it, and rehearse the process.

flowchart LR
    subgraph P[Primary region]
        PA[Account with zone redundancy]
    end
    subgraph S[Paired region]
        SA[Secondary copy]
    end
    PA -->|Asynchronous replication| SA
    APP[Applications] --> PA
    APP -.->|After failover| SA
        

What is not available with the hierarchical namespace

Backup options that work for ordinary blob storage are limited when the namespace is hierarchical. Blob versioning, point-in-time restore, change feed, and object replication are not supported on accounts with the hierarchical namespace. Snapshots are available in preview. Soft delete, however, is supported, as are container-level immutability policies on standard accounts. Always confirm the current feature matrix on the official page, since support expands over time.

Building your own backup

Since built-in point-in-time recovery is absent, many teams make periodic copies of critical zones to a second account, ideally in another subscription with different permissions and immutability turned on. Data pipeline tools or command-line copy can do this on a schedule. Table formats such as Delta keep a history of changes, which allows you to rewind a table without a separate backup, but this protects the table, not the account. Raw zone data can often be rebuilt from sources, which lowers how much needs backing up, while curated data that took costly work to build deserves a copy.

Storage is rarely the slow part when a lake is designed well. When things are slow, the cause is usually one of a handful of familiar patterns.

Parallelism is the main lever

The service scales by spreading data and requests across many servers, but each client must ask for work in parallel to benefit. Analytics engines do this automatically by splitting files and tasks. Simple copy tools may need settings that raise the number of concurrent transfers. A single thread reading one file is limited by round trips, however fast the network is.

File size and count

As discussed earlier, files of a few hundred megabytes up to a few gigabytes strike a balance. Too many small files add request overhead and slow listing. Too few huge files limit parallel readers. Partitioning data into folders by date or another common filter lets engines skip what they do not need.

Throttling

Each account has limits on request rate, ingress, and egress, and a heavy workload may hit them. When this happens, the service returns server-busy responses and clients should retry with growing delays. Frequent throttling is a sign to spread load across more accounts, reduce request counts by using larger files, or consider the premium tier. Request default limits from the official documentation, since they differ by region and can change, and raise a support request to increase them when justified.

Premium versus standard

Premium block blob accounts run on solid-state storage with low and steady delay, which helps interactive analytics, machine learning training with many small reads, and workloads with heavy transaction counts. They cost more per stored gigabyte but less per transaction, so busy workloads can come out cheaper. Standard accounts are better for large, mostly sequential data and for data that is stored for long periods.

Listing and directory operations

One of the big wins of the hierarchical namespace is that listing a directory and renaming it are cheap, because they work on a real directory object. Even so, listing a directory with millions of immediate children is slow in any system. Use partitioned subfolders so no single directory becomes gigantic, and let engines read table metadata instead of listing everything.

Scaling out with several accounts

When one account reaches its ceiling, the straightforward answer is more accounts. Split by domain, by environment, or by workload type so that a heavy batch job cannot starve interactive users. Engines can read from many accounts in one query, so users rarely notice the split. The cost of this approach is extra administration, which is why templates and policy matter. Measure first, because many teams add accounts when the real fix was larger files or better partitioning.

You cannot manage what you cannot see. Azure gives you metrics for health, logs for detail, and alerts for attention.

Metrics

Azure Monitor shows metrics for each account, including used capacity, transaction counts, ingress and egress volumes, success rate, end-to-end latency, and server latency. Compare end-to-end latency with server latency: if the first is much larger, time is being lost in the network or the client, not in storage. Break metrics down by response type to find throttling and authorisation failures.

Logs

Diagnostic settings can send resource logs for read, write, and delete operations to a Log Analytics workspace, an archive account, or an event stream. These logs show the caller, the operation, the path, the status code, and the authentication type. They are essential for security reviews and for understanding who deleted a folder.

flowchart LR
    ACC[Storage account] --> MET[Metrics]
    ACC --> LOG[Resource logs]
    MET --> AL[Alerts]
    LOG --> LA[Log Analytics workspace]
    LA --> AL
    LA --> WB[Workbooks and dashboards]
    AL --> TEAM[Responsible team]
        

Alerts worth having

Common problems and first checks

Building a short runbook

When an alert fires at night, the person on call should not have to guess. Write a one-page runbook for each important alert that says what the alert means, which metric or log to open first, the three most likely causes, who to contact, and how to confirm the fix. Review runbooks after each incident and remove steps nobody used. A tidy runbook turns scary alerts into routine tasks and keeps knowledge alive when people change teams.

Few teams start with an empty field. Most arrive with an older lake, a Hadoop cluster, or a pile of files in ordinary blob storage.

From Azure Data Lake Storage Gen1

Gen1 was retired by Microsoft in early 2024, so any remaining workload on it should already have moved. If you meet an old design that mentions Gen1, treat it as history and treat the migration as completed or urgent. Migration tools and guides moved data and access lists to Gen2. The main differences to plan for were a new endpoint format, a different permission model that combines roles with lists, and accounts that exist inside the general Azure storage platform.

From Hadoop and HDFS

On-premises Hadoop clusters tie storage to compute. Moving to Gen2 allows the two to scale separately. Because the ABFS driver offers a Hadoop-compatible file system, many existing jobs only need their paths changed. The data itself can move over the network with data copy tools, or physically with Data Box when volumes are very large. Plan for permission translation, since HDFS permissions map onto access lists but identities now come from Entra ID. Plan a period where both sides run in parallel, and verify results by comparing counts and checksums.

From existing blob storage

An existing blob account can be upgraded in place to enable the hierarchical namespace. The upgrade validates the account, then performs the conversion, during which the account is unavailable for writes. The process is one-way in practice: after it completes, you cannot simply switch the hierarchical namespace off, and you should check the feature support table because features such as versioning and point-in-time restore are not compatible and must be turned off before the upgrade. Test the process on a copy first. If incompatibility or risk worries you, copy data to a new account instead.

flowchart TD
    START[Existing data] --> Q1{Where is it today}
    Q1 -->|Hadoop cluster| H[Copy with data tools or Data Box and map permissions]
    Q1 -->|Blob account| Q2{Features incompatible with hierarchical namespace in use}
    Q2 -->|Yes| C[Copy to a new account]
    Q2 -->|No| U[Validate then upgrade in place]
    Q1 -->|Another cloud or on-premises files| F[Copy with pipelines or command line]
    H --> V[Verify counts and checksums]
    C --> V
    U --> V
    F --> V
    V --> CUT[Switch consumers and retire the old system]
        

Patterns are repeatable answers to repeated problems. Here are the ones that appear in almost every successful lake, along with the traps to avoid.

Helpful patterns

flowchart LR
    EXT[External partner] --> LND[Landing account]
    LND --> VAL[Validation pipeline]
    VAL --> CORE[Core lake account]
    CORE --> RAWZ[Raw]
    CORE --> CLNZ[Cleaned]
    CORE --> CURZ[Curated]
    CURZ --> USERS[Business users through groups]
        

When to use one account and when to use many

A single account is simple, but all zones share the same limits, the same network settings, and the same blast radius for a mistake. Many companies separate production and non-production, and sometimes split by business domain or sensitivity. More accounts mean more settings to manage, so automate their creation from templates. A good rule: split when you need different network rules, different keys, different regions, different owners, or separate capacity limits.

More anti-patterns

Choosing patterns by team size

A small team with a few datasets should keep things simple: one production account, three zones, group-based lists, and a weekly compaction job. A larger organisation with many domains benefits from separate accounts, a shared catalogue, central policy, and domain teams who own their own zones. Do not copy the pattern of a giant company if you are small, because complexity is also a cost. Start simple, measure the pain, and add structure only when the pain justifies it.

This chapter gathers the advice from all earlier chapters into short checklists you can use during design reviews and before go-live.

Best practices

Common mistakes and their fixes

A pre-launch checklist

Every design choice has a price in money, effort, and flexibility. Here is an honest look at what Azure Data Lake Storage Gen2 offers and what it asks in return.

How the bill is made

Costs come from several places. You pay for the amount of data stored, which depends on the tier and on redundancy. You pay for operations, such as reads, writes, and listings, with write operations usually costing more than reads and cooler tiers charging more for operations. You pay for data retrieval from cool and archive tiers, for early deletion before a tier’s minimum period ends, and for data transfer out of a region. Additional costs may come from geo-replication traffic, private endpoints, log storage, and the analytics services that read the data. Prices vary by region and change over time, so use the official pricing page and calculator for real numbers.

Ways to spend less without hurting the work

Cost is also a people question

The cheapest storage is the data you never keep. Ask owners to confirm that each dataset is still needed, and put an expiry tag on sandboxes. Show teams their own spend using tags and cost reports, because people who can see their bill tend to clean up after themselves.

These short stories show how the concepts fit together in situations you may recognise.

flowchart TD
    N[Need arises] --> T{Type of need}
    T -->|Analytics at scale| L[Lake with zones and columnar files]
    T -->|Strict security| S[Private endpoints, identity only, encryption keys]
    T -->|Streaming| ST[Event capture with compaction]
    T -->|Migration| M[Copy tools or upgrade after checks]
    T -->|Protection| P[Soft delete, locks, immutability, independent copies]
    L --> G[Governance and monitoring]
    S --> G
    ST --> G
    M --> G
    P --> G
        

What these stories have in common

In every scenario, success came from decisions made early and boring habits kept up over time. Teams that agreed on zones and naming found data easily. Teams that used groups and identity-based access changed permissions without panic. Teams that rehearsed restores recovered from mistakes in minutes instead of days. None of this needed exotic technology, only clear ownership, sensible defaults, and regular review.

Short, direct answers to questions that beginners and busy teams often ask.

No. It is a set of capabilities built on Azure Blob Storage. You get it by enabling the hierarchical namespace on a storage account, which adds real directories, access control lists, and the data lake interface.

Gen1 was retired in early 2024. New and existing workloads should use Gen2. Confirm details on the official Microsoft page.

It makes folders real objects instead of name prefixes, so renaming or deleting a folder is a single fast and atomic operation, and it allows access control lists on individual directories and files.

They are two doors to the same data. The blob endpoint uses the blob interface, and the dfs endpoint uses the data lake interface with directory operations. Many tools can use either, and a private endpoint is needed for each one you use.

It is the Hadoop-compatible driver that lets Spark and other engines treat the lake like a file system. Paths use the abfs scheme, or abfss for encrypted connections.

Not always. Roles are simple and broad, while lists give folder-level precision. Many teams use roles for platform administrators and lists for everyone else. Remember that roles are evaluated first and override lists.

The user probably lacks execute permission on one of the directories above the file, or an earlier matching entry or the mask limits the result. Check every directory along the path.

No. Lists apply to Entra identities. Requests using the account key or ordinary shared access signatures are not checked against lists, which is another reason to prefer identity-based access.

Not on an account with the hierarchical namespace. Use soft delete, locks, immutability, and your own copies instead, and check the feature table for current support.

In practice, no. Upgrading an existing blob account is one-way, so test on a copy and decide carefully.

Several hundred megabytes up to a few gigabytes is a common target for analytics. Avoid millions of tiny files and avoid a few huge ones.

No. Redundancy protects against hardware and site failure, but it copies deletions and corruption. Add soft delete, locks, immutability, and independent copies.

Use it when you need low and steady latency or very high transaction rates, such as interactive analytics or training with many small reads. For large sequential data stored for long periods, standard is usually cheaper.

Split accounts when you need different network rules, keys, regions, owners, or capacity limits, such as production versus development. Otherwise keep things simple and automate creation.

Right-size files, apply lifecycle rules, set retention, choose redundancy by data value, keep compute in the same region, and review cost reports regularly.

Azure Data Lake Storage Gen2 combines the scale and low price of object storage with the directory structure and fine-grained security that analytics need. The hierarchical namespace is the heart of it, turning folders into real objects that can be renamed atomically and protected with access control lists. A good lake is more than an account: it needs zones, folder and file layouts, group-based access, private networking, layered protection against mistakes, monitoring, lifecycle rules, and governance with clear owners. Know the limits too, such as the missing versioning and point-in-time restore on accounts with the hierarchical namespace, and plan your own safety nets. With these habits, the lake stays fast, safe, and affordable as it grows.