Hierarchical Namespace Controller in Kubernetes

Hierarchical Namespace Controller in Kubernetes

Kubernetes namespaces are a flat list, but real organizations are trees of companies, departments, teams and projects. The Hierarchical Namespace Controller teaches the cluster about parents and children, lets rules flow down the tree, and lets teams create their own namespaces safely. This guide explains how it works inside, and how to use it well.

Think of a large company’s filing room. At first there is a single long shelf with one folder per team: Payments, Search, Mobile. As the company grows, the shelf becomes crowded. Each department wants its own folders for development, testing and production. The company wants one rule, such as “all folders must be locked at night”, to apply to everything under a department without repeating it a hundred times. A flat shelf cannot express this. A filing cabinet with drawers inside drawers can. The Hierarchical Namespace Controller adds those inner drawers to Kubernetes.

1What the Hierarchical Namespace Controller Is

A short definition first, so that every later chapter has something solid to stand on.

The Hierarchical Namespace Controller, usually shortened to HNC, is an add-on for Kubernetes that lets namespaces form a tree. One namespace can be the parent of others, and the others are its children. A child can have children of its own. Once the relationships exist, HNC does two main things for you. It copies chosen objects, such as permissions and policies, from a parent down into all its descendants. And it lets people who control a parent namespace create new child namespaces without asking a cluster administrator.

HNC does not change Kubernetes itself. A namespace is still just a namespace as far as the core system is concerned, and the cluster still keeps them in a flat list. HNC adds a layer on top. It stores the parent and child links in its own objects, watches for changes, and keeps the right copies in the right places.

Simple Analogy

Imagine a school where every classroom has its own set of rules on the wall. A headteacher can walk into each room and write the same rule again and again, or the school can have a system where a rule written in the “Year 5” office automatically appears on the wall of every Year 5 classroom. HNC is that system for namespaces.

Term 1

Namespace

A named box inside a cluster that holds objects such as Pods and Services, and that can have its own permissions and limits.

Term 2

Parent and child

A parent namespace sits one level above a child. Each namespace can have at most one parent, and any number of children.

Term 3

Ancestors and descendants

Ancestors are all the namespaces above one, up to the top. Descendants are all the namespaces below it, at any depth.

Term 4

Propagation

The automatic copying of selected objects from a namespace down to its descendants.

Term 5

Subnamespace

A child namespace that is created by asking inside the parent, with no cluster-level rights needed.

i
Remember

HNC is a separate project from the core of Kubernetes. It lives in the Kubernetes community under the multi-tenancy effort. Before adopting it, check the current state of the project, the supported Kubernetes versions and how actively it is maintained.

2A Quick Refresher on Namespaces

To see what is missing, we first need to be clear about what namespaces give us today.

A namespace is a way to divide one cluster into several virtual sections. Most objects, such as Pods, Services, Deployments and ConfigMaps, belong to exactly one namespace. Names only need to be unique inside a namespace, so two teams can each have a Deployment called web without a clash. A few objects, such as Nodes and PersistentVolumes, are cluster-wide and do not live in any namespace.

Namespaces are also the main unit for control. Permissions are usually granted per namespace, using Roles and RoleBindings. Resource limits are applied per namespace, using ResourceQuotas. Network rules are written per namespace, using NetworkPolicies. A policy engine or an admission rule often looks at the namespace to decide what is allowed.

What namespaces give youExample
Name scopingTwo teams can both own a Service named api
Access control boundaryA team’s engineers can edit their namespace and nothing else
Resource limitsA quota caps how much CPU a team may use
Network boundaryPolicies can allow or block traffic between namespaces
Policy targetAdmission rules can be applied to selected namespaces

What namespaces do not give you

Namespaces have no relationship to each other. There is no such thing as a namespace inside a namespace. Each is an equal in a flat list. Rules written in one namespace do not affect any other. To apply the same rule to twenty namespaces, someone has to create twenty copies and keep them all up to date by hand or with automation.

Simple Analogy

A flat list of namespaces is like a street of identical houses. Each has its own front door and its own rules, but there is no concept of a neighborhood, a district or a city. If the city wants every house to have a smoke alarm, someone must visit each house one by one.

!
Common Misconception

Many people assume that putting dashes in names, like team-a-dev and team-a-test, creates a hierarchy. It does not. The dashes are only part of the name. Nothing in Kubernetes treats these two as related.

3The Pain That Hierarchy Removes

Hierarchy is not a luxury. It solves several very concrete problems that grow with the size of a cluster.

Repeated setup for every namespace

Every new namespace needs the same baseline: access rules for the team, a default network policy, a pull secret for the image registry, and often a quota. Doing this by hand is slow and error-prone, and automation scripts drift out of date.

A bottleneck at the platform team

In most clusters, only administrators may create namespaces, because a namespace is a cluster-level object. Every team that wants a new environment must file a ticket and wait. Platform teams become a queue.

Giving too much power to delegate creation

The alternative, giving team leads the right to create any namespace, is risky. They could create namespaces with names that collide with others, or take over names meant for other teams, and nobody controls where those new namespaces fit.

No way to say “everything under this team”

Policies often need to apply to a whole group of namespaces: all of Team A’s environments, or all production namespaces of a department. A flat list has no way to describe such groups except by relying on labels that people must remember to add.

Inconsistent permissions

When roles are copied by hand into many namespaces, copies fall out of step. One namespace still grants access to someone who left the team, because nobody remembered to update it.

Simple Analogy

Imagine a company where only the head of facilities may ever open a new office room. Every team that needs a meeting room must queue to ask. If the head of facilities gives out master keys to everyone to solve the queue, rooms are opened without any plan. A better arrangement lets each department head open rooms inside the department’s own floor, and nowhere else. That is what subnamespaces provide.

“HNC lets the organization chart of a company show up inside the cluster, and lets rules follow that chart.”

4The Core Idea: Trees of Namespaces

Everything HNC does grows from a very small idea about how namespaces are connected.

HNC arranges namespaces in a forest, which is simply a collection of trees. Each tree has one namespace at the top, called the root. Every other namespace in that tree has exactly one parent. A namespace with no parent is a root. A namespace with no children is a leaf. Cycles are not allowed, so a namespace can never be its own ancestor.

flowchart TD
  ORG["org: root"] --> TA["team-a"]
  ORG --> TB["team-b"]
  TA --> TAD["team-a-dev"]
  TA --> TAS["team-a-staging"]
  TA --> TAP["team-a-prod"]
  TB --> TBD["team-b-dev"]
  TB --> TBP["team-b-prod"]
        
FIG 1 — A forest made of one tree: an organization, two teams and their environments

Two simple rules about direction

Things flow in one direction only. Objects and rules move down the tree, from a parent to its children, grandchildren and so on. Nothing flows upward. A rule in a child never affects its parent, and a rule in one child never affects its sibling. This one-way flow keeps behavior easy to predict: to know everything that applies to a namespace, you look at it and at its ancestors, and nowhere else.

WordMeaningExample from the tree
RootNamespace with no parentorg
ParentThe namespace directly aboveteam-a is the parent of team-a-dev
ChildA namespace directly belowteam-a-prod is a child of team-a
AncestorAny namespace above, at any distanceorg and team-a are ancestors of team-a-dev
DescendantAny namespace below, at any distanceAll of the namespaces under org
SiblingAnother child of the same parentteam-a-dev and team-a-prod
LeafA namespace with no childrenteam-b-dev
Simple Analogy

A family tree has the same shape. Grandparents pass things down to parents and children, but a child cannot pass anything up, and cousins do not share with each other directly. Each person has exactly one set of parents in the tree.

5Architecture: Controller, Webhooks and the Forest

Now we open the box and look at the parts that make the tree real.

HNC is installed as a program that runs inside the cluster, in its own namespace, usually called hnc-system. It is built like most Kubernetes add-ons. It has custom resource definitions that add new object types. It has a set of controllers that watch objects and make changes. And it has admission webhooks that check changes before they are saved.

Part 1

Custom resource definitions

Add new types to the API: the hierarchy configuration, the subnamespace anchor and the HNC configuration.

Part 2

Controllers

Watch namespaces and objects, and create, update or delete copies so that the cluster matches the tree.

Part 3

Admission webhooks

Reject changes that would break the rules, such as a cycle in the tree or an edit to an inherited object.

Part 4

The in-memory forest

A data structure inside the controller that holds the whole tree of namespaces, so that decisions are made quickly without asking the API server each time.

flowchart LR
  U["User or automation"] --> API["API Server"]
  API -->|"Admission check"| W["HNC webhooks"]
  W --> API
  API -->|"Watch events"| C["HNC controllers"]
  C  F["In-memory forest"]
  C -->|"Create and update copies"| API
        
FIG 2 — The main parts of HNC and how they talk to the API server

The forest in memory

When HNC starts, it reads all namespaces and hierarchy objects from the API server and builds the forest in memory. After that, it keeps the forest up to date by listening for changes. Whenever a controller needs to know “who are the descendants of this namespace”, it asks the forest and gets an instant answer. This makes propagation quick. It also means that after a restart, HNC rebuilds its picture from the objects stored in the cluster, so the real source of truth stays in the API server.

Simple Analogy

A receptionist keeps a copy of the company’s organization chart on the desk, so that when someone asks “who works under Maria?” the answer is immediate. If the desk copy is lost, the receptionist rebuilds it from the official records in the archive. The forest is the desk copy, and the API server is the archive.

!
Important Detail

The webhooks are part of the path of requests. If HNC is unavailable, some operations on namespaces and on inherited objects may be delayed or blocked, depending on how the webhooks are configured. Plan for HNC’s availability like any other control plane component.

6Installing HNC and Using the kubectl Plugin

Getting started is straightforward, and a small command line plugin makes daily use friendly.

HNC is installed by applying the manifests published with each release, which create the controller, the definitions and the webhooks in the HNC namespace. Choose a release that supports your Kubernetes version, and read its notes. Because HNC adds webhooks that can affect namespaces and objects across the cluster, it is wise to try it first in a test cluster.

HNC also comes with a plugin for the kubectl command line tool, usually installed through the krew plugin manager. The plugin adds a command called kubectl hns with several subcommands. These do not do anything special that could not be done by editing objects directly, but they make common tasks much easier and their output much clearer.

Plugin commandWhat it doesWhen you use it
createCreates a subnamespace under a given parentA team lead wants a new environment
treePrints the tree of namespaces below a given one, with any problemsYou want to see the shape of the hierarchy
setChanges settings such as the parent of a namespaceYou attach an existing namespace to a tree
describeShows details of one namespace: its parent, children, conditions and inherited objectsYou are debugging why something did or did not propagate
configViews and changes which object types propagateYou want a new kind of object to flow down the tree

What changes after installation

Right after installation, nothing dramatic happens. Existing namespaces simply become roots of their own trees, because none of them has a parent yet. By default, RBAC objects, meaning Roles and RoleBindings, are already set up to propagate. From that point, you can start attaching namespaces together or creating subnamespaces.

i
Good Practice

Install HNC in a test cluster, create a small tree, and watch what happens when you add roles, delete namespaces and break rules on purpose. A hands-on feel for the behavior is worth hours of reading.

Simple Analogy

Installing HNC is like installing an intercom system in a building. On day one, every room still works exactly as before. The new capabilities appear only when you start connecting rooms into groups and sending announcements.

7Subnamespaces and Anchors

Subnamespaces are the part of HNC that makes self-service possible, and the trick behind them is clever.

Creating a namespace normally needs cluster-level permission, because a namespace is not inside any other namespace. HNC solves this with a small helper object called a subnamespace anchor. An anchor is an ordinary object that lives inside the parent namespace. Because it is namespaced, a person with rights in the parent can create it, with no cluster-level rights at all.

When the anchor appears, HNC reacts. Its controller, which does have cluster-level rights, creates a real namespace with the name given in the anchor, and records that the new namespace is a child of the parent. The person never needed the power to create namespaces. They only needed the ordinary power to create an anchor object in their own namespace, and HNC did the rest on their behalf.

sequenceDiagram
  participant T as Team lead
  participant A as API Server
  participant H as HNC controller
  T->>A: Create anchor named dev inside team-a
  A-->>H: Watch event: new anchor
  H->>A: Create namespace dev
  H->>A: Mark team-a as parent of dev
  H->>A: Update anchor status to Ok
  A-->>T: Anchor shows the new namespace is ready
        
FIG 3 — How an anchor in the parent causes a child namespace to be created
1

Rights are granted in the parent only

The platform team gives the team lead permission to create anchors in the team’s namespace. This is an ordinary namespaced permission.

2

The lead creates an anchor

The name of the anchor becomes the name of the new namespace.

3

HNC creates the namespace

It also marks the new namespace as a subnamespace, and records the parent link and an annotation that points back to the parent.

4

Inherited objects arrive

Roles, role bindings and any other configured objects flow down from the parent into the new namespace at once.

Simple Analogy

A manager who may not hire staff directly can fill in a request form kept in the department’s own folder. The human resources system watches that folder, and whenever a form appears, it hires the person and places them in the right team. The form is the anchor, and HR is the controller.

!
Important Detail

Namespace names are still unique across the whole cluster, because the core system keeps them in a flat list. A subnamespace called dev cannot be created if any other namespace is already called dev. A common habit is to include the parent’s name in the child’s name, such as team-a-dev, to avoid clashes.

8Full Namespaces: Attaching Existing Namespaces to a Tree

Subnamespaces are not the only way to build a tree. You can also connect namespaces that already exist.

HNC distinguishes two kinds of child namespaces. A subnamespace is created through an anchor, and is tied to its parent by that anchor. A full namespace is a normal namespace that was created in the usual way by an administrator, and then attached to a parent afterwards by setting its parent. Both kinds behave the same way for propagation. The difference is mostly in how they are created and how they are deleted.

AspectSubnamespaceFull namespace
Created byAn anchor object inside the parentNormal namespace creation by an administrator
Who needs cluster rightsNobody. HNC acts on the user’s behalfWhoever creates the namespace
Parent is set byHNC automatically, when it creates the namespaceExplicitly, by changing the hierarchy setting of the child
Can be moved to another parentLimited. It is tied to its anchorYes, by changing the parent, if permissions allow
Deleted byDeleting the anchor in the parentNormal namespace deletion
Typical useSelf-service environments for teamsBringing older namespaces into a tree

Who may set a parent

Setting the parent of a namespace is a powerful action, because it changes which rules flow into it. HNC therefore checks permissions on both sides. The person must be allowed to change the hierarchy setting of the child, and must also have the right to add children to the new parent. This prevents someone from attaching their own namespace to a powerful tree in order to inherit its permissions.

Simple Analogy

A new employee cannot simply decide to report to the chief executive. Both the employee’s current manager and the new boss must agree. HNC requires the same agreement from both sides before it accepts a new parent.

i
Good Practice

Use subnamespaces for everything that teams create on their own, and use full namespaces mainly for migration of existing namespaces. This keeps the creation path clean and keeps cluster-level rights away from teams.

!
Common Mistake

Attaching an existing namespace to a tree without checking what will flow into it. The moment it gets a parent, it receives all the inherited roles and objects, which may grant people access they should not have there.

9The HierarchyConfiguration Object

Each namespace in the tree has one small object that records its place in it.

The place of a namespace in the tree is stored in an object called the hierarchy configuration. There is at most one of these in each namespace, and it has a fixed name, so that it can always be found. Its spec, the part that people edit, holds the name of the parent and a setting about whether deleting this namespace’s anchor or parent should also remove everything below. Its status, the part that HNC fills in, lists the children of the namespace and a set of conditions that describe any problems.

Spec 1

Parent

The name of the parent namespace. Empty means this namespace is a root.

Spec 2

Allow cascading deletion

A switch that says whether removing this namespace should also remove all its descendants.

Status 1

Children

The list of direct children, maintained by HNC, which is used to draw the tree.

Status 2

Conditions

Notes about problems, such as a missing parent or a conflict, with a reason and a message for people to read.

Conditions are the health report

Conditions tell you what is wrong, and they are the first place to look when something seems off. A condition can warn that the named parent does not exist, that a cycle was detected, that an ancestor has a problem, or that an object could not be copied because of a conflict. Some conditions are only warnings about a single object. Others are serious, and cause HNC to stop activities in the affected part of the tree until the problem is fixed, to avoid making things worse.

Simple Analogy

Every employee file has a line saying who the manager is, a line listing direct reports, and a section of notes from human resources about anything unusual. The hierarchy configuration is that file for a namespace, and conditions are the notes from human resources.

i
Good Practice

Show conditions on a dashboard and alert on serious ones. A tree with a serious condition has frozen parts, and nobody should discover that by accident when a new subnamespace fails to appear.

!
Common Mistake

Editing the hierarchy configuration by hand without understanding the permission checks. Prefer the plugin commands, which show clear error messages when a change is refused.

10Propagation: What Flows Down the Tree

Propagation is the second big feature of HNC, and it is where most of the daily value comes from.

When you create an object in a namespace that is marked as a source, HNC makes a copy of it in every descendant of that namespace. If you change the original, the copies change too. If you delete the original, the copies are deleted. The copies are real objects in each descendant, so Kubernetes treats them like any other, and a Role copied into a child namespace grants real permissions there.

flowchart TD
  S["Source object created in team-a"] --> C1["Copy in team-a-dev"]
  S --> C2["Copy in team-a-staging"]
  S --> C3["Copy in team-a-prod"]
  C3 --> C4["Copy in any further child below team-a-prod"]
        
FIG 4 — An object in a parent is copied into every descendant

What kinds of objects can propagate

By default, two kinds of objects propagate: Roles and RoleBindings. These are the objects that control who may do what, and they are by far the most common reason for using a hierarchy. Other kinds of objects can be added to the list by configuration, and we will cover how in the next chapters. Typical additions are NetworkPolicies, ResourceQuotas, ConfigMaps, Secrets and limit ranges. Almost any namespaced object type can be added, including custom resources.

Object typePropagates by defaultTypical reason to propagate
Role and RoleBindingYesGive a team’s people the same access in all its environments
NetworkPolicyNo, can be enabledApply a baseline of network rules to every namespace in a tree
ResourceQuota and LimitRangeNo, can be enabledGive every child the same standard limits
ConfigMapNo, can be enabledShare standard settings such as certificate authority bundles
SecretNo, can be enabled with careShare an image registry pull secret with all environments
Custom resourcesNo, can be enabledShare policy or settings objects defined by operators
Simple Analogy

When a head office updates the safety poster in the company handbook, every branch office automatically receives the new poster, and nobody needs to print and mail them. If head office removes the poster, it disappears from every branch too.

!
Important Detail

Propagation always goes downward, and copies are never sent up or sideways. A Role created directly in a child stays in that child. This is what keeps the behavior predictable.

11Sharing Permissions Down the Tree with RBAC

The most loved feature of the controller is also the simplest: give someone access once, and every child namespace inherits it.

Kubernetes controls who may do what with a system called RBAC, short for role-based access control. A Role is a list of allowed actions, such as reading Pods or editing Deployments. A RoleBinding connects a Role to a person, a group or a service account inside one namespace. Without hierarchy, a RoleBinding in one namespace says nothing about any other namespace. If a team owns twelve namespaces, somebody must create twelve bindings and keep all twelve in sync.

The controller changes this by treating Roles and RoleBindings as things that flow downward by default. When you create a RoleBinding in a parent namespace, the controller creates an identical copy in every child, grandchild and deeper descendant. When you change the original, every copy changes. When you delete the original, every copy disappears. The people named in the binding therefore gain the same access across the whole subtree with one action.

Why this is a safe default

Access flows only downward. A person who is granted rights in a child gains nothing in the parent. This matches how most organizations think: a department head can see everything inside the department, while a single project member sees only that project. The direction of flow makes the permission model easy to explain and easy to audit.

Simple Analogy

Think of a master key at a school. The principal’s key opens the main gate, every hallway and every classroom. A teacher’s key opens only one classroom. When the school hands the principal a new key, nobody needs to visit each classroom to change a lock. Permissions that flow downward work the same way.

Roles and bindings travel together

A binding points to a Role by name. If only the binding were copied and the Role were left behind, the copy would point at nothing. For this reason both objects propagate together. A team can define a Role called “namespace editor” in the parent and bind it to the team’s group. Every child then has the Role and the binding, and they work at once.

SituationWithout hierarchyWith the controller
New team member joinsEdit bindings in every team namespaceEdit one binding in the parent
New environment namespace createdCopy all bindings by hand or by scriptBindings appear automatically
Person leaves the teamHunt for every binding that names themRemove the one original binding
Audit question: who can edit here?Read each namespace separatelyRead the namespace plus its ancestors
i
Good Practice

Bind groups, not individuals. When bindings name a group from your identity provider, joining or leaving the team is handled by group membership, and the hierarchy only carries the rule that the group has access.

!
Common Mistake

Forgetting that a permission placed high in the tree reaches every descendant, including namespaces created later by other people. Place broad rights only at levels where you are happy for them to reach everything below.

12Choosing More Types with HNCConfiguration

Roles and bindings are only the start. A single cluster-wide object tells the controller which other kinds of objects should travel.

The controller has one cluster-wide settings object called HNCConfiguration. There is exactly one of it, and it lists resource types together with a mode for each type. A resource type is a kind of object, such as ConfigMap, Secret, NetworkPolicy or ResourceQuota. The mode decides what the controller does with objects of that type.

The common modes

The names and details can change between versions, so check the documentation for your release. In general there are three ideas. The first is propagate: objects of this type that you place in a parent are copied down to descendants. The second is remove: the controller deletes inherited copies, which is useful when you want to stop a type from flowing without deleting the originals. The third is ignore: the controller does nothing at all for the type, which is the state of every type not on the list.

Mode

Propagate

Originals in a parent are copied into every descendant and kept in sync.

Mode

Remove

Inherited copies are cleaned up. The originals stay where they are.

Mode

Ignore

The controller leaves this type alone. Nothing flows and nothing is removed.

Simple Analogy

Imagine a school office with a photocopier and a list on the wall. The list says which papers get copied to every classroom: the fire plan, the timetable. Other papers are never copied. HNCConfiguration is that list on the wall. A new type of paper goes on the list only when someone decides it should travel.

Which types make good candidates

Good candidates are objects that every namespace in a subtree should share and that rarely differ. Examples include policies that apply to every workload, a shared configuration file with a company-wide setting, or a pull credential for a private image registry. Poor candidates are objects that hold unique data for each application, or objects that vary by environment.

Object typeGood to propagate?Reason
Role and RoleBindingYes, on by defaultShared access rules are the main purpose of hierarchy
NetworkPolicyOften yesBaseline traffic rules, such as default deny, should apply everywhere
ConfigMap with shared settingsSometimesUseful for company-wide values; avoid for per-app data
Secret for a registryCarefullyConvenient, but copies of secrets spread access to the secret
ResourceQuotaRarelyEach copy is a separate limit, so the totals add up in surprising ways
!
Common Mistake

Switching on a type without checking existing objects. When a type starts to propagate, every existing object of that type in every parent immediately flows down, which can surprise teams that already had a different object with the same name in a child.

13Selecting What Propagates and What Does Not

Sometimes one specific object should stay home, or should go only to some children. Annotations give you fine control.

Switching a whole type to propagate is a blunt tool. Real life has exceptions: a Role that is meant only for the parent, or a Secret that should reach just one branch of the tree. The controller offers small notes called annotations that you attach to the individual object. An annotation is a label-like key and value stored in the object’s metadata that tools can read. The controller reads a few of them to decide how that one object should travel.

The ideas behind the selectors

There are four main ideas. One note can say that the object should not propagate at all. Another can say it should propagate everywhere, even if the type is normally excluded in some situations. A third can pick children by their names or labels, in the style of a label selector. A fourth can pick whole subtrees through the tree labels you will meet soon. The exact annotation keys have changed over time, so read the documentation for the key names in your version rather than copying them from a blog post.

1

Type is listed to propagate

HNCConfiguration says Secrets travel, so the controller considers every Secret in a parent.

2

The object is checked for notes

The controller reads the object’s annotations to see whether it has been told to stay, to go everywhere or to go to chosen children only.

3

Each descendant is tested

For every descendant, the controller asks whether this object is allowed to reach it.

4

Copies are created or removed

Allowed descendants receive a copy. Disallowed ones have any old copy removed.

Simple Analogy

A teacher hands out a worksheet to the whole class by default. One sheet has a sticky note saying “only for group B”. The teacher reads the note and gives that sheet to group B only. Annotations are the sticky notes on individual objects.

Why selectors are powerful but risky

Selectors let you share a credential with exactly the right children. They also create invisible rules that live inside individual objects. A reader looking at a child namespace sees a copy but cannot tell why the other children did not get one. Keep a short written record of every selected object, and prefer the tree structure to express most of your intent.

i
Good Practice

Use the tree itself for the common case and annotations for rare exceptions. If many objects need annotations, the tree probably does not match how the organization works and deserves a redesign.

14Inherited Copies, Labels and Read-Only Rules

A copy in a child is not an ordinary object. It is marked, protected and tied to its source.

When the controller creates a copy of a parent object in a child, it labels that copy with a marker whose key is hnc.x-k8s.io/inherited-from and whose value is the name of the namespace where the original lives. This label is the passport of the copy. It tells people and tools that the object is inherited, and it tells the controller that this object is under its management.

Why copies are read-only

The controller’s webhooks refuse changes to inherited copies made by ordinary users. If anyone could edit a copy, the next sync would overwrite their change, or worse, the child would drift from the parent without anyone noticing. To change a copy, you change the original in the ancestor namespace. This is a strong guarantee. It means that what a child namespace receives from above is exactly what the owner above intended.

Simple Analogy

A photocopy of a school notice pinned in a classroom has a stamp that says “copy of the office notice”. Students may not write on it to change the rules. If the notice must change, the office issues a new one and all copies are replaced.

How a copy is kept up to date

The controller watches both the original and the copies. If the original changes, copies are updated. If a copy is somehow deleted or altered by an administrator who bypasses the webhook, the controller restores it. If the original is deleted, all the copies are removed. This reconciling loop is the same pattern that all Kubernetes controllers use, and it is why the system recovers from small accidents without human help.

Consequences you should know

Reading

Easy to find origin

Listing objects with the label quickly shows which ones are inherited and from where.

Writing

Edit at the source

Every change to a shared object must be made in the namespace that owns the original.

Naming

Names must be free

A child that already has an object with the same name and type blocks the copy, and a condition reports the clash.

!
Common Mistake

Trying to customize an inherited Role in a child by editing it. The edit is rejected. If a child needs a slightly different Role, create a new Role with another name in the child, and leave the inherited one alone.

15Tree Labels and Policies That Understand the Hierarchy

Besides copying objects, the controller stamps every namespace with labels that describe where it sits in the tree.

Many Kubernetes features select namespaces by label. A NetworkPolicy, for example, can say “allow traffic from namespaces that carry this label”. Core Kubernetes knows nothing about trees, so a plain policy cannot say “allow everything from the finance department and all its sub-teams”. The controller fills this gap by adding labels to each namespace that name its ancestors.

What the tree labels look like

For every ancestor of a namespace, including the namespace itself, the controller adds a label whose key includes the ancestor’s name followed by the suffix .tree.hnc.x-k8s.io/depth. The value is a number. Zero means the namespace itself, one means a child of that ancestor, two means a grandchild, and so on. A policy can then select every namespace that carries the label for the finance department, regardless of how deep the namespace sits.

flowchart TD
  F["finance (depth 0 label on itself)"] --> P["payroll (finance depth 1)"]
  F --> A["audit (finance depth 1)"]
  P --> PD["payroll-dev (finance depth 2, payroll depth 1)"]
  P --> PP["payroll-prod (finance depth 2, payroll depth 1)"]
        

Figure 5: Every namespace carries a label for each of its ancestors, with a number telling how far away that ancestor is.

In the figure, a policy that selects the finance label matches finance itself, payroll, audit and both payroll environments. A policy that selects the payroll label matches payroll and its two environments only. The same idea works for any feature that selects namespaces by label.

Simple Analogy

A family tree where each person wears a badge for every ancestor: “descendant of Grandma”, “descendant of Aunt Rosa”. If you invite “everyone wearing the Grandma badge” to a party, you get every child and grandchild without listing names.

Why this is valuable for network rules

A common requirement is that workloads inside one department may talk to each other, but not to other departments. Without tree labels, you list each namespace by name and update the list whenever a team adds a namespace. With tree labels, you write one rule per department. New namespaces join the department simply by being created under it.

GoalSelection by namesSelection by tree label
Allow traffic inside a departmentList every namespace and edit the list on each changeOne rule using the department label
Allow a shared service to be reached by all teamsAdd each new team by handOne rule using the organization label
Block one department from anotherMaintain two listsRules that match each department label
i
Good Practice

Remember that the policy engine, not the controller, enforces the rule. Test that your network plugin honors namespace selectors, and test with a real connection, not only by reading the objects.

16Passing Labels and Annotations to Child Namespaces

Some settings live on the namespace object itself. The controller can be told to hand selected ones down to new subnamespaces.

Namespaces carry labels and annotations of their own. Many cluster tools depend on them. A security feature might read a label that sets how strict the Pod rules are. A cost tool might read a label that names the team that pays. A service mesh might read a label that turns on automatic sidecars. If subnamespaces do not carry the same labels as their parent, those tools treat them differently, often by accident.

What the controller can do

Depending on version and startup settings, the controller can copy chosen labels and annotations from a parent namespace to its children, so that a team’s namespaces share the same cost label or security level. Which keys are allowed to propagate is a setting for the administrator, because blindly copying everything would be unsafe. Features in this area have changed across releases, so confirm what your version supports before depending on it.

Simple Analogy

A new student in a school class receives the same house badge as the rest of the class. The badge decides which colour team they play for and which fees are billed. If the badge is forgotten, the student is counted in no team at all.

Why it matters in practice

Cost

Team ownership

A cost label on the parent lets every child be charged to the same team.

Security

Pod safety level

A strict level on the parent stops children from starting with a weaker one.

Mesh

Sidecar injection

Children join the same mesh setting without manual labeling.

!
Common Mistake

Allowing too many keys to propagate. If a privileged label is allowed to flow, a person who can create a subnamespace may receive privileges they were never granted. Keep the list short and review it as carefully as any permission.

If your version does not support label propagation, you can reach a similar result with a policy engine that adds labels at creation time, or with automation that runs when a subnamespace appears. The key point is to decide on purpose how children get their labels, rather than leaving it to chance.

17Inside the Reconciliation Loop

Here we follow what the controller does, step by step, when a single object changes in a parent.

A reconciler is a piece of code that compares what should exist with what does exist and fixes the difference. The controller has several reconcilers. One looks after hierarchy objects and keeps the forest up to date. Another looks after anchors. Others look after each propagated type. They run whenever the API server tells them that something has changed.

sequenceDiagram
  participant U as User
  participant API as API server
  participant W as Webhook
  participant R as Reconciler
  participant F as Forest
  U->>API: Change a RoleBinding in the parent
  API->>W: Validate the change
  W->>F: Check hierarchy rules
  F-->>W: Allowed
  W-->>API: Accept
  API->>R: Notify of the change
  R->>F: Find all descendants
  F-->>R: List of children
  R->>API: Create or update copies in each child
        

Figure 6: A change in a parent reaches every descendant through a webhook check and then a reconciling step.

The steps in words

1

The event arrives

The API server notifies the controller that an object of a watched type changed.

2

The forest answers a question

The reconciler asks the in-memory forest which namespaces sit below the changed object’s namespace.

3

Each child is compared

For every descendant, the reconciler compares the existing copy with what the original now says.

4

Differences are written

Missing copies are created, stale ones updated, unwanted ones deleted.

5

Status is reported

If something went wrong, a condition is recorded so people can see it.

Simple Analogy

A school janitor walks the corridors with a checklist. Each time the head teacher posts a new notice, the janitor checks every classroom: is the notice there, is it the latest version, should an old one be taken down? Reconciliation is that walk, repeated whenever something changes.

Why the loop is forgiving

Because the reconciler compares the whole desired state to the whole actual state, it does not matter if it missed an event or crashed in the middle. After a restart it simply walks the corridors again. This idea, called level-based reconciliation, is what makes Kubernetes controllers resilient. It also means the controller can be slow for large trees, which we will discuss later.

18Conflicts, Cycles and Conditions

When something cannot be done safely, the controller refuses and tells you why, using conditions on the namespace.

A hierarchy can go wrong in several ways. Two people can try to give a namespace different parents at the same time. Someone can set a parent that does not exist yet. Someone can create a loop where A is a child of B and B is a child of A. Someone can try to copy an object into a child that already has a different object of the same name. The controller meets each of these with a clear rule.

ProblemWhat the controller doesHow you see it
Cycle in the treeWebhook rejects the change, or all activity in the loop is haltedA condition on the affected namespaces
Missing parentTreats the namespace as if it had no ancestors until the parent appearsA condition noting the parent is missing
Name conflict on copyDoes not overwrite the existing object in the childA condition describing the conflicting object
Ancestor in troubleStops propagating through the troubled namespaceA condition on descendants that mentions the ancestor
Forbidden parent changeWebhook denies the requestAn error message to the user
Simple Analogy

A railway signal turns red when two trains could meet on the same track. The signal does not fix the trains. It stops traffic and shows everyone that something needs attention. Conditions are the red signal of the hierarchy.

Halting is a safe choice

When a namespace is in trouble, the controller stops copying objects into and out of it. This is called halting. It may feel annoying, because teams stop receiving updates. It is much safer than guessing, because a wrong guess might give access to the wrong people. Once the problem is fixed, the condition clears and normal flow resumes.

i
Good Practice

Build an alert on namespaces that carry a problem condition for more than a few minutes. A condition that nobody sees is the same as a silent failure.

!
Common Mistake

Fixing the symptom, for example by deleting the conflicting object in the child, without asking why it existed. Sometimes the child owner created it for a reason, and the right answer is to rename the object in the parent.

19Deleting Namespaces and Cascading Deletion

Deleting a branch of the tree can remove a lot of work. The controller therefore adds guard rails.

In core Kubernetes, deleting a namespace deletes everything inside it. In a hierarchy, a parent may have many children, each full of applications. A careless deletion of the parent could, if cascading were automatic, remove the whole branch in seconds. The controller treats this very carefully.

Subnamespaces and their anchors

A subnamespace is owned by its anchor in the parent. If you delete the anchor, the controller deletes the subnamespace, but only when the subnamespace has no subnamespaces of its own, or when cascading deletion is allowed. This is a deliberate speed bump. It makes you stop and think when the deletion would affect several levels.

The cascading permission

The HierarchyConfiguration of a namespace has a setting, allowCascadingDeletion, which defaults to off. When it is off for a parent, the controller refuses to delete the parent’s subnamespace children by removing their anchors in a chain. When it is on, deletion may sweep downward through the whole subtree. The setting may be placed on a namespace or on its ancestors, and it lets the owner decide how much freedom to give.

Simple Analogy

A building has a master switch that turns off the power to every floor. The switch has a plastic cover so nobody flips it by accident. Cascading deletion is the cover. You can lift it on purpose, but it should not be opened casually.

ActionCascading offCascading on
Delete a leaf subnamespace anchorAllowedAllowed
Delete an anchor whose subnamespace has childrenRefusedAllowed, whole branch removed
Delete a full namespace that has childrenChildren are not deleted; they lose their parent linkDepends on the rules for that kind of child
!
Common Mistake

Turning on cascading deletion everywhere to make clean-up scripts work. Doing so removes the safety cover for every person on the team. Turn it on only on the namespaces and in the moments where clean-up truly needs it.

Remember too that deleting copied objects does nothing useful. The controller recreates them. To remove an inherited object you must delete the original in its owning namespace or change the settings that make it travel.

20Delegation: Who May Do What

The point of subnamespaces is to let teams act on their own. This chapter shows the exact line between the platform team and the tenants.

Delegation means handing part of a job to someone else together with the right to do it. In this system, the platform team prepares a team’s top namespace and gives the team permission to create anchors inside it. After that, the team can create, name and use subnamespaces without asking anyone, and the platform team does not need cluster-wide privileges for the work.

Three layers of permission

Layer 1

Cluster administrators

Install the controller, set HNCConfiguration and create top-level namespaces.

Layer 2

Team leads

Hold rights in their team namespace, including creating anchors, so they can create subnamespaces.

Layer 3

Team members

Work inside the subnamespaces with the rights that the hierarchy gives them.

Simple Analogy

A school principal gives each head of department a budget and the right to hire helpers inside that budget. The principal does not approve each pencil. The heads work freely inside their space, but cannot touch another department’s cupboard.

Where the guard rails are

The webhook checks that a person who sets a parent has the right to do so on both sides. Moving a namespace under a new parent needs authority over both the namespace and the new parent. This stops a person from attaching their namespace beneath someone else’s tree to inherit that tree’s permissions. Without this check, hierarchy would become a way to steal access.

i
Good Practice

Write down the three layers for your organization and review them every quarter. When the line between layers is clear, people stop asking for cluster-wide rights they do not need.

21Hierarchy, Quotas and Limits

People often expect a parent to share a budget of CPU and memory with its children. The truth is more careful.

A ResourceQuota in Kubernetes caps how much a single namespace may use: total CPU, memory, storage or number of objects. In a hierarchy the natural wish is to place one quota on a department and let the children share it. The project explored this idea in an early experimental version, but it is not part of the current stable features. Check the documentation of your release before planning around it.

What you can do today

You can propagate a ResourceQuota object downward, but each copy is its own separate limit. If a department has ten children and the quota says ten CPUs, the department can actually use a hundred CPUs. That is rarely what anyone wants. A better approach is to place a quota on each child by a different mechanism, such as an automation tool or policy engine that sets the right numbers at creation time, or a more advanced multi-tenant system that manages shared budgets.

Simple Analogy

A family gives each child a sweet jar with ten sweets. If the rule is copied to five children, the family hands out fifty sweets. If the family really wanted fifty sweets in total, they should have counted once, in the pantry, and handed out shares.

ApproachEffectTrade-off
Propagate a quota objectEach child gets an equal separate limitTotal can be much more than intended
Create quotas by automationEach child gets a planned shareExtra tooling to run
Use a platform with shared quotaOne budget for the whole subtreeAnother component with its own lifecycle
Use cost labels and reportsNo hard limit, only visibilityOveruse is found after the fact
!
Common Mistake

Assuming that a quota on a parent namespace limits its children. A quota counts only what runs inside the namespace where it sits.

22What the Controller Does Not Do

Knowing the limits is as useful as knowing the features. These are the common misunderstandings.

The controller is a small, focused tool. It adds a tree to namespaces and copies chosen objects. It is not a complete multi-tenancy platform. Many wrong plans come from expecting it to do things that belong to other tools.

Not

Not isolation

It does not separate workloads. Network rules, node pools and Pod security remain your job.

Not

Not a separate control plane

All namespaces share one API server. A cluster-wide problem affects every tenant.

Not

Not upward sharing

Nothing flows from child to parent. The flow is downward only.

Not

Not a cluster-wide object copier

Only types you list in the configuration travel, and only through the tree.

Not

Not a name system

Namespace names still must be unique across the cluster, whatever their position in the tree.

Not

Not a billing tool

It can help with labels, but it does not measure or report cost.

Simple Analogy

A hallway map on the school wall shows how rooms relate to one another. It does not lock the doors, count the pupils or pay the electricity bill. The map is very useful, and other systems still do the rest.

Hierarchy tells the cluster who belongs to whom. It does not decide what each group is allowed to do to the others.

This is why serious multi-tenant clusters combine the controller with network policies, Pod security settings, admission policies, quotas and sometimes dedicated nodes. The controller removes the repetition from these tools. It does not replace them.

23Worked Example: A Platform Team Serving Many Product Teams

Let us follow a made-up company called Northwind, which has one shared cluster and four product teams.

Northwind’s platform team runs the cluster. Four product teams, called checkout, search, payments and profile, each need several namespaces: a development one, a testing one and a production one. Before the controller, the platform team created twelve namespaces by hand, then copied the same access rules into each, then answered a steady stream of requests such as “please add a namespace for my experiment”.

The new design

The platform team creates one full namespace for each team, such as team-checkout. They bind the team’s group to a Role in that namespace that allows creating anchors and managing ordinary workloads. Then they step back. The checkout team creates subnamespaces called checkout-dev, checkout-test and checkout-prod by themselves. The access rules flow down at once. Network rules flow down too, because the platform team listed NetworkPolicy in HNCConfiguration.

flowchart TD
  NW["northwind-root (platform team)"] --> TC["team-checkout"]
  NW --> TS["team-search"]
  NW --> TP["team-payments"]
  NW --> TR["team-profile"]
  TC --> CD["checkout-dev"]
  TC --> CT["checkout-test"]
  TC --> CP["checkout-prod"]
  TP --> PD["payments-dev"]
  TP --> PP["payments-prod"]
        

Figure 7: Northwind’s tree. The platform team owns the top. Each team owns its own branch.

What changed for people

1

Requests dropped

Teams no longer wait for a namespace. They create it in seconds with their own rights.

2

Consistency improved

Every namespace receives the same baseline rules because they are inherited, not retyped.

3

Audits became shorter

Reviewers read one namespace per team plus the root, rather than twelve or more.

4

The platform team focused on the platform

Time once spent on tickets went to better monitoring and safer upgrades.

Simple Analogy

A landlord who once painted every tenant’s room now hands out a rulebook for the building and lets each tenant arrange their own flat. The landlord’s visits drop, and the house rules stay the same for everyone.

i
Good Practice

Start with one pilot team. Learn from their first month before you switch the other teams over.

24Worked Example: Sharing a Registry Credential and a Default-Deny Network Rule

Two chores that every new namespace needs, solved once at the top of the tree.

Every new namespace at Northwind needs two things before an application can run properly. First, a credential for the private image registry, because Pods cannot pull company images without it. Second, a default-deny network rule, which blocks all traffic by default so teams must add explicit allowances. Without hierarchy, both items were forgotten about as often as they were remembered.

How the platform team handles the credential

The platform team places one credential object in a platform namespace. They mark the type as one that propagates in HNCConfiguration, and they add a note so that the credential reaches only the product team branches, not other special namespaces. Each subnamespace receives a copy automatically. When the registry password rotates, the platform team replaces the one original and every copy updates within moments.

How the default-deny rule is handled

The default-deny NetworkPolicy is placed in each team’s top namespace. It propagates to every child. Teams then add their own allow-rules in their own namespaces to open the connections they need. Because the inherited rule cannot be edited by tenants, no team can remove the safety net by mistake.

ChoreWhere the original livesWhat teams doWhat the platform team does on a change
Registry credentialPlatform namespaceNothingReplace one object
Default-deny ruleTeam top namespace or rootAdd allow-rules for their needsEdit one object
Standard Role for developersTeam top namespaceAdd people to a groupEdit the Role once
Simple Analogy

A hotel gives every new guest a key card and a copy of the house rules at check-in, without the guest asking. When the hotel changes the Wi-Fi password, the front desk updates the card printer once.

!
Common Mistake

Treating a propagated secret as private to the platform team. Every copy is readable by whoever has permission to read secrets in that child. Check who holds that permission, and consider a registry credential that has pull access only.

25Working with GitOps and Policy Engines

Many teams keep cluster settings in a version-controlled repository. Inherited copies need a little care in that world.

GitOps means that a repository is the source of truth, and a tool in the cluster keeps the cluster equal to the repository. Such tools often have a pruning feature that deletes anything in the cluster that is not described in the repository. Inherited copies are created by the controller, not by the repository, so a pruning tool may see them as strangers and try to delete them. The controller then recreates them, and the two tools fight in a loop.

How to avoid the fight

Most GitOps tools allow you to exclude objects by label. Because every inherited copy carries the inherited-from label, you can tell the tool to ignore objects that have it. Another safe habit is to keep the originals in the repository and let the controller own all copies. Teams should never place the same object by hand into a child namespace when the parent already sends it.

flowchart LR
  G["Git repository"] --> S["GitOps tool"]
  S --> P["Original objects in parent namespaces"]
  P --> H["Hierarchical namespace controller"]
  H --> C["Inherited copies in children"]
  S -. "ignore objects labeled inherited-from" .-> C
        

Figure 8: The repository owns the originals. The controller owns the copies. The GitOps tool is told to leave the copies alone.

Policy engines

A policy engine, also called an admission controller, checks new objects against rules, for example “every Pod must have a team label”. The controller creates copies by writing to the API server like any other client, so the engine’s rules apply to those copies too. If your rules reject something the controller creates, you will see a condition about a failed copy. Make sure your policies allow what the controller needs, or exempt the controller’s service account where that is safe.

Simple Analogy

A school has a caretaker who tidies anything left in the corridor. A teacher who pins a notice on the corridor wall would see it thrown away unless the notice carries an approved stamp. The label on inherited copies is that stamp for the caretaker tool.

i
Good Practice

Test the whole chain in a small cluster first: repository, GitOps tool, policy engine and the controller together. Surprises show up in minutes there and in hours in production.

26Scale and Performance

The controller does work in proportion to the size of your trees and the number of objects that propagate.

Every change to a propagated object means copies in every descendant. A Role copied into ten children is ten writes. A Secret that changes every hour and travels to five hundred namespaces is five hundred writes every hour. Those writes go through the API server, are stored in etcd, and are watched by many other components. At small scale this is invisible. At large scale it can matter.

The factors that grow cost

Size

Number of namespaces

The forest lives in memory, and every namespace adds a few entries to it.

Fan-out

Children per parent

One change in a wide parent triggers many writes at the same moment.

Churn

Rate of changes

Objects that change often make more copies and more events.

Types

Number of propagated types

Each listed type adds its own watching and reconciling.

Simple Analogy

A teacher who photocopies one page for thirty pupils finishes in a minute. A teacher who must photocopy a new page every ten minutes for three hundred pupils spends the whole day at the copier. Both are the same job at different sizes.

Keeping it healthy

Keep propagated objects small and slow-changing. Avoid very deep trees; two to four levels are common and easier to reason about. Watch the controller’s CPU, memory and queue length, and the API server’s request rates. If you grow to thousands of namespaces, test in a staging cluster with similar numbers before going live, and read the project’s notes on tested limits for your release.

!
Common Mistake

Propagating an object that changes constantly, such as a token that is renewed every few minutes, to a very wide tree. Share the source of the secret in some other way instead.

27Observability: Metrics, Logs, Events and Conditions

A good operator can tell in a minute whether the hierarchy is healthy. Four sources of signals make that possible.

Observability means being able to understand what a system is doing from the outside. For this controller there are four natural places to look. Each answers a different question.

SourceQuestion it answersHow to use it
Conditions on namespacesIs any branch of the tree in trouble?List namespaces and alert on problem conditions
Controller metricsIs the controller busy, slow or failing?Graph reconcile counts, durations and errors
Controller logsWhat happened to this specific object?Search by object and namespace name
Kubernetes eventsWhat was the most recent action?Read events in the affected namespace
The tree view of the pluginWhat does the hierarchy look like right now?Print the tree for a root and compare with the plan
Simple Analogy

A car has a dashboard light for general trouble, a trip computer for numbers, a service history for detail and a mirror for recent events. You need all four to drive with confidence.

What to watch for first

1

Is the controller running?

Check that its Pod is ready and that its webhook answers requests quickly.

2

Are there problem conditions?

Any condition lasting more than a few minutes needs a person.

3

Are reconcile errors rising?

A rising error count often shows permission trouble or a rejected copy.

4

Is latency growing?

Slow propagation hints at too many copies or an overloaded API server.

i
Good Practice

Because the webhook sits in the path of namespace and hierarchy changes, track its latency and its error rate as carefully as you track the API server itself.

28Upgrades, Backups and Disaster Recovery

What do you need to save, and what happens if the controller disappears for an afternoon?

The controller holds no data of its own. The forest in its memory is rebuilt from the API server each time it starts. The real information lives in Kubernetes objects: the hierarchy settings, the anchors, the configuration and the originals. If you back up those objects as part of your normal cluster backup, you can rebuild the whole hierarchy.

What happens during an outage

If the controller is down, existing copies stay in place and keep working, because they are ordinary objects. Applications are not affected. New changes in parents do not reach children, and new anchors are not turned into namespaces until it returns. If the webhook is configured to fail closed, hierarchy changes may be refused during the outage. Choose that setting knowing the trade-off between safety and availability.

Simple Analogy

If the school photocopier breaks, the pages already pinned in classrooms stay up. Only the new pages cannot be delivered until the machine is repaired. Learning goes on.

ItemWhere it livesBack it up?
HierarchyConfiguration objectsIn each namespaceYes
SubnamespaceAnchor objectsIn parent namespacesYes
HNCConfigurationCluster-wideYes
Original propagated objectsIn parent namespacesYes
Inherited copiesIn childrenNot needed; they are rebuilt
The in-memory forestController memoryNo; it is rebuilt at start

Upgrade habits

Read the release notes before each upgrade, especially for changes to the custom resource definitions and to configuration names. Upgrade in a test cluster first. Take a backup. After the upgrade, check that conditions stay clear and that a test change in a parent reaches a test child.

!
Common Mistake

Restoring only some objects after a disaster. If anchors come back before their parents, or originals come back late, you may see temporary conditions and name conflicts. Restore namespaces first, then hierarchy, then originals, and let the controller settle.

29Security Considerations

A tool that copies permissions across namespaces deserves a careful security review.

The controller itself needs broad rights. To copy Roles and Secrets it must be able to read and write them across the cluster. Anyone who takes control of the controller could, in theory, create access in any namespace. Treat its service account, its Pod and its images as high-value assets.

Main risks and answers

Risk

Privilege creep

High-level bindings reach many children. Keep broad rights near the top and few.

Risk

Stolen controller identity

Restrict who can edit its Deployment, limit network access to it and pin its image.

Risk

Secret spread

Every copy of a Secret widens the circle of readers. Share only what is needed.

Risk

Re-parenting attacks

Moving a namespace under a powerful parent could inherit its rights. The webhook demands rights on both sides.

Risk

Webhook bypass

If the webhook is down and set to ignore failures, rules may not be checked. Choose the failure policy with care.

Simple Analogy

The person who cuts the master keys for a building must be trusted more than any tenant. The key-cutting shop, its tools and its records must be guarded better than any single flat.

i
Good Practice

Record who may create anchors and who may set parents. Review the list regularly, and use the audit log of the API server to watch for changes to HierarchyConfiguration objects outside normal work hours.

Remember too that hierarchy is a convenience for permissions, not a wall. Strong isolation needs network policies, Pod security rules and, for the strictest needs, separate clusters.

30Troubleshooting Guide

When something looks wrong, a calm order of questions finds the cause faster than guessing.

Start with the tree, then conditions, then the controller, then the object. This order moves from the widest view to the narrowest and avoids chasing small details before checking the big picture.

SymptomLikely causeFirst check
A child did not receive an objectType not configured, selector excludes it, or a name conflictLook at HNCConfiguration and the object’s annotations, then conditions on the child
Anchor stays in a waiting stateController not running, name already taken, or parent in troubleRead the anchor’s status and the controller logs
Copy keeps reappearing after deleteThe original still exists in an ancestorFind the origin using the inherited-from label
Cannot change a Role in a childIt is an inherited, read-only copyEdit the original in the owning namespace
Cannot set a parentMissing rights on one side, or it would form a cycleRead the webhook’s error message
Changes are slow to arriveLarge fan-out or overloaded API serverCheck controller metrics and API server load
GitOps tool keeps deleting copiesPruning sees copies as strangersExclude objects with the inherited-from label
Simple Analogy

A doctor first looks at the whole patient, then the chart, then the organ. A good engineer treats a cluster the same way: the map, the signals, the machine, the single object.

i
Good Practice

Keep a short runbook with these questions and the exact commands your team uses. Under stress, a checklist beats memory.

!
Common Mistake

Disabling the webhook to get past a refusal. The refusal usually protects something. Read the message, fix the cause, and keep the guard in place.

31Design Patterns and Anti-Patterns

These are patterns that work, and mistakes that appear again and again.

Patterns that work

Pattern

One root per organization

A single top namespace holds baseline rules for everything below it.

Pattern

Team branch with environments

A namespace per team, with dev, test and prod beneath it.

Pattern

Platform services branch

Shared services live under their own branch with a clear owner.

Anti-patterns

AP-1Avoid

Very deep trees. Six levels of nesting make permissions hard to reason about and slow to audit. Keep depth modest and flat where possible.

AP-2Avoid

Cluster-admin bindings in the root. Such a binding reaches every namespace beneath it. Use narrow Roles and bind them lower down.

AP-3Avoid

Propagating everything. Turning on many types makes the tree noisy, increases cost and creates unexpected name clashes. Add types one at a time, each with a reason.

AP-4Avoid

Mirroring the org chart exactly. Companies reorganize often, and namespaces are awkward to move. Model stable things such as products and environments instead of people and managers.

AP-5Avoid

Hand-editing inherited objects. The controller rejects or reverts such edits. Always change the original.

AP-6Avoid

Treating the tree as isolation. Hierarchy shares permissions. It does not stop one workload from calling another. Add network and Pod rules separately.

Simple Analogy

A good filing cabinet has few drawers, clear labels and the same layout every time. A bad one has twelve nested folders and names that change with every new manager.

32Advantages, Disadvantages and Alternatives

No tool is right for every cluster. Here is an honest comparison.

Advantages

  • Self-service namespaces without cluster-wide rights for teams.
  • Permissions and baseline rules defined once and inherited.
  • Tree labels make network and policy rules short and stable.
  • Small and simple: it adds a tree and copies selected objects.
  • Works with ordinary Kubernetes objects and tools.

Disadvantages

  • Namespaces still share one control plane and one cluster.
  • No built-in shared quota for a subtree in current features.
  • Extra moving parts: controller, webhooks and custom resources.
  • Large fan-out can strain the API server.
  • Check the project’s current maintenance status before adopting.

Alternatives to consider

OptionStrengthWeakness
Hierarchical namespace controllerLight and focused on trees and propagationLimited to namespaces in one cluster
Multi-tenancy operators that group namespaces into tenantsTenant concept with quotas and policies built inA different model, not a tree
Virtual clusters per teamStronger separation, own API server per tenantMore resources and more to operate
Separate clusters per teamStrongest isolationHighest cost and operations load
Own automation and policy enginesFits your exact rulesYou build and maintain it
Simple Analogy

Rooms in one house with a shared rulebook, flats in one building with separate locks, or separate houses on separate streets. Each costs more and separates more than the last. Pick the level that matches the risk.

i
Good Practice

Choose by the trust level of your tenants. Teams inside one company that trust each other usually fit trees of namespaces. Unknown outside customers usually need stronger separation.

33A Step-by-Step Adoption Plan

Introduce the controller slowly. Each step is small, safe and easy to undo.

1

Check fit and status

Confirm the project’s current maintenance state, supported Kubernetes versions and the features you need. Decide whether the alternatives suit you better.

2

Draw the tree on paper

Keep it shallow. Name each level by what it is for. Agree on owners.

3

Install in a test cluster

Add the controller and the plugin. Practice creating subnamespaces, setting parents and watching copies appear.

4

Choose the first propagated types

Begin with Roles and Bindings. Add network rules once the team is comfortable.

5

Add monitoring and alerts

Alert on problem conditions, webhook failures and reconcile errors before real teams depend on it.

6

Pilot with one team

Move one willing team into the tree. Ask what is confusing and write it down.

7

Adopt existing namespaces gradually

Set parents on old namespaces one at a time, checking for name conflicts each time.

8

Document and review

Write a short guide for teams and review the tree every quarter.

Simple Analogy

Moving a school to a new filing system is done one cabinet at a time. You do not carry every paper out of every room on the same Monday morning.

!
Common Mistake

Moving existing production namespaces under a parent before checking name conflicts and existing bindings. The first inherited copy may collide with something that production relies on.

34Where Teams Use It in Real Life

The pattern fits several kinds of organizations, each for a slightly different reason.

Internal developer platforms

Platform teams offer each product team a root namespace and let them create their own environments. The team gets speed and the platform team gets consistency.

Large enterprises with many departments

Each department receives a branch. Audit teams read access by branch, and compliance rules sit at the department level.

Training and workshop clusters

Each learner receives a subnamespace under a class namespace with the same baseline rules. At the end, the class anchor is removed and everything is cleaned up together.

CI preview environments

Each pull request creates a short-lived subnamespace under a CI namespace, which inherits the shared credentials and rules, and is deleted when the request closes.

Managed services and internal software vendors

A group that runs databases for others gives each customer team a subnamespace with the right access and network rules already in place.

Simple Analogy

A university gives each department a floor and lets the department arrange its own rooms. The same fire rules hang on every wall, and the building manager does not repaint a door for every student.

Where many small groups need the same rules and a little freedom, a tree of namespaces fits well.

35Naming and Designing Your Tree

A tree is easy to build and hard to rearrange. A little thought about names and levels at the start saves months of cleaning later.

Namespace names must be unique across the whole cluster. Position in the tree does not give a name its own private space. Two teams cannot both have a namespace called dev, even if they sit in different branches. The simplest way out is to put the team or product name inside the namespace name, so that checkout-dev and search-dev can live side by side.

Rules of thumb for names

Name

Start with the owner

Begin with the team or product, then add the purpose, such as checkout-prod. Sorted lists then group related namespaces together.

Name

Keep it stable

Choose words that will outlive reorganizations. Product names change less often than manager names.

Name

Keep it short

Tree labels use the ancestor name as part of a label key, and long names can hit length limits.

Simple Analogy

A school gives every locker a number with the class letter in front of it, such as 5B-17. A pupil never has to guess which hallway holds the locker, and two classes never receive the same number by mistake.

Choosing the levels

Each level in the tree should answer one question. A common three-level design asks: which organization, which team, which environment. A level that does not answer a clear question is probably unnecessary. Before you add a level, ask what rule or permission will be attached to it. If you cannot name one, leave the level out.

LevelQuestion it answersWhat usually lives here
RootWhat rules apply to everyone?Baseline network rules, auditor access, shared credentials
TeamWho owns this branch?Team group bindings, permission to create anchors
EnvironmentWhat is this space used for?The running workloads, stricter rules in production
!
Common Mistake

Putting environment above team, so that all production namespaces sit together. That looks tidy but gives the production owner power over every team’s production workloads, which often does not match who should be in charge.

i
Good Practice

Draw the tree and read it aloud to someone outside the platform team. If they cannot guess what each level is for, the design needs another pass.

36A Day in the Life of a Subnamespace

Let us follow one subnamespace from its birth to its removal and watch what the controller does at each step.

1

Morning: a developer asks for space

A developer in the payments team needs a place to test a new feature. She creates an anchor in the team namespace with the name she wants.

2

Seconds later: the namespace appears

The controller sees the anchor, creates the namespace, sets its parent and records it as a child of the team namespace.

3

Right after: rules arrive

The team’s bindings, the default-deny rule and the registry credential are copied in. The tree labels are added.

4

Midday: work happens

She starts test workloads and adds an allow-rule so they can talk to a test database. Her own objects are hers to change.

5

Afternoon: the platform team changes a rule

The credential is rotated at the top. Within moments her namespace holds the new copy, and she does nothing.

6

Evening: the test is done

She deletes the anchor. The controller removes the namespace and everything inside it, because it has no children of its own.

stateDiagram-v2
  [*] --> Requested: anchor created
  Requested --> Created: namespace made by controller
  Created --> Configured: inherited objects and labels added
  Configured --> InUse: team deploys workloads
  InUse --> InUse: parent changes flow in
  InUse --> Removing: anchor deleted
  Removing --> [*]: namespace removed
        

Figure 9: The life of a subnamespace, from the first request to its removal.

Simple Analogy

A hotel guest checks in, receives a key and the house rules, uses the room, hears about a new Wi-Fi password through a card slid under the door, and then checks out. The room is cleaned and ready for the next guest. Nobody had to ring the manager at any step.

What is worth noticing is how few people were involved. The developer acted on her own. The platform team acted once, at the top, for the credential change. Nobody with cluster-wide rights was needed for any step in her day.

37Testing and Validating Your Setup

A setup that works on the first day can silently break on the hundredth. Regular checks keep it honest.

Testing a tree means checking that the right things travel to the right places, and that the wrong things do not. It is useful to test the positive case, where an object should arrive, and the negative case, where it should not. Negative tests find most of the dangerous mistakes, since they show leaks of access.

A short list of checks

CheckWhat you expectWhy it matters
Create a test object in a parentA labeled copy appears in each descendantProves propagation works end to end
Change the test objectAll copies updateProves that updates flow
Delete the test objectAll copies disappearProves clean-up works
Create the same object in a sibling branchIt does not reach the other branchProves isolation between branches
Try to edit an inherited copy as a normal userThe change is refusedProves read-only protection
Try to create a cycleThe change is refusedProves the safety guard
Test a real network connection between two namespacesAllowed or blocked as the rules sayProves that tree labels match policies
Simple Analogy

A school holds a fire drill every term. The drill is dull, but it shows whether the alarms work and whether the doors open. Testing the tree is a fire drill for permissions.

Make the checks routine

Run the same list after every upgrade of the controller, after every change to HNCConfiguration, and on a regular schedule such as once a week from an automated job. Keep the results somewhere the whole team can read. A test that fails loudly the day after a change is far cheaper than a leak found during an audit.

i
Good Practice

Use a dedicated test branch of the tree with throwaway namespaces. That way tests never touch real team workloads, and anyone can run them without fear.

!
Common Mistake

Only testing that things arrive. A setup can pass every positive test and still leak a Secret to a branch that should never see it. Always include at least one test that proves something does not arrive.

38A Plain-Words Glossary

Keep this page nearby while you read other material about the controller. Every term is explained in everyday language.

TermPlain meaningEveryday picture
NamespaceA named area inside a cluster where objects liveA room in a house
HierarchyParents and children arranged as a treeA family tree
ForestThe set of all trees the controller knows aboutMany family trees in one town
Parent and childA namespace above another, and the one below itGrandmother and grandchild
Ancestor and descendantAny namespace above or below, at any distanceAll relatives up or down the line
SubnamespaceA child created by an anchor in its parentA room a team adds inside its own floor
AnchorAn object in the parent that asks for a subnamespaceA request slip handed to the office
PropagationCopying chosen objects from parent to descendantsA notice photocopied to every classroom
Inherited copyAn object in a child that came from an ancestorA stamped photocopy
ConditionA status message that reports a problem or stateA warning light
WebhookA check that runs before a change is acceptedA guard at the door
ReconcilerCode that compares wanted and actual state and fixes gapsA janitor with a checklist
Cascading deletionRemoving a whole branch with one actionCutting a branch with all its leaves
Simple Analogy

A glossary works like the legend on a map. The symbols mean nothing until someone tells you what they stand for, and then the whole map becomes easy to read.

If any of these words still feels unclear, return to the chapter that introduces it. Each chapter explains one idea with a story, so that the meaning stays in your memory.

39Frequently Asked Questions

Short answers to questions that newcomers ask most often.

Q1What is a hierarchical namespace in one sentence?

It is an ordinary Kubernetes namespace that has been placed in a tree, so that it can inherit selected objects and permissions from the namespaces above it.

Q2Does Kubernetes support this out of the box?

No. Core Kubernetes namespaces are flat. The tree is added by an extra controller that you install in the cluster.

Q3Does permission flow up or down?

Down only. Child namespaces inherit from ancestors. Nothing in a child affects its parents.

Q4What is the difference between a subnamespace and a full namespace?

A subnamespace is created through an anchor in the parent by someone with rights only in the parent. A full namespace is created the normal way by someone with cluster rights, and then attached to a parent. The tree treats both the same once attached.

Q5Can I edit an inherited object in a child?

No. Inherited copies are read-only to ordinary users. Change the original in the ancestor.

Q6Which object types propagate by default?

Roles and RoleBindings. Other types need to be added to the cluster-wide configuration.

Q7Does the tree share resource quotas?

Not in the current stable features. Propagating a quota object gives each child its own separate limit, so totals can add up to more than intended. Plan shared budgets with other tools.

Q8What happens if the controller stops running?

Existing copies stay in place and applications keep running. New changes do not propagate until it returns.

Q9Is a tree of namespaces secure isolation between tenants?

No. It shares permissions and rules but does not stop network traffic or workloads from affecting each other. Use network policies, Pod security rules and, where needed, separate clusters.

Q10Can two namespaces in different branches have the same name?

No. Namespace names are unique across the entire cluster, regardless of position in the tree.

Q11How deep should my tree be?

As shallow as works. Two to four levels are common. Deeper trees are harder to understand and audit.

Q12Is the project still maintained?

Project activity changes over time. Check the repository and release notes for the current state, supported Kubernetes versions and any announced changes before you plan on it.

40Summary and Key Takeaways

A tree of namespaces turns repeated manual work into a rule you write once.

We began with a flat list of namespaces and the headaches it creates: copied permissions, forgotten rules, endless requests. We then saw how the controller adds parents and children, lets teams create subnamespaces with anchors, and copies chosen objects downward while keeping them read-only. We looked at the controller’s inner workings, its conditions and its safety limits, then at real examples, risks and a careful path to adoption.

Key Takeaways
  • Core Kubernetes namespaces are flat. The hierarchical namespace controller adds parent and child relationships on top.
  • Roles and bindings flow down the tree by default. Other object types flow only when you list them in the cluster-wide configuration.
  • Inherited copies are labeled and read-only. Always change the original in the owning ancestor.
  • Subnamespaces let teams create their own namespaces through anchors without cluster-wide rights, and deletion has safety guards.
  • Tree labels let network and policy rules select a whole subtree by one label.
  • The tree shares rules and permissions, but it does not isolate workloads. Combine it with network policies, Pod security rules and quotas.
  • Verify the project’s current maintenance status and supported features before adopting, and roll out slowly with monitoring on conditions and the webhook.