GitOps Principles in Kubernetes: Git as the Single Source of Truth

GitOps Principles in Kubernetes: Git as the Single Source of Truth

A complete, plain-English guide to GitOps: the four core principles, how pull-based reconciliation really works, how to structure repositories, handle secrets, promote releases safely, and avoid the mistakes that make GitOps painful.

Picture a large restaurant kitchen where every dish must taste the same in every branch of the chain. One way to achieve this is to phone each chef and say “add salt now, lower the flame now, plate it now”. Soon nobody remembers who told whom to do what, and every kitchen drifts a little differently. The other way is to keep one master recipe book, locked in the head office, with every change signed and dated. Each branch’s kitchen manager checks the book regularly and adjusts the kitchen to match. GitOps is the second way. The recipe book is a Git repository, the kitchens are Kubernetes clusters, and the kitchen managers are small programs that keep checking and correcting.

In this guide we begin from zero. We will first remember what Kubernetes already does, then see why traditional deployment pipelines struggle, and then build up the GitOps idea piece by piece. By the end you will understand the four principles, the difference between push and pull, repository layouts, secrets, promotion across environments, progressive delivery, security, monitoring, and the traps to avoid.

1What Is GitOps?

GitOps is a way of running systems where the desired state lives in Git and software inside the cluster keeps reality matching it.

GitOps is an operating model for infrastructure and applications. You describe everything you want, such as which applications run, in which versions, with which settings, in plain text files stored in a Git repository. A software agent running next to your cluster reads those files and makes the cluster match them. If someone changes the cluster by hand, the agent notices and puts it back. If you want to change the cluster, you do not touch it directly; you change the files in Git, and the agent does the rest.

The name joins two words: Git, the version control tool that records every change to a set of files, and Ops, short for operations, the daily work of running systems. The term was popularized in 2017 by engineers at Weaveworks, who used it to describe how they ran their own Kubernetes platform. Since then it has grown into a community standard, with a shared set of principles maintained by the OpenGitOps working group under the Cloud Native Computing Foundation.

Simple Analogy

Think of a school timetable pinned to the notice board. Teachers do not decide on their own which room to go to. They read the timetable and follow it. If a teacher turns up in the wrong room, the head teacher gently sends them back. To change who teaches what, you edit the master timetable, with a record of who changed it and why, rather than whispering to individual teachers.

2017
Year the GitOps term was popularized
4
Core principles in the OpenGitOps standard
2
Leading GitOps tools, Argo CD and Flux, both graduated CNCF projects

It is worth noticing that GitOps is a way of working, not a product you buy. You can practice it with different tools, in different clouds, on different kinds of clusters. What makes something GitOps is not the tool but the rules it follows, and those rules are the subject of this guide.

2Why GitOps Exists: The Problems It Solves

Before GitOps, most teams shipped to Kubernetes with scripts and pipelines that had hidden weaknesses.

In a traditional setup, a continuous integration server builds your application, and then, as the last step, it runs commands to deploy to the cluster. This looks simple, but it creates several quiet problems. The pipeline needs powerful credentials to the cluster, so those credentials are stored in a build system that many people can reach. The pipeline only acts when it runs; between runs, anyone can change the cluster by hand and nobody will know. And the real state of the cluster is a mystery that must be discovered by asking the cluster, because no document reliably says what should be there.

i
Key Idea

The central problem is trust. Without GitOps, you cannot answer two basic questions with certainty: “What is supposed to be running?” and “Is that what is actually running?” GitOps gives a firm answer to both.

Problem 1

Configuration drift

Manual fixes at midnight slowly make clusters different from each other and from what anyone documented.

Problem 2

No reliable history

When something breaks, nobody can say exactly who changed what, when, and for what reason.

Problem 3

Risky credentials

Build servers hold cluster admin keys, making them a prime target for attackers.

Problem 4

Slow, scary recovery

Rebuilding a lost cluster means re-running old scripts, and hoping they still work.

Problem 5

Inconsistent environments

Testing and production slowly diverge, so a change that worked in testing behaves differently in production.

Problem 6

Approval gaps

Changes made directly to a cluster skip reviews, so quality and compliance depend on individual discipline.

3Foundations You Need First

A short refresher on the ideas GitOps builds on.

Term 1

Git Repository

A folder of files with a complete recorded history of every change, who made it, and when.

Term 2

Commit

A saved snapshot of a change, with an author, a timestamp, and a message explaining why.

Term 3

Pull Request

A proposal to merge a change, where teammates can review, comment, and approve before it takes effect.

Term 4

Manifest

A text file that describes a Kubernetes object, such as a Deployment or a Service, in a structured way.

Term 5

Controller

A program that compares desired state with actual state and acts to close the gap, over and over.

Term 6

Reconciliation

The act of comparing what you want with what exists and correcting any difference.

Simple Analogy

A Git repository is a shared notebook that can never lose a page. Each commit is a dated entry. A pull request is a draft entry that a colleague must read and approve before it is written into the notebook. Reconciliation is the act of looking at the notebook, looking at the real world, and tidying the world until they agree.

Kubernetes Is Already Declarative

Kubernetes is built around declaring a desired state. You tell it “I want three copies of this application” and its controllers work to make that true. GitOps takes that same idea one step further: it asks where the declarations themselves should live and who should be allowed to change them. The answer is Git, with reviews and history, and an in-cluster agent that applies them. In a sense, GitOps is the natural partner of Kubernetes, because both are based on the same love of declared intent and continuous correction.

It also helps to separate two kinds of software. Continuous integration is about building and testing code and producing artifacts such as container images. Continuous delivery is about getting those artifacts safely into running environments. GitOps is a particular, disciplined way of doing the delivery half. Your build server still builds; it simply stops reaching into the cluster.

4The Four GitOps Principles

The OpenGitOps standard describes four rules. A system is GitOps only if it follows all four.

1

Declarative

The whole system is described by stating the desired end state, not by listing the steps to reach it.

2

Versioned and Immutable

The desired state is stored in a way that keeps a complete, unalterable history and supports easy rollback.

3

Pulled Automatically

Software agents automatically fetch the desired state from the source; nobody pushes changes into the cluster from outside.

4

Continuously Reconciled

Agents constantly observe the real system and try to apply the desired state, correcting any drift.

flowchart LR
    A[1 Declarative description] --> B[2 Stored in Git with history]
    B --> C[3 Agent pulls automatically]
    C --> D[4 Agent reconciles continuously]
    D -->|drift detected| C
        
FIG 1 — The four principles working together as a cycle

Each principle supports the others. Declarative files are what make versioning meaningful, because you can compare two versions and see exactly what changed. Versioning is what makes pull safe, because the agent always fetches a known, reviewed state. Pull is what makes continuous reconciliation possible, because the agent lives inside the environment and can check as often as needed. And reconciliation is what makes the declarations truthful, because the live system is forced to match them.

“If it is not in Git, it does not exist. If it is in Git, it will exist.”

5Principle One: Declarative Desired State

Say what you want, not how to get there.

A declarative description states the end result. For example, “this namespace contains an application at version two with three replicas and a public address”. An imperative description lists commands: “create a namespace, then start a container, then scale it, then expose it”. The imperative way depends on the starting point. If you run the same commands twice, or on a cluster that already has some of the pieces, you may get errors or a different result. The declarative way does not care about the starting point, because the agent simply works out the difference between what exists and what you described.

Simple Analogy

Giving a taxi driver an address is declarative. Giving turn-by-turn directions is imperative. If a road is closed, the driver with an address finds another way. The driver with turn-by-turn directions gets lost the moment anything unexpected happens.

What Counts as Desired State

In Kubernetes, desired state is usually a set of manifests, but it can take other forms that eventually produce manifests. Teams use templating tools such as Helm charts, or overlay tools such as Kustomize, to avoid repeating the same text for different environments. What matters is that the final result is fully determined by files in the repository. If rendering a template pulls in values from a random web service at deploy time, the declaration is no longer fully in Git, and the principle is weakened.

!
Common Mistake

Mixing in imperative steps, such as “run this script after deploying to fix the database”. Anything that must happen for the system to work should itself be expressed declaratively, or handled by a controller that watches for it.

Everything as Code

The same idea extends beyond application manifests. Namespaces, access rules, network policies, quotas, and even the definitions of the clusters themselves can be written as declarations. Teams that go all the way, sometimes called “everything as code”, gain the ability to recreate an entire platform from scratch simply by pointing a fresh agent at the repository.

6Principle Two: Versioned and Immutable

Every change is recorded, reviewable, and reversible.

Git stores a permanent chain of commits. Each commit is identified by a unique fingerprint derived from its contents, which means history cannot be quietly rewritten without everyone noticing. This is the immutable part: once a state is recorded, it stays recorded. The versioned part means you can go back to any earlier state at any time. Together they give you an audit trail and a time machine in one.

Benefit 1

Audit trail

For every change, you know who made it, who approved it, when it happened, and the reason in the message.

Benefit 2

Easy rollback

Reverting a bad release is a matter of reverting a commit, and the agent restores the previous state automatically.

Benefit 3

Peer review

Every change goes through a pull request, so mistakes are caught before they reach a cluster.

Benefit 4

Compliance evidence

Auditors can read the history to see that approvals and separation of duties were followed.

Immutable Artifacts, Too

Immutability applies to what the manifests point at, not just the manifests. If a manifest says “use the image tagged latest”, then the same commit can mean different things on different days, because the tag can be moved to a different image. To keep the history honest, manifests should reference images by a fixed version tag or, even better, by the image’s content fingerprint called a digest. Then a given commit always means exactly one thing.

i
Practical Advice

Protect your main branch. Require reviews, require passing checks, forbid force-pushes, and sign commits where possible. In GitOps, the main branch is effectively the control panel for production, so it deserves the strongest protection you have.

7Principle Three: Pulled Automatically

The cluster reaches out to Git. Nobody reaches into the cluster.

In a push model, an external system, such as a build server, connects to the cluster and applies changes. In a pull model, an agent running inside or beside the cluster connects outward to the repository and fetches the desired state. This small change in direction has large effects, especially for security.

flowchart TD
    subgraph PUSH [Push model]
        P1[Developer] --> P2[CI server holds cluster credentials]
        P2 -->|pushes changes in| P3[Cluster]
    end
    subgraph PULL [Pull model]
        L1[Developer] --> L2[Git repository]
        L4[Agent inside cluster] -->|pulls from| L2
        L4 -->|applies to| L3[Cluster]
    end
        
FIG 2 — Push model versus pull model
AspectPush ModelPull Model
Where credentials liveIn the external build system, which can reach the clusterInside the cluster; only read access to Git is needed from outside
Network exposureCluster API must be reachable from the pipelineCluster only makes outbound connections
Drift handlingOnly corrected when the pipeline runs againCorrected continuously by the agent
Multiple clustersPipeline must hold keys for each clusterEach cluster runs its own agent
Source of truthPipeline plus whatever is in the clusterGit repository
Simple Analogy

Push is like giving a delivery person the key to your house so they can come in whenever they want. Pull is like you walking to the post office to collect your parcel. In the second case, nobody outside ever needs a key to your house.

The pull model is also friendlier to firewalls and private networks. A cluster in a locked-down environment can still make outbound connections to a repository, while it may be very difficult to allow inbound connections from a build server. For organizations running dozens of clusters in different places, the pull model scales naturally because each cluster looks after itself.

!
Common Misconception

Running a deployment command from a pipeline step that reads a Git repository is not GitOps, even though Git is involved. If the pipeline pushes the change into the cluster, it is the push model, and the continuous reconciliation benefits are lost.

8Principle Four: Continuously Reconciled

The agent never stops checking, so drift cannot hide.

Reconciliation is the loop at the heart of GitOps. The agent observes what is running in the cluster, compares it with the desired state from Git, and, if they differ, takes action to bring the cluster back to the declared state. Then it does this again, forever. The loop is the same idea that powers Kubernetes controllers, applied at the level of the whole environment.

sequenceDiagram
    participant G as Git Repository
    participant A as GitOps Agent
    participant K as Kubernetes API
    loop Every few minutes or on notification
        A->>G: Fetch latest desired state
        A->>K: Read live state
        A->>A: Compare desired and live
        alt Difference found
            A->>K: Apply changes to match Git
            A->>A: Record result and health
        else Same
            A->>A: Mark as in sync
        end
    end
        
FIG 3 — The continuous reconciliation loop of a GitOps agent

Drift: The Silent Enemy

Drift means the live system no longer matches the declared state. It happens when someone edits a resource by hand, when another automated tool changes a field, or when a resource is deleted by accident. Without reconciliation, drift accumulates unseen until it causes an outage or a failed audit. With reconciliation, drift is either undone automatically or at least reported clearly as “out of sync”.

Automatic Healing or Alert Only

Teams can choose how aggressively the agent reacts. In a fully automatic mode, the agent corrects any drift immediately, which is ideal for production consistency. In a cautious mode, the agent only reports differences and waits for a human to decide, which can be useful while a team is learning. Many teams start cautious in a few environments and move to automatic healing as confidence grows.

i
Emergency Changes

If an engineer must change something by hand during an incident, the agent may undo the fix a moment later. A good practice is to make the same fix in Git right away, or to temporarily pause synchronization for that one application, and to resume it once Git is updated.

“Reconciliation turns a repository from a document into a promise.”

9Architecture and Components of a GitOps System

A GitOps setup has a handful of cooperating parts.

flowchart TD
    DEV[Developer] -->|opens pull request| APPREPO[Application source repository]
    APPREPO --> CI[CI pipeline builds and tests]
    CI -->|pushes image| REG[(Container registry)]
    CI -->|updates version in| CFG[Configuration repository]
    CFG --> AGENT[GitOps agent in cluster]
    REG --> AGENT
    AGENT -->|applies| CLUSTER[Kubernetes cluster]
    AGENT -->|reports status| DASH[Dashboard and alerts]
        
FIG 4 — Typical GitOps architecture separating build from deployment
1

Application source repository

Holds the code of the application. Changes here trigger builds and tests, not deployments.

2

Continuous integration pipeline

Builds the application, runs tests, scans for problems, and publishes a versioned container image.

3

Container registry

Stores the immutable images that the manifests refer to.

4

Configuration repository

Holds the desired state of the environments: manifests, overlays, and version references. This is the true source of truth for what runs.

5

GitOps agent

Runs in the cluster, pulls from the configuration repository, renders manifests, and reconciles the cluster.

6

Observability layer

Dashboards, notifications, and alerts that show sync status, health, and failures.

Simple Analogy

Think of a publishing house. Authors write manuscripts (application repository). The printing press produces finished books (CI pipeline and registry). The catalogue lists which edition each bookshop should stock (configuration repository). Each bookshop’s manager (the agent) checks the catalogue and makes sure the shelves match it.

10Tools: Argo CD and Flux

Two mature, community-driven tools dominate GitOps on Kubernetes.

Argo CD and Flux are both graduated projects of the Cloud Native Computing Foundation. Both implement the four principles. Both run inside the cluster, pull from Git or from packaged artifact stores, render manifests, and reconcile continuously. They differ in style and user experience.

AspectArgo CDFlux
Main styleApplication-centric, with a rich web interfaceToolkit of small controllers, strongly command-line and API oriented
User interfaceBuilt-in visual dashboard showing live resource trees and diffsPrimarily through the command line and external dashboards
Multi-clusterOne control plane can deploy to many clustersOften one agent set per cluster, coordinated through Git
Templating supportPlain manifests, Kustomize, Helm, and other pluginsKustomize and Helm through dedicated controllers
ExtensionsApplication sets, sync waves, hooks, notificationsImage automation, notifications, dependency ordering
Good fitTeams wanting visibility and a central viewTeams wanting a lightweight, composable, Kubernetes-native toolkit
i
Choosing Wisely

Neither tool is wrong. Pick the one whose workflow your team finds natural, then invest in good repository design, which matters far more than the choice of agent.

Supporting Tools

Rendering

Kustomize and Helm

Produce environment-specific manifests from shared bases or packaged charts, keeping repetition low.

Policy

Policy engines

Tools that check manifests against company rules, either before merge or at admission to the cluster.

Delivery

Progressive delivery controllers

Roll out new versions gradually using canary or blue-green strategies with automatic analysis and rollback.

Secrets

Secret management tools

Encrypt secrets for storage in Git or sync them from external vaults, so plain-text secrets never sit in a repository.

11Repository Design and Environment Strategy

How you organize Git decides how pleasant GitOps will be for the next five years.

Separate Code from Configuration

A widely recommended practice is to keep application source code and deployment configuration in different repositories. The reasons are practical. Code changes often, in small steps, and many developers touch it. Configuration changes represent deliberate decisions about what runs where, and they have a different audience and approval needs. Separating them also stops a build from triggering itself in a loop: if the pipeline commits a new version number into the same repository that triggers the pipeline, you can create an endless cycle.

One Repository or Many

Single Shared Config Repository

  • One place to look for everything that runs.
  • Easy to make changes that span several applications.
  • Simple global policies and reviews.
  • Straightforward to bootstrap a new cluster.

Many Team Repositories

  • Teams own their configuration and move independently.
  • Access control can be tight per team.
  • Smaller blast radius for mistakes.
  • Harder to get a global overview without extra tooling.

Environments: Folders Beat Branches

Teams often ask whether to represent environments such as development, staging, and production as branches or as folders. Folders are generally preferred. With branches, the environments drift apart in confusing ways, merges become complicated, and it is hard to see differences at a glance. With folders on one main branch, each environment has its own directory, built from a shared base plus small overrides. Differences are easy to see, and promotion is as simple as changing a version number in the next folder.

Layer 1

Base

The common definition of an application, shared by every environment.

Layer 2

Overlays

Small environment-specific changes such as replica counts, resource limits, and addresses.

Layer 3

Cluster definitions

Which applications and which overlays belong on which cluster.

!
Common Mistake

Copying whole folders for each environment and editing them freely. Within weeks nobody knows which differences are intentional. Use a shared base and keep overrides as small as possible.

12Data Flow and Lifecycle of a Change

Let us follow one change, from a developer’s idea to a running update.

1

Code change merged

A developer’s pull request is reviewed and merged in the application repository.

2

Build and publish

The pipeline builds the application, tests it, scans it, and publishes a uniquely versioned image.

3

Configuration update proposed

A pull request in the configuration repository changes the image version for the first environment. This may be created by automation or by a person.

4

Review and checks

Automated checks validate the manifests and policies, and reviewers approve. Merging records the decision in history.

5

Agent detects and applies

The in-cluster agent notices the new commit, renders the manifests, and applies the difference to the cluster.

6

Health verification

The agent watches rollout health. Status shows synced and healthy, or degraded if something is wrong.

7

Promotion

After the change proves itself, another pull request promotes the same version to the next environment.

stateDiagram-v2
    [*] --> OutOfSync: New commit in Git
    OutOfSync --> Syncing: Agent starts applying
    Syncing --> Synced: Cluster matches Git
    Synced --> Healthy: Workloads pass health checks
    Syncing --> Failed: Apply error
    Failed --> OutOfSync: Fix committed
    Healthy --> OutOfSync: Drift or new commit
        
FIG 5 — States an application moves through under GitOps

Notice that synced and healthy are two different ideas. Synced means the cluster matches Git. Healthy means the things that were applied are actually working. An application can be perfectly synced to a broken configuration, which is why health checks and progressive delivery are important companions to GitOps.

13Promotion Across Environments

Moving a release from testing to production should be boring, visible, and repeatable.

In GitOps, promotion means changing the desired state of the next environment so that it matches the version already proven in the previous one. Because both are just files in Git, promotion becomes a small, reviewable change. The same artifact that passed testing is the one that reaches production; nothing is rebuilt along the way. This guarantees that what you tested is what you ship.

Style 1

Manual promotion

A person opens a pull request to bump the version in the next environment. Simple and fully controlled.

Style 2

Automated promotion

Automation raises the pull request once tests pass, and humans only approve. Faster with the same safety.

Style 3

Image update automation

An agent watches the registry and commits new versions into the lower environment automatically.

Style 4

Gated promotion

Promotion waits on checks, such as passing tests, quiet monitoring, or an approval window.

i
Practical Advice

Build once, promote many times. Never rebuild the image for production. Rebuilding can pick up different dependencies and silently change what you are shipping.

Rolling Back

Rollback in GitOps is delightfully simple: revert the commit that introduced the problem. The agent sees the new desired state and restores the earlier version. Because the revert is itself a commit, the history clearly shows that a problem occurred and was undone, which is valuable for later review. Remember that rollback only restores the declared state; it cannot undo irreversible changes such as data migrations, so those need separate care.

14Secrets Management in GitOps

Git must never contain plain-text passwords, yet the cluster needs them. Here is how teams reconcile the two.

Because Git is the source of truth and is widely readable, storing secrets in it as plain text would be a serious security failure. Even a private repository leaks through copies, backups, and former employees. GitOps teams use one of two broad approaches to solve this.

Encrypted Secrets in Git

  • Secrets are encrypted before being committed, and only the cluster or authorized people can decrypt them.
  • The repository remains the complete source of truth.
  • Changes to secrets go through the same review and history.
  • Key management becomes an important responsibility.

References to External Vaults

  • Git stores only a reference, such as “the database password from the vault”.
  • A controller in the cluster fetches the real value and creates the Secret.
  • Central rotation and detailed audit in the vault.
  • Adds a dependency on the external system being available.
Simple Analogy

Encrypted secrets are like sending a locked box through the mail: anyone can see the box, but only the receiver has the key. References to a vault are like leaving a note that says “the key is in the bank safe, box number seven”: the note is harmless without the bank’s cooperation.

!
Danger Zone

Encoding a secret in a text format such as base64 is not encryption. Anyone can reverse it in a second. Also remember that a secret committed once remains in Git history even after deletion, so rotate it immediately if it was ever exposed.

15Progressive Delivery and Policy as Code

GitOps decides what should run. These companions help decide how safely it gets there.

Progressive Delivery

Progressive delivery is the practice of releasing a new version to a small slice of traffic first, measuring how it behaves, and only then widening the rollout. In a canary release, a small percentage of users see the new version while the rest stay on the old one. In a blue-green release, a full copy of the new version runs beside the old one, and traffic is switched over once it looks healthy. Specialized controllers can automate the analysis, using metrics such as error rate and latency, and roll back automatically if results look bad. GitOps provides the desired state, and the progressive delivery controller handles the careful transition.

Simple Analogy

A canary release is like a restaurant testing a new dish on one table before adding it to the whole menu. If the first table sends it back, the kitchen changes course without upsetting the entire dining room.

Policy as Code

Policy as code means writing your organization’s rules as machine-checkable definitions. Examples include “no container may run as the root user”, “every workload must declare resource limits”, and “images must come from the approved registry”. These rules can be checked at two moments. Before merge, a pipeline validates the proposed manifests and blocks the pull request if it violates policy. At admission, the cluster itself rejects any object that breaks policy, even if it somehow got past the first check. Using both gives defence in depth.

Check 1

Schema validation

Confirms manifests are well-formed and match the Kubernetes API before they are merged.

Check 2

Security policy

Blocks privileged containers, host access, and missing security settings.

Check 3

Cost and quota policy

Requires resource requests and limits and prevents oversized workloads.

Check 4

Image policy

Allows only signed, scanned images from approved registries.

16Multi-Cluster and Platform Scale

GitOps shines brightest when you have many clusters to keep consistent.

Organizations rarely stop at one cluster. They have clusters per environment, per region, per business unit, or per customer. Managing these by hand is a recipe for drift. With GitOps, each cluster runs an agent, or a central agent manages several, and every cluster is simply a list of what it should contain. Adding a new cluster means pointing a fresh agent at the repository; within minutes, it installs everything it should have.

flowchart TD
    REPO[Central configuration repository] --> A1[Agent in cluster Dev]
    REPO --> A2[Agent in cluster Staging]
    REPO --> A3[Agent in cluster Prod Region 1]
    REPO --> A4[Agent in cluster Prod Region 2]
    A1 --> C1[Dev cluster]
    A2 --> C2[Staging cluster]
    A3 --> C3[Prod Region 1 cluster]
    A4 --> C4[Prod Region 2 cluster]
        
FIG 6 — One repository driving many clusters through independent agents

Fleet Patterns

Cluster bootstrap

A new cluster starts with a single bootstrap step that installs the agent. The agent then pulls a root definition that installs everything else: networking, monitoring, policies, and applications.

App of apps and generators

A parent definition lists child applications, or a generator creates one application per cluster or per team from a template. This avoids writing hundreds of near-identical definitions.

Staged fleet rollouts

Changes are applied first to a canary cluster, then to a small group, then to the whole fleet, with checks between stages.

Disaster recovery

If a cluster is lost, a replacement is created and pointed at the same repository. Stateless workloads return automatically, and only data needs to be restored from backups.

17Security Considerations

GitOps improves security in many ways, but it also moves the crown jewels into Git. Protect them accordingly.

With GitOps, access to the main branch of the configuration repository is effectively access to production. That makes the repository and its approval rules a primary security boundary. At the same time, the pull model improves the security posture by removing cluster credentials from external systems and by shrinking the number of people who need direct cluster access to almost none.

Control 1

Branch protection

Require reviews, status checks, and signed commits, and forbid direct pushes to main.

Control 2

Least-privilege agents

Limit what each agent may create. An agent for one team should not be able to change cluster-wide settings.

Control 3

Read-only repository access

Agents only need to read from Git. Give them read-only credentials or deploy keys.

Control 4

Break-glass access

Keep a controlled, audited emergency path for direct cluster access, and reconcile Git afterwards.

Control 5

Supply chain checks

Scan images, verify signatures, and pin versions or digests so only trusted artifacts are deployed.

Control 6

Separation of duties

The author of a change should not be its only approver, especially for production folders.

!
Privilege Escalation Risk

If a GitOps agent runs with cluster administrator rights and any developer can merge to its repository, then any developer can effectively become a cluster administrator. Use per-team scopes, restricted folders, and code ownership rules so each team can only change what it owns.

18Monitoring, Logging, and Observability

A GitOps system that fails silently is worse than none, because everyone assumes Git is being obeyed.

The agent is itself a critical piece of infrastructure. If it stops working, the cluster quietly stops following Git, yet nobody may notice until something goes wrong. For this reason, teams monitor both the agent and the applications it manages.

Signal 1

Sync status

How many applications are synced, out of sync, or failed. Out of sync for long periods deserves attention.

Signal 2

Reconciliation duration

How long each loop takes. Rising times can signal repository slowness or oversized configurations.

Signal 3

Repository fetch errors

Authentication failures or network issues that stop the agent from reading Git.

Signal 4

Health of workloads

Whether the applications that were applied are actually ready and serving traffic.

Signal 5

Drift events

How often and where manual changes are detected, which points to process gaps.

Signal 6

Deployment frequency and lead time

Delivery metrics measured straight from Git history, such as time from merge to running.

Notifications That Matter

Connect the agent to chat and incident tools so that failures reach people quickly. Useful alerts include failed syncs, applications that remain out of sync beyond a set time, health degradations right after a rollout, and agent downtime. Because every change is a commit, you can also link deployment events back to the exact pull request and author, which makes investigating incidents much faster.

i
Observability Tip

Mark deployments on your monitoring graphs using commit information. When a graph changes shape, one glance shows which commit coincided with the change.

19Advantages, Disadvantages, and Trade-offs

GitOps is powerful, yet it asks for discipline and brings its own costs.

Advantages

  • A single, trustworthy source of truth for what should run.
  • Complete audit trail and easy rollback through Git history.
  • Drift is detected and corrected continuously.
  • Stronger security by removing cluster credentials from outside systems.
  • Faster, calmer recovery and repeatable cluster creation.
  • Familiar developer workflow based on pull requests and reviews.

Disadvantages and Trade-offs

  • Upfront effort to design repositories, pipelines, and conventions.
  • Secrets need careful, extra tooling.
  • Emergency fixes feel slower because they must go through Git.
  • Not everything fits declarative files, such as one-time data migrations.
  • The agent and the repository become critical dependencies to protect.

GitOps Compared with Other Approaches

ApproachHow Changes Reach the ClusterDetects Drift?Source of Truth
Manual commandsEngineer types commands directlyNoWhatever is in the cluster
Scripted pipeline pushCI server applies changes at the end of a buildOnly when the pipeline runsPipeline plus cluster
Package manager onlyChart installed or upgraded by a person or scriptNoChart plus release records
GitOpsAgent in the cluster pulls from Git and reconcilesYes, continuouslyGit repository
i
When It May Be Overkill

For a tiny personal project with a single cluster and one developer, a simple script may be enough. GitOps pays off as the number of people, applications, and clusters grows.

20Real-World Adoption Patterns

Organizations of all sizes use the same handful of GitOps patterns.

Platform teams offering self-service

Platform engineers maintain the base definitions, policies, and shared services. Application teams change only their own folders through pull requests, getting safe, quick delivery without needing cluster access.

Regulated industries

Banks and healthcare companies use Git history and mandatory approvals as evidence of change control. Auditors can trace every production change to a reviewed commit without chasing screenshots and tickets.

Edge and retail fleets

Companies with hundreds of small clusters in stores or factories rely on pull-based agents, because each site only needs outbound connectivity and can fix itself after outages.

Software vendors shipping to customers

Vendors package their configuration so that customer clusters pull updates from controlled sources, keeping each installation consistent and upgradable.

Infrastructure beyond Kubernetes

Teams extend the same pattern to cloud resources by describing them as Kubernetes objects reconciled by controllers, so databases, networks, and buckets follow the same pull request flow.

A Typical Adoption Journey

1

Start small

Pick one non-critical application and one environment. Learn the workflow without risk.

2

Alert only, then heal

Begin by reporting drift. Once the team trusts the setup, turn on automatic correction.

3

Add guardrails

Introduce policy checks, branch protection, and secret handling before expanding.

4

Expand to production

Move critical workloads, with progressive delivery and strong monitoring.

5

Scale to the fleet

Use generators and standard templates to onboard teams and clusters quickly.

21Best Practices, Anti-Patterns, and Common Mistakes

Habits that keep GitOps smooth, and traps that make it painful.

Best Practices

  • Keep application code and configuration in separate repositories.
  • Use folders for environments, with a shared base and small overlays.
  • Reference images by fixed versions or digests, never by moving tags.
  • Build once and promote the same artifact through environments.
  • Protect the main branch and require reviews and checks.
  • Monitor the agent as seriously as the applications.
  • Practice recreating a cluster from Git before you need to.

Common Mistakes

  • Allowing routine manual edits in production.
  • Committing plain-text or merely encoded secrets.
  • Giving every agent cluster administrator rights.
  • Having automation commit back into the same repository that triggers it.
  • Ignoring out-of-sync warnings until they pile up.
  • Treating GitOps as only a tool choice rather than a working discipline.
ANTI-PATTERN-01 Avoid
Problem

Engineers regularly fix problems directly on the cluster and plan to “update Git later”.

Why It’s Harmful

Git stops being the truth. The agent may revert the fix at the worst moment, or the changes are forgotten and lost at the next rebuild.

Correct Approach

Make the change in Git first, or pause synchronization for that application during an emergency and commit the same change immediately afterwards.

ANTI-PATTERN-02 Avoid
Problem

Each environment is a long-lived branch, and changes are merged between branches to promote them.

Why It’s Harmful

Branches drift, merges bring surprises, and nobody can easily compare what differs between environments.

Correct Approach

Use one main branch with a folder per environment, and promote by changing a version in the next folder.

ANTI-PATTERN-03 Avoid
Problem

The deployment tool is treated as the goal, and the pipeline still pushes directly to the cluster with a Git step added for show.

Why It’s Harmful

You get the complexity of GitOps without continuous reconciliation or the security benefits of pull.

Correct Approach

Let the in-cluster agent be the only actor that applies changes, and make the pipeline stop at publishing artifacts and proposing configuration changes.

ANTI-PATTERN-04 Avoid
Problem

A single agent with unlimited rights deploys everything for every team from one shared repository with no ownership rules.

Why It’s Harmful

Any contributor can affect any workload or cluster setting, and one mistake can spread widely.

Correct Approach

Scope agents and folders per team, use code owners for approvals, and apply policy checks to every change.

Handling Resources Managed by Other Controllers

Some fields in a live resource are legitimately changed by other controllers. For example, an autoscaler adjusts replica counts, and a service mesh may inject extra containers. If the agent insists that every field match Git, it will fight with these controllers. Good practice is to tell the agent which fields to ignore, or to leave those fields out of the declaration, so each actor has clear ownership.

22Frequently Asked Questions

Q1Is GitOps only for Kubernetes?

No. The ideas apply to any system that can be described declaratively and reconciled by an agent. Kubernetes is the most common home because its controllers are built for exactly this model, but the same pattern is used for cloud resources and other platforms.

Q2Does GitOps replace continuous integration?

No. Continuous integration still builds, tests, and packages your software. GitOps replaces the final step of pushing into the cluster with a pull-based, reconciled delivery.

Q3What happens if Git is unavailable?

The cluster keeps running whatever it already has. The agent simply cannot see new changes until Git returns. Because the running workloads do not depend on Git, an outage there delays changes but does not stop applications.

Q4What if the agent itself fails?

Workloads continue to run, but drift will not be corrected and new changes will not be applied. That is why the agent should be monitored, run with high availability where possible, and covered by alerts.

Q5Should every change really go through a pull request?

Yes for anything that should persist. Pull requests give review, history, and accountability. For true emergencies, define a controlled break-glass process and bring Git back in line immediately afterwards.

Q6How is GitOps different from infrastructure as code?

Infrastructure as code is the idea of writing infrastructure in files. GitOps adds specific rules on top: the files live in version control, an agent pulls them automatically, and reconciliation continues forever. You can have infrastructure as code without GitOps, but not GitOps without it.

Q7How often does the agent check Git?

Typically every few minutes by default, and many setups add notifications from the Git platform so that changes are applied within seconds. The interval is a balance between speed and load.

Q8Can developers still look at the cluster directly?

Yes, read-only access is common and helpful for troubleshooting. What GitOps discourages is making changes there. Dashboards and read-only roles give visibility without risking drift.

Q9How do I handle database migrations?

Treat them as part of the release process, not as ad-hoc commands. Many teams run migrations through a controlled job triggered as part of a rollout, with careful ordering and backward-compatible changes so rollback remains safe.

Q10Which tool should a beginner start with?

Either Argo CD or Flux is a solid choice. Beginners often find a visual dashboard helpful for learning, while others like the lightweight, command-line-first style. Try both on a small cluster and keep the one that feels natural to your team.

23Summary and Key Takeaways

GitOps brings the best habits of software development, review, history, and automation, to the running of systems. You declare what you want in Git, an agent inside the cluster pulls it, and a continuous loop keeps reality matching the declaration. The result is a platform where changes are visible, reversible, and accountable, and where the question “what is running in production?” can be answered by reading a repository.

To succeed with GitOps, treat it as a discipline rather than a tool installation. Design your repositories carefully, protect the main branch as you would a production control panel, handle secrets with proper tooling, promote the same artifact through every environment, and monitor the agent itself. Start small, grow with confidence, and let Git become the calm center of how your team delivers software.

Key Takeaways

  • Four principles define GitOps — declarative, versioned and immutable, pulled automatically, and continuously reconciled. Miss one and you lose the benefits.
  • Pull beats push — an agent inside the cluster removes powerful credentials from outside systems and corrects drift continuously.
  • Separate code from configuration — use one repository for application source and another for desired state, with folders for environments.
  • Build once, promote many times — the same immutable artifact moves from testing to production through reviewed configuration changes.
  • Secrets never go to Git in plain text — use encryption or external vault references, and remember encoding is not encryption.
  • Git access is production access — protect branches, require reviews, scope agents, and apply policy as code before and at admission.
  • Watch the watcher — monitor sync status, health, and agent availability so silent failure cannot hide, and start small before scaling to the fleet.