GitOps Principles in Kubernetes: Git as the Single Source of Truth
A complete, plain-English guide to GitOps: the four core principles, how pull-based reconciliation really works, how to structure repositories, handle secrets, promote releases safely, and avoid the mistakes that make GitOps painful.
Picture a large restaurant kitchen where every dish must taste the same in every branch of the chain. One way to achieve this is to phone each chef and say “add salt now, lower the flame now, plate it now”. Soon nobody remembers who told whom to do what, and every kitchen drifts a little differently. The other way is to keep one master recipe book, locked in the head office, with every change signed and dated. Each branch’s kitchen manager checks the book regularly and adjusts the kitchen to match. GitOps is the second way. The recipe book is a Git repository, the kitchens are Kubernetes clusters, and the kitchen managers are small programs that keep checking and correcting.
In this guide we begin from zero. We will first remember what Kubernetes already does, then see why traditional deployment pipelines struggle, and then build up the GitOps idea piece by piece. By the end you will understand the four principles, the difference between push and pull, repository layouts, secrets, promotion across environments, progressive delivery, security, monitoring, and the traps to avoid.
1What Is GitOps?
GitOps is a way of running systems where the desired state lives in Git and software inside the cluster keeps reality matching it.
GitOps is an operating model for infrastructure and applications. You describe everything you want, such as which applications run, in which versions, with which settings, in plain text files stored in a Git repository. A software agent running next to your cluster reads those files and makes the cluster match them. If someone changes the cluster by hand, the agent notices and puts it back. If you want to change the cluster, you do not touch it directly; you change the files in Git, and the agent does the rest.
The name joins two words: Git, the version control tool that records every change to a set of files, and Ops, short for operations, the daily work of running systems. The term was popularized in 2017 by engineers at Weaveworks, who used it to describe how they ran their own Kubernetes platform. Since then it has grown into a community standard, with a shared set of principles maintained by the OpenGitOps working group under the Cloud Native Computing Foundation.
Think of a school timetable pinned to the notice board. Teachers do not decide on their own which room to go to. They read the timetable and follow it. If a teacher turns up in the wrong room, the head teacher gently sends them back. To change who teaches what, you edit the master timetable, with a record of who changed it and why, rather than whispering to individual teachers.
It is worth noticing that GitOps is a way of working, not a product you buy. You can practice it with different tools, in different clouds, on different kinds of clusters. What makes something GitOps is not the tool but the rules it follows, and those rules are the subject of this guide.
2Why GitOps Exists: The Problems It Solves
Before GitOps, most teams shipped to Kubernetes with scripts and pipelines that had hidden weaknesses.
In a traditional setup, a continuous integration server builds your application, and then, as the last step, it runs commands to deploy to the cluster. This looks simple, but it creates several quiet problems. The pipeline needs powerful credentials to the cluster, so those credentials are stored in a build system that many people can reach. The pipeline only acts when it runs; between runs, anyone can change the cluster by hand and nobody will know. And the real state of the cluster is a mystery that must be discovered by asking the cluster, because no document reliably says what should be there.
The central problem is trust. Without GitOps, you cannot answer two basic questions with certainty: “What is supposed to be running?” and “Is that what is actually running?” GitOps gives a firm answer to both.
Configuration drift
Manual fixes at midnight slowly make clusters different from each other and from what anyone documented.
No reliable history
When something breaks, nobody can say exactly who changed what, when, and for what reason.
Risky credentials
Build servers hold cluster admin keys, making them a prime target for attackers.
Slow, scary recovery
Rebuilding a lost cluster means re-running old scripts, and hoping they still work.
Inconsistent environments
Testing and production slowly diverge, so a change that worked in testing behaves differently in production.
Approval gaps
Changes made directly to a cluster skip reviews, so quality and compliance depend on individual discipline.
3Foundations You Need First
A short refresher on the ideas GitOps builds on.
Git Repository
A folder of files with a complete recorded history of every change, who made it, and when.
Commit
A saved snapshot of a change, with an author, a timestamp, and a message explaining why.
Pull Request
A proposal to merge a change, where teammates can review, comment, and approve before it takes effect.
Manifest
A text file that describes a Kubernetes object, such as a Deployment or a Service, in a structured way.
Controller
A program that compares desired state with actual state and acts to close the gap, over and over.
Reconciliation
The act of comparing what you want with what exists and correcting any difference.
A Git repository is a shared notebook that can never lose a page. Each commit is a dated entry. A pull request is a draft entry that a colleague must read and approve before it is written into the notebook. Reconciliation is the act of looking at the notebook, looking at the real world, and tidying the world until they agree.
Kubernetes Is Already Declarative
Kubernetes is built around declaring a desired state. You tell it “I want three copies of this application” and its controllers work to make that true. GitOps takes that same idea one step further: it asks where the declarations themselves should live and who should be allowed to change them. The answer is Git, with reviews and history, and an in-cluster agent that applies them. In a sense, GitOps is the natural partner of Kubernetes, because both are based on the same love of declared intent and continuous correction.
It also helps to separate two kinds of software. Continuous integration is about building and testing code and producing artifacts such as container images. Continuous delivery is about getting those artifacts safely into running environments. GitOps is a particular, disciplined way of doing the delivery half. Your build server still builds; it simply stops reaching into the cluster.
4The Four GitOps Principles
The OpenGitOps standard describes four rules. A system is GitOps only if it follows all four.
Declarative
The whole system is described by stating the desired end state, not by listing the steps to reach it.
Versioned and Immutable
The desired state is stored in a way that keeps a complete, unalterable history and supports easy rollback.
Pulled Automatically
Software agents automatically fetch the desired state from the source; nobody pushes changes into the cluster from outside.
Continuously Reconciled
Agents constantly observe the real system and try to apply the desired state, correcting any drift.
flowchart LR
A[1 Declarative description] --> B[2 Stored in Git with history]
B --> C[3 Agent pulls automatically]
C --> D[4 Agent reconciles continuously]
D -->|drift detected| C
Each principle supports the others. Declarative files are what make versioning meaningful, because you can compare two versions and see exactly what changed. Versioning is what makes pull safe, because the agent always fetches a known, reviewed state. Pull is what makes continuous reconciliation possible, because the agent lives inside the environment and can check as often as needed. And reconciliation is what makes the declarations truthful, because the live system is forced to match them.
5Principle One: Declarative Desired State
Say what you want, not how to get there.
A declarative description states the end result. For example, “this namespace contains an application at version two with three replicas and a public address”. An imperative description lists commands: “create a namespace, then start a container, then scale it, then expose it”. The imperative way depends on the starting point. If you run the same commands twice, or on a cluster that already has some of the pieces, you may get errors or a different result. The declarative way does not care about the starting point, because the agent simply works out the difference between what exists and what you described.
Giving a taxi driver an address is declarative. Giving turn-by-turn directions is imperative. If a road is closed, the driver with an address finds another way. The driver with turn-by-turn directions gets lost the moment anything unexpected happens.
What Counts as Desired State
In Kubernetes, desired state is usually a set of manifests, but it can take other forms that eventually produce manifests. Teams use templating tools such as Helm charts, or overlay tools such as Kustomize, to avoid repeating the same text for different environments. What matters is that the final result is fully determined by files in the repository. If rendering a template pulls in values from a random web service at deploy time, the declaration is no longer fully in Git, and the principle is weakened.
Mixing in imperative steps, such as “run this script after deploying to fix the database”. Anything that must happen for the system to work should itself be expressed declaratively, or handled by a controller that watches for it.
Everything as Code
The same idea extends beyond application manifests. Namespaces, access rules, network policies, quotas, and even the definitions of the clusters themselves can be written as declarations. Teams that go all the way, sometimes called “everything as code”, gain the ability to recreate an entire platform from scratch simply by pointing a fresh agent at the repository.
6Principle Two: Versioned and Immutable
Every change is recorded, reviewable, and reversible.
Git stores a permanent chain of commits. Each commit is identified by a unique fingerprint derived from its contents, which means history cannot be quietly rewritten without everyone noticing. This is the immutable part: once a state is recorded, it stays recorded. The versioned part means you can go back to any earlier state at any time. Together they give you an audit trail and a time machine in one.
Audit trail
For every change, you know who made it, who approved it, when it happened, and the reason in the message.
Easy rollback
Reverting a bad release is a matter of reverting a commit, and the agent restores the previous state automatically.
Peer review
Every change goes through a pull request, so mistakes are caught before they reach a cluster.
Compliance evidence
Auditors can read the history to see that approvals and separation of duties were followed.
Immutable Artifacts, Too
Immutability applies to what the manifests point at, not just the manifests. If a manifest says “use the image tagged latest”, then the same commit can mean different things on different days, because the tag can be moved to a different image. To keep the history honest, manifests should reference images by a fixed version tag or, even better, by the image’s content fingerprint called a digest. Then a given commit always means exactly one thing.
Protect your main branch. Require reviews, require passing checks, forbid force-pushes, and sign commits where possible. In GitOps, the main branch is effectively the control panel for production, so it deserves the strongest protection you have.
7Principle Three: Pulled Automatically
The cluster reaches out to Git. Nobody reaches into the cluster.
In a push model, an external system, such as a build server, connects to the cluster and applies changes. In a pull model, an agent running inside or beside the cluster connects outward to the repository and fetches the desired state. This small change in direction has large effects, especially for security.
flowchart TD
subgraph PUSH [Push model]
P1[Developer] --> P2[CI server holds cluster credentials]
P2 -->|pushes changes in| P3[Cluster]
end
subgraph PULL [Pull model]
L1[Developer] --> L2[Git repository]
L4[Agent inside cluster] -->|pulls from| L2
L4 -->|applies to| L3[Cluster]
end
| Aspect | Push Model | Pull Model |
|---|---|---|
| Where credentials live | In the external build system, which can reach the cluster | Inside the cluster; only read access to Git is needed from outside |
| Network exposure | Cluster API must be reachable from the pipeline | Cluster only makes outbound connections |
| Drift handling | Only corrected when the pipeline runs again | Corrected continuously by the agent |
| Multiple clusters | Pipeline must hold keys for each cluster | Each cluster runs its own agent |
| Source of truth | Pipeline plus whatever is in the cluster | Git repository |
Push is like giving a delivery person the key to your house so they can come in whenever they want. Pull is like you walking to the post office to collect your parcel. In the second case, nobody outside ever needs a key to your house.
The pull model is also friendlier to firewalls and private networks. A cluster in a locked-down environment can still make outbound connections to a repository, while it may be very difficult to allow inbound connections from a build server. For organizations running dozens of clusters in different places, the pull model scales naturally because each cluster looks after itself.
Running a deployment command from a pipeline step that reads a Git repository is not GitOps, even though Git is involved. If the pipeline pushes the change into the cluster, it is the push model, and the continuous reconciliation benefits are lost.
8Principle Four: Continuously Reconciled
The agent never stops checking, so drift cannot hide.
Reconciliation is the loop at the heart of GitOps. The agent observes what is running in the cluster, compares it with the desired state from Git, and, if they differ, takes action to bring the cluster back to the declared state. Then it does this again, forever. The loop is the same idea that powers Kubernetes controllers, applied at the level of the whole environment.
sequenceDiagram
participant G as Git Repository
participant A as GitOps Agent
participant K as Kubernetes API
loop Every few minutes or on notification
A->>G: Fetch latest desired state
A->>K: Read live state
A->>A: Compare desired and live
alt Difference found
A->>K: Apply changes to match Git
A->>A: Record result and health
else Same
A->>A: Mark as in sync
end
end
Drift: The Silent Enemy
Drift means the live system no longer matches the declared state. It happens when someone edits a resource by hand, when another automated tool changes a field, or when a resource is deleted by accident. Without reconciliation, drift accumulates unseen until it causes an outage or a failed audit. With reconciliation, drift is either undone automatically or at least reported clearly as “out of sync”.
Automatic Healing or Alert Only
Teams can choose how aggressively the agent reacts. In a fully automatic mode, the agent corrects any drift immediately, which is ideal for production consistency. In a cautious mode, the agent only reports differences and waits for a human to decide, which can be useful while a team is learning. Many teams start cautious in a few environments and move to automatic healing as confidence grows.
If an engineer must change something by hand during an incident, the agent may undo the fix a moment later. A good practice is to make the same fix in Git right away, or to temporarily pause synchronization for that one application, and to resume it once Git is updated.
9Architecture and Components of a GitOps System
A GitOps setup has a handful of cooperating parts.
flowchart TD
DEV[Developer] -->|opens pull request| APPREPO[Application source repository]
APPREPO --> CI[CI pipeline builds and tests]
CI -->|pushes image| REG[(Container registry)]
CI -->|updates version in| CFG[Configuration repository]
CFG --> AGENT[GitOps agent in cluster]
REG --> AGENT
AGENT -->|applies| CLUSTER[Kubernetes cluster]
AGENT -->|reports status| DASH[Dashboard and alerts]
Application source repository
Holds the code of the application. Changes here trigger builds and tests, not deployments.
Continuous integration pipeline
Builds the application, runs tests, scans for problems, and publishes a versioned container image.
Container registry
Stores the immutable images that the manifests refer to.
Configuration repository
Holds the desired state of the environments: manifests, overlays, and version references. This is the true source of truth for what runs.
GitOps agent
Runs in the cluster, pulls from the configuration repository, renders manifests, and reconciles the cluster.
Observability layer
Dashboards, notifications, and alerts that show sync status, health, and failures.
Think of a publishing house. Authors write manuscripts (application repository). The printing press produces finished books (CI pipeline and registry). The catalogue lists which edition each bookshop should stock (configuration repository). Each bookshop’s manager (the agent) checks the catalogue and makes sure the shelves match it.
10Tools: Argo CD and Flux
Two mature, community-driven tools dominate GitOps on Kubernetes.
Argo CD and Flux are both graduated projects of the Cloud Native Computing Foundation. Both implement the four principles. Both run inside the cluster, pull from Git or from packaged artifact stores, render manifests, and reconcile continuously. They differ in style and user experience.
| Aspect | Argo CD | Flux |
|---|---|---|
| Main style | Application-centric, with a rich web interface | Toolkit of small controllers, strongly command-line and API oriented |
| User interface | Built-in visual dashboard showing live resource trees and diffs | Primarily through the command line and external dashboards |
| Multi-cluster | One control plane can deploy to many clusters | Often one agent set per cluster, coordinated through Git |
| Templating support | Plain manifests, Kustomize, Helm, and other plugins | Kustomize and Helm through dedicated controllers |
| Extensions | Application sets, sync waves, hooks, notifications | Image automation, notifications, dependency ordering |
| Good fit | Teams wanting visibility and a central view | Teams wanting a lightweight, composable, Kubernetes-native toolkit |
Neither tool is wrong. Pick the one whose workflow your team finds natural, then invest in good repository design, which matters far more than the choice of agent.
Supporting Tools
Kustomize and Helm
Produce environment-specific manifests from shared bases or packaged charts, keeping repetition low.
Policy engines
Tools that check manifests against company rules, either before merge or at admission to the cluster.
Progressive delivery controllers
Roll out new versions gradually using canary or blue-green strategies with automatic analysis and rollback.
Secret management tools
Encrypt secrets for storage in Git or sync them from external vaults, so plain-text secrets never sit in a repository.
11Repository Design and Environment Strategy
How you organize Git decides how pleasant GitOps will be for the next five years.
Separate Code from Configuration
A widely recommended practice is to keep application source code and deployment configuration in different repositories. The reasons are practical. Code changes often, in small steps, and many developers touch it. Configuration changes represent deliberate decisions about what runs where, and they have a different audience and approval needs. Separating them also stops a build from triggering itself in a loop: if the pipeline commits a new version number into the same repository that triggers the pipeline, you can create an endless cycle.
One Repository or Many
Single Shared Config Repository
- One place to look for everything that runs.
- Easy to make changes that span several applications.
- Simple global policies and reviews.
- Straightforward to bootstrap a new cluster.
Many Team Repositories
- Teams own their configuration and move independently.
- Access control can be tight per team.
- Smaller blast radius for mistakes.
- Harder to get a global overview without extra tooling.
Environments: Folders Beat Branches
Teams often ask whether to represent environments such as development, staging, and production as branches or as folders. Folders are generally preferred. With branches, the environments drift apart in confusing ways, merges become complicated, and it is hard to see differences at a glance. With folders on one main branch, each environment has its own directory, built from a shared base plus small overrides. Differences are easy to see, and promotion is as simple as changing a version number in the next folder.
Base
The common definition of an application, shared by every environment.
Overlays
Small environment-specific changes such as replica counts, resource limits, and addresses.
Cluster definitions
Which applications and which overlays belong on which cluster.
Copying whole folders for each environment and editing them freely. Within weeks nobody knows which differences are intentional. Use a shared base and keep overrides as small as possible.
12Data Flow and Lifecycle of a Change
Let us follow one change, from a developer’s idea to a running update.
Code change merged
A developer’s pull request is reviewed and merged in the application repository.
Build and publish
The pipeline builds the application, tests it, scans it, and publishes a uniquely versioned image.
Configuration update proposed
A pull request in the configuration repository changes the image version for the first environment. This may be created by automation or by a person.
Review and checks
Automated checks validate the manifests and policies, and reviewers approve. Merging records the decision in history.
Agent detects and applies
The in-cluster agent notices the new commit, renders the manifests, and applies the difference to the cluster.
Health verification
The agent watches rollout health. Status shows synced and healthy, or degraded if something is wrong.
Promotion
After the change proves itself, another pull request promotes the same version to the next environment.
stateDiagram-v2
[*] --> OutOfSync: New commit in Git
OutOfSync --> Syncing: Agent starts applying
Syncing --> Synced: Cluster matches Git
Synced --> Healthy: Workloads pass health checks
Syncing --> Failed: Apply error
Failed --> OutOfSync: Fix committed
Healthy --> OutOfSync: Drift or new commit
Notice that synced and healthy are two different ideas. Synced means the cluster matches Git. Healthy means the things that were applied are actually working. An application can be perfectly synced to a broken configuration, which is why health checks and progressive delivery are important companions to GitOps.
13Promotion Across Environments
Moving a release from testing to production should be boring, visible, and repeatable.
In GitOps, promotion means changing the desired state of the next environment so that it matches the version already proven in the previous one. Because both are just files in Git, promotion becomes a small, reviewable change. The same artifact that passed testing is the one that reaches production; nothing is rebuilt along the way. This guarantees that what you tested is what you ship.
Manual promotion
A person opens a pull request to bump the version in the next environment. Simple and fully controlled.
Automated promotion
Automation raises the pull request once tests pass, and humans only approve. Faster with the same safety.
Image update automation
An agent watches the registry and commits new versions into the lower environment automatically.
Gated promotion
Promotion waits on checks, such as passing tests, quiet monitoring, or an approval window.
Build once, promote many times. Never rebuild the image for production. Rebuilding can pick up different dependencies and silently change what you are shipping.
Rolling Back
Rollback in GitOps is delightfully simple: revert the commit that introduced the problem. The agent sees the new desired state and restores the earlier version. Because the revert is itself a commit, the history clearly shows that a problem occurred and was undone, which is valuable for later review. Remember that rollback only restores the declared state; it cannot undo irreversible changes such as data migrations, so those need separate care.
14Secrets Management in GitOps
Git must never contain plain-text passwords, yet the cluster needs them. Here is how teams reconcile the two.
Because Git is the source of truth and is widely readable, storing secrets in it as plain text would be a serious security failure. Even a private repository leaks through copies, backups, and former employees. GitOps teams use one of two broad approaches to solve this.
Encrypted Secrets in Git
- Secrets are encrypted before being committed, and only the cluster or authorized people can decrypt them.
- The repository remains the complete source of truth.
- Changes to secrets go through the same review and history.
- Key management becomes an important responsibility.
References to External Vaults
- Git stores only a reference, such as “the database password from the vault”.
- A controller in the cluster fetches the real value and creates the Secret.
- Central rotation and detailed audit in the vault.
- Adds a dependency on the external system being available.
Encrypted secrets are like sending a locked box through the mail: anyone can see the box, but only the receiver has the key. References to a vault are like leaving a note that says “the key is in the bank safe, box number seven”: the note is harmless without the bank’s cooperation.
Encoding a secret in a text format such as base64 is not encryption. Anyone can reverse it in a second. Also remember that a secret committed once remains in Git history even after deletion, so rotate it immediately if it was ever exposed.
15Progressive Delivery and Policy as Code
GitOps decides what should run. These companions help decide how safely it gets there.
Progressive Delivery
Progressive delivery is the practice of releasing a new version to a small slice of traffic first, measuring how it behaves, and only then widening the rollout. In a canary release, a small percentage of users see the new version while the rest stay on the old one. In a blue-green release, a full copy of the new version runs beside the old one, and traffic is switched over once it looks healthy. Specialized controllers can automate the analysis, using metrics such as error rate and latency, and roll back automatically if results look bad. GitOps provides the desired state, and the progressive delivery controller handles the careful transition.
A canary release is like a restaurant testing a new dish on one table before adding it to the whole menu. If the first table sends it back, the kitchen changes course without upsetting the entire dining room.
Policy as Code
Policy as code means writing your organization’s rules as machine-checkable definitions. Examples include “no container may run as the root user”, “every workload must declare resource limits”, and “images must come from the approved registry”. These rules can be checked at two moments. Before merge, a pipeline validates the proposed manifests and blocks the pull request if it violates policy. At admission, the cluster itself rejects any object that breaks policy, even if it somehow got past the first check. Using both gives defence in depth.
Schema validation
Confirms manifests are well-formed and match the Kubernetes API before they are merged.
Security policy
Blocks privileged containers, host access, and missing security settings.
Cost and quota policy
Requires resource requests and limits and prevents oversized workloads.
Image policy
Allows only signed, scanned images from approved registries.
16Multi-Cluster and Platform Scale
GitOps shines brightest when you have many clusters to keep consistent.
Organizations rarely stop at one cluster. They have clusters per environment, per region, per business unit, or per customer. Managing these by hand is a recipe for drift. With GitOps, each cluster runs an agent, or a central agent manages several, and every cluster is simply a list of what it should contain. Adding a new cluster means pointing a fresh agent at the repository; within minutes, it installs everything it should have.
flowchart TD
REPO[Central configuration repository] --> A1[Agent in cluster Dev]
REPO --> A2[Agent in cluster Staging]
REPO --> A3[Agent in cluster Prod Region 1]
REPO --> A4[Agent in cluster Prod Region 2]
A1 --> C1[Dev cluster]
A2 --> C2[Staging cluster]
A3 --> C3[Prod Region 1 cluster]
A4 --> C4[Prod Region 2 cluster]
Fleet Patterns
Cluster bootstrap
A new cluster starts with a single bootstrap step that installs the agent. The agent then pulls a root definition that installs everything else: networking, monitoring, policies, and applications.
App of apps and generators
A parent definition lists child applications, or a generator creates one application per cluster or per team from a template. This avoids writing hundreds of near-identical definitions.
Staged fleet rollouts
Changes are applied first to a canary cluster, then to a small group, then to the whole fleet, with checks between stages.
Disaster recovery
If a cluster is lost, a replacement is created and pointed at the same repository. Stateless workloads return automatically, and only data needs to be restored from backups.
17Security Considerations
GitOps improves security in many ways, but it also moves the crown jewels into Git. Protect them accordingly.
With GitOps, access to the main branch of the configuration repository is effectively access to production. That makes the repository and its approval rules a primary security boundary. At the same time, the pull model improves the security posture by removing cluster credentials from external systems and by shrinking the number of people who need direct cluster access to almost none.
Branch protection
Require reviews, status checks, and signed commits, and forbid direct pushes to main.
Least-privilege agents
Limit what each agent may create. An agent for one team should not be able to change cluster-wide settings.
Read-only repository access
Agents only need to read from Git. Give them read-only credentials or deploy keys.
Break-glass access
Keep a controlled, audited emergency path for direct cluster access, and reconcile Git afterwards.
Supply chain checks
Scan images, verify signatures, and pin versions or digests so only trusted artifacts are deployed.
Separation of duties
The author of a change should not be its only approver, especially for production folders.
If a GitOps agent runs with cluster administrator rights and any developer can merge to its repository, then any developer can effectively become a cluster administrator. Use per-team scopes, restricted folders, and code ownership rules so each team can only change what it owns.
18Monitoring, Logging, and Observability
A GitOps system that fails silently is worse than none, because everyone assumes Git is being obeyed.
The agent is itself a critical piece of infrastructure. If it stops working, the cluster quietly stops following Git, yet nobody may notice until something goes wrong. For this reason, teams monitor both the agent and the applications it manages.
Sync status
How many applications are synced, out of sync, or failed. Out of sync for long periods deserves attention.
Reconciliation duration
How long each loop takes. Rising times can signal repository slowness or oversized configurations.
Repository fetch errors
Authentication failures or network issues that stop the agent from reading Git.
Health of workloads
Whether the applications that were applied are actually ready and serving traffic.
Drift events
How often and where manual changes are detected, which points to process gaps.
Deployment frequency and lead time
Delivery metrics measured straight from Git history, such as time from merge to running.
Notifications That Matter
Connect the agent to chat and incident tools so that failures reach people quickly. Useful alerts include failed syncs, applications that remain out of sync beyond a set time, health degradations right after a rollout, and agent downtime. Because every change is a commit, you can also link deployment events back to the exact pull request and author, which makes investigating incidents much faster.
Mark deployments on your monitoring graphs using commit information. When a graph changes shape, one glance shows which commit coincided with the change.
19Advantages, Disadvantages, and Trade-offs
GitOps is powerful, yet it asks for discipline and brings its own costs.
Advantages
- A single, trustworthy source of truth for what should run.
- Complete audit trail and easy rollback through Git history.
- Drift is detected and corrected continuously.
- Stronger security by removing cluster credentials from outside systems.
- Faster, calmer recovery and repeatable cluster creation.
- Familiar developer workflow based on pull requests and reviews.
Disadvantages and Trade-offs
- Upfront effort to design repositories, pipelines, and conventions.
- Secrets need careful, extra tooling.
- Emergency fixes feel slower because they must go through Git.
- Not everything fits declarative files, such as one-time data migrations.
- The agent and the repository become critical dependencies to protect.
GitOps Compared with Other Approaches
| Approach | How Changes Reach the Cluster | Detects Drift? | Source of Truth |
|---|---|---|---|
| Manual commands | Engineer types commands directly | No | Whatever is in the cluster |
| Scripted pipeline push | CI server applies changes at the end of a build | Only when the pipeline runs | Pipeline plus cluster |
| Package manager only | Chart installed or upgraded by a person or script | No | Chart plus release records |
| GitOps | Agent in the cluster pulls from Git and reconciles | Yes, continuously | Git repository |
For a tiny personal project with a single cluster and one developer, a simple script may be enough. GitOps pays off as the number of people, applications, and clusters grows.
20Real-World Adoption Patterns
Organizations of all sizes use the same handful of GitOps patterns.
Platform teams offering self-service
Platform engineers maintain the base definitions, policies, and shared services. Application teams change only their own folders through pull requests, getting safe, quick delivery without needing cluster access.
Regulated industries
Banks and healthcare companies use Git history and mandatory approvals as evidence of change control. Auditors can trace every production change to a reviewed commit without chasing screenshots and tickets.
Edge and retail fleets
Companies with hundreds of small clusters in stores or factories rely on pull-based agents, because each site only needs outbound connectivity and can fix itself after outages.
Software vendors shipping to customers
Vendors package their configuration so that customer clusters pull updates from controlled sources, keeping each installation consistent and upgradable.
Infrastructure beyond Kubernetes
Teams extend the same pattern to cloud resources by describing them as Kubernetes objects reconciled by controllers, so databases, networks, and buckets follow the same pull request flow.
A Typical Adoption Journey
Start small
Pick one non-critical application and one environment. Learn the workflow without risk.
Alert only, then heal
Begin by reporting drift. Once the team trusts the setup, turn on automatic correction.
Add guardrails
Introduce policy checks, branch protection, and secret handling before expanding.
Expand to production
Move critical workloads, with progressive delivery and strong monitoring.
Scale to the fleet
Use generators and standard templates to onboard teams and clusters quickly.
21Best Practices, Anti-Patterns, and Common Mistakes
Habits that keep GitOps smooth, and traps that make it painful.
Best Practices
- Keep application code and configuration in separate repositories.
- Use folders for environments, with a shared base and small overlays.
- Reference images by fixed versions or digests, never by moving tags.
- Build once and promote the same artifact through environments.
- Protect the main branch and require reviews and checks.
- Monitor the agent as seriously as the applications.
- Practice recreating a cluster from Git before you need to.
Common Mistakes
- Allowing routine manual edits in production.
- Committing plain-text or merely encoded secrets.
- Giving every agent cluster administrator rights.
- Having automation commit back into the same repository that triggers it.
- Ignoring out-of-sync warnings until they pile up.
- Treating GitOps as only a tool choice rather than a working discipline.
Problem
Engineers regularly fix problems directly on the cluster and plan to “update Git later”.
Why It’s Harmful
Git stops being the truth. The agent may revert the fix at the worst moment, or the changes are forgotten and lost at the next rebuild.
Correct Approach
Make the change in Git first, or pause synchronization for that application during an emergency and commit the same change immediately afterwards.
Problem
Each environment is a long-lived branch, and changes are merged between branches to promote them.
Why It’s Harmful
Branches drift, merges bring surprises, and nobody can easily compare what differs between environments.
Correct Approach
Use one main branch with a folder per environment, and promote by changing a version in the next folder.
Problem
The deployment tool is treated as the goal, and the pipeline still pushes directly to the cluster with a Git step added for show.
Why It’s Harmful
You get the complexity of GitOps without continuous reconciliation or the security benefits of pull.
Correct Approach
Let the in-cluster agent be the only actor that applies changes, and make the pipeline stop at publishing artifacts and proposing configuration changes.
Problem
A single agent with unlimited rights deploys everything for every team from one shared repository with no ownership rules.
Why It’s Harmful
Any contributor can affect any workload or cluster setting, and one mistake can spread widely.
Correct Approach
Scope agents and folders per team, use code owners for approvals, and apply policy checks to every change.
Handling Resources Managed by Other Controllers
Some fields in a live resource are legitimately changed by other controllers. For example, an autoscaler adjusts replica counts, and a service mesh may inject extra containers. If the agent insists that every field match Git, it will fight with these controllers. Good practice is to tell the agent which fields to ignore, or to leave those fields out of the declaration, so each actor has clear ownership.
22Frequently Asked Questions
No. The ideas apply to any system that can be described declaratively and reconciled by an agent. Kubernetes is the most common home because its controllers are built for exactly this model, but the same pattern is used for cloud resources and other platforms.
No. Continuous integration still builds, tests, and packages your software. GitOps replaces the final step of pushing into the cluster with a pull-based, reconciled delivery.
The cluster keeps running whatever it already has. The agent simply cannot see new changes until Git returns. Because the running workloads do not depend on Git, an outage there delays changes but does not stop applications.
Workloads continue to run, but drift will not be corrected and new changes will not be applied. That is why the agent should be monitored, run with high availability where possible, and covered by alerts.
Yes for anything that should persist. Pull requests give review, history, and accountability. For true emergencies, define a controlled break-glass process and bring Git back in line immediately afterwards.
Infrastructure as code is the idea of writing infrastructure in files. GitOps adds specific rules on top: the files live in version control, an agent pulls them automatically, and reconciliation continues forever. You can have infrastructure as code without GitOps, but not GitOps without it.
Typically every few minutes by default, and many setups add notifications from the Git platform so that changes are applied within seconds. The interval is a balance between speed and load.
Yes, read-only access is common and helpful for troubleshooting. What GitOps discourages is making changes there. Dashboards and read-only roles give visibility without risking drift.
Treat them as part of the release process, not as ad-hoc commands. Many teams run migrations through a controlled job triggered as part of a rollout, with careful ordering and backward-compatible changes so rollback remains safe.
Either Argo CD or Flux is a solid choice. Beginners often find a visual dashboard helpful for learning, while others like the lightweight, command-line-first style. Try both on a small cluster and keep the one that feels natural to your team.
23Summary and Key Takeaways
GitOps brings the best habits of software development, review, history, and automation, to the running of systems. You declare what you want in Git, an agent inside the cluster pulls it, and a continuous loop keeps reality matching the declaration. The result is a platform where changes are visible, reversible, and accountable, and where the question “what is running in production?” can be answered by reading a repository.
To succeed with GitOps, treat it as a discipline rather than a tool installation. Design your repositories carefully, protect the main branch as you would a production control panel, handle secrets with proper tooling, promote the same artifact through every environment, and monitor the agent itself. Start small, grow with confidence, and let Git become the calm center of how your team delivers software.
Key Takeaways
- Four principles define GitOps — declarative, versioned and immutable, pulled automatically, and continuously reconciled. Miss one and you lose the benefits.
- Pull beats push — an agent inside the cluster removes powerful credentials from outside systems and corrects drift continuously.
- Separate code from configuration — use one repository for application source and another for desired state, with folders for environments.
- Build once, promote many times — the same immutable artifact moves from testing to production through reviewed configuration changes.
- Secrets never go to Git in plain text — use encryption or external vault references, and remember encoding is not encryption.
- Git access is production access — protect branches, require reviews, scope agents, and apply policy as code before and at admission.
- Watch the watcher — monitor sync status, health, and agent availability so silent failure cannot hide, and start small before scaling to the fleet.