Secret Manager in Google Cloud
A friendly, step-by-step guide to the safe where Google Cloud keeps your passwords, keys, and tokens: what a secret is, how Secret Manager stores and protects it, who may open it, how it gets into your applications, and how to avoid the mistakes that cause real breaches.
Imagine you run a small shop. You have a spare key to the back door, the code to the cash safe, and a card that lets you order stock from a supplier. Where do you keep them? If you tape them to the front window, anyone walking past can use them. If you hide them under the doormat, a thief will find them in a minute. If you write them on a note and email the note to five staff members, you have just made five new copies that can be lost or stolen. A sensible owner puts them in a locked cabinet, gives each staff member only the key they need, and keeps a log of who opened the cabinet and when.
Secret Manager is that locked cabinet for Google Cloud. It stores the small, sensitive pieces of text that your software needs in order to work, and it guards them with encryption, strict permissions, and a written record of every visit. This guide assumes you know nothing about the topic. Every new word is explained when it first appears. By the end you will understand how Secret Manager works inside, how to set it up, how applications read secrets safely, and how real teams keep their keys out of trouble.
WWhat Is Secret Manager?
Start with the big picture, in plain words.
The simple definition
A secret, in computing, is any small piece of private information that gives access to something valuable. Common examples are database passwords, API keys, security tokens, private certificates, and encryption keys that your own application manages. If a stranger learned one of these, they could pretend to be your application and do harm.
Secret Manager is a Google Cloud service for storing, managing, and accessing secrets. You put a secret in once. Applications and people you approve can then fetch it when they need it. Everyone else is turned away. Google encrypts the stored data, records who touched it, and lets you keep older versions so that you can change a secret without breaking everything at once.
Think of a bank’s safe deposit room. The room itself is guarded. Every box has its own number. The bank staff check your identity before opening a box. Each visit is written in a logbook. If you move house and change your valuables, the bank lets you replace the contents while keeping a record of what was there before. Secret Manager works the same way, with software identities instead of customers at the counter.
Why it exists
Before services like this existed, teams scattered secrets in many risky places: inside source code, in configuration files, in chat messages, in spreadsheets, and in the notes of a favorite engineer. Each copy was a leak waiting to happen. Secret Manager gives you one controlled place to keep each secret, with a clear answer to the question, “Who can read this, and who did read it?”
What it is not
Secret Manager is not a general database and it is not a place for large files. Each secret version holds a small payload, which Google documents as having a size limit of tens of kilobytes. It is also not the same as Cloud KMS, the Key Management Service, which creates and uses cryptographic keys without ever handing the key itself to you. We will compare the two in a later chapter. For now, remember: Secret Manager stores things your software must read, while KMS performs operations with keys that never leave the vault.
Secret Manager does not make a weak secret strong. A short, guessable password stays guessable inside the safe. Its job is to protect storage and access. Choosing strong secrets is still your job.
TThe Problem: Why Secrets Leak
To value a safe, you must first see how doors get left open.
Hard-coded secrets
The most common mistake is writing a password straight into the source code. It feels convenient, because the program works instantly. But source code is copied everywhere: onto laptops, into backups, into build systems, and sometimes into public websites by accident. A password inside code is a password in every one of those places forever, because code history is kept.
Secrets in configuration files and environment variables
Moving the password into a configuration file is slightly better, but the file still gets copied, checked in, or pasted into tickets. An environment variable is a named setting handed to a program when it starts. It is popular and often fine, but it can leak through crash reports, debugging pages, and process listings. The question is not only “where is it stored” but “who can see it, and how long does it live?”
Secrets in chat, email, and documents
People share passwords in messages when they are in a hurry. Messages are searchable, forwarded, and retained for years. A secret shared this way has many copies and no log of who read it.
Secrets that never change
Many breaches succeed because a stolen password still works months or years later. If secrets are hard to change, nobody changes them. If changing is easy and safe, rotation becomes a habit. Secret Manager’s versions make changing less scary.
Over-sharing
A shared admin password used by twenty people gives you no way to know which of them did something. Per-identity access and audit trails fix that.
Inside code
Copied to every laptop, backup, and build system, and kept in history.
In config files
Checked in by accident or pasted into tickets and wikis.
In chat and email
Searchable, forwarded, and retained long after anyone remembers.
Never rotated
A stolen secret keeps working for years if nobody changes it.
Shared by many
No way to tell who used it or to remove one person’s access.
Printed in logs
Applications that print their settings during a crash leak them to log storage.
Production example: keys in a repository
Well-known incidents in the industry follow a familiar pattern. In a widely reported 2016 breach at Uber, attackers found cloud access credentials stored inside a code repository. In 2021, a compromised tool used by many build pipelines at Codecov exposed credentials that were available in those build environments. Different stories, same lesson: secrets that sit where code and build tools can see them are easy to steal, and the damage grows with how powerful and long-lived the secret is.
Removing a secret from the latest version of a code file does not remove it from history. If a real secret was ever committed, treat it as stolen and replace it immediately.
CCore Concepts of Secret Manager
The vocabulary of the service, explained one word at a time.
Project
A project is the basic container for resources in Google Cloud. It carries billing, permissions, and the list of switched-on services. Secrets live inside a project, and the project is the first boundary of who can reach them. The Secret Manager API must be enabled in the project before you can create anything.
Secret
In Secret Manager, a secret is a named container. It has a name, labels, a replication setting, and a list of permissions. Surprisingly, the secret itself does not hold the sensitive text. Think of it as an empty labeled drawer. The actual valuable content lives in versions.
Secret version
A secret version holds the actual sensitive content, called the payload. A version is immutable, which means once it is created, its payload can never be edited. If you need a new password, you do not change the old version. You add a new version with the new value. Versions are numbered 1, 2, 3, and so on.
Version states
Each version is in one of three states. Enabled means it can be read. Disabled means it is temporarily blocked but can be turned on again, like putting a document in a locked drawer. Destroyed means the payload has been permanently erased and cannot be recovered. Disabling first is a safe habit, because it is reversible.
Alias and the latest version
When you read a secret, you can ask for a specific number, such as version 3, or you can ask for the word latest, which means “the newest enabled version”. Newer features also let you define your own friendly aliases that point at a version number, so that “current” and “previous” can be named clearly. Using “latest” is convenient, but pinning a number is more predictable, and we will discuss the trade-off later.
Replication policy
The replication policy decides where Google stores copies of your secret. You can let Google choose automatically, or you can pick specific regions yourself. This choice affects availability and any rules about where data may physically live.
IAM and roles
IAM, short for Identity and Access Management, is the rulebook of who may do what in Google Cloud. A role is a bundle of permissions. For secrets, there are roles for reading payloads, adding versions, viewing metadata, and full administration. You can attach roles to a whole project or to one individual secret, which is the safer choice.
Labels, annotations, and topics
Labels are short key and value tags, such as “team: payments” or “env: prod”, used to organize and report. Annotations hold longer notes. A topic is a Pub/Sub channel, a messaging service, where Secret Manager can post notifications about events such as a version being added or a rotation being due.
Customer-managed encryption keys
By default, Google encrypts your secrets with keys it manages. If your organization requires more control, you can use customer-managed encryption keys, often shortened to CMEK, created in Cloud KMS. Then you decide when the key is used, rotated, or turned off.
| Term | Simple meaning | Can it change? |
|---|---|---|
| Secret | Named container with settings and permissions | Settings can change |
| Version | One saved value of the secret | Payload never changes |
| Payload | The actual sensitive text or bytes | No, add a new version instead |
| State | Enabled, disabled, or destroyed | Enabled and disabled can switch; destroyed is final |
| Replication | Where copies are stored | Chosen at creation, with limits on later changes |
| Role | Bundle of permissions | Granted and removed any time |
A secret is a labeled folder in a filing cabinet. Each version is a sheet of paper inside, stamped with a number. You never erase a sheet. You add a newer one on top. Old sheets can be locked away (disabled) or shredded (destroyed). The cabinet rules say who may read which folder.
AArchitecture: How the Pieces Fit Together
Follow a request from an application to the stored secret and back.
The system has a simple shape. Something that wants a secret, called the caller, proves its identity to Google. Google checks the permission rulebook. If the answer is yes, Secret Manager fetches the encrypted data, decrypts it, and returns it over a secure connection. Every step can be logged.
flowchart LR
A["Caller: application or person"] --> B["Identity proof: service account or user login"]
B --> C["Secret Manager API"]
C --> D["IAM permission check"]
D -- "Allowed" --> E["Secret metadata and version lookup"]
D -- "Denied" --> X["Permission denied error"]
E --> F["Encrypted payload in regional storage"]
F --> G["Decryption using Google-managed or customer-managed key"]
G --> H["Payload returned over HTTPS"]
C --> L["Audit log entry"]
Reading the diagram
Caller. This can be a program running on Cloud Run, Google Kubernetes Engine, a virtual machine, a build pipeline, or a human engineer using a command line. Each caller needs an identity.
Identity proof. Programs normally use a service account, an identity made for software. People use their own Google login. Google verifies the proof and attaches a short-lived token to the request, like a visitor badge that expires.
Secret Manager API. The request arrives at Google’s front door. This is an ordinary secure web request.
IAM permission check. Before touching any data, Google asks IAM whether this identity may perform this action on this secret. If not, the request stops with a permission denied message.
Metadata and version lookup. Secret Manager finds the secret, resolves “latest” or an alias to a concrete version number, and confirms that version is enabled.
Encrypted storage and decryption. The payload is stored encrypted. It is decrypted only at the moment it is returned to an authorized caller.
Audit log entry. Google writes a record of the call, who made it, and what it asked for, but never the secret value itself.
Secret Manager is a pull system. Applications ask for secrets when they need them. Nothing pushes a secret into an application unless a platform feature you configured fetches it on the application’s behalf.
It works like a library’s reference desk. You show your card (identity). The librarian checks whether your card allows access to the restricted shelf (IAM). If yes, the book is brought out, you read what you need, and the librarian notes the visit in a register (audit log). The librarian never reads the book aloud to anyone who has not shown a valid card.
RReplication and Locations
Where your secret lives matters for availability and for the law.
A region is a geographic area where Google runs data centers, such as a city region in Asia or Europe. A zone is a separate building inside a region. Secret Manager lets you decide how widely your secret is copied, using a replication policy.
Automatic replication
With the automatic policy, Google chooses the locations and stores copies so that the secret stays available even if one place has trouble. This is the simplest choice and works well for most teams that have no special rules about where data may live.
User-managed replication
With the user-managed policy, you list the exact regions that should hold copies. This is useful when a law, a contract, or a company rule says data must stay within certain countries. It is also useful when your applications run in a few known regions and you want secrets close to them. The more regions you choose, the better the availability, but the higher the number of stored copies.
Regional secrets
Google has also introduced regional secrets, which are tied to one location from the start and are accessed through a regional endpoint. They suit strict data residency needs, where the data and the requests must stay inside one region. Because this feature set has been growing, check the current documentation for the supported regions and any differences in features compared with the global kind.
| Choice | Best for | Trade-off |
|---|---|---|
| Automatic | Most teams, simple setup, strong availability | You do not pick the locations |
| User-managed, several regions | Data residency rules plus good availability | You must choose and maintain the list |
| User-managed, one region | Strict locality | Less protection if that region has trouble |
| Regional secret | Strict residency for data and requests | Features and endpoints differ from global secrets |
flowchart TD
A["Create secret"] --> B{"Any rule about where data may live?"}
B -- "No" --> C["Automatic replication"]
B -- "Yes, several allowed regions" --> D["User-managed replication in those regions"]
B -- "Yes, one region only" --> E["Regional secret or single-region policy"]
D --> F["Choose encryption: Google-managed or customer-managed key"]
C --> F
E --> F
Replication is like keeping copies of an important document. You can let a trusted courier decide where to store spare copies around the country (automatic). Or you can say “only in my own city and my brother’s city” (user-managed). Or you may say “never leave this building” (regional). More copies in more places mean it is harder to lose, but you must follow your own rules about who holds them.
Decide the replication approach before you create many secrets. Changing it later can be limited or require creating new secrets, so a quick early decision saves rework.
IInternal Working: Encryption and Storage
Lift the lid and watch the locks turn.
Encryption at rest
Encryption scrambles data so that only someone holding the right key can read it. At rest means while the data is sitting in storage. Google encrypts secret payloads before writing them, so someone who steals a disk sees only scrambled bytes.
Envelope encryption
Google uses a layered method called envelope encryption. Imagine locking a letter in a small box with a key, and then locking that key inside a bigger, stronger vault with a master key. In practice, each piece of data is encrypted with a data encryption key. That small key is itself encrypted by a key encryption key that lives in a dedicated key management system. This design makes it practical to protect huge amounts of data, because only the small keys need heavy protection, and master keys can be rotated without re-encrypting everything.
Google-managed versus customer-managed keys
By default, the key encryption key is managed by Google. With CMEK, you create a key in Cloud KMS and tell Secret Manager to use it. Then you control the lifecycle of that key. If you disable the key, Secret Manager can no longer decrypt the matching secrets, which gives you a powerful emergency switch. It also means that a mistake with the key can make your secrets unreadable, so protect the key carefully.
Encryption in transit
In transit means while moving across a network. Calls to Secret Manager use HTTPS, which encrypts the connection. An eavesdropper on the network sees only scrambled traffic.
Immutability and the version chain
Because a version’s payload cannot be edited, there is no race where two people overwrite each other’s change. Adding a version always creates a new numbered entry. This gives you a built-in history, and it gives applications a stable target: version 5 will mean the same bytes tomorrow as it does today.
What happens on delete
Destroying a version erases its payload. Google also supports a delayed destruction setting, where a destroy request is held for a chosen period before the data is truly removed. This is a safety net against accidents. During that window, the version can still be recovered by cancelling the destruction, which is useful when someone destroys the wrong version on a busy day.
sequenceDiagram
participant App as "Application"
participant SM as "Secret Manager"
participant IAM as "IAM"
participant KMS as "Key management system"
participant Store as "Encrypted storage"
App->>SM: "Access version 3 of db-password"
SM->>IAM: "May this service account access this secret?"
IAM-->>SM: "Yes"
SM->>Store: "Fetch encrypted payload and wrapped data key"
Store-->>SM: "Encrypted payload and wrapped key"
SM->>KMS: "Unwrap the data key"
KMS-->>SM: "Data key"
SM->>SM: "Decrypt payload in memory"
SM-->>App: "Payload over HTTPS"
SM->>SM: "Write audit log entry without the value"
Envelope encryption is like a hotel. Each guest room has its own key. Those room keys are kept in a locked cabinet at the front desk, and the cabinet is opened by the manager’s master key. If one guest loses a key, you change one lock. If the manager’s key must change, you replace the master key without re-cutting every room key.
You do not manage any of this machinery. But understanding it helps you reason about risks. For example, if you use a customer-managed key and delete it, no one at Google can bring your secrets back. That is a feature when you want control, and a danger when you act without care.
AAccess Control with IAM
Decide exactly who may do what, and nothing more.
The principle of least privilege
The principle of least privilege says every identity should hold only the permissions it truly needs. An application that reads one database password should not be able to read every secret in the project, and it should certainly not be able to delete secrets.
The main roles
| Role | What it allows | Who should have it |
|---|---|---|
| Secret Manager Secret Accessor | Read the payload of versions | Applications that use the secret |
| Secret Manager Secret Version Adder | Add new versions but not read old ones | Rotation jobs and automation |
| Secret Manager Secret Version Manager | Disable, enable, and destroy versions | Secret owners and operators |
| Secret Manager Viewer | See metadata but not payloads | Auditors and support staff |
| Secret Manager Admin | Full control, including permissions | A very small group of administrators |
Grant at the secret level, not the project level
IAM permissions flow downward. If you give an application the accessor role on the whole project, it can read every secret there, including ones added next year. If you give the role on a single secret, the permission reaches only that secret. Narrow grants take a little more work to set up, but they shrink the damage of any mistake or compromise.
Split reading from writing
A rotation job that creates new passwords needs to add versions, but it does not need to read old ones. The application needs to read, but it does not need to add. Splitting these jobs across different identities means that stealing one identity does not give the thief everything.
IAM conditions
Google also supports conditions on role grants. A condition can limit a permission to certain secret name patterns, to certain times, or to a short period. Time-limited access is useful for emergencies: grant a person read access for two hours, and it disappears automatically.
People versus programs
Day to day, people should rarely read production secrets. If an engineer needs to inspect a value during an incident, use short, approved, logged access instead of a standing permission. Programs read secrets all the time, so their identities deserve the narrowest roles and the best protection.
One secret, one grant
Give accessor roles on individual secrets rather than the entire project.
Separate duties
Those who add versions are not the same identities that read them.
Use groups
Grant roles to groups so that team changes are simple and tidy.
Time-limit exceptions
Use conditions for emergency access that expires on its own.
Review regularly
List who has access and remove stale grants every quarter.
Small admin circle
Keep the full admin role to a handful of trusted people.
Giving an application the broad Owner or Editor role “to make it work” also lets it read every secret in the project. Always grant the narrow accessor role instead.
Production example: split duties in a payments team
A payments team runs a nightly job that creates a fresh API token with a payment partner and stores it as a new secret version. That job’s identity holds only the version adder role. The payment service holds only the accessor role. If the job’s identity is ever stolen, the thief can add versions but cannot read the current token. If the service is compromised, the attacker can read the token but cannot change or destroy it.
UUsing Secret Manager: Ways to Work With It
The same safe, opened by several different doors.
The Google Cloud Console
The Console is the web interface. You can create secrets, add versions, see who has access, and view audit information with a few clicks. It is great for learning and for one-time tasks. For safety, remember that anyone who can open the payload in the Console can see the plain text on screen, so avoid sharing screens while doing so.
The gcloud command line
The gcloud tool lets you manage secrets from a terminal and in scripts. There is a command group for secrets, with actions to create a secret, add a version, access a version, list versions, disable, enable, and destroy. Be careful with scripts that print payloads, since terminal output can be saved in history or in build logs.
Client libraries
Google publishes client libraries for popular languages, such as Python, Java, Go, Node.js, and C#. They let your application call Secret Manager directly. A typical program asks for a secret when it starts, holds the value in memory, and uses it to connect to a database or call another service.
The REST and gRPC APIs
Beneath everything, Secret Manager exposes an API that you can call directly. The tools above are friendly wrappers around it.
Infrastructure as code
Tools like Terraform can create the secret container, its labels, replication, and permissions as part of a reviewed plan. A good rule is that code should create the container and permissions, but the sensitive value should be added separately, so that the value never lands in a code repository or a plan file.
A typical day-one workflow
Enable the API
Turn on the Secret Manager API in the project that will hold your secrets.
Create the secret container
Choose a clear name, labels, and a replication policy.
Add the first version
Provide the payload from a secure source, not by typing it into a shared screen or a chat.
Create the application identity
Give the application its own service account.
Grant narrow access
Give that service account the accessor role on this one secret.
Connect the application
Mount the secret, or have the application fetch it at start-up.
Verify and watch
Check that the application works, and confirm that access appears in audit logs.
Naming secrets
Good names make life easier. A common style is to put the application, the environment, and the purpose in the name, such as “payments-prod-db-password”. Names must be unique within a project and use only letters, numbers, dashes, and underscores. Never put the secret value itself, or hints about it, into the name or labels, because metadata is visible to people with the viewer role.
Create a small written convention for names and labels at the start. Consistent names let you search, report, and write permission rules with patterns.
IGetting Secrets into Your Applications
Different platforms offer different ways to hand a secret to your code.
There are two broad styles. In the first, the platform delivers the secret to your application, for example as an environment variable or as a file. In the second, the application fetches the secret itself by calling the API with a client library. Each has strengths.
Cloud Run
Cloud Run is Google’s service that runs containers without servers for you to manage. It can expose a secret to your container either as an environment variable or as a file in a mounted folder. The service account that runs the Cloud Run service needs the accessor role on the secret. When mounted as a file with the “latest” setting, new versions can be picked up, while environment variables are generally fixed when an instance starts. Because exact behaviors can change, confirm the current details in the documentation when you design your rollout.
Google Kubernetes Engine
Kubernetes runs many containers across many machines. Google Kubernetes Engine, or GKE, is the managed version. Kubernetes has its own built-in object for secrets, but by default it only stores them in a lightly protected form. Google offers integrations, such as a Secret Manager add-on that uses the Container Storage Interface driver, which mounts secrets from Secret Manager into pods as files. Combined with Workload Identity, each pod receives a short-lived identity linked to a Google service account, so no key file is needed.
Cloud Functions and App Engine
These serverless platforms also support referencing secrets in their configuration, so that the secret becomes available as a setting at runtime. The runtime identity must hold the accessor role.
Compute Engine virtual machines
A virtual machine is a pretend computer running inside a real one. Programs on it use the attached service account to call the API with a client library, usually at start-up. This is the “application fetches” style.
CI/CD pipelines
A pipeline is an automated factory line that builds, tests, and delivers software. Services such as Cloud Build can read secrets from Secret Manager during a build, so that tokens for publishing or deploying never sit in the pipeline’s definition file. Pipelines outside Google Cloud should use workload identity federation to obtain short-lived access instead of storing a long-lived key.
Outside Google Cloud
A program on another cloud or a company data center can still call the API if it has valid credentials. Prefer federation over key files.
flowchart TD
A["Where does the application run?"] --> B{"Platform feature available?"}
B -- "Cloud Run, Functions, App Engine" --> C["Reference the secret in service configuration"]
B -- "GKE" --> D["Mount through CSI driver with Workload Identity"]
B -- "VM or custom runtime" --> E["Fetch with client library at start-up"]
B -- "Outside Google Cloud" --> F["Federated identity then client library"]
C --> G["Application reads value"]
D --> G
E --> G
F --> G
Environment variable or file?
Mounted file
- Less likely to appear in crash dumps and process listings
- Can be refreshed without restarting in some setups
- Natural fit for certificates and multi-line values
Environment variable
- Very easy for applications to read
- May leak through debug pages and error reports
- Usually fixed until the instance restarts
Fetch once, then cache carefully
Calling the API on every user request wastes time and adds cost. A common pattern is to fetch the secret at start-up and keep it in memory, perhaps refreshing it every several minutes. Never write the value to disk, to logs, or to a shared cache that others can read.
Handing a secret to an application is like giving a delivery driver the gate code. You can write it on a card that the guard hands over at the gate each morning (platform delivery). Or the driver can phone the office and ask each time (application fetch). Either way, the code is not painted on the truck.
Production example: a containerized web shop
A web shop runs on Cloud Run. The container image contains no passwords at all. The service configuration references three secrets: a database password, a payment partner key, and an email service token. Each is mounted using the service’s own service account, which holds the accessor role only on those three secrets. The same image runs unchanged in testing and production; only the secrets differ.
VVersions, Rotation, and Expiry
The healthiest secret is one that changes on a schedule.
What rotation means
Rotation means replacing a secret with a fresh one, then retiring the old one. It limits the damage of a silent leak: even if a thief once copied the old password, it will stop working. It also proves that your systems can survive a change, which is valuable practice before a real emergency.
How Secret Manager helps
Secret Manager does not change the password inside your database for you. Instead, it can store a rotation schedule and publish a message to a Pub/Sub topic when a rotation is due. A small piece of your own automation, such as a Cloud Function or a Cloud Run job, listens for that message, creates the new credential in the target system, and adds it as a new secret version. Think of Secret Manager as the alarm clock and the filing cabinet, and your automation as the worker who actually changes the lock.
A safe rotation recipe
Rotation reminder fires
Secret Manager publishes a message to the topic you chose.
Automation creates the new credential
The job sets a new password or key in the target system. Ideally, the old one stays valid for a while.
Add a new secret version
The job stores the new value as the next version number.
Applications pick it up
Applications read the newest version when they restart or on their next refresh.
Verify health
Check that errors have not risen and that connections use the new credential.
Disable the old version
Switch it off, but do not destroy it yet, in case you need to roll back.
Destroy after a safe wait
When you are confident, destroy the old version, using delayed destruction as a safety net.
stateDiagram-v2
[*] --> Enabled: "Version created"
Enabled --> Disabled: "Disable"
Disabled --> Enabled: "Re-enable"
Enabled --> Destroyed: "Destroy"
Disabled --> Destroyed: "Destroy"
Destroyed --> [*]
The overlap trick
Changing a password in two places at the same instant is nearly impossible. The safe way is an overlap period in which both the old and the new credential work. Applications can switch over gradually, and nothing breaks. After everyone has moved, you retire the old one. Some systems support two active credentials at once, which makes this easy. For others, you might alternate between two accounts, called blue and green.
Expiration and time to live
A secret can be given an expiration time or a time to live, after which Google deletes it automatically. This suits temporary credentials, such as one issued for a short project. It is wonderful for cleanliness and dangerous for forgetfulness, so use it only when the lifetime is truly known.
“Latest” versus a pinned version
| Approach | Benefit | Risk |
|---|---|---|
| Always read “latest” | New values arrive without changing configuration | A bad new version can break all instances at once |
| Pin a specific number | Predictable, easy to roll back, easy to test | You must update the reference during each rotation |
| Use your own alias such as current | Controlled promotion without editing every consumer | Requires discipline to move the alias safely |
Many teams read “latest” in development, where speed matters, and pin versions or use an alias in production, where predictability matters more. Choose one approach per environment and write it down.
Rotating a secret is like changing the locks on a building while people still work inside. You do not swap every lock overnight. You cut new keys, hand them out, keep the old keys working for a short while, confirm everyone has the new one, and only then retire the old lock.
DData Flow and Lifecycle of a Secret
From birth to burial, step by step.
Generation
A strong secret is created, ideally by a tool or a random generator, not invented by a person typing something memorable.
Storage
The secret container is created and the first version is added. The payload is encrypted and stored in the chosen locations.
Distribution
Permissions are granted to specific identities. Platforms or applications fetch the value when needed.
Use
The application holds the value in memory and uses it to connect to a database or call a service.
Monitoring
Audit logs and alerts record and watch access. Unusual reads raise questions.
Rotation
On a schedule, or after any suspicion, a new version replaces the old one.
Retirement
Old versions are disabled and later destroyed. When the application no longer exists, the whole secret is deleted.
Where the value exists along the way
A secret exists in several places during its life: the generator’s memory, the encrypted store, the memory of the Secret Manager service while decrypting, the network connection in transit, and the application’s memory. A good design reduces how many people and systems ever see the plain value. In the best case, no human ever sees it at all, since the generator writes directly into Secret Manager and the application reads it directly from there.
flowchart LR
A["Automated generator"] --> B["New secret version in Secret Manager"]
B --> C["Encrypted at rest in storage"]
C --> D["Authorized application reads over HTTPS"]
D --> E["Value held in application memory"]
E --> F["Used to connect to database or service"]
B --> G["Audit log records the write"]
D --> H["Audit log records the read"]
The best secret is one that is created by a machine, stored by a machine, read by a machine, and replaced by a machine. People then manage the rules, not the values.
AAdvantages, Disadvantages, and Alternatives
An honest look at when Secret Manager is the right tool, and when it is not.
Advantages
- Central control: one managed place for each secret.
- Strong protection: encryption at rest and in transit by default.
- Fine-grained access: permissions per secret and per identity.
- Versioning: safe changes and easy rollback.
- Audit trail: records of who did what.
- Native integration: works with Cloud Run, GKE, Functions, and more.
- Fully managed: no servers to patch or scale.
Disadvantages
- Small payloads only: not meant for large files.
- No automatic password change: rotation needs your own automation.
- Google Cloud centric: less natural for multi-cloud estates.
- Cost per access and version: heavy polling can add up.
- Dependency: an application that fetches at start-up depends on the service being reachable.
- Misconfiguration risk: broad roles can undo the benefits.
Secret Manager versus Cloud KMS
Both belong to Google Cloud’s security family, and beginners often confuse them. Secret Manager is for values your software needs to read, such as a database password. Cloud KMS is for cryptographic keys that stay inside the service. You ask KMS to encrypt, decrypt, or sign something for you, and the key itself is never handed out. They also cooperate: Secret Manager can use a KMS key to protect its stored secrets.
| Option | Best for | Weak spot |
|---|---|---|
| Secret Manager | Passwords, API keys, tokens, certificates that applications must read | Small size limit, Google Cloud focus |
| Cloud KMS | Encrypting data or signing without exposing the key | Not a place to store arbitrary values for reading |
| Environment variables set by hand | Quick tests | No versioning, weak audit, easy to leak |
| Kubernetes secrets alone | Simple in-cluster needs | Weaker default protection and limited audit |
| Self-hosted vault tools such as HashiCorp Vault | Multi-cloud, dynamic short-lived credentials, advanced features | You run and secure the system yourself |
| Other clouds’ secret services | Workloads living in those clouds | Different identity model and tooling |
Secret Manager is the staff key cabinet: employees borrow a key, use it, and the cabinet notes who took it. Cloud KMS is the bank’s cash-counting machine: you hand over your money and instructions, the machine does the work inside a locked box, and nobody ever takes the machine’s inner workings home.
Large secrets
If the thing you must protect is large, such as a big configuration file, the usual pattern is to encrypt the file with a key from Cloud KMS and keep the encrypted file in Cloud Storage, while keeping only a small reference or the small wrapping secret in Secret Manager. This keeps each tool in the job it does best.
SSecurity: Hardening the Safe
The cabinet is strong, but the way you use it decides the outcome.
Audit logs
Google Cloud writes audit logs, which are records of who did what and when. Administrative actions, such as creating secrets or changing permissions, are logged by default. Reading the payload of a secret is a data access event, and data access logs usually need to be switched on deliberately for the Secret Manager service. Turn them on for sensitive projects, so you can answer questions such as “Which identity read this token last Tuesday at three in the morning?” The logs record the fact of access, not the secret value.
Alerts on unusual access
Logs are only useful if somebody looks. Create alerts for patterns that deserve attention: a sudden burst of reads, access from an identity that never read the secret before, permission changes on a sensitive secret, or a version destroyed outside business hours.
Customer-managed encryption keys
For regulated industries, CMEK gives you control of the key that protects your secrets. You can rotate it, restrict who may use it, and disable it in an emergency. Treat the key as at least as important as the secrets it protects.
VPC Service Controls
VPC Service Controls draw a protective boundary around Google Cloud services in your projects. Even a person with the right IAM permission can be blocked from reading secrets when the request comes from outside the boundary. This reduces the risk of stolen credentials being used from the open internet.
Organization policies
Administrators can set organization policies, company-wide rules that apply to all projects below them. Examples include limiting the locations where resources may be created, which also affects secret replication, and requiring customer-managed keys for supported services.
Protecting the identities that read secrets
A secret is only as safe as the identity that can read it. Avoid downloaded service account key files. Prefer attached service accounts, Workload Identity on Kubernetes, and federation for outside systems. Give each application its own service account so that you can see and limit its behavior.
Keeping secrets out of the wrong places
- Do not print secret values in logs or error messages.
- Do not pass secrets as command-line arguments, which can be visible in process listings.
- Do not put secrets in container images, build arguments, or version control.
- Do not share secrets in tickets, chats, or screen shares.
- Scan code repositories automatically for accidentally committed secrets.
Audit logs
Turn on data access logs and review them for unusual reads.
Alerts
Notify the team about bursts, new readers, and permission changes.
CMEK
Control the encryption key when regulations or policy require it.
Service controls
Block reads from outside the trusted boundary.
Organization policy
Apply company-wide guardrails on locations and keys.
Keyless access
Use attached identities and federation instead of key files.
Granting the accessor role to an entire team group on a production secret “for convenience”. Convenient access is also convenient for attackers. Keep production readers to the application identity and a tiny break-glass group.
Production example: break-glass access
A financial services team gives nobody standing access to production secrets. During a serious incident, an on-call engineer requests access, a second person approves, and a time-limited grant is created that expires after two hours. Every read appears in the audit log, and the access-review meeting next morning goes through each event. Emergencies stay possible, and silent misuse stays very hard.
MMonitoring, Logging, and Troubleshooting
When something breaks, ask the right questions in the right order.
What to watch
Cloud Monitoring can track how many requests your project makes to Secret Manager and how many fail. Cloud Logging holds the audit entries. Pub/Sub notifications tell you about secret events such as new versions. Combine them to answer three everyday questions: Is access working? Is anyone accessing it unusually? Did anything change that should not have?
A troubleshooting checklist
Is the API enabled?
Check that Secret Manager is switched on in the project that owns the secret.
Which identity is calling?
Find the exact service account the running application uses, not your own account.
Does that identity hold the accessor role?
Check the role on the specific secret and on its parent project and folders.
Does the version exist and is it enabled?
A disabled or destroyed version returns an error, as does a typo in the number.
Is the secret in the project you think it is?
Secrets are named inside a project. The wrong project returns “not found”.
Is a network rule or perimeter blocking you?
VPC Service Controls and proxy rules can reject otherwise valid requests.
Is a customer-managed key available?
If the encryption key is disabled or permissions to use it are missing, reads fail.
| Symptom | Likely cause | First thing to try |
|---|---|---|
| Permission denied | Missing accessor role or wrong identity | Check the service account and the grant on this secret |
| Not found | Wrong project, name, or version | List secrets and versions in the intended project |
| API not enabled error | Service switched off | Enable Secret Manager in the project and retry shortly |
| Version in a disabled or destroyed state | Rotation retired it too early | Re-enable if disabled, or point to a newer version |
| Works on laptop, fails in production | Different identity | Compare the identities and their roles |
| Suddenly failing across all instances | Bad new “latest” version or revoked permission | Check recent version and audit entries, then roll back |
| Slow start-up | Too many secret fetches in sequence | Fetch only needed secrets, in parallel where possible |
| Quota exceeded | Reading on every request | Cache the value in memory |
Investigating a suspected leak
If you think a secret has leaked, act in a calm order. First, add a new version so that a fresh credential exists. Second, switch applications to it and retire the old credential in the target system. Third, disable and later destroy the old version. Fourth, review audit logs to see who read the secret and when. Fifth, find how it leaked and fix that cause. Replacing the credential comes before detective work, because every minute the old one remains valid is a minute of risk.
If an application works with one secret but fails with another, compare the grants on the two. The usual difference is a missing role on the second secret, not a problem with the application.
PPerformance, Availability, Quotas, and Cost
Practical limits and how to live comfortably within them.
Speed and availability
Secret Manager is a managed service designed for high availability, with replicated storage across locations. A single read is a fast network call, but it is still a network call. If your application fetches secrets many times per second, you add latency and load for no benefit. The usual approach is to fetch once at start-up, keep the value in memory, and refresh it periodically.
What if Secret Manager is unreachable?
An application that fetches a secret at start-up cannot start if it cannot reach the service at that moment. Two habits reduce this risk. First, make start-up retry with growing pauses, a technique called exponential backoff. Second, for a running application, keep the last known value in memory so that a brief outage does not stop work. Platform features that mount secrets also have their own behaviors during outages, so test them.
Quotas and size limits
Google sets quotas, which are limits on how many requests you can make per minute and on how many secrets and versions you can hold. Each secret version payload has a size limit of tens of kilobytes. The exact numbers can change, so read the current quota page before large rollouts, and request increases early if you expect heavy use.
How the bill works
Pricing details change over time, so use Google’s pricing page as the authority. In general, the cost model has three parts: a charge for each active secret version stored per location per month, a charge for access operations beyond a free allowance, and a charge for rotation notifications. The practical conclusions are easy to remember. Do not keep hundreds of old versions you no longer need. Do not read the same secret on every request. And note that replicating to many regions multiplies the stored copies.
flowchart TD
A["Application starts"] --> B["Fetch secret with retry and backoff"]
B --> C{"Fetch succeeded?"}
C -- "Yes" --> D["Keep value in memory"]
C -- "No after retries" --> E["Fail start-up and alert team"]
D --> F["Serve traffic"]
F --> G["Refresh value every few minutes"]
G --> H{"Refresh succeeded?"}
H -- "Yes" --> D
H -- "No" --> I["Keep last known value and log a warning"]
I --> G
You do not phone the bank every time you need to know your account number. You write it in your own notebook once, and you check with the bank only when the number changes. Fetching once and caching in memory is that notebook habit.
Memory is the only safe place to cache. Never write secrets to disk caches, shared files, or logs for the sake of speed.
DDesign Patterns and Anti-Patterns
Habits that help, and habits that hurt.
Helpful patterns
One service account per application
Gives clear access, clear audit trails, and clear blast radius.
Per-secret grants
Permit only the exact secrets an application uses.
Separate project for secrets
Keep sensitive secrets in a locked-down project with its own tight rules.
Automated rotation
Use scheduled notifications and automation to change values regularly.
Overlap then retire
Keep old and new valid briefly, then disable and destroy the old one.
Generate and store by machine
Let tools create values and write them straight into Secret Manager.
Anti-patterns to avoid
What it looks like
Using Secret Manager but also keeping the same secret copied in a configuration file or a team document “just in case”.
Why it hurts
The extra copies leak, drift out of date, and defeat the purpose of a single protected home.
Do this instead
Keep exactly one source of truth and delete every other copy.
What it looks like
Granting the accessor role at the project level to every service.
Why it hurts
Any compromised service can read every secret, including ones it never needed.
Do this instead
Grant the role on individual secrets, one application identity at a time.
What it looks like
Reading the secret from the API on every incoming request.
Why it hurts
It adds latency, risks quota errors, and increases cost.
Do this instead
Fetch at start-up, hold in memory, and refresh on a calm schedule.
What it looks like
Never rotating, and keeping the same password for years because “it works”.
Why it hurts
A secret that leaked long ago still opens the door today, and nobody knows how to change it safely.
Do this instead
Rotate on a schedule so that changing becomes a routine, practiced skill.
What it looks like
Storing the secret value inside infrastructure-as-code files, plan outputs, or container build arguments.
Why it hurts
Those files are copied, shared, and logged widely, and the value spreads far beyond the safe.
Do this instead
Create the container and permissions in code, and add the value separately through a controlled process.
What it looks like
Using a downloaded service account key to reach Secret Manager from outside Google Cloud, then storing that key in a file next to the application.
Why it hurts
The key becomes the new secret that guards all other secrets, and it is stored in the open.
Do this instead
Use workload identity federation or an attached identity so there is no key file.
BBest Practices and Common Mistakes
A short, practical rulebook you can pin to the wall.
Best practices
- Treat every secret as temporary. Plan its replacement from the day you create it.
- Use least privilege everywhere. Narrow roles, narrow scope, short duration.
- Give each application its own identity. Shared identities blur responsibility.
- Prefer keyless authentication. Attached identities, Workload Identity, and federation.
- Turn on data access audit logs for sensitive projects and review them.
- Alert on unusual access. Bursts, new readers, permission changes, and destructions.
- Pin versions or use aliases in production. Avoid surprises from a bad “latest”.
- Disable before destroying. Keep a way back until you are certain.
- Use delayed destruction as a safety net against accidents.
- Label and name consistently. Organization pays off in every audit.
- Scan code and build logs for leaks. Catch mistakes before attackers do.
- Practice rotation and recovery. A drill in calm weather prevents panic in a storm.
Common mistakes beginners make
Mistakes
- Forgetting to enable the API
- Granting access to the wrong service account
- Giving broad Owner or Editor roles
- Printing secrets in logs while debugging
- Destroying a version still in use
- Deleting a customer-managed key without checking what depends on it
- Putting hints about the secret in its name or labels
Better habits
- Follow the day-one workflow in order
- Check which identity the running app really uses
- Grant the single accessor role on single secrets
- Mask values in logs and never print payloads
- Disable first, watch, then destroy
- Map dependencies before touching any key
- Keep names descriptive but never revealing
Secret Manager protects the storage and delivery of secrets. It cannot protect a secret that your own application prints, forwards, or stores elsewhere. Good tools need good habits around them.
RReal-World Usage Patterns
How different teams put the safe to work.
The startup with a small team
A five-person startup runs its product on Cloud Run. They keep about a dozen secrets: a database password, an email service key, and a few partner tokens. Each secret has its own grant to the single service account of the service that uses it. Rotation is a calendar reminder at first, then a small scheduled job once they have time. Their biggest win is simple: no password ever appears in the repository, so new engineers can clone the code on day one without receiving any sensitive information.
The e-commerce platform at scale
Large online retailers run hundreds of services, each with its own set of credentials for databases, payment providers, and shipping partners. A platform team defines a standard: every service gets a dedicated service account, secrets are named by service and environment, and a template creates secrets with labels and the right grants. Rotation notifications trigger automation that rotates database users on a schedule. Security teams review audit logs for unusual reads, especially around sales events.
The financial services team with strict rules
A bank-style team stores secrets with customer-managed encryption keys, in regions allowed by its regulators, inside a perimeter built with VPC Service Controls. Nobody holds standing read access to production secrets. During incidents, a break-glass process grants time-limited access with a second approver, and every read is reviewed the next day. The extra steps feel heavy, but they turn a feared audit into a routine report.
The data team with pipelines
A data team runs nightly jobs that call external data providers. Each provider token is a secret, read by a job’s identity. When a provider asks customers to rotate tokens, the team adds a new version and the next night’s job picks it up automatically. No pipeline definition or notebook ever contains a token, so sharing notebooks with colleagues is safe.
The incident that was caught early
An alert on unusual access fires at night: a service account that normally reads one secret once a day has made hundreds of reads from an unfamiliar network. The on-call engineer adds a new secret version, rotates the credential in the target system, disables the old version, and investigates. Because audit logs, narrow permissions, and rotation were already in place, the incident closes in an hour with no data loss.
flowchart TD
A["Platform team defines secret standards"] --> B["Template creates secret, labels, and grants"]
B --> C["Service reads secret through its own identity"]
C --> D["Audit logs and alerts watch access"]
D --> E["Rotation notification fires on schedule"]
E --> F["Automation adds new version and retires old one"]
F --> C
D --> G["Unusual access triggers incident response"]
G --> E
Notice what these stories share. Secret Manager is not the whole defense. It is the foundation that makes the other defenses possible: narrow permissions, audit trails, rotation, and fast recovery. The tool is simple. The habits around it create the safety.
FFrequently Asked Questions
Quick answers to the questions beginners ask most.
It is a Google Cloud service that stores small sensitive values, such as passwords and tokens, encrypts them, controls who can read them, and keeps versions and audit records.
A secret is the named container with settings and permissions. A secret version holds one actual value. You add new versions to change what the secret contains.
No. Versions are immutable. To change the value, add a new version, then disable or destroy the old one when you are ready.
Its payload is permanently erased and cannot be recovered. If you set delayed destruction, you have a waiting period during which the destruction can be cancelled.
The Secret Accessor role, granted on the specific secret it needs, to the application’s own service account. Avoid broad roles such as Owner or Editor.
Not by itself. It can send a notification on a schedule, and your own automation then changes the credential in the target system and adds a new version.
Latest is convenient, but a bad new version can affect every instance at once. Many teams pin versions or use an alias in production and use latest in development.
Secret Manager stores values that applications must read. Cloud KMS holds cryptographic keys that stay inside the service and performs encryption or signing for you. They can work together.
They are common and often acceptable, but they can leak through crash reports and debugging pages. Mounted files are often preferred. In both cases, the value should come from Secret Manager, not from a hand-typed setting.
No. Each version has a small size limit. For large data, encrypt it with a key in Cloud KMS and store the encrypted file elsewhere, such as Cloud Storage.
Yes. They are encrypted at rest and in transit by default. You may also choose customer-managed encryption keys if you need more control.
Access is controlled by Google’s security and compliance practices, and you can add controls such as customer-managed keys and access logging. For the exact commitments, read Google’s current security and compliance documentation.
Pricing changes, so check the official pricing page. In general, you pay for active secret versions per location, for access operations beyond a free allowance, and for rotation notifications.
Yes, any program that can reach the API with valid credentials can use it. Prefer workload identity federation to downloaded key files.
Create a new credential, add it as a new version, switch applications to it, retire the old credential in the target system, disable then destroy the old version, and review audit logs to understand what happened.
SSummary and Key Takeaways
Everything important, in one place.
Key Takeaways
- Secret Manager is a managed safe for small sensitive values. It encrypts them, controls access with IAM, keeps versions, and records activity in audit logs.
- A secret is a container and versions hold the values. Versions are immutable, so you add a new one instead of editing, and you can enable, disable, or destroy them.
- Grant the narrowest access possible. Give each application its own service account and the accessor role on only the secrets it needs. Keep admin rights to a tiny group.
- Prefer keyless identities. Use attached service accounts, Workload Identity, and federation, so that no key file becomes the new secret to protect.
- Deliver and cache with care. Mount or fetch at start-up, hold values in memory, refresh calmly, and never write them to disk, logs, or code.
- Rotate on a schedule. Secret Manager sends the reminder, your automation changes the credential, and an overlap period keeps everything running while you switch.
- Watch and plan. Turn on data access logs, alert on unusual reads, choose replication and encryption early, and rehearse your response to a leak before you need it.