AWS Secrets Manager – Rotation, Versioning and the Life of a Secret
A deep look at how AWS Secrets Manager actually versions, rotates, and delivers credentials — the staging labels, rotation lifecycle, and access patterns that separate a secrets strategy that survives an audit from one that doesn't.
Picture a bank vault that never actually hands anyone the master key — instead, every time someone needs access, the vault mints a brand-new, single-purpose key, quietly retires the previous one on a schedule, and keeps a perfect written log of every single request. That’s a fair mental model for AWS Secrets Manager. Most engineers know it as “the place credentials live instead of a config file.” Fewer understand how it actually tracks multiple versions of the same secret simultaneously, how automatic rotation avoids ever having a moment where no valid credential exists, or how its access model interacts with KMS and IAM in ways that shape real production design. This tutorial goes past “store your secret here” and into the internal lifecycle that makes Secrets Manager trustworthy enough to hold a database password nobody ever has to see.
1Core Concepts Beyond the Basics
Once you know Secrets Manager “stores credentials securely,” the next layer of understanding is that a secret is never really just one value — it’s a small, versioned history.
Every secret in Secrets Manager is actually a container for multiple versions, each holding its own encrypted value, and each tagged with one or more staging labels that describe its role. Applications don’t fetch “the secret” in the abstract — they fetch the version currently carrying the label AWSCURRENT. This design exists specifically so that rotation can prepare a brand-new credential without disturbing the one still in active use, until the moment the switch is deliberately made.
Think of staging labels like sticky notes on a set of house keys hanging in a key box. One key has a note saying “use this one” (AWSCURRENT), another might have a note saying “new key, not handed out yet” (AWSPENDING), and an old one might say “just retired, keep for now” (AWSPREVIOUS). The physical keys don’t change — only which sticky note is attached to which key changes, and that’s what determines what gets handed to whoever asks.
Secrets versus parameters: a distinction worth being precise about
AWS Systems Manager Parameter Store can also store sensitive strings, and the two services are frequently confused. Parameter Store is a general-purpose key-value configuration store with an optional secure string type; Secrets Manager is purpose-built specifically around the lifecycle of credentials — native rotation, versioning with staging labels, and direct integration with services like Amazon RDS. Choosing between them is really a question of whether you need that specific lifecycle machinery or just an encrypted value.
A secret’s value can be structured JSON containing several key-value pairs (username, password, host, port) rather than a single string — this is the default shape for database credentials and is what native rotation functions expect to work with.
2Architecture and Core Components
A handful of distinct pieces work together every time a secret is created, read, or rotated.
Secret
A named resource holding metadata, a resource policy, and a history of versions — never the plaintext value itself at rest.
Version
An individual, encrypted snapshot of a secret’s value, identified by a unique version ID and tracked with staging labels.
Rotation Lambda Function
A function, either AWS-provided or custom, that implements the four-step rotation process described in Chapter 3.
AWS KMS Key
Every secret is encrypted with a KMS key, either the AWS-managed default or a customer-managed key for tighter control over who can decrypt it.
graph TD
App[Application] -->|GetSecretValue| SM[Secrets Manager]
SM --> KMS[AWS KMS Key]
SM --> V1[Version: AWSCURRENT]
SM --> V2[Version: AWSPREVIOUS]
SM --> V3[Version: AWSPENDING]
Rotator[Rotation Lambda] --> SM
Rotator --> DB[(Target Database)]
Resource policies: access control attached to the secret itself
Beyond IAM policies attached to users and roles, a secret can carry its own resource-based policy, defining exactly which principals — including principals in other AWS accounts — may access it. This dual model (identity-based and resource-based) is what makes cross-account secret sharing possible without duplicating the secret itself into every account that needs it.
3Internal Working: The Four-Step Rotation Process
Rotation isn’t a single atomic swap — it’s a deliberate, staged sequence designed so a failure at any step never leaves an application without a working credential.
createSecret
The rotation function generates a brand-new credential value and stores it as a new version, labeled AWSPENDING — the old AWSCURRENT version is untouched.
setSecret
The function applies the new credential to the actual target system — for example, creating the new password inside the database engine itself.
testSecret
The function verifies the new credential actually works by attempting a real connection or operation against the target system using the AWSPENDING version.
finishSecret
Only after a successful test does the function move the AWSCURRENT label onto the new version, and the old version becomes AWSPREVIOUS.
People sometimes assume rotation deletes the old credential immediately. It doesn’t — the AWSPREVIOUS version is deliberately kept around specifically to support rollback and to tolerate a brief window where some application instances may still be caching the old value.
Why the test step exists at all
Without the testSecret step, a rotation function could label a broken credential as AWSCURRENT and every application using it would start failing simultaneously. Requiring a successful real-world verification before the label ever moves is the single design decision that makes automated, unattended rotation safe enough to run on a schedule without a human watching.
Native versus custom rotation
AWS provides ready-made rotation Lambda functions for RDS, Redshift, and DocumentDB credentials that already implement this four-step process correctly for those engines. For anything else — a third-party API key, a custom application credential — teams write their own rotation function following the same four-step contract, since Secrets Manager itself only orchestrates the steps; it doesn’t know how to talk to an arbitrary external system.
4Data Flow and Lifecycle
Following a single secret retrieval, and then a rotation event, shows how these two independent flows interact.
Retrieval flow
An application calls GetSecretValue, optionally specifying a version stage; if none is given, AWSCURRENT is returned by default. Secrets Manager checks the caller’s IAM policy and the secret’s resource policy, decrypts the requested version using its associated KMS key, and returns the plaintext value over an encrypted connection — the plaintext is never persisted anywhere by the service itself outside of that response.
sequenceDiagram
participant App as Application
participant SM as Secrets Manager
participant KMS as AWS KMS
App->>SM: GetSecretValue (AWSCURRENT)
SM->>SM: Check IAM policy + resource policy
SM->>KMS: Decrypt version
KMS-->>SM: Plaintext value
SM-->>App: Return secret value
Rotation lifecycle over time
On a configured schedule, Secrets Manager invokes the rotation function, which runs the four steps from Chapter 3. Applications that call GetSecretValue for AWSCURRENT immediately after a successful rotation get the new credential automatically — no code deployment, no restart, no manual coordination required, since the label move is what redirects future requests.
Client-side caching changes this picture
To avoid an API call on every single secret use, applications typically use a caching client (such as the AWS-provided Secrets Manager caching libraries) that holds the secret in memory for a configurable interval. This means there’s a real, bounded window after rotation during which some application instances may still be using the AWSPREVIOUS value — a window that connection retry logic and the kept-around AWSPREVIOUS version are both specifically designed to tolerate.
5Advantages, Disadvantages and Trade-offs
Secrets Manager’s rotation and versioning machinery is powerful specifically because it’s opinionated — and that opinion isn’t free.
Advantages
- Native, tested rotation integrations for common database engines remove a large class of manual credential-rotation toil.
- Staging labels make rotation safe by design, never leaving a moment with no valid credential.
- Resource policies enable clean cross-account secret sharing without duplicating values.
- Every access and change is recorded in CloudTrail automatically, supporting audit requirements.
- Encryption via KMS is mandatory and automatic — there’s no accidental unencrypted storage path.
Disadvantages / Trade-offs
- Costs a monthly fee per secret plus a per-API-call charge, which adds up for very large numbers of small secrets.
- Custom rotation functions require real engineering effort to write and test correctly for non-native systems.
- Client-side caching, while necessary for cost and latency, introduces a window of potential staleness after rotation.
- Cross-region access requires explicit multi-region replication configuration; it isn’t automatic.
The trade-off in one sentence
Secrets Manager trades the simplicity and lower cost of a plain encrypted key-value store for a genuinely safer, auditable credential lifecycle — a trade that pays for itself the first time a rotation event would otherwise have required a coordinated deployment across dozens of services.
6Performance and Scalability
Secrets Manager scales well as a store, but its API request model means the biggest performance lever lives on the client side, not the service side.
API request limits and why caching isn’t optional
GetSecretValue has an account-level request rate limit shared across all secrets. An application fetching a secret on every single database connection, rather than caching it, can realistically hit that limit under real production load — this is precisely why AWS ships caching client libraries for common languages instead of leaving every team to build their own.
Scaling secret count, not just secret usage
Organizations with hundreds of microservices sometimes end up with thousands of secrets, and the per-secret monthly cost becomes a real line item worth optimizing — for example, by consolidating multiple related credentials into one structured JSON secret rather than creating a separate secret per field, where that pattern genuinely fits the access model.
Pin application code to a specific version stage rather than always assuming AWSCURRENT is instantaneous everywhere — during high-throughput rotation events, staggering when different service fleets pick up the new label reduces the chance of an entire fleet failing at once if something is subtly wrong.
7High Availability and Reliability
Secrets Manager is a regional, highly available service by default, but disaster recovery across regions requires an explicit decision.
Regional durability
Within a region, Secrets Manager is built as a highly available, multi-AZ managed service — there’s no instance or capacity to provision, and no single Availability Zone failure should interrupt secret retrieval or rotation for a properly configured secret.
Multi-Region Secrets for cross-region resilience
For applications that fail over across AWS regions, Secrets Manager supports replicating a secret to one or more secondary regions, keeping the replica’s value automatically synchronized with the primary. This is what allows a disaster-recovery region to read the same credential locally, with low latency, rather than making a cross-region API call during a failover event when every millisecond and every dependency matters.
Assuming a secret is automatically available in a disaster-recovery region just because the application is deployed there. Without explicit multi-region replication configured, the secret simply doesn’t exist in that region, and the failover plan silently breaks at the worst possible moment.
Rotation failure handling
If a rotation function fails partway through — say, the new credential is created but the test step fails — Secrets Manager leaves AWSCURRENT untouched, so applications keep working with the old, still-valid credential while the failure is investigated. This graceful-failure behavior is a direct consequence of the staged, label-based design covered in Chapter 3.
8Security
Secrets Manager layers three independent controls on every access: identity policy, resource policy, and encryption key policy — all three have to agree.
| Control | Attached To | Decides |
|---|---|---|
| IAM Identity Policy | User or role | Whether this principal can call Secrets Manager actions at all |
| Resource Policy | The secret itself | Which principals, including cross-account ones, may access this specific secret |
| KMS Key Policy | The encryption key | Whether this principal is allowed to use the key needed to decrypt the value |
Conditional access with IAM policy conditions
IAM policies attached to roles can include conditions scoping access further — for example, restricting access to secrets tagged with a specific project, or requiring the request to originate from within a specific VPC endpoint. Combined with resource policies, this supports fine-grained patterns like “only the payments service role, only from within the payments VPC, may ever read this particular secret.”
Private access via VPC endpoints
A VPC interface endpoint for Secrets Manager, powered by PrivateLink, lets resources in private subnets retrieve secrets without any traffic touching the public internet or needing a NAT gateway — consistent with the same VPC endpoint pattern used across many other AWS services.
Problem
Granting a broad secretsmanager:GetSecretValue permission on all resources (*) to an application role, rather than scoping it to the specific secret ARN the application actually needs.
Why It’s Harmful
A single compromised application role with wildcard access can read every secret in the account, turning one vulnerability into a full credential-exposure incident across unrelated systems.
Correct Approach
Scope IAM policies to specific secret ARNs or tag-based conditions, and pair that with a resource policy on sensitive secrets that explicitly enumerates the few roles allowed to read them.
9Monitoring, Logging and Metrics
Because secrets are high-value targets, monitoring here is less about performance and mostly about accountability — knowing exactly who touched what, and when.
AWS CloudTrail
Records every GetSecretValue, PutSecretValue, and administrative API call, including the calling principal — the primary tool for answering “who read this secret and when.”
CloudWatch Events / EventBridge
Emits events on rotation success or failure, which can trigger alerts so a failed rotation is caught immediately rather than discovered when a credential later expires.
CloudWatch Metrics
Tracks API call volume, useful for spotting an application that isn’t caching correctly and is hammering the service far more than expected.
Setting up automatic rotation and never configuring an alert for rotation failure. A silently failing rotation function means a credential simply stops rotating, quietly reintroducing the exact long-lived-credential risk the whole system was built to avoid.
10Deployment and Cloud Integration
Secrets Manager is rarely used in isolation — its value shows up most clearly in how tightly it integrates with the rest of the deployment pipeline.
Dynamic References
CloudFormation and similar tools can reference a secret’s current value at deployment time without ever hardcoding it into a template file.
ECS / EKS Secret Injection
Container task definitions can pull a secret’s value directly into an environment variable at container start, keeping it out of image layers and source code.
Resource Policies + RAM patterns
A shared services account can host secrets that multiple application accounts read via resource policy, avoiding duplicated credential values across accounts.
Choosing Secrets Manager versus Parameter Store in a real deployment
A practical rule many teams settle on: use Secrets Manager for anything that needs rotation, cross-account sharing, or is a genuine credential (database passwords, API keys); use Parameter Store for general configuration values and non-rotating secrets where the lower cost matters more than the extra lifecycle machinery.
11Design Patterns and Anti-patterns
A small number of recurring patterns account for the majority of well-run Secrets Manager deployments.
Pattern: Structured JSON Secrets
Storing an entire connection profile (username, password, host, port, database name) as one JSON secret, so a single rotation event updates everything an application needs in one atomic label move.
Pattern: Least-Privilege Resource Policies
Attaching a resource policy that explicitly names the handful of roles allowed to read a sensitive secret, rather than relying on identity policies alone to carry the entire access decision.
Pattern: Caching Client with Bounded TTL
Using a caching library with a short, deliberate refresh interval balances API cost against how quickly a fleet picks up a newly rotated credential.
Problem
Building a custom rotation Lambda that skips the testSecret step, or treats it as an optional formality, to make development faster.
Why It’s Harmful
Without a real verification step, a broken new credential can be promoted to AWSCURRENT, and every application depending on that secret starts failing at once, exactly the outage scenario the four-step design exists to prevent.
Correct Approach
Always implement a genuine connectivity or functional test against the target system using the AWSPENDING credential before allowing finishSecret to run, even if it adds development time upfront.
12Best Practices and Common Mistakes
Most Secrets Manager problems in production trace back to a handful of predictable oversights rather than unusual edge cases.
Advantages
- Scope IAM and resource policies to specific secret ARNs rather than wildcards.
- Alert on rotation failure events, not just on successful rotations.
- Use a caching client library instead of calling GetSecretValue on every operation.
- Enable multi-region replication explicitly for any secret needed in a disaster-recovery region.
Disadvantages / Trade-offs
- Hardcoding a specific version ID instead of using AWSCURRENT, silently breaking automatic rotation pickup.
- Creating one secret per individual field instead of a structured JSON secret, multiplying both cost and rotation complexity.
- Skipping resource policies on sensitive, cross-account-relevant secrets and relying on identity policies alone.
- Forgetting that customer-managed KMS keys need their own key policy granting the application role decrypt permission, separate from the secret’s own policy.
Testing a rotation manually in a non-production secret before enabling a schedule in production catches most rotation function bugs long before they can cause a real outage.
13Real-world and Industry Examples
The organizations that get the most value from Secrets Manager are usually the ones that had a painful manual-rotation incident before adopting it.
Financial Services Credential Rotation Compliance
Regulated institutions frequently use Secrets Manager’s scheduled rotation specifically to satisfy compliance requirements mandating periodic credential changes, with CloudTrail logs serving as the audit evidence.
SaaS Platforms with Per-Tenant Database Credentials
Multi-tenant SaaS platforms managing a separate database credential per tenant use structured JSON secrets and native RDS rotation to keep hundreds or thousands of tenant credentials rotating on schedule without manual intervention per tenant.
Enterprises Consolidating Third-Party API Keys
Enterprises integrating with many external SaaS APIs use Secrets Manager as a central store for those API keys, paired with custom rotation functions where the third party supports programmatic key rotation.
14Frequently Asked Questions
It isn’t deleted — it’s relabeled AWSPREVIOUS and kept, both to support rollback and to tolerate any application instance that hasn’t yet refreshed its cached copy of the secret.
The design is built to avoid that: AWSCURRENT only moves to the new version after the testSecret step succeeds. If any step fails earlier, the old credential remains AWSCURRENT and applications continue working with it.
No — AWS provides ready-made rotation functions for common database engines like RDS, Redshift, and DocumentDB. A custom function is only needed for credentials outside those native integrations, such as third-party API keys.
No — secrets are regional by default. Cross-region availability requires explicitly configuring multi-region replication for that specific secret.
Use Secrets Manager for genuine credentials needing rotation or cross-account sharing; Parameter Store is usually a better fit for general configuration values or secrets that don’t need automatic rotation, largely due to cost differences.
Through three layered checks that all must pass: the caller’s IAM identity policy, the secret’s own resource policy, and the KMS key policy governing decryption — a gap in any one of the three still results in denied access.
15Summary and Key Takeaways
AWS Secrets Manager’s real value isn’t just encrypted storage — plenty of services offer that. It’s the versioned, staged-label model that makes rotation safe to automate, and the layered access model that makes sharing a secret across accounts and services auditable rather than ad hoc. Understanding staging labels, the four-step rotation contract, and how identity policies, resource policies, and KMS key policies must all agree turns Secrets Manager from “a place to hide passwords” into a genuine credential lifecycle system that a security team can actually stand behind during an audit.
Key Takeaways
- A secret is a versioned history, not a single value — staging labels like AWSCURRENT and AWSPENDING decide what applications actually receive.
- Rotation follows a strict four-step contract — createSecret, setSecret, testSecret, finishSecret — designed so a failure never removes a working credential.
- Access requires three layers to agree: IAM identity policy, the secret’s resource policy, and the KMS key policy.
- Client-side caching is expected, not optional — it’s what keeps API usage sustainable, at the cost of a bounded staleness window after rotation.
- Multi-region availability is opt-in — a secret used in a disaster-recovery region needs explicit replication configured ahead of time.
- Structured JSON secrets simplify rotation for multi-field credentials like full database connection profiles.
- Every access is logged in CloudTrail, making Secrets Manager a strong fit for environments with real audit and compliance obligations.