Three ideas, put together: authentication, credential, and limited access. Once each is clear, the rest follows naturally.

Elasticsearch is a search and data engine. Programs talk to it over the network to store documents and to ask questions. Because the data is valuable, Elasticsearch has to answer two questions for every request. The first is authentication: “who are you?” The second is authorization: “are you allowed to do this particular thing?” A credential is the proof you show in answer to the first question. A username with a password is one kind of credential. An API key is another kind.

An API key is a credential that Elasticsearch itself creates and remembers. When you ask for one, Elasticsearch generates a random secret, gives the key an identifier, records the key in its own security storage, and shows you the secret one time. From then on, any program that presents the id and the secret together is recognized as holding that key. The program does not need a username, and it does not need anyone’s password.

The most useful part is the third idea: limited access. When you create a key, you can describe exactly what it may do, such as “write documents into the logs indexes and nothing else.” You can also give it an expiry time. And later you can cancel just that one key without touching anything else.

What a key is made of

The one-sentence version

If you remember one line from this chapter, make it this: an Elasticsearch API key is a revocable, optionally expiring, permission-limited credential, created and verified by Elasticsearch itself, that lets a program authenticate without using anyone’s password.

Every security feature is born from a painful situation. Here is the one that produced API keys.

Picture a company with fifty small programs that all need Elasticsearch: a website search box, an order tracker, three log shippers, a nightly report, and many more. The quick way to connect them is to create one user and give every program the same username and password. It works on day one, and then the trouble begins.

• Too much power. That shared user often has broad rights, because it must satisfy the needs of every program. Each program therefore holds far more power than it uses.
• No way to tell them apart. When something odd happens in the logs, every program looks like the same user. You cannot tell which one did it.
• Painful rotation. If the password leaks, you must change it, and then every single program breaks at once until each is updated.
• Tied to a person. Sometimes the password belongs to an actual employee. When that person leaves, or is forced to change the password, production stops.
• No expiry. Passwords live until someone remembers to change them, which in practice means years.

API keys solve these problems together. Each program gets its own key with only the rights it needs. Each key has its own id, so logs show which key did what. A leaked key is cancelled by itself, and the other forty-nine programs carry on. A key can be given an end date at birth, so forgotten credentials clean themselves up.

A short history

The exact release for each step is worth double-checking in the release notes of your version. The direction of travel is what matters: from “programs borrow a person’s identity” toward “programs carry their own small, limited identity.”

Why not just make a new user for every program?

You can, and some teams do. A user needs a password, a role, and a home in some authentication system, and creating many of them is heavy work. API keys are lighter. They can be created by a program, in seconds, on demand, with permissions attached at creation. They fit the way modern systems create and destroy small workers all the time.

Three small stories about shared passwords

These stories are invented, but every security team has seen something like them.

The departing engineer. A developer sets up the nightly reporting job using her own account, because it is the quickest way to start. Two years later she moves to another company. Her account is disabled as part of the normal exit process, and the next morning every report is empty. It takes a day to find out why, because nobody remembers that the job was using her login.

The noisy neighbor. Ten services share one user. One of them has a bug and starts deleting documents it should not touch. The logs show only the shared user name, so the team spends hours checking each service in turn. If each service had its own key, the log would have named the guilty one at once.

The leaked configuration. A configuration file with a shared password is copied into a public code repository by mistake. Changing the password would break all ten services, so the team hesitates for two days while they prepare every service for the change. During those two days, the leaked password is still valid. With separate keys, only the one exposed key would have needed to be cancelled, in seconds.

A dozen words explain nearly everything about API keys. Learn them here and the later chapters will read like common sense.

Authentication versus authorization, in one story

At an airport, you show your passport at the check-in desk. That is authentication: the desk learns who you are. Then the agent looks at your ticket and decides whether you may board this flight, sit in this class, and enter the lounge. That is authorization. Two separate checks, done one after the other. An API key does both jobs in a special way. Presenting the key answers the first question. The limits written into the key answer the second.

The owner rule: a key cannot outgrow its maker

This is the most important safety rule to learn early. A key is always created by an authenticated user, its owner. The key may be given fewer permissions than the owner has, but never more. If a user can only read the sales indexes, any key that user creates can, at most, read the sales indexes. This prevents someone with modest rights from creating a super-powerful key and using it to climb higher.

Two things a key is not

• A key is not a user. It does not have a password, it does not show up in the user list, and it cannot log in to Kibana as a person.
• A key is not a session. It does not expire after a short idle time. It stays valid until it reaches its expiry time, if any, or until someone cancels it.

Quick glossary

Where does the key check happen in a real cluster, and which parts work together to make it possible?

Every request that reaches Elasticsearch first meets the security layer. That layer looks at the request’s headers to find out how the caller is proving their identity. If the header says the caller is using an API key, the request is handed to the API key service. Otherwise it goes through the chain of realms, such as the native user store or a single sign-on provider. Once identity is settled, the request moves on to the authorization step and only then to the actual data work.

flowchart TB
    App["Application or agent holding an API key"] -->|HTTPS request with Authorization header| GW["Elasticsearch HTTP layer"]
    GW --> AUTH["Authentication service"]
    AUTH --> AK{"Header scheme is ApiKey?"}
    AK -- Yes --> AKS["API key service: check cache, then security index"]
    AK -- No --> RLM["Realm chain: native, directory, single sign-on, and others"]
    AKS --> VER["Verify secret, expiry, and invalidation"]
    RLM --> VER2["Verify password or token"]
    VER --> AUTHZ["Authorization: key role descriptors and owner snapshot"]
    VER2 --> AUTHZ2["Authorization: roles of the user"]
    AUTHZ --> ACT["Requested action runs on the data"]
    AUTHZ2 --> ACT
    

Read the picture from top to bottom. The application sends its request over an encrypted connection, adding an authorization header that names the scheme and carries the encoded key. The authentication service reads the scheme. For an API key, it asks the API key service, which first looks in an in-memory cache and, if needed, in the security index, and then verifies that the secret matches, the key has not expired, and the key has not been cancelled. Only then does the request proceed to authorization, which combines the limits stored in the key with the permissions of its owner to decide what is allowed.

The main components

Walking one request through the machine

Now we open the box. Two short stories explain everything: the birth of a key and the daily life of using it.

Story one: creating a key

Story two: using a key

Each time a program sends a request, the process described in Chapter 4 runs from start to finish. The interesting internal details are these.

Why the secret is hashed slowly

If a thief managed to read the security index, a plain-text secret would be an instant disaster. That is why only a fingerprint is stored. The fingerprint is made with a method that is intentionally slow, so that trying billions of guesses would take unreasonably long. The price is that every verification costs real processor time. For that reason Elasticsearch remembers successful verifications for a while, in memory.

The cache

Suppose one application sends a thousand requests per second with the same key. Running a slow hash comparison a thousand times per second would waste a great deal of effort. So each node keeps a cache of recently verified keys. On the first request the node does the full check and stores the outcome. Later requests find the key in the cache and skip the costly step. The cache has a time-to-live and a size limit, both configurable. When a key is cancelled, the cluster arranges for the cached results to be discarded quickly, so that a cancelled key does not keep working for long.

How permissions are computed

Authorization does not simply trust the key’s own description. It combines two sets. The first is the set of role descriptors stored in the key, which say what the key should be allowed to do. The second is a snapshot of the owner’s permissions, which say what the owner was allowed to do when the key was created or last updated. The key’s effective permissions are the overlap of the two. Anything outside either set is denied.

flowchart LR
    O["Owner's permissions: snapshot at creation or last update"] --> X(("Overlap"))
    K["Key's role descriptors: what you asked for"] --> X
    X --> E["Effective permissions of the key"]
    

If a key is created without any role descriptors of its own, it receives the owner’s permissions as they were at that time. In current versions, the way owner changes are reflected in existing keys depends on the version and on whether the key is later updated, so confirm the behavior in the documentation of the version you run, and test it before you rely on it.

Where the data lives

Keys are stored in the protected security index. This has two consequences. First, keys are as durable as that index, which is replicated like other indexes and can be included in snapshots of the cluster’s security data. Second, key operations depend on that index being available, which Chapter 13 discusses.

What expiry and cancellation really do

Expiry is checked when a key is used, comparing the stored time to the current time. Cancellation marks the record so that it can no longer authenticate. Neither removes the record immediately. Expired and cancelled keys are kept for a retention period and then removed by a regular cleanup job, which keeps recent history available for investigation while stopping the security index from growing forever. The exact retention and cleanup timing are configurable.

Version notes and upgrade checks

The way keys are stored, updated, and reported has grown over several releases. Some operations, such as updating a key in place, searching through keys, and cross-cluster keys, exist only in newer versions. The default values for cache and cleanup settings can also differ. Before an upgrade, read the release notes for security changes, confirm that existing keys still authenticate on a test cluster, and check that any automation that reads or creates keys still understands the response format. After the upgrade, run a small test with a real key from each critical application, before declaring the work done.

A key is born, used, changed, replaced, and eventually removed. Knowing this life story helps you plan for each stage instead of being surprised by it.

flowchart LR
    C["Create: id and secret issued, secret shown once"] --> U["In use: applications authenticate"]
    U --> P{"What changes?"}
    P -- "Permissions or notes" --> UP["Update the key"]
    UP --> U
    P -- "Time to rotate" --> N["Create a new key and deploy it"]
    N --> I["Invalidate the old key"]
    P -- "Expiry time reached" --> E["Key expires"]
    E --> R["Kept for a retention period"]
    I --> R
    R --> D["Removed by automatic cleanup"]
    

The stages in order

The rotation dance

The safest way to replace a key is to make the new one before removing the old one. If you cancel first, the application fails until the new secret arrives. If you create first, both keys work for a short overlap, the application switches at its own pace, and you cancel the old key once you see the new one in use. This is the same idea as a relay race: the second runner starts moving before the first hands over the baton.

What happens when the owner changes

A key is linked to the user who created it. That link has consequences. If the owner’s account is removed, or the owner’s roles change, existing keys may behave differently from what you expect, depending on your version. For this reason, many organizations create long-lived keys under a dedicated service user, not under a person’s account, so that keys do not quietly break when an employee changes jobs. Test this behavior in your version before you rely on it.

The security index and upgrades

Because keys live in the cluster’s internal security data, they travel with it during ordinary maintenance, such as rolling restarts and upgrades. They are lost if you rebuild a cluster from scratch without restoring that data. Plan for this in disaster recovery: either restore the security data from a snapshot, or be ready to re-issue keys to every application.

The real strength of an API key is not that it identifies a program, but that it limits that program. Here is how the limits are written and how they combine.

When you create a key, you can include one or more role descriptors. A role descriptor is a small description of allowed access. It has the same ingredients as a role in Elasticsearch, but it lives inside the key. It can name three kinds of things.

Effective permissions: the overlap rule

As Chapter 5 showed, what the key can actually do is the overlap between its role descriptors and the snapshot of its owner’s permissions. This produces a comforting property: whatever mistakes you make when writing role descriptors, the key can never exceed the owner. It also produces a common surprise: if your key seems too weak, the reason may be that the owner never had the permission in the first place.

Typical permission sets, in words

Use aliases and patterns thoughtfully

Index privileges can name a pattern such as “logs-” followed by a star. That is convenient, since new daily indexes automatically fall under the key’s permissions. But a broad star can also cover more than you intended. Review each pattern: would a new index that matches it be something this program should touch? Where possible, grant access to an alias that points at the right indexes, so that your access description stays short and clear.

Start from nothing and add

The most reliable way to write role descriptors is to start with no permissions and add only what the program demonstrably needs. The opposite habit, which is to start with a powerful set and remove things, tends to leave surplus rights behind. If a program breaks because a permission is missing, the “unauthorized” answer names the action that was refused, and you add just that one privilege.

Keys with no role descriptors

If you do not include role descriptors, the key simply carries the owner’s permissions. This is convenient and dangerous. If the owner is an administrator, every such key becomes an administrator key. Even if it is fast to create, this habit undoes the main benefit of the feature. Treat “no role descriptors” as an exception, not a default.

Who may create keys

Creating keys is itself a privilege. One cluster privilege lets a user manage every key in the cluster, and a narrower one lets a user manage only their own keys. Hand out the narrow one whenever you can. A person or a tool that can create keys with any permission and any owner can quietly build a long-lived back door, so this privilege deserves the same care as a master password.

Limiting fields and documents (license-dependent)

Beyond index names and actions, some subscription levels let a role descriptor limit which fields a key may see and which documents it may read. This is called field-level and document-level security. It is very useful when you must share part of an index. For example, a vendor may read a product index but see only the public fields, and only the documents that belong to that vendor. Because these features depend on the license, check what your subscription allows before designing around them.

Do not treat these limits as a substitute for good data design. If a whole category of data must never be visible to a certain program, keeping it in a separate index that the key cannot touch is simpler and easier to prove than a filter on a shared index.

A short tour of the operations you will use, and the habits that keep a growing collection of keys understandable.

These operations are available through the Elasticsearch security interface and through the Kibana management screens, where you can see a list of keys, their owners, and their status, and create or cancel them with a few clicks. For repeatable work, such as onboarding a new application, scripting the creation is better than clicking, because the script documents what was done.

Give every key a good name

The name is not required to be unique, which means it is easy to end up with twenty keys called “test.” A good naming convention includes the application, the purpose, and sometimes the environment. For instance, a key named “checkout-service-prod-orders-writer” tells you a lot at a glance. When you review keys six months later, names like that save real time.

Use metadata as a label

Metadata is a free-form set of notes stored with the key. Use it to record things a future reader will need: the owning team, a contact address, a ticket reference, the date the key was reviewed, and the reason it exists. Never place secrets in metadata, because metadata is visible to anyone who can read the key’s details.

Set an expiry when you can

An expiry turns forgotten credentials into self-cleaning ones. For temporary work, such as a one-off migration, an expiry of days is natural. For long-running services, an expiry of a few months combined with a rotation process is better than a key that lives forever, because the rotation process is regularly tested and the key never gets old enough to be forgotten. Remember that an expiry can cause an outage when it arrives, so calendar the rotation before it happens and alert on keys nearing their end date.

Keys created by tools

Many Elastic tools, such as the agents and shippers, can create or use keys for you as part of their setup. These keys are real keys and appear in the same list. Their names often reveal the tool that made them. Avoid cancelling a key you do not recognize before you find out who owns it, since it may be feeding a live pipeline.

Cleaning up

Once a quarter, review the list of keys. Ask of each one: does the owner still exist, is the application still running, is the access still the smallest that works, and is the expiry sensible? Cancel what fails the review. This ten-minute habit prevents the slow buildup of unused, powerful credentials that later becomes a serious finding in a security audit.

Automating key creation safely

As the number of applications grows, creating keys by hand does not scale, and people start copying old keys “just this once.” A better answer is a small provisioning process that creates keys on request. A good version of it has these properties.

• It runs under a narrow account. The account holds only the right to manage its own keys, not the right to manage all keys.
• It uses templates. Each application type has an approved permission template, such as “log writer” or “read-only search,” so that people request a type, not free-form access.
• It records who asked. The requester, the team, and the reason go into the metadata.
• It delivers the secret directly to a secrets manager. The secret never appears in tickets, chat messages, or build logs.
• It sets the expiry itself. The requester cannot choose “never.”

Continuous integration systems deserve special mention. Build logs are widely readable and often kept for a long time. If a script prints the response of the creation operation, the secret ends up in the logs. Configure such scripts to send the secret straight to the secrets system and to mask it in output.

Creating a key is half the job. Presenting it correctly, from every kind of client, is the other half.

The header format

Requests carry an authorization header. For an API key, the header’s value is the word ApiKey, a space, and then the encoded value that Elasticsearch returned when the key was created. That encoded value is simply the id and the secret, joined by a colon, and turned into text using a standard encoding. If you only saved the id and the secret separately, you can rebuild the encoded value yourself, but using the ready-made value avoids mistakes.

Official client libraries

The official Elasticsearch libraries for languages such as Java, Python, JavaScript, Go, and others let you configure a key once, when you create the client. After that, every request carries the header automatically. Read the key from an environment variable or a secrets system when the program starts, and never hard-code it in source files.

Shippers, agents, and pipelines

Tools that send data to Elasticsearch, such as Beats, Elastic Agent, and Logstash, can be configured to authenticate with a key, usually in their output settings. Where a tool asks for the key in a combined form, it commonly expects the id and the secret joined by a colon. Read the tool’s documentation, because the accepted format differs. Use one key per shipper or per group of shippers with the same job, not one key for the whole fleet, so that a single compromised machine does not force you to replace every credential.

Proxies and load balancers

Some proxies and gateways strip or rewrite authorization headers, especially those that apply their own authentication. If a key works when you contact a node directly but not through the gateway, look at how the gateway handles that header. Also make sure that every hop from your program to the cluster uses encryption.

Keys in browsers and mobile apps

A key placed in a web page or a mobile app is visible to anyone who looks. It is therefore not a secret at all. If a browser must talk to Elasticsearch directly, then that key should be as weak as possible: read access to a single alias, no write or management rights, and a short expiry. A safer design puts a small backend service in between. The service holds the real key and exposes only the specific searches the browser is allowed to run, so that the key never leaves your servers.

Checking who you are

Elasticsearch has a small operation that answers the question “who am I?” for whatever credential you send. When called with an API key, it reports that the authentication type is an API key and shows the key’s id and name. It is the fastest way to confirm that the right key is in use and that it is recognized, before you start debugging permissions.

Keys in containers and orchestration platforms

Modern applications often run in containers, and orchestration platforms offer built-in ways to hand secrets to them. The choices differ in safety.

Whichever way you choose, make sure the application can pick up a new key without restarting, or that a restart is cheap and safe. Rotation that needs a coordinated restart of many services will be postponed until the day it becomes an emergency.

API keys are one of several credentials. Choosing the right one for each situation is part of good design.

flowchart TD
    A["Who or what needs to authenticate?"] --> B{"A person using a browser or Kibana?"}
    B -- Yes --> C["Single sign-on or a user account"]
    B -- No --> D{"An Elastic component such as Kibana or Fleet Server?"}
    D -- Yes --> E["Service account token"]
    D -- No --> F{"An application, script, or pipeline?"}
    F -- Yes --> G["API key with least-privilege role descriptors"]
    F -- No --> H["Review the design and choose a matching realm"]
    

Choosing between an API key and a user account for a program

If the program is a long-lived service with a well-known job, both work, but keys let you split its access into small pieces and cancel them separately. If the program is a temporary task, a key with a short expiry is nearly always better. If a person is running a script from their own laptop for their own purposes, their own key, created under their own account and with a short expiry, avoids putting their password into a script.

API key versus service account token

Service account tokens exist for components that Elastic supplies, and they are tied to a fixed set of permissions defined for that component. Your own applications are not service accounts in this sense. For your own software, the flexible choice is the API key, which you shape to your needs.

Keys with third-party systems

When a third-party tool must call your cluster, give it a dedicated key with the narrowest access that lets it work, and a fixed expiry, so that a vendor problem never turns into a breach of your data. Record the vendor in the key’s metadata so that any reviewer knows why the key exists.

A path from a shared user to individual keys

Many teams start with one shared user and want to move to keys without breaking anything. A calm, step-by-step plan works better than a big-bang change.

Checking a key is a security step that runs on every request. How costly is it, and what keeps it from becoming a bottleneck?

Verifying an API key involves three pieces of work: finding the key’s record, checking the secret against its stored fingerprint, and working out the permissions. Of these, the secret check is the heaviest, because the fingerprint method is slow on purpose. If the system did this from scratch on every request, busy applications would burn a lot of processor time on security. That is why caching is central to how keys perform.

A numbers walkthrough

Let us use invented numbers to feel the difference. Suppose one full verification of a secret costs 20 milliseconds of processor time. An application sends 1,000 requests per second with the same key. If every request needed a full verification, that would be 20 seconds of processor time for each second on the clock, which is about 20 processor cores doing nothing but security.

Now imagine that each node remembers a successful verification for a while. The first request pays the 20 milliseconds. The next thousands find the answer in memory at almost no cost. The security work per second drops from about 20 cores to a tiny fraction of one core. The figures in real systems will differ, but the shape is what matters: without the cache, keys would be too expensive to use at scale, and with it, the cost is negligible.

What influences the cost

Scaling with the cluster

Each node keeps its own cache, so the cost of verification scales with the number of nodes handling requests. When a cluster grows, more nodes share the requests, and each node still remembers the keys it has seen. The first request for a key on a fresh node pays the full price, so a burst of new nodes joining, or a restart of many nodes at once, causes a short period with more processor use, as each node rebuilds its cache. It is a mild effect, but it is a good reason to restart nodes in a rolling way.

Cancellation and the cache

There is a balance between speed and revocation. A long cache lifetime means better performance but also that a cancelled key might, in theory, continue to work on a node until its cache entry is discarded. Elasticsearch handles this by clearing the relevant cache entries across the cluster when a key is cancelled, so that the delay is short in practice. For urgent cases, there is also an operation to clear the key cache on demand.

No security mechanism is free. Here is the honest balance sheet.

How strict should you be?

Security that makes your system fragile is a bad trade. Here is how to keep key-based access dependable.

Keys depend on the security index

Every key record lives in the internal security index. That index is replicated like other data, so the loss of a single node does not lose keys. But if the index becomes unavailable, for example when the cluster is in a serious failure state, authentication that needs to read key records can fail. Results already stored in the cache may still let recently used keys keep working for a while, but you should not count on that. The practical lesson is that the health of the security index is part of the health of your applications.

Backups and disaster recovery

Snapshots of a cluster can include its security data, and this is what allows keys to survive a rebuild. When you plan disaster recovery, decide explicitly whether security data is part of the snapshots, test a restore in a safe environment, and confirm that keys still authenticate afterward. If you decide not to restore security data, write down the process for re-issuing every key to every application, and estimate how long it would take.

Multiple clusters

Keys exist inside one cluster. A key made in cluster A means nothing to cluster B. If you run several clusters, such as development, staging, and production, you will have separate keys in each, and you must never reuse a secret across them. For work that spans clusters, such as searching or replicating between them, newer versions provide a special type of key designed for cross-cluster access, which is scoped to that purpose. Check the documentation for the version you use.

Avoiding expiry outages

A key that expires on a Sunday night at midnight will stop the application on a Sunday night at midnight. That is the intended behavior, and also a common cause of surprise incidents. Prevent it with three habits.

Owner changes and continuity

People move teams and leave companies. If a key is owned by a person’s account, then the change of that person can quietly break or endanger the key. Prefer dedicated service users to own long-lived keys, keep the owner’s role stable, and record a contact team in the key’s metadata. When someone leaves, review the keys they created, and re-issue important ones under a stable owner.

Graceful behavior in the client

A well-built client treats an “unauthorized” answer as a distinct event. It logs the key’s name, never the secret, raises an alert, and tries to load a fresh key. It does not hammer the cluster with retries. Meanwhile it should keep any data it could not send in a safe local buffer, so that the outage costs delay, not loss.

A runbook for rebuilding a cluster

Suppose a cluster must be rebuilt from scratch. Keys are one of the things that are easiest to forget in the plan. A good runbook answers a few questions in advance.

• Are security data and key records included in the snapshots, and has a restore been tested recently?
• If not, where is the list of every application and the access it needs, so that new keys can be made quickly?
• Who has the right to create keys in the rebuilt cluster, and how is the first administrator credential protected?
• How will new secrets reach each application, and in what order should applications be reconnected?
• How will you confirm that no old key from the previous cluster is still in use anywhere, since old secrets should not be reused?

Practicing this once in a test environment is far cheaper than discovering the gaps during a real outage.

An API key is a security tool, and it is also a security risk. Knowing where it can go wrong is what keeps it safe.

The key is a bearer secret

Anyone who holds the key can act as it, from anywhere, with nothing more to prove. There is no second factor and no check of the caller’s location built in. That makes the protection of the secret the most important task. Treat it with the same care as a password to a production database.

Where secrets leak

How the cluster protects the secret

Elasticsearch stores only a salted, deliberately slow fingerprint of each secret, so that reading the security index does not reveal usable keys. The secret is shown once and cannot be retrieved. Requests should travel over encrypted connections, and communication between the nodes themselves should also be encrypted. These are the foundations, and they only help if the security features are turned on and configured.

Least privilege and blast radius

The single most effective control is to make each key weak. A key that can only write to one index can leak without turning into a data breach. When you review key permissions, imagine an attacker holding the key and ask what they could do with it. If the answer is troubling, narrow the key.

Who can create keys

Anyone who can create keys with broad access, under an account with wide permissions, can create a long-lived credential that outlives their own employment. Limit key management to the people and automation that need it, prefer the narrower “own keys only” privilege, and review the list of users who can manage all keys regularly. Automation that creates keys should itself be tightly controlled and logged.

Auditing

Audit logging records security events, including which key was used to make a request, subject to the features of your subscription level. With audit logs you can answer questions such as “which key deleted this index?” and “is this old key still being used?” Without them, you rely on application logs and the key’s metadata. If your data is sensitive, plan for audit logging from the start, and keep the logs where the people you audit cannot change them.

When a key leaks: an incident playbook

Compliance and data protection

Many organizations must show auditors that access to sensitive data is controlled, limited, and reviewed. API keys can help you tell that story, because each key has an owner, a stated purpose, a narrow set of rights, and an end date. Auditors typically ask a handful of questions: who can create credentials, how are they stored, how often are they rotated, how quickly can access be withdrawn, and where is the evidence? Good answers are a written key policy, a list of keys with owners, records of rotations, and audit logs of key usage.

If the data in the cluster is personal or regulated, add two more habits. First, keep keys that can read that data to a minimum, and review them more often than the rest. Second, avoid giving any key access to more personal data than its task requires, because the amount of data reachable through a key is the measure of the harm if it leaks.

You cannot manage what you cannot see. These are the signals to watch, and the questions that solve most problems.

What you can ask the cluster

• Who am I? The authentication check shows whether the current credential is an API key, and gives its id and name.
• What keys exist? The key listing shows names, owners, creation and expiry times, and whether each key is still active. It can be filtered to show only active keys.
• Which keys are about to expire? In newer versions the key search operation can filter by expiry time, which powers a reminder job.
• What are a key’s limits? The detailed view of a key can show the role descriptors and, optionally, the owner’s permission snapshot that limits it.

Troubleshooting: 401 versus 403

The two most common problems have different answers, and telling them apart saves hours.

Logging that helps

Log the key’s name or id in your application’s messages, never the secret. When something goes wrong, you can then match a failure to a specific key. In the cluster, audit logs, where available, record authentication successes and failures with the key’s details. Keep them long enough to investigate incidents, and protect them from tampering.

A monthly review checklist

Good habits make keys a strength. Bad habits turn them into the weakest link.

Patterns that work

Anti-patterns to avoid

A short list you could hand to a new teammate on their first day.

Best practices

Common mistakes

• Losing the secret. It is shown once. If you did not save it, you must create a new key.
• Using the id where the encoded value is needed. The id alone is not a credential.
• Assuming the key can exceed its owner. Effective access is always limited by the owner.
• Forgetting the owner’s lifecycle. Keys under personal accounts can surprise you when that person moves on.
• Reusing secrets across environments. A leak in development should never reach production.
• Broad index patterns. A pattern with a star may cover indexes you never intended.
• Ignoring expiry dates. A calendar-driven outage is entirely avoidable.
• Treating metadata as private. Anyone who can view a key’s details can read its metadata.

Myths versus facts

Where do API keys quietly do their job?

Many well-known organizations have said publicly that Elasticsearch is part of their systems, including Wikipedia’s search, Netflix, Uber, and Stack Overflow. They rarely publish their security configurations, so the scenarios below are typical patterns for this kind of workload, not descriptions of anyone’s private setup.

Short answers to the questions that come up most often.

If you skim only one part of this guide, make it this one.

An Elasticsearch API key is a small, purpose-made credential that lets a program prove who it is without borrowing anyone’s password. It can be limited to exactly the access the program needs, given an end date, and cancelled without disturbing anything else. Used with care, it makes systems safer and easier to run. Used carelessly, it becomes one more forgotten secret.