Runtime Security Monitoring in Kubernetes with Falco
Your cluster passed every scan before deployment, yet attackers act after containers start running. This guide explains, from first principles, how Falco watches live behavior inside Kubernetes, raises an alarm when something looks wrong, and fits into a complete defense strategy.
Imagine a large apartment building with a strong front door, a guard who checks every visitor’s ID, and a thick lock on every flat. That feels safe. But what happens if someone sneaks in through a window, copies a key, or a trusted neighbor suddenly starts behaving strangely at three in the morning? Locks and ID checks only protect the moment of entry. To catch trouble that is already inside, the building needs cameras and an alert system that notices unusual behavior while it is happening.
A Kubernetes cluster is very much like that building. We scan container images before they ship, we write admission rules that decide what is allowed to start, and we restrict who can talk to the API. These are the front door, the guard, and the locks. But once a container is running, thousands of small actions happen every second, and any one of them could be an attacker at work. Runtime security is the camera system. Falco is the most widely adopted open-source camera system for the cloud-native world, and this tutorial teaches you how it works, why it is built the way it is, and how to use it well in production.
1What Is Runtime Security and Why Does Kubernetes Need It?
Security has a before, a during, and an after. Runtime security is the “during”.
The three moments of protecting software
Every piece of software has a life with three stages. First it is built, when source code and dependencies are put together into a container image. Second it is deployed, when Kubernetes decides where and how that image will run. Third it is running, when the application is alive, handling real users and real data. Each stage has its own security tools, and a mature team uses all of them because each one catches problems the others cannot.
Build-time security looks for known weaknesses before anything ships, such as outdated libraries with published vulnerabilities. Deploy-time security checks the plan before it is carried out, for example refusing a workload that asks for dangerous permissions. Runtime security looks at what the workload actually does once it is alive. This last stage matters because the first two work with what is known and declared, while attackers specialize in what is unknown and undeclared.
Think of airport security. Checking luggage at the entrance is build-time and deploy-time security. Watching passengers behind the gate for strange behavior is runtime security. A bag can pass every scanner and still be carried by someone who acts suspiciously later, and only the second kind of watching will notice.
Why scans and policies are not enough
A vulnerability scanner can only report weaknesses that someone has already discovered and published. A brand-new flaw, often called a zero-day, does not appear in any database on the day it is exploited. Admission policies can only judge what a workload asks for at the start. They cannot see what a program decides to do an hour later. A perfectly clean image can be tricked into downloading a malicious tool, a stolen password can let an intruder open a shell inside a healthy pod, and a compromised dependency can quietly start contacting a stranger’s server.
In all of these cases nothing was wrong at deployment time. The danger appears only through behavior: a process that should never start, a file that should never be read, a network connection that should never be opened. Runtime security exists to see that behavior and call attention to it quickly, ideally within seconds.
Why containers make runtime visibility harder
Traditional servers ran a handful of long-lived programs, and administrators knew them well. A Kubernetes node may host dozens of short-lived containers from many teams, started and stopped automatically at any hour. They all share the same operating system kernel on the node, which means one compromised container can sometimes reach for the kernel to affect its neighbors. Because containers are so numerous and so temporary, nobody can log into each one to look around. We need an automated observer that understands containers, pods, and namespaces, and that works at the one place all containers share: the kernel.
Many beginners believe that a container is a tiny separate computer. It is not. A container is an ordinary process on the host with some walls around it. Those walls are strong but not perfect, which is exactly why watching what happens at the kernel level is so valuable.
2Meet Falco: The Cloud-Native Runtime Watchdog
Falco is a free, open-source tool that listens to what your machines and containers do and shouts when behavior breaks the rules.
What Falco is, in plain words
Falco is a behavioral detection engine. It continuously collects events from the Linux kernel, compares each event against a list of rules that describe suspicious behavior, and sends out an alert whenever a rule matches. It does not scan files, it does not look for virus signatures, and it does not require changes to your applications. It simply observes what programs do and judges whether that behavior is normal.
The name comes from the falcon, a bird known for sharp eyesight and fast reactions, which is a fitting picture for a tool whose whole job is to notice problems quickly from above.
A short history
Falco was created in 2016 by the company Sysdig, which had already built deep expertise in capturing and understanding system activity inside containers. In 2018 the project was donated to the Cloud Native Computing Foundation, the same neutral home that hosts Kubernetes and Prometheus. It moved from the early sandbox stage to incubating in 2020, and in February 2024 it reached the highest maturity level, called graduated. Graduation is a strong signal that the project has a healthy community, contributors from many different companies, solid governance, and real production adoption.
The project also publishes a public roadmap toward a 1.0 release, with themes such as making the modern eBPF driver the default, offering a lighter image that ships without a bundled driver, clearer guidance for first-time users, and a more extensible plugin framework. Those themes tell us where the community sees the tool heading: easier to install, easier to operate, and easier to extend.
Falco
The detection engine itself. It reads events, applies rules, and produces alerts.
Falcosidekick
A companion that takes Falco’s alerts and forwards them to dozens of destinations such as chat tools and incident systems.
Falco Talon
A response engine that can take automatic action, such as isolating or terminating a suspicious workload, when an alert arrives.
Falcoctl
A command-line helper that installs and updates rule bundles and plugins so you do not have to manage them by hand.
Plugins
Add-ons that teach Falco to read new sources of events, such as the Kubernetes audit log or cloud activity logs.
Rules Repository
A community-maintained collection of ready-made detection rules, grouped by how mature and well tested they are.
Falco is a detection tool first. Its natural job is to notice and report. Taking automatic action is possible, but it is done by companion tools that listen to Falco’s alerts, which keeps the watching part simple and dependable.
Falco is a security camera with a very smart brain behind it. The camera sees everything the kernel sees. The brain holds a rulebook that says “a person entering the vault room after midnight is suspicious”. When the rule matches, a bell rings. Whether anyone runs to the vault is decided by the people, or the robots, listening to that bell.
3How Falco Works: The Big Picture
Four layers turn raw kernel activity into a meaningful security alert.
From a single action to a single alert
Whenever a program on a Linux machine wants to do something that touches the outside world, such as opening a file, starting another program, or connecting to a network address, it must ask the kernel for permission through a request called a system call. Falco sits quietly next to the kernel and receives a copy of those requests. It never blocks or slows the original request in a noticeable way. It only watches.
The journey from request to alert has four stages. A driver inside the kernel captures the event. A userspace library collects the events and enriches them with context, such as which container, pod, and namespace the program belongs to. A rules engine checks each enriched event against the loaded rules. Finally, output channels deliver any resulting alert to people and systems.
flowchart LR
A["Application in a container"] --> B["System call to the kernel"]
B --> C["Falco driver captures a copy"]
C --> D["Userspace library adds container and Kubernetes context"]
D --> E["Rules engine evaluates the event"]
E -->|"Rule matches"| F["Alert produced"]
E -->|"No match"| G["Event discarded"]
F --> H["Output channels: logs, webhooks, Falcosidekick"]
The four layers explained
Capture
The driver lives inside the kernel and records each system call along with its arguments. Because it runs in the kernel, it sees everything, even activity that tries to hide from tools running inside a container.
Enrich
A raw event only says that a process number opened a file. Falco adds the story: which container, which image, which pod, which namespace, which user, and which parent process started it. Context turns noise into evidence.
Evaluate
The rules engine runs each event through the loaded rules. A rule is a condition plus a message. If the condition is true, an alert is created with a priority level.
Deliver
Alerts are written to standard output, to files, or sent over the network. This is where companion tools take over to notify people or trigger a response.
Why watching the kernel is such a strong idea
There are two common places to observe a running program. One is inside the application, using an agent or library that the application loads. The other is from the outside, at the operating system. The first approach needs changes in every application and can be tampered with by the very program being watched. The second approach needs no change to the applications, works for every language and framework, and is much harder for an attacker to switch off. Falco takes the second approach, and that single design choice explains most of its strengths.
4System Calls: The Language Falco Listens To
To understand Falco, you only need to understand one idea: programs must ask the kernel for everything.
What is a system call?
A program running on Linux lives in a restricted area often called user space. It cannot directly touch the disk, the network card, or other programs. When it needs any of those, it politely asks the kernel, which lives in a privileged area. That polite request is a system call. Reading a file is a system call. Starting a new program is a system call. Opening a network connection is a system call. Changing a file’s permissions is a system call.
Picture a restaurant. The customers in the dining room are programs. The kitchen is the kernel. Customers are never allowed to walk into the kitchen. They write an order on a slip and hand it to a waiter. Every slip is a system call. If you stand next to the pass where the slips arrive, you can learn everything happening in the restaurant without ever entering the kitchen. That is exactly where Falco stands.
Why system calls reveal attacks
Almost every attack, no matter how clever, eventually has to do something observable through system calls. To steal a password file, the attacker must read it. To install a cryptocurrency miner, the attacker must write a new program to disk and start it. To phone home, the attacker must open a network connection. To dig deeper into the machine, the attacker must change permissions or load special kernel features. The attacker may hide their tools well, but they cannot avoid asking the kernel for help.
File activity
Opening, reading, writing, renaming, and deleting files. Useful for spotting access to sensitive credentials.
Process activity
Starting new programs and spawning shells. Useful for spotting unexpected tools appearing inside a container.
Network activity
Creating sockets and connecting to remote addresses. Useful for spotting calls to unfamiliar servers.
Privilege changes
Switching users, changing capabilities, or adjusting permissions. Useful for spotting attempts to gain more power.
A busy node can produce hundreds of thousands of system calls per second. Falco copes by filtering early and cheaply, keeping only the event types its loaded rules actually care about, so the cost grows with what you ask it to watch rather than with everything the machine does.
The blind spot Falco covers
Network firewalls watch traffic between machines, but they cannot see what happens inside a machine. Application logs show only what developers decided to record. System calls, by contrast, form a complete and honest diary of what every process actually did. This is why Falco is often described as giving you “ground truth” about your workloads.
5The Driver: Falco’s Eyes Inside the Kernel
The driver is the small piece that actually collects events. Choosing the right one matters.
What a driver does
The driver is a tiny program that runs inside the kernel, taps into system calls, and passes compact copies of each event up to Falco running in user space. It is designed to do as little work as possible, because anything that runs in the kernel must be extremely careful not to slow the machine down or cause instability.
The driver options
Falco has supported several kinds of drivers over the years. They all produce the same events, but they differ in how they are built, how portable they are, and how much trust they require.
| Driver | How it works | Strengths | Limitations |
|---|---|---|---|
| Kernel module | A loadable piece of kernel code built for the specific kernel version | Long track record, wide kernel coverage | Must be compiled or downloaded to match each kernel; a bug in a kernel module can affect the whole machine |
| Legacy eBPF probe | A sandboxed program loaded into the kernel, built against kernel headers | Safer than a module because the kernel verifies it first | Still needs kernel headers or a prebuilt probe for each kernel |
| Modern eBPF probe | A sandboxed program built once and adapted at load time to the running kernel, using a technique known as Compile Once, Run Everywhere | No kernel headers, no compilation on the node, ships inside the Falco program, works well on immutable operating systems | Needs a reasonably recent kernel, commonly cited as version 5.8 or newer |
What eBPF means, without the jargon
eBPF is a feature of the Linux kernel that lets small, carefully checked programs run safely inside the kernel. Before such a program is allowed to start, the kernel runs it through a strict inspector called the verifier. The verifier refuses programs that could crash the machine, loop forever, or read memory they should not touch. This gives eBPF a safety record that custom kernel modules cannot match, and it is the reason eBPF has become the foundation of modern observability and security tools.
A kernel module is like letting a contractor knock down walls in your house on their own judgment. An eBPF program is like hiring the same contractor but requiring that a building inspector approve every plan first. Both get work done, but the second approach protects the house.
Current guidance from the project and the community favors the modern eBPF probe as the recommended choice on recent distributions, because it removes the most painful operational chore: keeping a driver in step with every kernel update. The project roadmap even lists making it the default. Always confirm the kernel version of your nodes before deciding, and keep an older driver option in mind only for nodes running older kernels.
Managed Kubernetes services and minimal container operating systems often lock down what you can load into the kernel. Verify that your chosen node image supports the driver you plan to use before rolling Falco out to a whole fleet. Testing on one node pool first saves a lot of confusion.
6Inside the Engine: Turning Raw Events into Understanding
A bare event is almost meaningless. Context is what makes an alert useful.
Two libraries do the heavy lifting
Falco is built on top of two shared libraries maintained by the same project. The first, often called libscap, is responsible for capturing events from the driver and also from saved capture files. The second, often called libsinsp, takes those events and builds a live picture of the machine: every running process, every open file, every network connection, and how they relate to each other. When the rules engine evaluates a condition such as “the parent of this process is a web server”, it is libsinsp that remembers who the parent is.
Why enrichment matters so much
Suppose an alert says only that process 48213 opened a sensitive file. That is nearly useless to an engineer on call. Now suppose the alert says that a shell started by the command line tool kubectl exec, inside the payments pod in the production namespace, running image version 2.4, as the root user, just read the password file. That alert tells a complete story in one sentence. The second alert is possible because Falco enriches events with container and Kubernetes details.
Process lineage
The program, its command line, and the chain of parents that started it, which reveals how it came to exist.
Container identity
Container name, identifier, image name, and image tag, so you know exactly what software was running.
Cluster identity
Pod name, namespace, and labels, so you can route the alert to the right owning team.
User and session
The user the process runs as, and whether it came from an interactive terminal, which is rare for healthy production workloads.
Where Kubernetes details come from
Falco learns container details by talking to the container runtime on the node, and it learns cluster details through a Kubernetes metadata component that collects information about pods and namespaces and hands it to the engine. This is how an event that began as a low-level kernel record ends up labeled with the human-friendly names your team already uses.
flowchart TD
K["Kernel events"] --> S["Capture library"]
S --> I["Inspection library: processes, files, connections"]
R["Container runtime metadata"] --> I
M["Kubernetes metadata: pods, namespaces, labels"] --> I
I --> E["Rules engine"]
E --> O["Enriched alert"]
Short-lived containers can disappear before metadata is fully collected, which occasionally leaves a field blank in an alert. This is normal behavior for very fast workloads and is one reason alert pipelines should not depend on every field always being present.
7Rules: Teaching Falco What Suspicious Looks Like
Rules are the heart of Falco. They are simple to read, even for non-programmers.
What a rule is
A Falco rule is a short, human-readable description of behavior that deserves attention. Each rule has a name, a description, a condition, an output message, a priority, and optional tags. The condition is the interesting part. It is a sentence-like test over the fields of an event, for example “a new process was started, and it is a shell, and it is inside a container, and it is attached to an interactive terminal”. When the test is true, the rule fires.
Condition
The test that decides whether an event is suspicious.
Output
The message template that explains what happened, filled in with details from the event.
Priority
How serious the finding is, from informational up to emergency.
Tags
Labels such as the matching technique in the MITRE ATT&CK framework, which help analysts and reports.
Building blocks that keep rules tidy
Writing long conditions again and again would be tiring and error-prone, so the rule language offers reusable pieces. A list is a named collection of values, such as the names of all common shells. A macro is a named, reusable piece of condition, such as “this event happened inside a container”. Rules then combine lists and macros like building blocks, which makes them short, readable, and easy to maintain. When a macro is improved, every rule that uses it improves too.
Think of a school rulebook. A list is the roster of approved field-trip destinations. A macro is a standard phrase like “during school hours”. The rule itself is a sentence such as “no visitors without a badge during school hours”. You reuse the roster and the phrase across many rules instead of rewriting them each time.
Exceptions: telling Falco about legitimate behavior
Every real environment has legitimate tools that look suspicious in isolation. A backup agent may read many sensitive paths. A debugging container may start shells on purpose. Instead of deleting a useful rule, you add an exception that describes the specific, known-good situation, for example the name of the container image allowed to do this one thing. The rule keeps guarding everyone else while the trusted case stays quiet.
Where rules come from
Most teams start with the community-maintained default rules, which cover common threats across hosts, containers, and Kubernetes. The project sorts shared rules into maturity levels, so you can choose between only the most battle-tested rules or a wider set that includes newer ideas. On top of that, teams add their own custom rules for behavior specific to their own applications. Rule bundles can be installed and updated with the lifecycle helper, so improvements arrive without manual copying.
| Rule source | Best for | Effort | Risk |
|---|---|---|---|
| Default stable rules | A safe, low-noise starting point | Very low | May miss threats specific to your environment |
| Extended community rules | Wider coverage once you have tuned the basics | Low to medium | More tuning needed to control noise |
| Your own custom rules | Protecting business-specific assets and workflows | Medium | Poorly written rules can be noisy or too narrow |
Editing the shipped default rule files directly makes upgrades painful because your changes can be overwritten or conflict with new versions. Keep your own rules and overrides in separate files that load after the defaults.
8Priorities and Outputs: What Happens When a Rule Fires
An alert is only useful if the right person sees it at the right urgency.
The priority ladder
Falco uses the classic syslog severity ladder. From most to least urgent the levels are emergency, alert, critical, error, warning, notice, informational, and debug. In practice most detection rules sit at notice, warning, error, or critical. Good teams decide in advance what each level means for their organization, such as “critical pages the on-call engineer immediately, warning goes to a chat channel, notice is kept for investigations”.
Critical and above
Strong signs of compromise, such as a reverse shell or tampering with security tools. These should interrupt a human right away.
Error and warning
Behavior that is probably wrong or risky, such as a shell opened in a production container. Worth prompt attention during working hours or through a chat alert.
Notice and informational
Unusual but often harmless events, kept as breadcrumbs. They shine during investigations when you want to reconstruct a timeline.
Where alerts can go
Falco can write alerts to standard output, which is the natural choice on Kubernetes because the cluster’s logging system already collects container output. It can also write to files, send to the system log, run a program, or send messages to a web address. In most clusters the simplest durable pattern is this: write alerts as structured JSON to standard output, let your existing log collector pick them up, and use Falcosidekick for fan-out to chat, paging, and storage destinations.
Prefer structured, machine-readable alert output over plain sentences. Structured fields let your log platform filter by namespace, rule name, or priority, build dashboards, and create reliable automation without fragile text matching.
What makes an alert actionable
A strong alert answers five questions at a glance: what happened, where it happened, who or what did it, when it happened, and how serious it is. If an alert leaves out any of these, responders waste precious minutes hunting for details. When tuning your outputs, review a sample of real alerts and ask whether a tired engineer at three in the morning could decide what to do within one minute.
9Beyond System Calls: Kubernetes Audit Events and Plugins
System calls show what happens on nodes. The Kubernetes audit log shows what happens to the cluster itself.
The second source of truth
Some dangerous actions never touch a node in an obvious way. Creating a powerful role binding, reading a secret through the API, or launching a privileged pod are all requests sent to the Kubernetes API server. Kubernetes can record such requests in an audit log. Falco can read that log through a plugin and apply rules to it, in the same way it applies rules to system calls.
flowchart LR
U["User or service account"] --> A["Kubernetes API server"]
A --> L["Audit log events"]
L --> P["Falco audit plugin"]
P --> E["Rules engine"]
E --> X["Alert: risky cluster change"]
N["Node system calls"] --> D["Falco driver"]
D --> E
What audit rules can catch
Unexpected access to secrets
Alert when an identity that normally never reads secrets suddenly lists or fetches many of them, which is a classic step after stealing a credential.
Privileged or host-level pods
Alert when a new pod asks for extra privileges or mounts the node’s own file system, because those settings weaken container walls.
Permission escalation
Alert when someone grants broad cluster-wide administrative rights, or when a new binding appears outside your normal change process.
Anonymous or unusual API access
Alert when requests arrive without proper identity, or from sources you have never seen before.
Plugins: extending what Falco can read
The plugin framework lets Falco consume other event streams beyond the kernel, such as cloud provider activity logs and software supply chain events. Each plugin brings its own fields and matching rules, but the experience for the operator stays the same: one engine, one rule language, one alert pipeline. This is why many organizations describe Falco as a single place for threat detection across hosts, containers, clusters, and cloud accounts.
Audit-based detection depends on your cluster’s audit configuration. On managed Kubernetes platforms, audit logs may be exposed differently from self-managed clusters, so check your provider’s approach before planning this part of your coverage.
10Deploying Falco on a Kubernetes Cluster
Falco must run on every node, because it watches the kernel of the node it lives on.
The daemon set pattern
Because each node has its own kernel, one Falco instance is needed per node. Kubernetes has a built-in workload type for exactly this need, called a DaemonSet, which automatically places one copy of a pod on every eligible node, and starts another copy whenever a new node joins the cluster. The official installation method uses a package manager for Kubernetes, known as Helm, which bundles Falco, its settings, its rules, and optional companions into a single, repeatable installation.
flowchart TD
subgraph C["Kubernetes cluster"]
subgraph N1["Node 1"]
F1["Falco pod"]
W1["Workload pods"]
end
subgraph N2["Node 2"]
F2["Falco pod"]
W2["Workload pods"]
end
subgraph N3["Node 3"]
F3["Falco pod"]
W3["Workload pods"]
end
end
F1 --> S["Falcosidekick"]
F2 --> S
F3 --> S
S --> D["Chat, paging, SIEM, storage"]
Permissions and trust
Because Falco loads an observer into the kernel, it needs elevated permissions on the node. Older drivers required fully privileged containers. The modern eBPF approach can run with a narrower set of specific capabilities, which follows the principle of least privilege more closely. Either way, Falco itself becomes a high-value target, so its namespace, service account, and configuration should be locked down and watched carefully.
A sensible rollout plan
Pilot on a non-production cluster
Install with default rules, confirm the driver loads on your node images, and watch the alert stream for a week to learn what normal looks like.
Tune before you trust
Add exceptions for known-good tools until the stream becomes quiet enough that every alert deserves a look.
Connect alert delivery
Route high-priority alerts to the on-call channel and everything else to searchable storage. Test that a deliberate harmless trigger actually reaches a human.
Roll out gradually to production
Start with one node pool or namespace group, measure resource use, then expand. Always keep the rules and configuration in version control.
Write the response playbook
For each critical rule, document who is paged, what they check first, and when to isolate or terminate a workload.
Teams sometimes skip the tuning stage and send raw default output straight to a paging system. The result is alert fatigue within days, and people start ignoring everything, including the one alert that mattered. Tune first, page second.
11Falcosidekick: Getting Alerts to the Right Place
Falco decides that something is wrong. Falcosidekick makes sure the right people and systems hear about it.
The problem it solves
Every team uses a different mix of tools: one chat app for conversation, another system for paging, a log platform for search, a security information system for compliance, and maybe an object store for long-term records. If Falco had to speak directly to every one of them, its core would grow enormous. Instead, Falco speaks one simple language, and a companion service called Falcosidekick translates and forwards alerts to a very large number of destinations, commonly described as more than fifty.
Falco is the fire alarm sensor. Falcosidekick is the dispatch center that receives the signal and then calls the fire department, texts the building manager, and posts a notice on the staff board, all at once, each in the format that recipient understands.
What it adds beyond simple forwarding
Priority filtering
Send only critical alerts to the paging tool while sending everything to storage.
Many destinations at once
One alert can reach chat, a log platform, a ticket system, and an archive in parallel.
Counting and graphing
Expose alert counts to your monitoring system so you can chart trends and set thresholds.
Optional web interface
A simple user interface for browsing recent alerts, handy during pilots and small teams.
Design advice for alert routing
Separate urgency from record-keeping
Keep one path that interrupts humans and a different path that stores everything. People should see only what needs action, while investigators can still search the full history.
Add owner information
Use namespace or label information to send alerts to the team that owns the workload, instead of one overloaded security inbox.
Protect the delivery path
Alerts can contain sensitive details such as command lines. Use encrypted connections and restrict who can read the destinations.
12From Detection to Response: Falco Talon and Automation
Spotting an intruder in seconds is wasted if the response takes hours.
Why automate any response at all?
Attackers move quickly. A cryptocurrency miner can be running within moments of entry, and a data thief can copy files in minutes. Waiting for a human to read an alert, log in, and act may take far longer than the attack. Automated response closes that gap for situations where the right action is clear and the cost of acting wrongly is low.
What Falco Talon does
Falco Talon is a response engine that listens to Falco’s alerts and runs predefined actions when specific rules fire. Typical actions include labeling a pod so it can be isolated by network policy, terminating the offending pod so the controller recreates a clean one, or collecting evidence from the workload before it disappears. Each action is tied to specific rules and conditions that you define in advance, so behavior stays predictable.
flowchart LR
A["Falco alert"] --> B["Response engine"]
B --> C{"Matching response policy?"}
C -->|"Low risk, clear case"| D["Automatic action: label, isolate, or terminate pod"]
C -->|"High impact or unclear"| E["Notify human for decision"]
D --> F["Record what was done"]
E --> F
Choosing the right level of automation
| Level | What happens | Good for | Danger |
|---|---|---|---|
| Notify only | Humans receive the alert and decide | Ambiguous or business-critical workloads | Slow reaction time |
| Contain | Workload is isolated from the network but kept alive for study | Suspected compromise where evidence matters | May interrupt a legitimate service if the alert was wrong |
| Terminate | Pod is stopped and replaced by a fresh one | Clear-cut cases such as a miner in a stateless service | Destroys evidence unless captured first; can cause flapping if the cause is in the image |
Automatically killing pods on a rule that is still noisy can turn a false alert into an outage. Only automate on rules that have been tuned and proven reliable, and begin with the gentlest action, such as isolation.
13What Falco Catches: Common Attack Scenarios
Understanding the attacker’s story makes the rules feel natural.
A typical intrusion, step by step
Most container attacks follow a recognizable path. The attacker finds a way in, usually through a vulnerable web application or a leaked credential. They get command execution inside a container. They look around, fetch tools, try to reach sensitive data or credentials, attempt to escape or move sideways, and finally pursue their goal, such as stealing data or mining cryptocurrency. Falco can raise an alert at many of these steps, which is valuable because even if the attacker evades one rule, another may catch them.
flowchart LR
A["Initial access: vulnerable app"] --> B["Command execution in container"]
B --> C["Discovery: reading config and credentials"]
C --> D["Tool download and new executable"]
D --> E["Lateral movement or escape attempt"]
E --> F["Impact: data theft or crypto mining"]
Behavior patterns and the rules that watch for them
Problem
A terminal shell starts inside a running production container. Healthy application containers almost never need one.
Why It’s Harmful
A shell is the attacker’s workbench. From it they can run any tool the container allows, and it often means someone gained command execution, either maliciously or by an engineer bypassing normal change control.
Correct Approach
Alert on interactive shells in production workloads, add narrow exceptions for approved debugging practices, and require debugging to happen through audited, time-limited channels.
Problem
A process reads files that hold credentials, such as the system password store or service account tokens.
Why It’s Harmful
Stolen credentials let an attacker impersonate legitimate users or services and widen their reach across the cluster or into cloud accounts.
Correct Approach
Alert on reads by unexpected programs, allow known system tools, and keep tokens short-lived and narrowly scoped so a stolen one is worth little.
Problem
A new executable appears and runs inside a container even though it was not part of the original image.
Why It’s Harmful
Containers are supposed to be immutable. A program that was downloaded or written after startup is a strong sign that someone is adding their own tools.
Correct Approach
Alert when a program runs that was created after the container started. Combine with read-only file systems so that most such attempts fail outright.
Problem
A workload connects to an address or service it has never used, such as a mining pool or an unknown remote server.
Why It’s Harmful
Outbound connections are how attackers receive instructions and send stolen data out.
Correct Approach
Alert on unexpected destinations and, where possible, back detection with egress network policies that simply refuse unapproved traffic.
Problem
A process writes to directories that hold system programs or configuration, or tries to change security settings.
Why It’s Harmful
Attackers modify system areas to hide, to persist across restarts, or to disable the very tools that watch them.
Correct Approach
Alert on writes to protected locations and on any attempt to stop or alter security agents, and treat such alerts as high priority.
No single rule is a silver bullet. The strength of runtime detection comes from layering many small, precise rules so an attacker has to avoid all of them, while a defender only needs one to fire.
14Taming the Noise: Tuning Falco for Real Environments
A detection tool that cries wolf is worse than none, because people stop listening.
Why noise happens
Default rules are written to be broadly useful, which means they cannot know what is normal in your cluster. Your monitoring agent, your service mesh sidecar, your backup job, and your deployment tooling may all do things that match generic suspicious patterns. If you do not teach Falco about them, it will report them again and again.
A repeatable tuning loop
Observe
Collect a week or two of alerts. Group them by rule name, namespace, and image so patterns stand out.
Classify
For each frequent pattern decide whether it is truly benign, risky but accepted, or genuinely suspicious. Involve the owning team in this judgment.
Narrow
For benign patterns add the smallest possible exception, such as a specific image and a specific program, rather than silencing the whole rule.
Review
Revisit exceptions on a schedule. Applications change, and an exception that was safe last year may hide something today.
Good tuning habits
- Narrow exceptions tied to specific images or namespaces
- A short written reason for every exception
- Rules and exceptions stored in version control
- Regular reviews with application owners
- Measuring alert volume per rule over time
Bad tuning habits
- Disabling a noisy rule entirely
- Broad exceptions that match whole categories
- Undocumented changes made directly on a running cluster
- Never revisiting old exceptions
A smoke detector that goes off every time you make toast gets its battery removed, and then it fails you during a real fire. Tuning is how you keep the battery in. You teach the detector the difference between toast and fire instead of switching it off.
15Performance, Reliability, and Watching the Watcher
A security tool that slows your applications or silently stops working is a liability.
Resource cost
Falco uses processor time and memory on every node. The amount depends mainly on how busy the node is and how many rules are loaded. Because filtering happens early, adding rules that match only a few event types costs little, while very broad rules that inspect every event cost more. Plan resource requests and limits from measurements taken in your own environment, not from guesses, and test on your busiest nodes, since they set the worst case.
Dropped events: the silent risk
If the node produces events faster than Falco can read them, the buffer that carries events from the kernel to user space can fill up, and some events are dropped. A dropped event is an event Falco never judged, which could be the very one that mattered. Falco reports drop counts so you can detect this condition. Treat sustained drops as an operational incident: investigate overly broad rules, increase resources, or adjust buffer sizes.
Event drops
The most important health signal. Any sustained non-zero value deserves attention.
Resource usage
Processor and memory of each Falco pod, compared against its requests and limits.
Rules loaded and matched
Shows which rules fire most and confirms that your latest rule bundle really loaded.
Pod liveness per node
Confirms that every node has a healthy Falco pod, since a missing pod means a blind node.
Kernel upgrades and driver health
When nodes upgrade their kernel, the driver must still work with it. The modern eBPF approach greatly reduces this concern because it adapts at load time instead of needing a matching prebuilt file. Even so, the safest practice is to upgrade nodes gradually and confirm that Falco is healthy on the first upgraded nodes before continuing.
Create an alert on the absence of Falco itself. If a pod crashes or a node joins the cluster without Falco, nobody will notice by looking at a quiet alert stream, because quiet looks exactly like safe. Monitor the monitor.
16Falco Among Other Kubernetes Security Tools
Falco is one layer in a stack. Knowing what the other layers do shows you where Falco fits.
Detect, prevent, and harden
Security tools for Kubernetes fall into three families. Preventive tools stop bad things before they start, like admission controllers that refuse unsafe workloads. Hardening tools shrink what a workload is allowed to do, like profiles that restrict system calls or file access. Detective tools, where Falco belongs, watch what happens and report it. A strong program combines all three, because prevention can be bypassed, hardening can be too loose, and detection is the safety net under both.
| Tool type | Examples | When it acts | Main strength | Main limit |
|---|---|---|---|---|
| Image scanners | Vulnerability scanners for images | Build time | Finds known flaws early and cheaply | Blind to unknown flaws and to runtime behavior |
| Admission policy engines | Policy controllers using declarative rules | Deploy time | Blocks unsafe configuration before it runs | Cannot see what a running program does |
| Kernel-level confinement | System call filters, mandatory access control profiles | Run time, preventive | Strictly limits what a process can attempt | Profiles take effort to write and maintain |
| Runtime detection | Falco | Run time, detective | Broad behavioral visibility with simple rules | Reports rather than blocks by itself |
| Runtime enforcement | eBPF tools that can also block actions | Run time, preventive and detective | Can stop a malicious action in progress | Higher risk of disrupting good workloads if rules are wrong |
Falco compared with newer eBPF-based tools
Other open-source projects also use eBPF for observability and security, and some can enforce policy directly in the kernel by blocking an action rather than only reporting it. The trade-off is a familiar one. Enforcement is powerful but riskier, because a wrong rule can break a healthy service. Detection is gentler and easier to adopt, and its rule language and ecosystem of integrations are mature. Many organizations start with detection to learn their environment and add enforcement later for a small number of well-understood behaviors.
17Advantages, Limits, and Honest Trade-offs
Every tool has a shape. Knowing Falco’s shape helps you use it where it shines.
Advantages
- Watches behavior at the kernel, so it covers every language and framework without changing applications
- Readable rule language that security and platform teams can both understand
- Understands containers, pods, and namespaces, which makes alerts meaningful
- Large community, shared rules, and a mature ecosystem of integrations
- Open source with neutral foundation governance, avoiding vendor lock-in
- Also reads Kubernetes audit events and other sources through plugins
Disadvantages / Trade-offs
- Detects and reports; blocking or responding needs companion tools or your own automation
- Needs privileged access to node kernels, so Falco itself must be well protected
- Default rules need tuning, and tuning takes ongoing human effort
- Driver support depends on node operating system and kernel version
- Very heavy event loads can cause dropped events if resources are too small
- A determined attacker who compromises the kernel itself may evade any in-kernel observer
Reading the trade-offs correctly
None of these limits are reasons to avoid Falco. They are reasons to deploy it thoughtfully. The need for tuning, for instance, is true of every detection product in the world; it simply cannot know your normal behavior until you tell it. The privilege requirement is the price of seeing what the kernel sees, and it is manageable with proper isolation of the Falco namespace. And the fact that Falco reports instead of blocks is a deliberate design choice that keeps the observer simple, safe, and easy to trust.
18Best Practices for Running Falco in Production
Teams that succeed with Falco tend to follow the same handful of habits.
Treat rules as code
Store rules, exceptions, and settings in version control, review changes like application changes, and roll them out through the same pipeline you use for other cluster configuration.
Start broad, then specialize
Begin with the community default rules to get coverage quickly, then add custom rules for the assets that matter most to your business, such as payment services or data stores.
Map alerts to owners
Use namespace and label information so that each alert reaches the team that can actually judge it, with a shared security channel only for the highest severities.
Pair detection with prevention
Use read-only file systems, restricted network egress, and minimal permissions so that many attacks fail on their own, and let Falco catch what slips through.
Rehearse the response
Run small drills where a harmless test action triggers a rule, and confirm the alert reaches the right person and that they know the next step.
Secure Falco itself
Limit who can change its configuration, protect its alert channels, and keep it updated, since it holds powerful access and produces sensitive data.
Measure and improve
Track alert volume, time to acknowledge, false positive rate, and dropped events. Use these numbers to decide where to tune and where to add coverage.
Link rule tags to the MITRE ATT&CK framework in your reports. It gives security leaders a shared vocabulary for coverage, such as “we have detections for credential access and execution, but fewer for defense evasion”, and it points directly to where to invest next.
19Anti-patterns: Mistakes That Undermine Runtime Security
Learning from common failures is cheaper than learning from your own.
Problem
Installing Falco and forwarding every alert straight to a paging system on day one.
Why It’s Harmful
The volume overwhelms responders, trust in the tool collapses, and real incidents get lost among the noise.
Correct Approach
Pilot quietly, tune with application owners, and page humans only for rules that have proven precise.
Problem
Disabling a noisy rule instead of adding a precise exception.
Why It’s Harmful
You lose protection for the whole cluster to quiet one tool. Attackers benefit from every disabled rule.
Correct Approach
Describe the exact known-good behavior as an exception and keep the rule active for everything else.
Problem
Deploying Falco on only some nodes, or forgetting nodes that join later.
Why It’s Harmful
An attacker only needs to land on one unprotected node. Partial coverage gives a false sense of safety.
Correct Approach
Use the per-node workload pattern so every node, including future ones, receives a Falco pod automatically, and alert if any node lacks one.
Problem
Automatic termination of pods on rules that have not been validated.
Why It’s Harmful
False positives turn into self-inflicted outages, and the destroyed pod takes the evidence with it.
Correct Approach
Start with isolation and evidence capture, automate only well-understood cases, and keep a human in the loop for anything ambiguous.
Problem
Treating Falco as a replacement for scanning, admission control, and least-privilege design.
Why It’s Harmful
Detection alone means attackers get in and you merely learn about it. The result is more incidents and heavier workloads for responders.
Correct Approach
Use Falco as the detective layer inside a layered strategy that includes prevention and hardening.
20Real-World Scenarios Where Falco Earns Its Keep
These illustrative scenarios show how different organizations typically apply runtime detection.
Online payments platform
Teams handling card data use Falco to watch for shells, unexpected file reads, and unfamiliar outbound connections around payment services. Alerts feed audit evidence showing that production behavior is continuously monitored, which supports the monitoring and logging expectations found in many payment security standards.
Multi-tenant software provider
A provider hosting many customers on shared clusters watches for container escape signals and cross-namespace access. Because tenants share nodes, the early warning from kernel-level visibility is especially valuable.
Healthcare data service
An organization storing sensitive records watches for unusual reads of data paths and unexpected network egress. Detection complements encryption and access control by showing when the other safeguards are being tested.
Platform engineering team
A central team offering a shared cluster to many application teams uses Falco rules plus label-based routing, so each team gets its own alerts while the platform team keeps an overall view of fleet health and rule coverage.
Cost and abuse protection
Teams running public-facing workloads use rules that detect cryptocurrency miners and unexpected compute spikes, catching abuse that would otherwise show up first as a surprising cloud bill.
Regulated and audited environments
Organizations under frameworks that require continuous monitoring use Falco’s alerts and archives as part of their evidence trail, together with change records for rules and exceptions.
Tools support compliance but do not by themselves satisfy it. Auditors look for people, process, and evidence together. Use Falco to produce reliable evidence, and document who reviews alerts and how incidents are handled.
21Designing Your Own Rules Thoughtfully
Default rules give you coverage. Custom rules give you protection that fits your business.
Start from behavior, not from tools
New rule authors often begin by asking, “Which attack tool should I look for?” That question leads to fragile rules, because attackers rename tools, repackage them, and switch to others. A sturdier approach is to ask, “What behavior would my service never legitimately show?” A payment service should never start a shell. A static web server should never open a connection to a database it has no business with. A batch job should never write into system directories. Rules built on behavior survive changes in attacker tooling, because the behavior they describe stays unacceptable no matter which tool performs it.
A bank does not keep a list of every known bank robber’s face and hope the next robber is on it. It watches for behavior: someone lingering near the vault, someone wearing a mask indoors, someone testing locked doors. Behavior-based rules are the vault-watching approach, and they catch strangers as well as known faces.
A checklist for each new rule
State the threat in one sentence
If you cannot describe in plain words what bad thing the rule catches and why it matters, the rule is not ready to write.
Scope it tightly
Limit the rule to the namespaces, images, or labels where the behavior is truly unacceptable. A rule scoped to your payment workloads is far more precise than one applied to everything.
Write an output a stranger could act on
Include the user, program, command line, container, pod, and namespace in the message so the person who reads it at night needs no extra digging.
Pick an honest priority
Reserve the highest levels for strong signs of compromise. Over-rating rules teaches responders to ignore the top of the ladder.
Test in a safe place first
Run the rule in a staging cluster, deliberately trigger it with harmless activity, and watch for surprises from normal traffic before it reaches production.
Tag and document
Add tags such as the matching ATT&CK technique, and write a short note on the response expected when it fires.
Common pitfalls in rule design
Matching on names alone
Attackers can rename programs. Combine names with context such as parent process, location, and container identity.
Too broad a scope
A rule that applies everywhere will meet every legitimate exception in the cluster and become noisy.
Forgetting maintenance
Applications evolve. Rules written for last year’s architecture may miss new behavior or flag harmless new tools.
Costly conditions
Conditions that must examine every event are more expensive than those that first check a cheap, selective field. Put the most selective checks first.
Consider building a small library of reusable macros for your own organization, such as “inside a production namespace” or “payment workload”. Your custom rules then read like business policy, and a single edit updates every rule that relies on it.
22Operating Falco Across Many Clusters
One cluster is a project. A fleet is an operation, and it needs consistency.
The fleet challenge
Organizations rarely have just one cluster. They have development, staging, and production, often in several regions and sometimes across several cloud providers. If each cluster runs a slightly different Falco configuration, coverage becomes uneven and nobody can say with confidence what is protected. The goal of fleet operations is to make the same security intent apply everywhere, with controlled differences where the environment truly demands them.
Declare, version, and roll out
The most reliable pattern is to describe your Falco setup, including rules, exceptions, driver choice, and outputs, as configuration stored in version control. A deployment tool then applies that configuration to every cluster from one source of truth. Changes go through review, are tested in a low-risk environment, and are promoted gradually. This approach, commonly called GitOps, also gives you a history of who changed what and why, which is valuable during audits and investigations.
flowchart LR
A["Rule change proposed"] --> B["Peer review"]
B --> C["Test in staging cluster"]
C --> D["Canary: small production group"]
D --> E["Full production rollout"]
E --> F["Monitor alert volume and drops"]
F -->|"Problem found"| G["Roll back to previous version"]
Handling differences between environments
Development clusters
Engineers need freedom to debug, so interactive shells may be permitted and noted at a low priority, while credential theft and miner detection stay fully active.
Production clusters
The strictest rule set applies. Interactive shells, unexpected network destinations, and new executables raise high-priority alerts to the owning team.
Regulated workloads
Extra rules and longer alert retention apply to namespaces that handle sensitive data, with evidence kept according to your retention policy.
Central visibility
Alerts from every cluster should include a cluster name so they can be sent to one central place without confusion. A central store lets analysts search across the entire fleet, spot an attacker moving between environments, and compare alert volumes to find clusters that are unusually noisy or suspiciously quiet.
Letting engineers hand-edit Falco settings directly on a live cluster creates silent drift. Within months, no two clusters match, and nobody knows which one is the real baseline. Make the version-controlled configuration the only way changes are made.
23Protecting the Protector: Securing Falco Itself
A powerful security tool is also an attractive target. It deserves the same care you give your most sensitive systems.
Why Falco is a high-value target
Falco runs with elevated access on every node, sees the commands and file paths of every workload, and holds the credentials needed to send alerts to other systems. An attacker who can silence Falco gains freedom to act unseen. An attacker who can read its alerts learns how much you know about them. An attacker who can change its rules can create blind spots on purpose. So the protection of Falco is not an afterthought; it is part of the design.
Layers of protection
Restrict who can change it
Only a small, named group should be able to edit Falco’s namespace, configuration, and rule files, and every change should be traceable.
Grant the minimum
Prefer the narrow capability set that the modern eBPF approach allows over fully privileged containers, and give the service account only the permissions it truly needs.
Trust only verified artifacts
Pull Falco images and rule bundles from trusted sources, pin versions, and verify integrity where signatures are offered.
Encrypt alert delivery
Use secure connections to every destination and keep delivery credentials in the cluster’s secret management rather than in plain configuration.
Detect tampering
Alert when Falco’s configuration changes, when its pods restart unexpectedly, or when a node lacks a Falco pod.
Separate the storage
Send alerts off the node to a store the workloads cannot reach, so an intruder cannot erase the evidence of their own entry.
Alerts contain sensitive information
Command lines, file paths, user names, and image names can reveal sensitive details, and command lines sometimes contain passwords or tokens that users typed. Treat the alert stream as confidential data. Limit who can read it, set retention to match your policy and legal obligations, and consider whether certain fields should be masked before long-term storage.
Keep a copy of your alerts somewhere that is write-once or otherwise protected from deletion. In an incident, the integrity of your evidence matters as much as its content.
24When an Alert Fires: A Calm Response Workflow
The best time to decide how to react is before the alert arrives.
The first five minutes
When a serious alert arrives, the responder faces three questions in rapid succession. Is this real? How far has it gone? What should be done right now? A good workflow turns these into a short, repeatable sequence so that stress does not lead to improvisation.
Read the alert fully
Note the rule, the pod, the namespace, the program, the user, and the exact time. Check whether the owning team has a known change or debugging session in progress.
Look for surrounding events
Search the alert history for other rules that fired on the same pod or node in the minutes before and after. A single alert may be a fluke, while a chain of alerts tells a story.
Decide: benign, suspicious, or confirmed
Use the context and the owner’s input to classify the event. When uncertain, treat it as suspicious until proven otherwise.
Contain
Isolate the workload from the network so it cannot reach other systems or send data out, while keeping it running if you need to study it.
Preserve evidence
Capture logs, record the running state, and note the image version before terminating anything. Once a pod is gone, much of the evidence goes with it.
Eradicate and recover
Replace the workload with a clean one, rotate any credentials that may have been exposed, and fix the weakness that allowed entry.
Learn and improve
After the incident, ask which earlier alert could have caught it sooner, then adjust rules, exceptions, and response steps accordingly.
flowchart TD
A["Alert received"] --> B{"Known benign activity?"}
B -->|"Yes"| C["Add precise exception and close"]
B -->|"No or unsure"| D["Check related alerts"]
D --> E{"Signs of compromise?"}
E -->|"No"| F["Monitor and record findings"]
E -->|"Yes"| G["Contain workload"]
G --> H["Preserve evidence"]
H --> I["Replace workload and rotate credentials"]
I --> J["Improve rules and prevention"]
Deleting a suspicious pod immediately feels decisive, but it often destroys the only clues about how the attacker got in. Contain first, collect evidence second, and replace the workload third.
25Growing Up: A Maturity Path and the Numbers to Watch
You do not need to do everything at once. A staged path lets value arrive early and risk stay low.
Three stages of adoption
| Stage | Focus | Typical activities | Success looks like |
|---|---|---|---|
| Crawl: See | Visibility | Install on a pilot cluster, use default rules, collect alerts in a searchable place, learn what normal looks like | The team can answer “what is running and what is it doing” for the pilot |
| Walk: Understand | Signal quality | Tune with owners, add custom rules for critical assets, route alerts by team, connect paging for top severities | Alerts are rare enough that each one is read, and owners trust them |
| Run: Respond | Speed and coverage | Fleet-wide rollout through version control, careful automated responses, response drills, coverage reports mapped to attacker techniques | Typical incidents are contained within minutes with a recorded, repeatable process |
Metrics that tell the truth
False positive rate
The share of alerts closed as harmless. A high rate means tuning is overdue.
Time to acknowledge
How long between an alert firing and a human or automation taking charge.
Time to contain
How long until a confirmed incident is isolated. This is the number attackers care about most.
Node and rule coverage
Percentage of nodes running a healthy Falco pod and the attacker techniques your rules cover.
Dropped event ratio
The share of events never examined. It should stay at or very near zero.
Exception age and count
How many exceptions exist and when each was last reviewed. Stale exceptions hide risk.
Thinking about cost honestly
Falco itself is free, but running it is not. Costs include the processor and memory it uses on each node, the storage for alert history, the engineering time to tune and maintain rules, and the people who respond to alerts. The largest cost, by far, is usually human attention. This is why quality matters more than quantity: a smaller number of precise, trusted alerts saves more money than a flood of noisy ones, and tuning work pays for itself quickly.
Review your metrics monthly with both the security team and a few application owners. Owners know which behavior is normal, and security knows which behavior is dangerous. The most useful tuning decisions come from putting both perspectives in the same conversation.
26A Kubernetes Threat Catalog: What Attackers Do to Clusters
Kubernetes has its own special doors and windows. Knowing them tells you what to watch.
Why Kubernetes threats are different from ordinary server threats
A traditional server attack usually stays on one machine. A Kubernetes attack can spread through the orchestration layer itself. Every pod normally receives a credential that lets it talk to the cluster interface. Every node trusts the control plane. Every namespace shares the same underlying network unless told otherwise. An attacker who understands these relationships can turn a small foothold in one pod into influence over many workloads. Falco helps because it can see both the low-level activity inside the pod and the high-level cluster actions recorded in audit events.
Problem
Every pod can receive a service account credential stored in a well-known location. A compromised application can read it and use it to call the cluster interface.
Why It’s Harmful
If that credential has broad rights, the attacker can list secrets, read configuration, or create new workloads, all while appearing to be a legitimate service.
Correct Approach
Alert when unexpected programs read the credential location, remove credentials from pods that do not need them, and scope rights to the minimum. Pair this with audit rules that catch unusual cluster actions by service accounts.
Problem
A pod is created with extra privileges or with access to the node’s own file system or process space.
Why It’s Harmful
The walls between container and node become thin or disappear. From such a pod, an attacker can read files belonging to other workloads or manipulate the node.
Correct Approach
Use admission policies to refuse such pods outside a tiny allowed list, and use Falco to alert on any that appear anyway or on suspicious activity from the few that are allowed.
Problem
A process inside a container attempts to break out, for example by abusing a kernel flaw, a mounted host path, or a powerful capability.
Why It’s Harmful
A successful escape gives the attacker the node itself, and with it every workload and secret on that node.
Correct Approach
Watch for behaviors that precede escapes, such as changes to namespaces, attempts to load kernel features, or writes to sensitive host locations. Keep nodes patched and use the narrowest possible container privileges.
Problem
A workload starts scanning internal network addresses, querying the cluster interface for everything it can see, or probing other services.
Why It’s Harmful
Reconnaissance is how an intruder maps targets before moving sideways. It is often the quietest and earliest sign of an intrusion.
Correct Approach
Alert on unusual connection patterns and large numbers of cluster queries from identities that normally make few, and use network policies to limit which pods can talk to which.
Problem
An attacker runs cryptocurrency mining software inside a compromised workload, or creates new workloads purely for mining.
Why It’s Harmful
It burns your compute budget, starves legitimate workloads, and proves the attacker already has a foothold they could use for worse purposes.
Correct Approach
Alert on connections to mining pools, on known mining program behavior, and on newly created executables. Combine with resource limits and budget alerts as a second line of defense.
Problem
A trusted image or dependency has been altered so that it behaves normally at first and then performs hidden actions.
Why It’s Harmful
Because the software was trusted, scans and admission checks may pass it. Only its runtime behavior reveals the difference.
Correct Approach
Watch for unexpected network destinations, file reads, and process launches from otherwise trusted images. Runtime detection is one of the few ways to catch tampering that scanners cannot yet recognize.
Notice how every entry above ends with a pairing: Falco for detection plus something preventive. This is no accident. Reliable cluster defense comes from combining a detective layer with limits that make attacks harder in the first place.
27Mapping Detection to Attacker Techniques
A shared map of attacker behavior helps you see what you can catch and what you cannot.
What the framework is
MITRE ATT&CK is a public catalog that describes how real attackers behave, organized into stages called tactics, each containing specific techniques. Tactics include gaining initial access, executing commands, persisting after restarts, escalating privileges, evading defenses, stealing credentials, exploring the environment, moving sideways, and finally causing impact. Many Falco rules carry tags that point to the matching technique, so your alerts can be grouped and counted by attacker stage.
Imagine a coach’s whiteboard showing every play the opposing team might run, grouped by phase of the game. Your defense can be reviewed against that board: “We are strong against the opening plays and weak against the late-game moves.” The framework is that whiteboard for cyber attacks.
Reading your coverage honestly
| Attacker stage | What Falco can observe | Typical complementary control |
|---|---|---|
| Initial access | Limited. It sees the aftermath, such as unexpected processes in a web service | Image scanning, web application protection, strong authentication |
| Execution | Strong. Shells, new programs, and unusual command lines | Read-only file systems, minimal images without shells |
| Persistence | Good. Writes to startup locations and new scheduled jobs | Immutable workloads that restart from clean images |
| Privilege escalation | Good. Capability changes, privileged pods, risky mounts | Admission policies and least-privilege defaults |
| Defense evasion | Moderate. Tampering with logs and security tools | Off-node log storage and integrity monitoring |
| Credential access | Strong. Reads of credential files and secret access via audit events | Short-lived credentials and secret management |
| Discovery and lateral movement | Moderate. Unusual connections and cluster queries | Network policies and service identity |
| Impact | Good. Mining, destructive file operations, large outbound transfers | Resource limits, backups, egress controls |
Using the map to decide what to build next
A coverage map is not meant to reach one hundred percent. It is meant to start conversations. If you discover that credential access is strongly covered but lateral movement is thin, you can decide whether to write more rules, add network policy, or accept the risk with a clear record of the decision. Reviewing the map each quarter keeps detection work aimed at real gaps instead of at whatever is easy to write.
A tag on a rule says what the rule is intended to detect, not that the detection is perfect. Test each important rule with safe simulated behavior to confirm it really fires in your environment, before you claim coverage in a report.
28Testing Your Detections: Treat Security Like Software
A rule that has never been tested is a hope, not a control.
Why rules deserve tests
Software teams would never ship an application feature without testing it, yet security rules are often deployed on faith. Rules can quietly stop working after an update, after a change in an image, or after a refactoring of shared macros. Testing gives you proof that important detections still fire and that harmless activity still stays quiet.
flowchart LR
A["Write or change rule"] --> B["Automated syntax and loading check"]
B --> C["Replay known-good activity: expect silence"]
C --> D["Simulate harmless suspicious behavior: expect alert"]
D --> E["Review results"]
E --> F["Promote to production"]
Three kinds of checks
Does it load?
Automatically confirm that rule files are valid and that the engine accepts them. This catches typing mistakes and broken references before deployment.
Does it fire?
Perform a harmless action that should trigger the rule in a test cluster and confirm the alert appears with the expected fields.
Does it stay quiet?
Replay or run normal application behavior and confirm no alerts are produced, which protects against noisy rules.
Does the alert arrive?
Follow the alert all the way to the person or system that should receive it, since a perfect rule is useless if delivery is broken.
Purple team exercises
A purple team exercise brings together people who play the attacker role and people who play the defender role, working side by side. The attackers perform agreed, safe simulations of real techniques in a controlled environment, and the defenders check whether their tools noticed. Each missed technique becomes a concrete task: write a rule, tune an exception, or fix an alert path. Repeating these exercises a few times each year steadily raises confidence and quality.
Scope and permission first
Run simulations only in environments you own and with explicit approval. Tell the people who monitor alerts so they can distinguish exercises from real incidents.
Record the results
Keep a simple table of technique tested, whether an alert fired, how long it took to reach a person, and what was changed afterward.
Close the loop
Retest every fix. A rule is only considered improved when a second simulation confirms it now behaves as intended.
29Troubleshooting: When Falco Misbehaves
Most problems fall into a few families. Recognizing the family speeds up the fix.
| Symptom | Likely causes | What to check first |
|---|---|---|
| Falco pod will not start on some nodes | Kernel too old for the chosen driver, missing permissions, a locked-down node image | The pod’s startup messages and the kernel version of the failing node compared with a working one |
| No alerts at all | Rules not loaded, output not connected, driver not delivering events | Whether rule files loaded successfully, and whether a deliberate harmless trigger produces any output on the node |
| Far too many alerts | Default rules not tuned, a new workload or tool recently deployed | Which rule and which image produce the most alerts, then add a narrow exception where justified |
| Alerts missing container or pod details | Metadata component not working, short-lived containers, runtime connection problems | Whether the metadata component is healthy and whether the affected containers lived only a few seconds |
| Dropped events reported | Overly broad rules, busy nodes, undersized resources or buffers | Which rules are the most expensive, and whether the busiest nodes need more resources |
| Alerts generated but never delivered | Forwarder misconfiguration, expired credentials, blocked network path | The forwarder’s own logs and a manual test message to each destination |
| Problems after a node upgrade | Driver incompatible with the new kernel | Whether the node runs a supported kernel and which driver type is in use |
A simple method for any problem
Find out whether it is one node or all nodes
A problem on a single node points to that node’s kernel, image, or permissions. A problem everywhere points to configuration or rules.
Find out where in the chain it breaks
Follow the path: driver captures, engine evaluates, output writes, forwarder delivers, destination receives. The first broken link is the one to fix.
Compare with a known-good node
Differences in kernel, operating system image, or settings often explain the failure within minutes.
Change one thing at a time
Making several changes at once hides which change worked, and may introduce a new problem while fixing the old one.
Keep a short internal runbook of the problems you have already met and how you solved them. Every team hits the same handful of issues, and a runbook turns each second occurrence into a five-minute fix.
30A Plain-English Glossary
Keep these short definitions handy while reading dashboards, rules, and incident reports.
Kernel
The core of the operating system. It controls hardware and decides what every program is allowed to do.
System call
A request from a program to the kernel, such as opening a file, starting a program, or connecting to a network address.
eBPF
A kernel feature that runs small, carefully verified programs safely inside the kernel, widely used for observability and security.
Driver
The small component that captures system calls inside the kernel and hands copies to Falco in user space.
Rule
A description of suspicious behavior made of a condition, an output message, a priority, and optional tags.
Macro
A named, reusable piece of a condition, so common tests are written once and shared by many rules.
List
A named collection of values, such as common shell names, used inside conditions.
Exception
A precise description of known-good behavior that stops a rule from firing for that specific case.
DaemonSet
A Kubernetes workload type that places one copy of a pod on every eligible node.
Audit log
A record of requests made to the Kubernetes API, useful for spotting risky cluster changes.
Plugin
An add-on that lets Falco read a new source of events, such as audit logs or cloud activity.
Falcosidekick
A companion service that forwards Falco alerts to many chat, paging, logging, and storage destinations.
Response engine
A tool that listens to alerts and carries out predefined actions, such as isolating or replacing a pod.
Container drift
Running a program inside a container that was not part of its original image, a sign of tampering.
Dropped event
An event the kernel produced that Falco never examined because its buffer was full.
Least privilege
The principle of giving each component only the access it needs and nothing more.
Share this glossary with new team members and with application owners who will receive alerts. Shared vocabulary shortens every tuning conversation and every incident call, because people spend their time deciding what to do instead of asking what a word means.
31Frequently Asked Questions
Quick answers to the questions beginners ask most often.
On its own, Falco detects and reports. Blocking or reacting is done by companion tools such as a response engine, by your own automation, or by separate preventive controls like network policies and confinement profiles.
No. Falco watches the kernel on each node, so it works the same for any language or framework and requires no changes to application code or images.
The observation work is designed to be light, and filtering happens early. Real overhead depends on node activity and the number of rules, so measure in your own environment and set resource requests from those measurements.
On recent distributions the modern eBPF probe is generally the recommended choice because it needs no kernel headers or on-node compilation. Check your kernel version and node operating system, and keep an older option in mind for older kernels.
Often yes, but support depends on the provider and node image, especially for kernel-level access and audit log access. Test on a small pool first and read your provider’s guidance.
A scanner inspects software for known weaknesses before it runs. Falco watches live behavior while it runs. They answer different questions and work best together.
Start with the stable default set, tune it until it is quiet, then add more rules gradually. More rules are not automatically better; precise rules that people trust are better than many rules that people ignore.
If the event buffer overflows, some events are dropped and Falco reports the drop count. Monitor that metric and respond by simplifying overly broad rules, adding resources, or adjusting buffer settings.
Yes. With the audit plugin it can evaluate Kubernetes audit events using the same rule language, covering actions like secret access and privileged pod creation.
Send urgent alerts to the tool your on-call team uses, and send all alerts to a searchable store with a sensible retention period. The full history is invaluable when investigating incidents after the fact.
No. It also runs on ordinary Linux hosts and container hosts, and through plugins it can analyze cloud activity logs. Kubernetes is simply where it is most commonly deployed.
Run it as a per-node workload, monitor the number of healthy Falco pods against the number of nodes, and alert on any mismatch.
32Summary and Key Takeaways
Kubernetes security does not end when a workload passes its scans and policy checks. Containers live for hours or days, share a kernel with their neighbors, and face attackers who specialize in behavior that no earlier check could predict. Falco answers that challenge by watching system calls and Kubernetes audit events, enriching them with container and cluster context, and comparing them against readable rules that describe suspicious behavior. Its companions route alerts to the right people and, where it is safe, trigger a measured response. Used with tuning, ownership, and layered prevention, it turns an invisible runtime into one you can see and understand.
Key Takeaways
- Runtime is a separate stage of security — scans and admission rules protect build and deploy time, but only behavioral monitoring sees what a running workload actually does.
- Falco watches the kernel — by observing system calls it covers every language and framework without touching application code, and it is hard for an attacker to disable from inside a container.
- Context makes alerts useful — enrichment with process lineage, container, pod, and namespace turns raw events into stories a responder can act on quickly.
- Rules are readable building blocks — lists, macros, and exceptions let you start with community defaults and refine them precisely, instead of switching noisy rules off.
- Prefer the modern eBPF driver where kernels allow — it avoids kernel headers and on-node compilation, though you must confirm support for your node images first.
- Route, respond, and rehearse — use Falcosidekick for delivery, automate only well-proven responses, and practice the full path from alert to action.
- Monitor the monitor — watch for dropped events and missing pods, secure Falco’s own privileges, and combine detection with prevention for defense in depth.