Skipping “what is sensitive data discovery” — this chapter covers the concepts that only matter once Macie is running against a real, large estate.

Macie has two entirely different jobs, and conflating them causes bad architecture

Macie’s first job is bucket-level inventory and risk scoring — this runs continuously and automatically the moment Macie is enabled, at no incremental job cost, evaluating every bucket in scope for public accessibility, encryption status, sharing configuration, and a bucket-level “sensitivity score” derived from prior scan history. Macie’s second, entirely separate job is sensitive data discovery jobs — deliberate, scoped scans that actually open objects, sample their content, and classify what’s inside using machine learning models and pattern-matching against built-in and custom identifiers. Teams that assume “Macie is scanning everything all the time” because the dashboard shows bucket risk scores are misreading the product — bucket inventory is passive metadata analysis; content classification is an explicit, billed, opt-in job against a chosen scope.

Sampling, not exhaustive scanning, is the default behavior at scale

When a discovery job runs against a large bucket, Macie does not necessarily open every single object — for cost and performance reasons, jobs can be configured to sample a percentage of eligible objects rather than scan 100%, and even at 100% sampling, Macie applies object-size and object-count eligibility rules (extremely large objects are chunked and only representative portions are classified in some scenarios) before spending compute on classification. This is the single most misunderstood aspect of Macie by teams new to the service: a “clean” scan result on a sampled job is a statistical statement about likely exposure, not an absolute guarantee that zero sensitive data exists anywhere in the bucket.

Managed data identifiers vs. custom data identifiers are fundamentally different detection mechanisms

Managed data identifiers are AWS-built, continuously maintained detectors for common sensitive data types (credit card numbers, national ID formats across dozens of countries, AWS credentials, private keys) that combine regular-expression pattern matching with contextual keyword proximity and, for select types, machine learning models trained to reduce false positives — for example, distinguishing a real US Social Security Number from a random nine-digit invoice number sitting near unrelated text. Custom data identifiers, by contrast, are regex patterns you define yourself, with optional keyword proximity rules and a maximum match distance — they carry no ML-based false-positive suppression at all, meaning a poorly scoped custom identifier can flood findings with noise at scale in a way a managed identifier is specifically engineered to avoid.

Findings are not the same thing as “sensitive data exists”

Macie emits two categories of findings: policy findings (about the bucket itself — public access, missing encryption, shared outside the account) and sensitive data findings (about content discovered inside objects during a job). A policy finding can exist with zero sensitive data present — a perfectly empty bucket can still generate a policy finding for being publicly readable. Conflating the two finding types is a common source of miscommunication between security and platform teams during incident triage.

graph TB
    subgraph Continuous["Always-On: Bucket Inventory"]
        S3B[S3 Buckets] --> INV[Macie Automated
Sensitivity & Risk Scoring]
        INV --> PF[Policy Findings
public access, no encryption, sharing]
    end
    subgraph OnDemand["On-Demand: Discovery Jobs"]
        JOB[Scoped Discovery Job] --> SAMPLE[Sampling & Eligibility Rules]
        SAMPLE --> MDI[Managed Data Identifiers
+ ML false-positive suppression]
        SAMPLE --> CDI[Custom Data Identifiers
regex only, no ML]
        MDI --> SDF[Sensitive Data Findings]
        CDI --> SDF
    end
        

What actually happens, mechanically, between scheduling a discovery job and a finding appearing in the console.

Job planning phase: object selection before a single byte is read

When a discovery job starts, Macie’s control plane first enumerates the scoped buckets and prefixes, applies any include/exclude rules (file extension, object age, tags), then applies the sampling configuration to select a working set of objects — this planning phase produces a manifest of exactly what will be classified before any classification compute is spent, which is also how Macie estimates job cost up front for you.

Extraction and format-aware parsing

For each selected object, Macie must first determine whether it can extract text at all — it natively understands plain text, a range of structured formats (CSV, JSON, Parquet, Avro), common document formats (Word, PDF, Excel), and compressed archives (it will decompress and inspect contents of supported archive formats up to configured depth and size limits). Encrypted objects it lacks a key for, unsupported binary formats, and archives nested beyond configured limits are marked as “unable to process” rather than silently skipped — this distinction matters enormously for audit purposes, since “scanned, found nothing” and “could not be scanned” are very different security postures.

Classification: pattern matching plus contextual scoring

Extracted text is run against the selected managed and custom data identifiers. For managed identifiers with ML components, a raw regex match (say, a 16-digit number matching a credit card pattern) is scored against surrounding context — nearby keywords like “card,” “exp,” or “cvv” increase confidence, while a 16-digit number embedded in what’s clearly a product SKU list decreases it. This contextual scoring is precisely what keeps Macie’s managed identifiers usable at scale — without it, any large numeric dataset would generate overwhelming false-positive noise.

Aggregation into findings, not per-match records

Macie does not emit one finding per matched string — it aggregates matches at the object level into a single sensitive data finding per object per job, with counts and examples of what was found, categorized by type (financial, personal, credentials) and severity. This aggregation is a deliberate design choice to keep finding volume manageable even when an object contains thousands of matching instances.

sequenceDiagram
    participant J as Discovery Job
    participant Plan as Planning & Sampling Engine
    participant Ext as Extraction Layer
    participant Class as Classification Engine
    participant F as Findings Store

    J->>Plan: Enumerate scoped buckets/prefixes
    Plan->>Plan: Apply sampling %, size/type filters
    Plan-->>J: Manifest of objects to scan
    J->>Ext: Fetch & extract text per object
    Ext-->>J: Extracted content or "unable to process"
    J->>Class: Run managed + custom identifiers
    Class->>Class: Context scoring (ML-assisted for managed types)
    Class-->>J: Aggregated per-object matches
    J->>F: Emit sensitive data finding
        

Tracing a bucket from first enablement through ongoing monitoring, and where the lifecycle silently degrades in a growing estate.

Enablement is account-wide and organization-aware

Enabling Macie activates continuous bucket inventory for every bucket the enabling principal has visibility into, immediately. In an AWS Organizations setup, a delegated Macie administrator account can enable Macie centrally across every member account, aggregating findings into one place — this is the standard enterprise pattern, since running Macie independently per account produces fragmented visibility exactly when centralized visibility is the entire point of running it.

New object lifecycle: inventory updates continuously, discovery does not

Bucket-level inventory metrics (object count, size, public access status) refresh on a rolling basis as S3 changes. Sensitive data findings, however, only reflect the state of objects at the time a discovery job last scanned them — a bucket can accumulate large volumes of new, unscanned sensitive data between job runs, and nothing in the bucket inventory layer will surface that, because inventory doesn’t look inside objects at all. This gap is why mature Macie deployments schedule recurring discovery jobs (daily or weekly against high-churn buckets) rather than relying on one-time scans.

Finding lifecycle: suppression, archival, and retention

Findings can be manually or automatically suppressed via suppression rules (for known, accepted risks — a test bucket with intentionally synthetic PII, for example) so they stop generating alert noise without being permanently deleted from the record. Findings are retained in Macie for 90 days by default; anything needed longer must be exported — typically to S3 via the findings export configuration, or forwarded to Security Hub / EventBridge for longer-lived downstream storage. Teams that don’t configure export discover, months later during an audit, that their finding history simply isn’t there anymore.

Macie’s scaling story is about job cost and duration curves against object count and size, not raw throughput limits.

Job duration scales with object count more than with total bytes

A bucket with ten million small objects generally takes longer to classify than a bucket with the same total byte volume held in a handful of very large files, because per-object overhead (fetch, extract, classify, aggregate) dominates at high object counts. Advanced teams profile their estate’s object-size distribution before scoping a job, since a “small” bucket by byte count can still be an expensive, slow scan if it’s millions of tiny objects.

Sampling percentage is the primary cost/coverage lever

Because job cost scales with objects and bytes evaluated, sampling percentage is the single biggest lever for controlling both cost and job duration. A common pattern for very large, low-risk buckets is a low sampling percentage (5–10%) on a recurring schedule, trending sensitivity over time, paired with 100% coverage triggered specifically for buckets that a policy finding flags as newly public or newly shared — directing expensive full scans precisely at newly elevated risk rather than uniformly everywhere.

Custom data identifier complexity affects classification latency

Overly broad or poorly anchored custom regex identifiers (a pattern with excessive backtracking, or no keyword proximity constraint at all) can measurably slow classification throughput per object, in addition to generating more findings noise. Well-scoped custom identifiers with tight proximity windows perform closer to managed identifier speed.

Macie is regional, and cross-region visibility requires deliberate aggregation

Macie operates per region — buckets in different regions are inventoried and scanned by the Macie instance in their own region. Organization-wide, multi-region visibility requires the delegated administrator account to have Macie enabled and configured in every region where member accounts hold data, and findings must be aggregated centrally (typically via Security Hub, which does support cross-region aggregation) if a single unified view across regions is required — Macie itself does not merge findings across regions automatically.

Job reliability: partial completion is a real, expected state

Large discovery jobs against buckets with millions of objects can encounter individual object-level failures (permission issues, unsupported formats, transient S3 errors) without failing the entire job — Macie tracks per-object processing status and reports how many objects were successfully processed, skipped, or errored, rather than treating any single object failure as a job-wide failure. Reliability engineering here means monitoring the *skipped/errored* count per job, not just whether the job status shows “complete,” since “complete” can still mean a meaningful fraction of objects were never actually classified.

Inventory availability during account or service disruption

Because bucket-level inventory is derived from S3 metadata Macie has already indexed, a transient Macie service disruption does not retroactively invalidate previously computed bucket risk scores — the console continues to show the last known state. New discovery jobs simply cannot be started or completed until service is restored, which is a lower-severity failure mode than, say, losing visibility into already-known risk.

graph LR
    A[Discovery job runs
across millions of objects] --> B{Per-object outcome}
    B -->|Success| C[Classified & aggregated
into finding]
    B -->|Permission/format issue| D[Marked unable to process
job continues]
    B -->|Transient S3 error| E[Retried, then marked
errored if exhausted]
    C --> F[Job reports: processed / skipped / errored counts]
    D --> F
    E --> F
        

Macie needs read access to the data it’s protecting — and that’s a real risk surface

For Macie to classify object content, its service-linked role (or the delegated administrator’s cross-account role, in an Organizations setup) must be able to read object content across every in-scope bucket. This means the Macie execution path is itself a high-value target — a security control whose job is finding sensitive data necessarily has broad read access to that same sensitive data, and the IAM policies governing who can configure Macie jobs, export destinations, and suppression rules deserve the same scrutiny as any other broad-read-access service.

Findings export destinations are a data exfiltration surface if misconfigured

Because sensitive data findings can include example snippets of the actual sensitive content matched (configurable, and something advanced teams deliberately disable in most cases), the S3 bucket or Security Hub/EventBridge pipeline receiving exported findings must be locked down at least as tightly as the source data itself — an exported findings bucket with example snippets and loose access controls can become a lower-friction path to the exact sensitive data Macie was deployed to protect.

Suppression rules are a governance risk if ungoverned

Any principal with rights to create suppression rules can silence findings for a bucket indefinitely — a legitimate need for known false-positive patterns, but also a mechanism a malicious or careless actor could use to hide a genuine exposure from ongoing monitoring. Advanced deployments restrict suppression-rule creation to a small, audited set of principals and periodically review active suppression rules as part of a security review cycle, rather than treating them as fire-and-forget configuration.

KMS-encrypted objects require explicit key access

Objects encrypted with customer-managed KMS keys are only classifiable if Macie’s role is explicitly granted decrypt permission on those keys — this is not automatic, by design, since blanket decrypt access across every KMS key in an account would be its own significant security concern. Teams frequently discover gaps here only when a discovery job reports a cluster of “unable to process” objects that all trace back to a KMS key policy that was never updated to include Macie’s role.

EventBridge is the integration point for real-time response

Macie publishes findings to EventBridge as they’re generated, which is how mature deployments wire policy findings (a bucket just became public) into near-real-time automated response — for example, an EventBridge rule that triggers a Lambda function to immediately re-apply a blocking bucket policy the moment a “publicly accessible” policy finding fires, closing the exposure window from hours to seconds.

Security Hub aggregation for cross-account, cross-region rollups

Security Hub is the standard destination for consolidating Macie findings alongside GuardDuty, Config, and other security services into one place, and it’s the mechanism that solves Macie’s per-region isolation described in Chapter Six — Security Hub can aggregate findings from multiple regions and accounts into a single administrator view, which native Macie configuration alone cannot do.

What to actually track: job health, not just finding counts

Beyond raw finding counts (which trend up simply as an estate grows and tell you little on their own), advanced monitoring tracks: the skipped/errored object percentage per job, the trend of “unable to process” objects tied to KMS key access gaps, sampling percentage relative to bucket sensitivity tier, and days-since-last-scan per bucket — a bucket that hasn’t had a discovery job run against it in six months while accumulating new objects daily is a coverage gap the finding-count dashboard alone will never surface.

Organizations-based centralized deployment is the enterprise default

A delegated administrator account enables Macie across all member accounts in an AWS Organization, with member accounts able to opt in or be centrally enrolled depending on organizational policy. This single-pane deployment model is what makes Macie viable for organizations with dozens or hundreds of accounts — running it independently per account, with no central rollup, quickly becomes unmanageable and defeats the purpose of unified sensitive-data visibility.

Infrastructure-as-code patterns for job scheduling

Discovery jobs, custom data identifiers, and allow/suppression lists are all API-addressable resources, and advanced teams manage them via CloudFormation, Terraform, or CDK rather than console clicks — this is what makes the “tiered scanning by risk signal” pattern from Chapter Five actually maintainable: job definitions checked into version control, reviewed like any other infrastructure change, rather than living as tribal knowledge in a console someone configured once.

Integration with data lake governance tooling

In data lake architectures using AWS Lake Formation, Macie findings are commonly used as an input signal to Lake Formation’s fine-grained access control decisions — a prefix flagged with high-severity sensitive data findings can trigger tightened column- or row-level permissions in Lake Formation, effectively closing the loop between “we found sensitive data here” and “access to that data is now more tightly controlled,” rather than leaving that connection as a manual follow-up step for a human to remember.

graph TD
    ORG[AWS Organizations] --> DA[Delegated Macie
Administrator Account]
    DA --> MA[Member Account A]
    DA --> MB[Member Account B]
    DA --> MC[Member Account C]
    MA --> S3A[S3 Buckets]
    MB --> S3B[S3 Buckets]
    MC --> S3C[S3 Buckets]
    DA --> SH[Security Hub
cross-region rollup]
    DA --> EB[EventBridge
real-time automation]
    DA --> LF[Lake Formation
governance signal]
        

Pattern: Risk-triggered escalation from sample to full scan

Baseline low-sample recurring scans across the estate, with an automated EventBridge-driven trigger that escalates a bucket to a 100%-coverage job the moment its policy findings or object metadata change in a way that raises risk (newly public, newly shared cross-account, encryption removed). This concentrates the most expensive scanning exactly where it’s most needed.

Pattern: Findings as a governance feedback loop, not just a dashboard

High-severity sensitive data findings feed directly into automated remediation (tightening bucket policies, updating Lake Formation permissions, triggering a ticket in the security team’s queue) rather than sitting in a console someone checks occasionally. The goal is closing the loop between detection and action programmatically.

Pattern: Custom identifiers scoped narrowly with keyword proximity

Rather than a broad regex intended to “catch everything,” effective custom data identifiers pair a tight pattern with a required nearby keyword and a short maximum match distance — trading a small amount of recall for a large reduction in false-positive noise, which keeps the finding stream trustworthy enough that people keep paying attention to it.

Anti-Pattern: Enabling Macie and never scheduling a discovery job

Bucket-level inventory alone gives zero visibility into actual content — teams that enable Macie, see the dashboard populate with bucket risk scores, and stop there have effectively no sensitive-data discovery coverage at all, despite believing otherwise.

Anti-Pattern: One-time scan treated as permanent assurance

A single discovery job run once, months or years ago, against a bucket that has since accumulated enormous volumes of new data, provides essentially no assurance about current state. Discovery must be recurring and scheduled against the actual rate of change of the underlying data.