We’re assuming you already know that Rekognition can detect objects, faces, and text in images. Here we go past that into the concepts that determine how a Rekognition deployment behaves and costs money in production.

Confidence scores, not booleans

Every Rekognition API response — whether it’s a detected label, a face match, or a piece of moderated content — comes back with a confidence score between 0 and 100, not a simple true/false. Rekognition never claims certainty; it reports how strongly its underlying model believes a given detection is correct. The API itself never decides what counts as “detected” — your application does, by choosing a minimum confidence threshold and discarding anything below it. This single design decision is the source of most Rekognition production bugs: a threshold set too low produces false positives (flagging safe content as inappropriate, matching the wrong face), while a threshold set too high produces false negatives (missing real matches or genuine violations).

Collections: vectors, not photos

A Rekognition Collection is a container used for face search, but it’s crucial to understand what it actually stores: not the original images, but face vectors — mathematical representations (feature embeddings) extracted from each detected face, along with a unique face ID and any external metadata you attach. When you later search a collection with a new photo, Rekognition extracts a vector from the new face and compares it mathematically against the stored vectors, returning similarity scores. Because the original image is never retained inside the collection itself, deleting an indexed face’s source photo elsewhere in your system does not remove it from the collection — you must explicitly delete the face by its face ID.

You may likeAWS Direct Connect→

Pre-trained APIs vs. Custom Labels

Rekognition ships two fundamentally different capabilities under one product name. The pre-trained APIs — label detection, face detection/comparison, text detection, celebrity recognition, content moderation, PPE detection — are models AWS already trained on massive general-purpose datasets; you call them, you don’t train them. Rekognition Custom Labels, by contrast, lets you train a model on your own labeled images to recognize domain-specific objects a general model was never trained to know — a specific machine part on a manufacturing line, a particular product logo, or a defect type unique to your industry. Custom Labels uses transfer learning under the hood, meaning it starts from Rekognition’s existing visual understanding and fine-tunes only the final layers on your smaller, specific dataset, which is why it can produce usable models from a few hundred images rather than the millions a model would need if trained from scratch.

Image analysis vs. video analysis: two different pipelines

Single-image analysis is a stateless, synchronous request-response pattern: you send bytes, you get a JSON response back within the same call. Video analysis is fundamentally different — it is inherently asynchronous and stateful, because a video can be hours long and needs frame-by-frame processing over time. For stored video (in S3), Rekognition Video runs as a job you start and poll or receive a completion notification for. For live streaming video, Rekognition Video connects to a Kinesis Video Stream and continuously emits results as the stream plays, which is an entirely different integration pattern from calling a single synchronous API.

Content moderation categories and hierarchy

Rekognition’s content moderation model doesn’t return a single “is this safe” flag — it returns a hierarchical taxonomy of moderation labels (top-level categories like “Explicit Nudity” or “Violence,” each with more specific sub-labels beneath them), each with its own confidence score. This lets an application make nuanced policy decisions — perhaps blocking one sub-category outright while only flagging another for human review — rather than being forced into a single binary safe/unsafe decision for all content types.

Bounding boxes, landmarks, and image coordinates

When Rekognition detects an object, face, or piece of text, it doesn’t just tell you that something was found — it tells you where, via a bounding box expressed as relative coordinates (a fraction of the image’s width and height, from 0 to 1, rather than absolute pixels). This makes the coordinates independent of the original image resolution, so the same bounding box math works whether the source photo was a phone snapshot or a high-resolution studio image. For faces specifically, Rekognition goes further and returns facial landmarks — precise points for eyes, nose tip, and mouth corners — along with pose estimates for roll, yaw, and pitch, which many applications use to filter out faces photographed at extreme angles before attempting a comparison, since accuracy degrades predictably as facial pose moves further from front-on.

Face attributes vs. face identity

It’s worth being precise about a distinction Rekognition draws internally: detecting a face and analyzing its attributes (estimated age range, emotion, whether eyes are open, presence of a smile) is a completely separate concern from establishing identity (matching a face against a Collection or another photo). The DetectFaces operation returns attributes without any notion of who the person is; only IndexFaces, SearchFaces, and CompareFaces deal with identity matching. Conflating these two — for example, assuming an emotion attribute says anything about whether two photos show the same person — is a common source of confused application logic for teams new to the API surface.

Rekognition is really two parallel systems wearing one name: an image pipeline and a video pipeline, sharing the same underlying models.

graph TD
    APP["Your Application"] --> IMGAPI["Rekognition Image API
(synchronous, stateless)"]
    APP --> VIDAPI["Rekognition Video API
(asynchronous, job-based)"]
    IMGAPI --> S3IN["S3 Bucket or Inline Bytes"]
    IMGAPI --> MODELS["Pre-trained Vision Models
(labels, faces, text, moderation, PPE)"]
    IMGAPI --> COLL["Face Collections
(vector store, not images)"]
    VIDAPI --> S3VID["Stored Video in S3"]
    VIDAPI --> KVS["Kinesis Video Streams
(live streaming input)"]
    VIDAPI --> SNS["Amazon SNS
(job completion notification)"]
    VIDAPI --> MODELS
    CUSTOM["Rekognition Custom Labels"] --> TRAIN["Training Pipeline
(transfer learning on your images)"]
    TRAIN --> ENDPOINT["Inference Endpoint
(started/stopped per project)"]
    APP --> CUSTOM
        

A detail worth surfacing explicitly: Rekognition Image and Rekognition Video are not simply “the same models applied to a different input format.” Video-specific operations like person tracking, activity segmentation across a timeline, and scene detection have no direct equivalent in the image API at all, because they require temporal context a single frame can never provide. Conversely, some image-only operations — celebrity recognition on a static photo, for instance — have video equivalents that behave subtly differently, returning a timeline of appearances rather than a single yes/no match. Treating the two APIs as interchangeable beyond their shared underlying vision models is a common early misunderstanding that leads teams to reach for the wrong operation for a given problem.

Regional availability and model versioning are two architectural details that quietly matter more than they first appear. Rekognition’s underlying models are versioned independently of the API itself, and AWS periodically rolls out improved model versions behind the same API surface — meaning the exact confidence score returned for a given image can shift slightly over time even though your code hasn’t changed. Applications with strict reproducibility requirements (for example, a compliance system that needs to explain a historical moderation decision) should log the model version alongside every stored result, since Rekognition exposes this information in its responses specifically to support that kind of auditability.

Not every Rekognition operation is available in every AWS region, and Custom Labels in particular has historically launched in a narrower set of regions than the core pre-trained APIs. Confirming regional availability for every operation a design depends on — early, before committing to a specific account and region layout — avoids the awkward discovery midway through a build that a needed capability simply isn’t offered where the rest of an application’s infrastructure already lives.

What actually happens between submitting an image and receiving a list of labels or a face match?

For a synchronous image call, Rekognition first decodes and normalizes the submitted image, then runs it through the specific model pipeline relevant to the API called — object/scene detection uses a convolutional neural network trained for multi-label classification across thousands of categories; face detection uses a separate model specialized for locating facial bounding boxes and landmark points (eyes, nose, mouth corners) before a distinct embedding model converts the detected face into a numeric vector. For a face comparison or search call, this vector is then compared against either a second submitted image’s vector (for CompareFaces) or every vector in a Collection (for SearchFacesByImage), using a distance metric that Rekognition converts into the 0–100 similarity score you see in the response.

For text detection, a different specialized model locates regions of an image likely to contain text, then runs an OCR-style recognition step on each region, returning both individual words and the lines they belong to, each with its own bounding box and confidence score — which is why text results come back structured, not as one flat block of extracted text.

For video, the internal working adds a temporal dimension: rather than analyzing one static frame, Rekognition Video samples frames across the timeline and tracks entities (a face, a person, a piece of text) across consecutive frames, returning timestamps for when each detection appears and disappears — which is what lets you ask “at what point in this two-hour video does this person first appear” instead of only “does this person appear somewhere in this video.”

Person tracking deserves particular attention because it solves a harder problem than frame-by-frame face detection alone: Rekognition Video assigns a consistent tracking ID to a person as they move through a scene, even across frames where their face briefly isn’t visible (they turn away from the camera, or another person momentarily blocks the view). This is achieved by combining face-level detection with body-position and movement continuity across nearby frames, rather than relying purely on repeated face matches — which is why person tracking can maintain identity continuity through brief occlusions that would otherwise break a naive frame-by-frame face-matching approach.

The path data takes through Rekognition looks very different depending on whether you’re indexing a face, moderating an upload, or training a custom model.

sequenceDiagram
    participant App as Application
    participant S3 as Amazon S3
    participant Rek as Rekognition
    participant Coll as Face Collection
    participant SNS as Amazon SNS

    App->>S3: Upload image/video
    App->>Rek: IndexFaces / DetectLabels / StartLabelDetection
    Rek->>S3: Read object (for S3-based calls)
    Rek->>Rek: Run model pipeline, extract vectors/labels
    Rek->>Coll: Store face vector + face ID (IndexFaces only)
    Rek-->>App: Synchronous response (image APIs)
    Rek->>SNS: Job completion notification (video APIs)
    SNS-->>App: Notify job done
    App->>Rek: GetLabelDetection (retrieve results)
        

Three lifecycle stages are worth calling out specifically because they’re where production issues tend to concentrate:

It’s worth being explicit that Rekognition itself is not a data store for your images — beyond the face vectors held in a Collection, Rekognition does not retain the media you send it after a request completes. Your own S3 buckets and databases remain the source of truth; Rekognition is purely an inference layer sitting on top of data you continue to own and manage.

This “stateless by default, stateful only where you explicitly opt in” design has a practical consequence for how teams build retry logic. Because a failed synchronous image call has no lingering side effect to clean up, retrying it is safe and simple. A failed IndexFaces call, however, might have partially succeeded from Rekognition’s perspective even if your application never received the response — for example, due to a network timeout after the vector was already stored. Idempotency-conscious pipelines handle this by checking whether a face matching the same source image already exists in the Collection before blindly retrying an IndexFaces call, avoiding the duplicate-vector problem described earlier in the reliability discussion.

Custom Labels training also has its own lifecycle nuance worth understanding before a first project: dataset versioning. Each time you add, remove, or relabel images in a Custom Labels project, you create a new dataset version rather than mutating the existing one in place, and a training job always references a specific dataset version explicitly. This means you can always trace exactly which images and labels produced a given trained model, and re-run training against an earlier dataset version if a later data addition turns out to have introduced mislabeled or low-quality images that hurt accuracy — a safety net that mirrors why software teams keep version history for code rather than only ever working against a single mutable copy.

Custom Labels introduces a fourth lifecycle worth tracing on its own: dataset preparation, training, evaluation, and deployment. You assemble a labeled image dataset (either manually annotated or imported with existing bounding-box labels), split it into training and test sets, and start a training job that Rekognition runs as a managed process, producing precision, recall, and F1 metrics per label once complete. Only after reviewing these metrics and deciding the model meets your bar do you start an inference endpoint to actually serve predictions — a deliberately gated process that keeps a barely-trained or under-performing model from silently reaching production traffic.

The trade-off is worth stating plainly for teams comparing Rekognition against a self-hosted open-source vision stack: a self-hosted approach offers full control over model architecture, training data, and inference infrastructure, but every one of those knobs becomes something your own team must build, monitor, and keep current as vision research advances. Rekognition’s pre-trained APIs will never match a bespoke model tuned exhaustively for one narrow task, but they typically get a team to a usable production feature in days rather than months — and Custom Labels exists specifically to close part of that gap for the cases where a general model genuinely isn’t enough.

Image and video workloads scale under very different constraints, and Custom Labels adds a third.

Synchronous image APIs scale primarily by request throughput — each account has default request-per-second limits per API operation, which AWS will raise on request as volume grows, but which must be accounted for in application-level retry and backoff logic during traffic spikes (a product launch, a viral upload event). Image size and complexity have a modest effect on latency, but the dominant scaling variable is simply how many calls per second your application generates.

Video processing scales along a different axis entirely: job concurrency and video duration. Rekognition Video can process multiple stored-video jobs in parallel, but each individual job’s completion time scales with the video’s length and the complexity of what’s being detected (face tracking across a two-hour video takes meaningfully longer than label detection on the same file). For live streaming analysis via Kinesis Video Streams, throughput is bounded by how many concurrent streams your account is provisioned to analyze simultaneously, which is a capacity dimension worth planning for explicitly if you’re building something like a multi-camera live monitoring system.

Custom Labels performance is bounded by the inference endpoint’s own provisioned capacity, which — unlike the pre-trained APIs — you explicitly start and can scale by running multiple inference units for higher-throughput batch scoring. Because you pay for endpoint uptime rather than only per-prediction, right-sizing how long an endpoint runs (start it before a batch job, stop it immediately after) is one of the most direct cost-versus-performance levers available in the entire service.

Batching strategy matters more for image workloads than it might first appear. Rekognition’s synchronous APIs process one image per call, so a system needing to analyze a large historical archive of millions of photos should think carefully about controlled concurrency — issuing enough parallel requests to use available throughput fully, without exceeding account-level rate limits and triggering throttling that then requires backoff and retry, which slows the overall job down more than a well-tuned concurrency level would have in the first place. Many production ingestion pipelines use a queue (Amazon SQS) in front of a fleet of Lambda functions specifically to smooth out this concurrency control automatically rather than managing it by hand.

Image preprocessing is an underrated performance and accuracy lever that has nothing to do with Rekognition’s own configuration at all. Extremely large source images add latency for no accuracy benefit beyond a certain resolution threshold, since the underlying models operate on a fixed internal input size regardless of how large the original file is — downscaling oversized images before submission reduces upload time and transfer cost without meaningfully harming detection quality. Conversely, images that are too small, heavily compressed, or poorly lit measurably reduce confidence scores and detection reliability, which is why production pipelines dealing with inconsistent-quality source images (security camera stills, old scanned photos) often include a basic image-quality gate before submitting to Rekognition at all, rejecting or flagging inputs unlikely to produce a trustworthy result rather than letting a low-quality image silently produce a low-confidence, easily-missed detection downstream.

Cold-start latency is generally not a concern for the pre-trained synchronous APIs, since they run on infrastructure AWS keeps continuously warm across all customers. Custom Labels inference endpoints behave differently: the first request immediately after starting an endpoint can carry noticeably higher latency while the model finishes loading into memory, which is worth accounting for in any workflow that starts an endpoint, immediately sends a single time-sensitive request, and expects instant results — a brief warm-up call after starting the endpoint, before real traffic arrives, is a simple mitigation many teams adopt.

You may likeAWS Certificate Manager→

As a fully managed AWS service, Rekognition’s own infrastructure availability is AWS’s responsibility, distributed across multiple Availability Zones within a region without any configuration required on your part. Your reliability responsibilities concentrate around three areas instead: handling API throttling gracefully, ensuring video job notifications are reliably received, and maintaining the integrity of your Face Collections over time.

For Face Collections specifically, reliability also has a data-integrity dimension: because a Collection is a live, mutable index, accidental duplicate indexing of the same face (from repeated uploads of similar photos) gradually degrades search precision over time by cluttering the vector space with near-duplicate entries. Production face-search systems typically implement de-duplication logic — checking whether a very similar face already exists before indexing a new one — as part of their own reliability practice, since Rekognition itself does not prevent duplicate indexing.

Regional failover is also worth planning for explicitly rather than assuming automatically. Because Face Collections and Custom Labels models are regional resources, a disaster recovery plan for a Rekognition-dependent system generally means maintaining a secondary Collection (kept in sync via your own replication logic, since Rekognition provides no native cross-region Collection replication) or re-training a Custom Labels model in a backup region ahead of time, rather than discovering during an actual regional outage that recovery requires rebuilding a vector index from scratch under pressure.

Graceful handling of partial video job failures is another reliability dimension specific to the video pipeline. A stored-video job can occasionally fail partway through processing due to an unsupported codec, a corrupted segment, or an unexpected content type — and because the job runs asynchronously, this failure surfaces only when your application checks status or receives the SNS notification, not immediately at submission time. Production video pipelines validate basic file integrity and format compatibility before submission where practical, and always check the job status field explicitly rather than assuming that receiving any SNS notification at all means the job succeeded.

Security for a computer vision service spans data access, biometric data handling, and increasingly, jurisdiction-specific legal obligations.

Least-privilege IAM design deserves special emphasis for any Rekognition workload touching faces. Because IndexFaces, SearchFaces, and DeleteFaces together constitute the full lifecycle of biometric data handling, granting broad Rekognition access to a service role that only ever needs DetectLabels for general object detection is an unnecessary expansion of what that role could do if compromised. Scoping IAM policies down to the specific operations each component of a pipeline actually calls — rather than a blanket AmazonRekognitionFullAccess-style policy — is a small amount of extra configuration effort that meaningfully reduces the blast radius of any credential leak.

Data residency and cross-border transfer are worth surfacing for organizations operating under regulations that restrict where personal data, including face vectors, may be processed or stored. Because Rekognition Collections and Custom Labels training data live in a specific AWS region, choosing that region deliberately — and ensuring source images referenced by S3 path are also stored in a compliant region — is a foundational compliance decision made once at design time, not something correctable easily after a system has already been indexing faces in the wrong region for months.

You may likeAWS Wavelength→

Rekognition integrates with Amazon CloudWatch for operational metrics — request counts, error rates, and throttling events per API operation — which should be the first place to look when diagnosing intermittent failures at scale. AWS CloudTrail separately logs management-plane and API-call activity for auditing who called which operation and when, which matters particularly for any workload touching face recognition or content moderation where an audit trail may be a compliance requirement rather than a nice-to-have.

For Custom Labels specifically, model performance metrics (precision, recall, and F1 score per label) are surfaced after training completes and should be tracked release over release as you retrain — a model that appeared to perform well in initial testing can degrade against real-world images that look meaningfully different from the original training set.

Structured logging of every inference decision — the input reference, the returned confidence scores, the threshold applied, and the resulting application action — is worth building deliberately rather than as an afterthought. Beyond satisfying audit requirements for sensitive use cases like moderation or face matching, this log becomes the dataset you’ll eventually need to answer questions like “how many false positives did our threshold produce last month” or “has accuracy degraded since we changed camera hardware” — questions that are impossible to answer retroactively if only the final pass/fail decision was ever recorded.

Cost monitoring belongs in the same conversation as operational monitoring, since Rekognition’s usage-based pricing means a runaway retry loop, an unexpectedly large batch job, or an idle Custom Labels endpoint left running overnight all show up as cost anomalies well before they show up as anything else. Tagging API calls by originating pipeline or feature (where the SDK and account structure support it) and reviewing cost by tag on a regular cadence catches this class of problem far earlier than waiting for a monthly bill to look unusually large.

Rekognition is almost always deployed as one stage in a larger event-driven pipeline rather than called in isolation. A typical pattern: an S3 upload triggers an AWS Lambda function via an S3 event notification, the Lambda function calls the relevant Rekognition API, and the result is written to a database or triggers a downstream workflow (approve/reject content, tag a photo, alert a moderator) via Amazon EventBridge or Step Functions for more complex multi-stage logic.

Serverless orchestration frameworks are a natural fit for the more elaborate moderation and review pipelines Rekognition commonly sits inside. AWS Step Functions state machines are frequently used to model the full decision tree — call Rekognition, branch on confidence score into “auto-approve,” “auto-reject,” or “send to human review” paths, write the outcome to a database, and notify a downstream system — as an explicit, visualizable workflow rather than as scattered conditional logic buried inside a single Lambda function. This becomes especially valuable once a moderation pipeline grows multiple stages (an initial automated pass, followed by a secondary specialized check for borderline content, followed by human review), since a state machine keeps that growing complexity legible to whoever maintains it later.

Infrastructure-as-code coverage for Rekognition tends to be partial in practice, worth flagging so teams don’t assume otherwise: Collections, Custom Labels projects, and IAM roles are generally well supported by CloudFormation and CDK, but the actual training dataset content and the images indexed into a Collection are data, not infrastructure, and require their own separate data-management and backup discipline outside the usual infrastructure-as-code pipeline.

Custom Labels projects are typically managed through infrastructure-as-code alongside the training dataset stored in S3, with a defined promotion path: train and validate a model version in a development project, then deploy the same model artifact’s inference endpoint in production once accuracy metrics clear an agreed threshold — mirroring the general MLOps discipline applied to any custom-trained model, not just Rekognition’s.

Multi-account architectures deserve a specific mention because they’re common for organizations with strict data governance requirements. A frequent pattern places the source images and video in a data-owning account, with a separate analytics account holding the Rekognition Collections, Custom Labels projects, and downstream processing logic, connected via cross-account IAM roles scoped narrowly to exactly the S3 prefixes and Rekognition operations required. This isolation limits the blast radius if credentials in the analytics account were ever compromised, and keeps usage-based Rekognition billing cleanly separated from the cost of storing the underlying media itself.

Blue/green deployment for Custom Labels models is worth planning for even though Rekognition doesn’t provide it automatically. Because starting a new inference endpoint version alongside the currently running one is fully supported, teams can route a small percentage of production traffic to a newly trained model version, compare its precision and recall against the incumbent on live data, and only fully cut over once confident — rather than replacing a working production model in a single all-or-nothing swap that offers no easy rollback if the new version underperforms in ways the offline evaluation metrics didn’t fully capture.

You may likeAmazon Route 53→

A pattern worth calling out on its own is graceful degradation under partial detection: rather than treating “Rekognition returned no faces” or “text detection found nothing” as an outright failure state, well-designed pipelines treat absence of detection as a valid, expected outcome and route it down a distinct code path (flag for manual review, apply a fallback rule) instead of throwing an error. Real-world images are messy — poor lighting, unusual angles, obscured subjects — and a pipeline that only handles the happy path of confident, clean detections will fail in visible, embarrassing ways precisely when it matters most, such as a security system that silently stops working the first time a camera lens gets slightly dirty.