We skip what sentiment analysis or entity recognition is in general. Instead, we look at the mechanics that only matter once you are training and operating custom models: annotation formats, multi-class versus multi-label classification, inference units, and the flywheel continuous-training model.

Custom Classification: Multi-Class Versus Multi-Label

Comprehend Custom Classification supports two distinct modes that behave very differently at training and inference time. Multi-class assigns exactly one label per document out of a fixed label set — useful for routing a support ticket to exactly one department. Multi-label allows a document to receive zero, one, or several labels simultaneously — useful for tagging a product review with any combination of “shipping,” “quality,” and “pricing” complaints at once. The training data format differs accordingly: multi-class training CSVs pair one label per row with the document text, while multi-label training CSVs use a pipe-delimited label column to express multiple simultaneous tags per document.

Custom Entity Recognition: Annotations Versus Entity Lists

Training a custom entity recognizer can use one of two data strategies. An entity list is a simple plain-text list of known entity values (for example, a list of every valid product SKU) paired with unannotated raw documents — Comprehend learns to generalize from exact-match occurrences, which works well when your entities are a closed, enumerable vocabulary. Annotations instead provide documents with explicit character-offset labels marking exactly where each entity appears in context — this is required when entities are open-ended (like extracting “claim reference numbers” that follow a pattern rather than existing in a fixed list) because the model needs surrounding context, not just string matching, to generalize to unseen values.

Inference Units and Real-Time Endpoint Capacity

A real-time custom model endpoint is provisioned in units of “inference units” (IUs), where each IU guarantees a fixed throughput ceiling — roughly 100 characters processed per second per document type at time of writing, though AWS periodically adjusts these figures. Unlike Lambda or many serverless AWS services, a Comprehend endpoint is not scale-to-zero: you provision a number of IUs and pay for them continuously while the endpoint exists, whether or not traffic is flowing, which fundamentally changes the cost-versus-latency trade-off compared to the asynchronous batch job model covered later.

Flywheels: Continuous Model Improvement

A Comprehend flywheel is a managed wrapper around the classification or entity recognition training lifecycle that automatically versions models, evaluates new training data against the currently active model, and promotes a new model version only if it outperforms the active one on held-out evaluation data. This removes the manual “train, evaluate, decide whether to deploy” loop that teams previously scripted themselves using the raw training and model-comparison APIs.

What happens inside AWS’s infrastructure between submitting a training job and getting a working, servable model.

The Training Pipeline

When you submit a custom classification or entity recognition training job, Comprehend provisions managed compute behind the scenes, splits your training data into an internal train/validation partition (unless you explicitly supply your own held-out test set), fine-tunes a managed transformer-based architecture on your labeled examples, and produces a versioned model artifact along with a performance report covering precision, recall, and F1 score per label. You never see or manage the underlying compute instances — this entire pipeline is opaque infrastructure, which is precisely the trade-off Comprehend makes against something like a self-managed SageMaker training job where you would control the instance type, framework version, and hyperparameters directly.

flowchart LR
    S3T[(S3 - Training Data)] --> TJ[Training Job]
    TJ --> SPLIT[Internal Train / Validation Split]
    SPLIT --> FT[Fine-Tune Managed Model]
    FT --> MV[Versioned Model Artifact]
    MV --> REP[Performance Report - Precision/Recall/F1]
    MV --> EP[Real-Time Endpoint]
    MV --> BJ[Async Batch Job]
    EP -->|IU-provisioned| RT[Real-Time Inference]
    BJ -->|reads/writes S3| S3O[(S3 - Batch Output)]
    

Confidence Scores and Threshold Tuning

Every prediction Comprehend returns — whether a built-in sentiment label or a custom entity match — includes a confidence score between 0 and 1. Comprehend itself applies no business threshold; it returns everything above its internal minimum detection bar. Production systems almost always add their own threshold logic downstream (for example, only auto-routing a ticket if classification confidence exceeds 0.85, otherwise routing to human review), because the cost of a wrong automated action is rarely symmetric with the cost of an unnecessary human review step.

Tracing a document’s complete life, from raw text to a stored, actionable prediction.

Why Language Detection Placement Matters

A document processed with the wrong language model does not typically fail loudly — it silently produces low-quality or nonsensical entity and sentiment results, because the tokenizer and learned patterns assume a different language’s grammar and vocabulary. Production pipelines treat dominant-language detection as a mandatory first stage, not an optional nicety, precisely because failures here are silent rather than loud.

The Central Trade-off: Managed Simplicity Versus Model Control

Comprehend trades away the flexibility of choosing your own base model, framework, or training regime for a fully managed pipeline that a small team can operate without dedicated ML infrastructure expertise. Teams that need architecture-level control — custom transformer variants, non-standard loss functions, or multi-task heads — outgrow Comprehend and move to SageMaker, where they take back full control at the cost of taking back full operational responsibility.

How real-time endpoints and asynchronous batch jobs scale differently, and why choosing the wrong one for your traffic shape causes either latency pain or cost pain.

Real-Time Endpoints: Provisioned, Not Elastic by Default

A real-time custom model endpoint’s throughput ceiling is a direct function of how many inference units you provision. Traffic above that ceiling is throttled, not auto-scaled, unless you explicitly configure endpoint auto-scaling (where supported) with a minimum and maximum IU range. This makes real-time endpoints best suited to workloads with a known, relatively steady request rate — an inbound customer message classifier running continuously, for example — rather than highly spiky workloads.

Asynchronous Batch Jobs: Throughput Over Latency

Batch jobs (custom classification jobs, entity detection jobs, or built-in analysis jobs for sentiment, key phrases, and PII) trade latency — results can take minutes depending on corpus size — for a pay-per-job-run cost model with no continuously provisioned capacity. This makes batch analysis the correct default for periodic, high-volume workloads such as nightly processing of the day’s accumulated support tickets, reserving real-time endpoints specifically for interactive, low-latency use cases.

As a fully managed regional service, Comprehend’s availability is AWS’s responsibility for the underlying infrastructure — you do not manage or patch any serving hosts. Real-time endpoints are backed by managed, redundant compute within the region, and AWS handles host failure and replacement transparently, similarly to how a managed database service abstracts away individual node failures from the application layer.

What Reliability Comprehend Does Not Give You

Comprehend is a regional service with no built-in cross-region failover for custom models or endpoints. Teams requiring multi-region resilience must explicitly replicate training data and re-train or re-import model versions into a secondary region’s endpoint, and route traffic between regions themselves — there is no managed active-active mode analogous to a globally replicated database.

Encryption

Comprehend supports encryption at rest for training data, model artifacts, and batch job output using customer-managed AWS KMS keys, and all API traffic uses TLS in transit by default. For custom models, you can specify a KMS key both for the volume used during training and for the storage of the resulting model artifact, giving you separate control over data-at-rest protection at each pipeline stage.

Built-In PII Detection and Redaction

Comprehend includes a dedicated PII detection capability that identifies categories like names, addresses, credit card numbers, and social security numbers within text, and can optionally redact them (replacing detected spans with a placeholder) as part of an asynchronous analysis job. This is frequently used as a pre-processing stage before other text analytics run, ensuring downstream classification or entity extraction never operates on raw sensitive data unnecessarily.

IAM and VPC Isolation

IAM policies scope Comprehend permissions per action — training, endpoint management, and inference can each be granted independently, allowing a data science role to train and version models while a separate application role is only permitted to invoke inference against an already-published endpoint. For workloads requiring network isolation, Comprehend supports VPC configuration for training and inference jobs, keeping traffic on private AWS network paths rather than the public internet.

The handful of CloudWatch metrics and CloudTrail events that actually predict problems before they become customer-visible.

Model Quality Drift Is Not a CloudWatch Metric

CloudWatch tells you whether the service is healthy; it tells you nothing about whether your model’s predictions are still accurate against today’s real-world text. Because language and business categories drift over time, teams typically build a separate, application-level monitoring loop — periodically sampling live predictions for human review, or tracking a proxy signal like the rate of low-confidence predictions — since Comprehend itself has no built-in drift detection for custom models.

Comprehend is rarely called in isolation — its value comes from sitting inside a larger text-processing pipeline. The most common production topology lands raw documents in S3, triggers an asynchronous batch job (or a Step Functions workflow orchestrating multiple Comprehend calls in sequence), and writes structured results back to S3 or a downstream data store for search, analytics, or routing.

flowchart TB
    SRC[Support Tickets / Reviews / Documents] --> S3IN[(S3 - Raw Input)]
    S3IN --> SF[Step Functions Workflow]
    SF --> LANG[DetectDominantLanguage]
    SF --> PII[PII Detection & Redaction]
    SF --> CLS[Custom Classification Job]
    CLS --> S3OUT[(S3 - Structured Output)]
    S3OUT --> LAMBDA[Lambda - Route by Label]
    LAMBDA --> HELPDESK[Helpdesk / CRM System]
    

Comprehend Medical: A Separate, Specialized Service

It is worth distinguishing standard Comprehend from Comprehend Medical, a related but separately billed and separately trained service purpose-built for clinical text — extracting medical conditions, medications, and protected health information with domain-specific accuracy that general-purpose Comprehend models are not tuned for. Teams processing clinical notes should default to Comprehend Medical rather than attempting to train a general Comprehend custom entity recognizer for the same task.

“A custom model is only as good as the boundary case it has never seen — that’s what confidence thresholds and human-review fallbacks are actually for.”