Skipping “what is a foundation model,” this chapter builds the vocabulary you need before touching architecture: how Bedrock actually organizes models, requests, and customization.

Foundation Models as a Catalog, Not a Single Product

Bedrock does not host one model — it hosts a catalog of foundation models from multiple providers, each identified by a model ID such as a Claude or Llama variant. Every model in the catalog exposes the same outer request/response contract (the Converse API and the InvokeModel API), even though each provider’s model has its own internal input schema, context window size, and pricing. This uniform contract is what lets an application swap the underlying model with a one-line configuration change rather than a full code rewrite.

On-Demand, Provisioned Throughput, and Batch Inference

Bedrock offers three distinct ways to pay for and consume inference capacity. On-Demand mode charges per token processed, with no capacity reservation, ideal for unpredictable or low-to-moderate traffic. Provisioned Throughput reserves a guaranteed level of model capacity for a fixed hourly cost, needed when an application requires consistent, predictable latency at high volume. Batch inference processes large sets of prompts asynchronously at a lower price point, well suited to offline workloads like bulk document summarization that don’t need real-time responses.

Embeddings vs. Generative Models

Bedrock’s catalog spans two fundamentally different model categories. Generative (text, image) models produce new content from a prompt. Embedding models instead convert text or images into dense numerical vectors that capture semantic meaning — two sentences with similar meaning produce vectors that sit close together in vector space. This distinction matters because embeddings are the foundation of retrieval, not generation, and the two categories are typically combined rather than used interchangeably.

Bedrock is a serverless front door, but behind that door sits a set of distinct, composable services.

Custom Model Import and Fine-Tuning

Beyond the base catalog, Bedrock supports two customization paths: fine-tuning a supported foundation model on your own labeled examples to shift its behavior toward a specific task, and Custom Model Import, which lets you bring a compatible model you’ve already trained or fine-tuned elsewhere and serve it through Bedrock’s standard inference infrastructure without managing GPU servers yourself.

Vector Stores as a Pluggable Backend

Knowledge Bases for Bedrock doesn’t include its own proprietary vector database — it plugs into one of several supported vector stores, including Amazon OpenSearch Service, Amazon Aurora with the pgvector extension, Pinecone, and Redis. This pluggable design means teams already running one of these stores for other purposes can reuse it rather than standing up a new dedicated system.

graph TD
    App[Application] --> API[Bedrock API Layer]
    API --> Guard[Guardrails]
    Guard --> Model[Foundation Model Runtime]
    API --> KB[Knowledge Bases]
    KB --> Vec[Vector Store]
    API --> Agent[Bedrock Agents]
    Agent --> Lambda[Action Group - Lambda]
    Agent --> KB
        

A single “chat” call actually triggers a small pipeline of distinct processing stages, most of them invisible from the outside.

Tokenization and Prompt Assembly

Before a model sees your prompt, it’s converted into tokens using that specific model’s tokenizer — different model families use different tokenization schemes, which is one reason the same text can consume a different number of tokens (and therefore cost) depending on which model processes it. System prompts, conversation history, and any retrieved context from a Knowledge Base are all assembled into a single structured input before this tokenization step.

Guardrails Evaluation, Twice

When Guardrails are attached to a request, evaluation happens at two separate points: once on the incoming prompt (blocking disallowed topics or detecting prompt injection attempts before the model ever runs) and once on the outgoing response (filtering harmful content or redacting sensitive data the model may have generated). This dual-checkpoint design means a request can be blocked before incurring the cost of a full model generation, while a response can still be intercepted even if the model itself produces something it shouldn’t.

Streaming vs. Synchronous Responses

Bedrock supports both a synchronous response, where the caller waits for the entire generation to complete, and a streaming response, where tokens are sent back incrementally as they’re generated. Streaming dramatically improves perceived latency for chat interfaces — the user sees the first words almost instantly rather than waiting for the full answer — even though the total generation time is roughly the same either way.

Retrieval-Augmented Generation, Step by Step

When a Knowledge Base is attached, an incoming query is first converted into an embedding vector, that vector is used to search the vector store for the most semantically similar document chunks, and those chunks are injected into the prompt as context before the model generates its final answer. This lets a general-purpose foundation model answer accurately about private, proprietary documents it was never trained on — the model isn’t memorizing your documents, it’s being handed the relevant excerpts at request time.

From raw documents to a grounded answer, data passes through a well-defined ingestion and retrieval pipeline.

The Knowledge Base Ingestion Pipeline

Source documents in Amazon S3 are first split into smaller, overlapping chunks — a necessary step because embedding models and context windows have size limits, and overlapping chunk boundaries prevent a fact from being awkwardly split across two disconnected chunks. Each chunk is then passed through an embedding model to produce a vector, and both the vector and the original text are written to the configured vector store, ready for similarity search.

sequenceDiagram
    participant S3 as S3 Documents
    participant Ing as Ingestion Job
    participant Emb as Embedding Model
    participant Vec as Vector Store
    participant App as Application Query
    S3->>Ing: Sync source documents
    Ing->>Ing: Split into chunks
    Ing->>Emb: Send each chunk
    Emb->>Vec: Store vector + text
    App->>Vec: Query with embedded question
    Vec-->>App: Return top matching chunks
        

Automated Re-Sync and Incremental Updates

A Knowledge Base data source can be re-synced on demand or on a schedule; Bedrock tracks which documents have changed since the last sync and only re-processes the delta, avoiding the cost of re-embedding an entire document corpus every time a handful of files are updated.

Model Invocation Logging

Bedrock can be configured to log every invocation’s request and response payloads to Amazon S3 and Amazon CloudWatch Logs, which becomes the backbone for later auditing, debugging unexpected model outputs, and building evaluation datasets from real production traffic.

Scaling generative AI workloads is less about servers and more about managing tokens, context, and concurrency.

Context Window Management

Every additional token of context — conversation history, retrieved chunks, system instructions — adds to both latency and cost, and eventually competes for space against the model’s fixed context window limit. Production systems typically implement context truncation or summarization strategies, trimming or condensing older conversation turns rather than naively appending the entire history to every request.

Provisioned Throughput for Predictable Latency

On-Demand inference shares underlying capacity across many customers, which can introduce variable latency during periods of high overall demand. Provisioned Throughput reserves dedicated capacity measured in model units, guaranteeing a consistent, predictable throughput ceiling — the right choice once an application’s traffic is stable enough to justify a fixed hourly commitment.

You may likeAWS Secrets Manager→

Caching Prompts to Cut Latency and Cost

Prompt caching allows Bedrock to reuse the internal computation for a repeated prefix — such as a long, unchanging system prompt or a large retrieved document — across multiple requests, avoiding the need to reprocess that same prefix from scratch every single call. For applications that send the same lengthy context repeatedly with only the final user question changing, this can meaningfully cut both cost and response latency.

Batching for Throughput-Oriented Workloads

Batch inference processes a large collection of prompts asynchronously in the background, trading immediate responses for significantly higher throughput per dollar — a natural fit for workloads like classifying a backlog of a million support tickets, where no individual result is needed in real time.

Because Bedrock is serverless, availability design shifts from managing servers to managing regions, retries, and fallback strategies.

Multi-Region Model Access

Since Bedrock is a regional service and not every model is available in every Region, resilient architectures often configure a primary Region and a fallback Region (or even a fallback model) so that a Region-wide disruption or a specific model’s temporary unavailability doesn’t take down the whole application.

Cross-Region Inference for Load Distribution

Cross-Region inference profiles let a single logical request be automatically routed across multiple Regions’ capacity pools, smoothing out demand spikes and improving effective availability without the application needing to manually implement its own multi-Region routing logic.

Idempotency in Agent-Driven Workflows

When Agents for Bedrock call external actions — like placing an order or sending an email through a Lambda-backed action group — those actions need their own idempotency safeguards, because a retried or duplicated agent step should not cause the same real-world action to happen twice.

Generative AI systems introduce new attack surfaces — prompt injection, data leakage through retrieval — on top of standard cloud security concerns.

IAM-Based Access Control

Every Bedrock API call is authorized through standard IAM policies, letting you restrict which principals can invoke which specific models, create Knowledge Bases, or configure Guardrails — the same familiar least-privilege model used across the rest of AWS.

VPC Endpoints for Private Connectivity

Bedrock supports AWS PrivateLink, allowing applications running inside a VPC to call Bedrock APIs without traffic ever traversing the public internet, which matters for workloads handling regulated or highly sensitive data.

Prompt Injection as a New Class of Risk

Because Agents and Knowledge Bases feed external content (retrieved documents, tool outputs) back into the model’s context, a malicious instruction hidden inside that external content can attempt to hijack the model’s behavior — a risk category unique to generative AI systems. Guardrails’ input filtering, combined with careful action-group permission scoping, is the primary defense against this.

Observability for generative AI has to cover both traditional operational health and model-specific behavior quality.

CloudWatch Metrics That Matter

Model Evaluation

Bedrock includes built-in model evaluation jobs that can score model outputs against a dataset using either automatic metrics or human review, letting teams compare candidate models or prompt variants objectively before choosing one for production — treating model selection as a measurable experiment rather than a subjective guess.

Invocation Logging for Audit and Debugging

With invocation logging enabled, every prompt and response can be captured in S3 and CloudWatch Logs, which is invaluable both for reconstructing exactly what happened during an incident and for building a labeled dataset to later fine-tune a model or tune Guardrail policies.

Because Bedrock is serverless by default, “deployment” here is mostly a question of capacity mode and integration surface, not server provisioning.

Choosing an Inference Mode by Workload Shape

Integration Surfaces

Beyond direct API calls, Bedrock integrates with Amazon SageMaker for combined classical machine learning and generative AI pipelines, with AWS Lambda as the backing compute for Agent action groups, and with Amazon Q (built on Bedrock) for ready-made business applications that don’t require custom orchestration code at all.

Multi-Account and Multi-Team Governance

Larger organizations typically centralize Bedrock access through a shared account or a set of IAM roles with per-team model allowlists, so cost and model usage can be governed centrally even as individual product teams build independently on top of it.

Pattern: Model Router

Rather than sending every request to the most capable (and most expensive) model, a lightweight routing layer classifies incoming requests by complexity and sends simple ones to a smaller, cheaper model while reserving the largest model for genuinely complex tasks — a pattern that can cut overall inference cost substantially without a noticeable quality drop for the bulk of traffic.

Pattern: Guardrails as a Shared Policy Layer

Rather than re-implementing content-safety logic inside every individual application, teams define a small number of shared Guardrail configurations centrally and attach them across all Bedrock-based applications, ensuring consistent policy enforcement without duplicated effort.

No — data submitted through Bedrock, including prompts, responses, and any content used for fine-tuning, is not used to train or improve the base foundation models offered to other customers.

A Knowledge Base retrieves relevant information to ground a single response; an Agent orchestrates a multi-step process that can call multiple tools, reason between steps, and optionally use one or more Knowledge Bases along the way. Knowledge Bases answer “what do we know”; Agents answer “what should happen next.”

Not necessarily — many production workloads run entirely on On-Demand inference. Provisioned Throughput becomes necessary specifically when you need guaranteed capacity and predictable latency at a volume where On-Demand throttling risk becomes unacceptable.

Yes — Knowledge Bases for Bedrock support several vector store backends including OpenSearch, Aurora with pgvector, Pinecone, and Redis, so you can reuse existing infrastructure instead of adopting a new dedicated system.

Guardrails’ input-side filtering helps catch overtly malicious instructions, but the strongest defense is architectural: scoping Agent action permissions tightly and requiring confirmation for high-impact actions, so even a successfully injected instruction has limited blast radius.

Amazon Bedrock’s real value isn’t just “access to many models” — it’s a consistent, serverless operational layer that wraps inference, retrieval, orchestration, and safety policy into composable, managed building blocks. Understanding what happens inside a single request — tokenization, the dual Guardrails checkpoints, retrieval-augmented context assembly — pays off directly when making the decisions that matter in production: which inference mode to pick, how to scope Agent permissions, and how to keep token cost proportional to actual value delivered.