AWS DMS: Inside the Replication Engine

AWS DMS: Inside the Replication Engine

A deep, advanced-only tour of how AWS Database Migration Service actually moves and continuously replicates data — its task engine internals, log-based change capture, LOB handling, validation architecture, and the design decisions that let it migrate databases with near-zero downtime.

Imagine renovating a building’s entire plumbing system while water is still running through every tap in every apartment, and nobody is allowed to notice a drop in pressure. That is the problem AWS Database Migration Service exists to solve — moving a live, actively-written-to database from one engine or one environment to another, while applications keep reading and writing against the source the whole time, until the exact moment a cutover happens. This tutorial assumes you already know DMS exists and roughly what it is for — this is not an introduction. Instead, we go under the hood: how the replication instance’s task engine actually reads a full load and a change stream in parallel, how log-based CDC differs across source engines, how LOB columns are handled without blowing out memory, and the architectural trade-offs that separate a clean migration from a stalled one.

1Core Architecture: Instances, Endpoints, and Tasks

DMS is built from three independently addressable primitives whose separation is what makes its architecture flexible rather than a monolithic migration tool.

The Replication Instance as a Managed Compute Layer

A replication instance is a fully managed Amazon EC2 instance, provisioned and patched by AWS, whose sole job is to run the DMS task engine — the process that actually reads from a source, transforms rows in flight, and writes to a target. It is not the source database and not the target database; it sits logically between the two, which is why its instance class (from small burstable classes up to memory-optimized classes with hundreds of gigabytes of RAM) is sized based on the volume and complexity of in-flight transformation work, not based on either endpoint’s own size.

Endpoints as Reusable, Engine-Aware Connection Definitions

A source or target endpoint is a stored, tested connection definition — engine type, connection details, and a specific set of engine-aware extra connection attributes that materially change behavior (for example, LOB truncation limits, or whether CDC should capture DDL changes). Endpoints are deliberately decoupled from tasks: the same source endpoint can be reused across several tasks migrating different schemas to different targets, and testing an endpoint’s connectivity is a standalone operation that does not require a task to exist yet.

Tasks as the Unit of Migration Work

A replication task binds one source endpoint, one target endpoint, a replication instance, table mapping rules (which schemas/tables to include, exclude, or rename), and a migration type — full load only, CDC only, or full load plus ongoing CDC. The task is where actual execution state lives: load progress per table, CDC latency, and error counts are all task-scoped, which is why the same pair of endpoints can be migrated through entirely independent tasks for different table subsets, each progressing and failing independently of the others.

Compute

Replication Instance

Managed EC2 host running the DMS task engine; sized for transformation workload, not endpoint size.

Connection

Endpoints

Reusable, engine-aware source/target definitions with extra connection attributes controlling fine-grained behavior.

Execution

Replication Task

Binds instance, endpoints, and table mappings; owns all progress, latency, and error state.

Serverless

DMS Serverless

Removes explicit instance sizing; the service scales replication capacity automatically based on workload.

flowchart LR
    S[(Source Database)] -->|Full Load: bulk read| RI[Replication Instance / Task Engine]
    S -->|CDC: log-based change stream| RI
    RI -->|Transform + type-map| RI
    RI -->|Apply rows / statements| T[(Target Database or Data Store)]
    RI -.->|Metrics, latency, errors| CW[CloudWatch]
        
FIG 1 — Full load and CDC both flow through the same task engine on the replication instance
Simple Analogy

The replication instance is like a professional translator standing between two people who speak different languages, relaying a live conversation in real time — while also working through a backlog of everything already said before the translator arrived. The translator is not either speaker; they are the dedicated relay in the middle.

The Task State Machine

A task moves through a well-defined set of states — Creating, Ready, Starting, Running, Stopping, Stopped, and Failed — and understanding this state machine matters for automation built around DMS, since certain operations are only valid from certain states. A task must be in Ready before it can start, and a Running task must first transition through Stopping before it reaches Stopped; attempting to modify table mappings or certain settings while a task is Running is rejected, which is why programmatic migration tooling typically checks current task state before issuing a modification call rather than assuming the call will simply succeed or fail silently.

2Internal Working: The Task Engine and Migration Types

A task’s migration type is not a cosmetic setting — it determines which internal subsystems of the task engine are activated and in what order.

Full Load Internals: Table-Parallel Bulk Extraction

In full-load mode, the task engine does not migrate tables strictly one at a time by default. It opens multiple parallel table-load processes (bounded by a configurable maximum, commonly defaulting to eight concurrent tables), each of which bulk-extracts rows from the source using engine-native bulk-read mechanisms, buffers them in memory in configurable-size chunks, and streams them to the target using the target’s most efficient bulk-load path — for example, COPY for PostgreSQL-family targets or bulk INSERT batching elsewhere. Primary key and unique index creation on the target is deferred until after bulk load completes for many target types, because building indexes incrementally during a large bulk insert is dramatically slower than building them once against a fully loaded table.

Full Load Plus CDC: The Critical Handoff Moment

When a task is configured for full load plus ongoing replication, the engine does something that is easy to underappreciate: before starting the bulk extraction, it first establishes and records a precise position in the source’s change stream (a log sequence number, binlog coordinate, or equivalent depending on engine). Full load then proceeds from a consistent snapshot as of that position. Any changes that occur to already-migrated tables during the full load are not lost — they are captured into the CDC change stream and cached, then applied in order once the corresponding table’s full load completes. This caching-then-catch-up mechanism is what prevents the “double apply or lost write” race condition that would otherwise occur if full load and live changes touched the same rows without coordination.

CDC-Only Mode and Starting-Position Precision

CDC-only tasks skip full load entirely and require an explicit starting point — either “now” or a specific timestamp/log position — which is commonly used when a separate, faster bulk-copy mechanism (a native export/import, or a prior full-load task) has already seeded the target and DMS only needs to pick up ongoing changes from that known point forward.

1

Change-Stream Position Captured

The task engine records the exact log position before bulk extraction begins, establishing the consistent snapshot boundary.

2

Parallel Table-Load Processes Run

Multiple tables extract and load concurrently, each independently tracked for progress and errors.

3

Concurrent Changes Are Cached

Writes to tables still mid-load are captured from the change stream and held rather than applied immediately.

4

Cached Changes Applied Per Table

Once a specific table’s full load finishes, its cached changes are applied in original order before live CDC continues for that table.

5

Steady-State CDC

Once every table has transitioned, the task settles into continuous, low-latency change application across the whole set.

!
Common Misconception

Full load parallelism is per-table, not per-row within a single table by default for most engines — a handful of very large tables in an otherwise small schema will not automatically speed up just because the concurrent-table limit is raised, since those large tables are still each a single sequential extraction stream unless table-level parallel load is explicitly configured.

Error Handling and Stop-Task-On-Error Behavior

Task-level error behavior settings determine whether a data error on a single row or table halts the entire task or is logged and skipped so the rest of the migration continues. For full load, a common production configuration lets individual table failures be logged without stopping the overall task, so one problematic table does not block progress on hundreds of healthy ones — while CDC-phase apply errors are more often configured to stop the task, since silently skipping a change record during ongoing replication risks a target that has quietly drifted out of sync in a way that would not be obvious until a validation pass or an application-level discrepancy surfaces it.

3Data Flow and Lifecycle: LOBs, Transformations, and Batching

Three data-flow behaviors — LOB handling, in-flight transformation, and target-side batching — shape performance and correctness more than almost any other setting.

LOB Column Handling: Full, Limited, and Inline Modes

Large object columns cannot simply be buffered the same way as fixed-width columns without risking memory exhaustion on very large BLOBs or CLOBs. DMS exposes three strategies: full LOB mode, which migrates LOBs of any size but processes them in a separate, slower pass with a two-step read-then-update pattern per row; limited LOB mode, which truncates LOBs above a configured maximum size but allows them to flow through the same fast path as ordinary columns, dramatically improving throughput when the application can tolerate truncation or LOBs are known to be small; and inline LOB mode, a hybrid that migrates LOBs under a threshold inline with the row and falls back to the full-LOB two-step path only for the rows that exceed it, minimizing the performance penalty to just the subset of rows that actually need it.

In-Flight Transformation Rules

Table mapping supports transformation rules that execute inside the task engine during migration — renaming schemas, tables, or columns; adding a computed prefix or suffix; and applying built-in expression-based transformations without needing a downstream ETL step. Because these run row-by-row inside the engine’s transform stage, they add per-row CPU cost on the replication instance, which is part of why heavily-transformed migrations often need a larger instance class than a comparable untransformed migration of the same data volume.

Target-Side Batch Apply

During CDC, applying changes one statement at a time against the target is safe but slow at high change volumes. Batch apply mode groups multiple change records into fewer, larger transactions against the target, net of net-change collapsing (multiple updates to the same row within a batch window can be collapsed into the row’s final state before being applied), trading a small amount of apply latency for substantially higher CDC throughput — a trade-off that matters for high-write-volume sources trying to keep replication lag near zero.

sequenceDiagram
    participant Src as Source Change Stream
    participant Eng as Task Engine
    participant Tgt as Target Database
    Src->>Eng: Stream of row-level changes
    Eng->>Eng: Buffer into batch window
    Eng->>Eng: Collapse multiple changes to same row (net change)
    Eng->>Tgt: Apply collapsed batch as fewer transactions
    Tgt-->>Eng: Acknowledge apply
    Eng->>Eng: Advance CDC checkpoint position
        
FIG 2 — Batch apply collapsing net changes before writing to the target
Simple Analogy

Limited LOB mode is like a moving company that ships every box that fits on the truck in the normal load, but sends anything larger than a certain size on a separate, slower specialized truck — most boxes travel fast, and only the oversized furniture takes the careful, dedicated route.

Data Type Mapping and Precision Handling

Every source-target engine pair has a defined default data-type mapping — how a source’s numeric, date, and string types map onto the closest equivalent target types — and these defaults are not always lossless. High-precision decimal types, engine-specific date/time ranges, and certain character-set edge cases can require explicit override mapping rules rather than accepting the default, particularly for heterogeneous migrations where the source and target type systems do not line up one-to-one. Reviewing the default type mapping against a schema’s actual column definitions before full load begins catches precision-loss issues while they are still a configuration change, rather than after they have already silently truncated production data on the target.

4Homogeneous vs. Heterogeneous Migrations and Schema Conversion

DMS itself never converts a schema — that is a distinct, purpose-built tool it works alongside, and understanding the boundary matters for planning any cross-engine migration.

Why DMS Is Schema-Agnostic by Design

DMS operates on data movement — reading rows, transforming them at the row and column level, writing them to a target — but it does not generate DDL for a target schema that differs structurally from the source. For homogeneous migrations (same engine family, such as Oracle to Oracle, or PostgreSQL to Aurora PostgreSQL), this is rarely an issue because the schema is largely compatible already. For heterogeneous migrations (Oracle to PostgreSQL, SQL Server to MySQL), the target schema, including data-type mapping decisions, stored procedure conversion, and function equivalents, is the job of the AWS Schema Conversion Tool, run as a separate, prior step whose output DMS then migrates data into.

The SCT-then-DMS Handoff

A well-run heterogeneous migration runs SCT first to assess conversion complexity, generate a converted target schema, and flag objects (typically complex stored procedures or proprietary functions) that require manual rework, and only then hands the already-created target schema to DMS purely for data movement and ongoing CDC. Treating this as two distinct phases with two distinct tools, rather than expecting DMS to somehow infer target structure, is the single most common planning gap in first-time heterogeneous migrations.

Homogeneous: On-Premises Oracle to Amazon RDS for Oracle

Minimal schema friction; DMS handles data movement and CDC directly against an essentially compatible target schema, often with SCT skipped entirely or used only for a light compatibility check.

Heterogeneous: SQL Server to Amazon Aurora PostgreSQL

SCT converts T-SQL-specific constructs, stored procedures, and data types to PostgreSQL equivalents first; DMS then migrates the actual row data and keeps it current via CDC while application cutover is planned.

Assessment Reports and Migration Readiness Scoring

Before either tool moves a single row, SCT’s assessment report scores each database object by conversion complexity — straightforward, medium effort requiring review, or high effort requiring manual rewrite — giving a quantified basis for estimating a heterogeneous migration’s timeline rather than relying on a rough guess. This scoring is what typically drives the wave-based sequencing decisions covered later in this tutorial: low-complexity objects migrate first to validate the pipeline, while high-complexity objects get dedicated engineering time budgeted in before their migration wave begins.

5Advantages, Disadvantages, and Trade-offs

DMS’s log-based, near-zero-downtime design is powerful, but it is not free of real operational costs.

Every mechanism already described — the change-stream handoff at full-load start, LOB mode selection, batch apply collapsing — exists to solve a specific migration problem, and each introduces its own operational consideration elsewhere in the system. Evaluating those trade-offs honestly against a specific migration’s requirements, rather than assuming DMS is uniformly the right tool for every data-movement problem, is what separates a smooth migration from a painful one.

Advantages

  • Near-zero-downtime migration through log-based CDC, allowing cutover at a chosen moment rather than during a long maintenance window.
  • Decoupled endpoints and tasks allow flexible reuse and independent scaling of migration work across large, multi-schema estates.
  • Built-in data validation compares source and target row-by-row without a separate reconciliation tool.
  • Broad engine support, including heterogeneous paths, when paired with the Schema Conversion Tool.
  • Fully managed replication instance removes the operational burden of patching and scaling the migration compute layer itself.

Disadvantages / Trade-offs

  • Schema and stored-procedure conversion for heterogeneous migrations is out of scope and must be handled separately, adding a coordination burden.
  • Full LOB mode’s two-step read-then-update pattern can become a serious throughput bottleneck on LOB-heavy schemas if left as the default without tuning.
  • Source-side change-log retention requirements (for example, binlog or archive-log retention windows) must be actively managed, or a long-running or paused task can lose its ability to resume CDC.
  • Replication instance sizing for heavily transformed or high-throughput workloads is non-trivial and often requires iterative tuning under real load.
  • Certain DDL changes on the source during active CDC are not automatically propagated and require manual task or schema intervention.

These trade-offs are rarely fatal on their own — each has a known mitigation covered elsewhere in this tutorial — but they are worth weighing deliberately against a specific migration’s constraints (source engine, LOB profile, dependency structure, and available maintenance windows) before committing to a timeline, rather than discovering them mid-migration.

6Performance and Scalability

At meaningful data volumes, a handful of tuning levers determine whether a migration finishes in hours or days.

Table-Level Parallel Load for Very Large Tables

Beyond the concurrent-table limit, individual very large tables can be split into parallel load segments using either a numeric range on a suitable column or a partition-based strategy that matches the source’s own physical partitioning. This turns a single multi-hour sequential extraction into several concurrent range-bounded extractions, which is usually the single highest-leverage tuning change for a schema dominated by one or two massive tables rather than many medium ones.

Commit-Rate and Batch-Apply Tuning for CDC Throughput

CDC throughput is governed by how aggressively the engine batches changes before applying them to the target. Increasing the batch-apply interval and batch size generally increases throughput at the cost of slightly higher apply latency, while very write-heavy sources with tight latency requirements may instead need a larger replication instance class to sustain low-latency, smaller-batch apply without falling behind.

Replication Instance Sizing as a Function of Transformation, Not Just Volume

A task with heavy in-flight transformation rules, extensive LOB traffic in full mode, or many parallel table loads needs proportionally more memory and CPU on the replication instance than raw data volume alone would suggest, because all of that work happens on the instance itself rather than on either endpoint. Sizing decisions made purely from source database size routinely under-provision the replication instance for transformation-heavy migrations.

Per-table
default parallelism unit for full load extraction
Range/Partition
strategies to parallelize a single very large table
Batch Apply
the primary lever for sustained high-volume CDC throughput

Network Throughput as a Frequently Overlooked Ceiling

Every performance lever inside the task engine assumes the network path between the replication instance and both endpoints can sustain the resulting data rate. Migrations that connect back to an on-premises source over a VPN or a constrained Direct Connect link commonly find that the actual full-load throughput ceiling is set by that network path’s available bandwidth, not by replication instance class or parallelism settings, which is why bandwidth capacity is worth validating explicitly before assuming a larger instance class alone will fix a slow full load against a remote source.

i
Scaling Practice

Separate large, high-volume tables into their own dedicated task rather than bundling them with hundreds of small tables in one task — an isolated task can be tuned, monitored, and restarted independently without affecting the migration progress of unrelated tables.

7High Availability and Reliability

A migration tool that itself becomes a single point of failure during a long-running CDC task defeats much of the purpose of near-zero-downtime migration.

Multi-AZ Replication Instances

A Multi-AZ replication instance maintains a synchronously replicated standby in a second Availability Zone; if the primary fails, DMS fails over automatically to the standby, which resumes the task from its last durably recorded checkpoint. This matters specifically for long-running CDC tasks, where an unplanned restart from scratch could mean re-establishing change-stream position from a potentially expired log retention window.

Task Recovery and Checkpoint Resumption

Regardless of Multi-AZ, every task periodically persists its CDC checkpoint (the durable position in the source change stream) independent of the replication instance’s own health. A task that is manually stopped, or that fails and is restarted, resumes from its last checkpoint rather than from the beginning, provided the source’s change log still retains data back to that checkpoint — which is precisely why change-log retention window management is a reliability concern, not merely a performance one.

Built-in Data Validation as a Reliability Backstop

DMS can optionally run continuous data validation alongside a task, comparing row counts and row-level content between source and target and flagging discrepancies without requiring a separate reconciliation job. Validation runs as its own background process on the replication instance, consuming additional resources, which is why it is commonly enabled for final cutover-readiness verification even when it is left disabled during earlier, high-throughput phases of a large migration to avoid contending for the same compute capacity as the load and CDC work itself.

RELIABILITY-PATTERN-01 Recommended
Problem

A long-running CDC task falls behind or is paused for an extended maintenance window, and the source’s change-log retention expires before the task resumes.

Why It’s Harmful

Once the required log position has aged out of retention, DMS cannot resume CDC from that checkpoint, forcing a fresh full load and re-synchronization rather than a simple resume.

Correct Approach

Size source change-log retention windows to comfortably exceed the longest expected task pause or lag, and alarm proactively on CDC latency approaching that retention boundary rather than discovering the gap after the fact.

Reading CDC From a Replica to Reduce Primary Load

For engines and configurations that support it, CDC can sometimes read its change stream from a read replica rather than the primary source, reducing the incremental load DMS places on a production primary that is already under application write pressure. This is not universally available or advisable for every engine and CDC mechanism, since replica lag itself can introduce additional latency into the migration’s own change stream, but for primaries already close to their own capacity ceiling, offloading the read-side burden of CDC can be the difference between a migration that coexists comfortably with production traffic and one that measurably degrades it.

Cross-Region Replication Instances for Disaster Recovery

Beyond Multi-AZ within a region, some organizations run a standby replication configuration in a second region specifically to protect the migration or ongoing-CDC pipeline itself against a full regional event, particularly for migrations that are expected to run as a long-lived continuous CDC pipeline rather than a one-time cutover — treating the replication layer with the same regional resilience posture as the production databases it connects.

8Security Architecture

Because a replication instance holds live connection credentials to both a production source and a production-bound target simultaneously, its security posture deserves the same rigor as either database itself.

Network Isolation and VPC Placement

Replication instances are deployed inside a VPC and commonly placed in private subnets with no direct internet route, reaching source and target endpoints either within the same VPC, across VPC peering or Transit Gateway for other VPCs, or through a VPN or Direct Connect link back to an on-premises source — meaning the instance itself never needs a public IP for typical production migration topologies.

Encryption of Endpoint Connections and At-Rest Data

Endpoint connections support SSL/TLS with certificate validation against most supported engines, and the replication instance’s own storage is encrypted using a KMS key, which by extension protects any data cached or staged on the instance during full load and CDC processing. Endpoint connection credentials themselves are stored encrypted and are never exposed in plaintext through the API or console after initial configuration.

IAM Scope for Task and Endpoint Management

IAM policies govern who can create, modify, start, and stop tasks, and separately who can create or modify endpoint connection details — a deliberate separation that lets an organization allow a broader set of engineers to manage task execution while restricting who can see or change the actual database credentials embedded in an endpoint’s connection configuration.

ConcernMechanismWhy It Matters
Network exposurePrivate subnet placement, VPC peering/Transit Gateway/VPNRemoves the need for a publicly reachable replication instance
Transit encryptionSSL/TLS on source and target endpoint connectionsProtects live production data in flight during migration
At-rest encryptionKMS-encrypted replication instance storageProtects data staged during full load and LOB processing
Credential exposureEncrypted endpoint connection storage, IAM-scoped accessLimits who can view or change database credentials embedded in endpoints

Secrets Manager Integration for Endpoint Credentials

Rather than storing database credentials directly within an endpoint’s configuration, endpoints can reference a secret stored in AWS Secrets Manager, with DMS retrieving the credential at connection time via a designated IAM role. This decouples credential rotation from endpoint configuration entirely — a rotated secret in Secrets Manager takes effect on the next connection attempt without any change to the endpoint definition itself, and it keeps the actual credential value out of the DMS configuration surface altogether.

9Monitoring, Logging, and Metrics

Migration health is best understood through three lenses: per-table progress, CDC latency, and validation state.

CloudWatch Metrics as the Primary Health Signal

DMS emits replication-instance-level metrics (CPU, memory, storage, network) and task-level metrics, most critically CDCLatencySource and CDCLatencyTarget, which respectively measure how far behind the task is in reading the source’s change stream versus how far behind it is in applying already-read changes to the target. Distinguishing the two matters operationally: rising source latency points to a read-side bottleneck (often source-side log generation volume outpacing extraction), while rising target latency points to an apply-side bottleneck (often target write capacity or batch-apply tuning).

Table Statistics for Per-Table Migration Progress

Beyond aggregate task metrics, DMS tracks per-table statistics — rows loaded, rows inserted/updated/deleted via CDC, and validation state per table — which is the primary tool for answering “which specific tables are lagging or failing” in a schema with hundreds of tables, rather than relying solely on an aggregate task-level percentage that can mask a handful of badly-behaving tables.

Task and Endpoint Logs for Root-Cause Diagnosis

Detailed task logs, streamed to CloudWatch Logs, record engine-level events including transformation errors, apply conflicts, and connection issues at a granularity well beyond what the summary metrics expose, and are the standard first stop when a task’s aggregate metrics show a problem but do not explain its cause.

Read-side

CDCLatencySource

How far behind the task is in reading the source’s change stream.

Apply-side

CDCLatencyTarget

How far behind the task is in applying already-captured changes to the target.

Per-table

Table Statistics

Row counts and validation state broken out per individual table, not just aggregate task progress.

Root-cause

Task Logs

Engine-level detail for transformation errors, apply conflicts, and connection failures.

i
Monitoring Practice

Alarm on CDCLatencySource and CDCLatencyTarget separately rather than a single combined latency figure — the remediation for a read-side bottleneck (source-side investigation) is different from the remediation for an apply-side bottleneck (target capacity or batch tuning), and a combined metric hides which one is actually happening.

Event Subscriptions for Task State Changes

Beyond continuous metrics, DMS emits discrete events — a task failing, a replication instance running low on storage, a task reaching a specific state transition — that can be subscribed to via Amazon SNS, giving operators a push-based notification path for state changes that would otherwise require actively polling metrics or the console to notice. This is particularly valuable for infrequent but urgent events, such as a task entering a failed state, where waiting for the next scheduled metrics check would introduce needless delay before anyone becomes aware.

10DMS Serverless, Fleet Advisor, and Deployment Patterns

Beyond the classic provisioned replication instance model, DMS offers serverless capacity management and fleet-wide discovery tooling for larger migration programs.

DMS Serverless: Automatic Capacity Management

DMS Serverless removes explicit replication-instance-class selection; instead, an operator specifies a capacity range expressed in DMS Capacity Units, and the service automatically provisions and scales compute within that range based on the actual workload of the task, including scaling down to a minimal footprint during idle CDC periods and scaling up during heavy full-load bursts. This is particularly well suited to migration programs running many small-to-medium tasks where manually right-sizing dozens of individual replication instances would be a significant ongoing operational burden.

Fleet Advisor for Large-Scale Migration Discovery

Organizations planning to migrate not one database but hundreds across a data center use DMS Fleet Advisor to first inventory the existing estate — collecting metadata and performance characteristics from source databases at scale — before a single migration task is created, producing a prioritized, complexity-scored migration plan rather than requiring each database to be manually assessed one at a time.

Cross-Account and Cross-Region Migration Topologies

A replication instance in one account and region can migrate data to a target endpoint in a different account or region, commonly used when consolidating acquired-company database estates into a central account, or when a migration also doubles as a deliberate cross-region relocation of the data footprint itself, with network connectivity established via VPC peering, Transit Gateway, or a public endpoint secured with strict security-group and SSL controls.

Rolling Fleet-Wide Migration Program

A large enterprise uses Fleet Advisor to assess three hundred on-premises databases, then runs waves of DMS Serverless tasks against the highest-priority, lowest-complexity subset first, using early waves to validate the process before tackling the more complex heterogeneous migrations later in the program.

Cost Model Considerations: Serverless Versus Provisioned

DMS Serverless bills based on capacity actually consumed within the configured range, which tends to favor workloads with meaningful idle periods or unpredictable bursts — a CDC-only pipeline that is mostly quiet punctuated by occasional bursts of source activity rarely benefits from paying for a continuously-running large provisioned instance sized for its peak. Conversely, a single, predictably high-throughput full-load-and-CDC task running near-continuously at a known utilization level can sometimes be more cost-efficient on a right-sized provisioned instance, since Serverless capacity pricing carries a premium for its elasticity that a steady, fully-utilized provisioned instance does not need to pay for.

11Log-Based Change Data Capture Internals Per Engine

CDC is not one universal mechanism — DMS adapts to each source engine’s own native change-log format, and those differences carry real operational implications for anyone responsible for both the source database and the migration pipeline reading from it.

MySQL and MariaDB: Binary Log Position Tracking

For MySQL-family sources, CDC reads the binary log (binlog) in row-based format, tracking a specific binlog file and position as its checkpoint. Binlog retention on the source is configured independently of DMS, which is why binlog expiration settings on the source instance are a direct operational dependency for any long-running or pausable CDC task against a MySQL-family source.

PostgreSQL: Logical Replication Slots

PostgreSQL sources use a logical replication slot, a server-side construct that guarantees the write-ahead log is retained at least back to the slot’s confirmed position — meaning, unlike binlog-based retention, PostgreSQL will not prune WAL data still needed by an active slot. The trade-off is the opposite risk: an abandoned or orphaned replication slot from a stopped task can cause unbounded WAL retention on the source, consuming disk space until the slot is explicitly dropped, which is why cleaning up unused replication slots is a standard post-migration and task-teardown step.

Oracle: LogMiner or Binary Reader Against Redo/Archive Logs

Oracle sources are read either through LogMiner, which parses redo and archive logs via the database’s own mining interface, or through a binary reader mode that parses the redo log format more directly for higher throughput on demanding workloads. Both require archive-log retention on the source to cover the task’s checkpoint window, mirroring the same retention-dependency pattern seen with MySQL binlogs.

SQL Server: Native CDC or Transaction Log Reading

SQL Server sources can be read either via SQL Server’s own native Change Data Capture feature, which must be explicitly enabled per database and table on the source, or through direct transaction log reading depending on configuration and licensing considerations, each with different setup prerequisites and slightly different latency characteristics.

Source EngineCDC MechanismRetention Dependency
MySQL / MariaDBBinary log, row-based formatBinlog expiration window on the source
PostgreSQLLogical replication slotSlot holds WAL retention; must be dropped on teardown
OracleLogMiner or binary reader on redo/archive logsArchive log retention window
SQL ServerNative CDC or transaction log readingLog retention and native CDC enablement per table
!
Common Mistake

Leaving an orphaned PostgreSQL logical replication slot behind after deleting a DMS task is one of the most common post-migration incidents — the slot silently pins WAL retention indefinitely until it is manually dropped, eventually filling source disk space.

12Table Mapping and Selection Rules Internals

Table mapping is evaluated as an ordered rule set, not a flat inclusion list, and the evaluation order matters for anyone building non-trivial selection logic.

Selection Rules as an Ordered Include/Exclude Evaluation

A task’s table mapping is a JSON document made up of selection rules and transformation rules, evaluated in the order they appear. Selection rules use wildcard patterns against schema and table names to include or exclude objects, and later rules can override the effect of earlier, broader rules for a more specific subset — a common pattern is an initial rule including an entire schema, followed by a later, more specific rule excluding a handful of named tables within it, rather than trying to express the same result as one giant enumerated inclusion list. This ordered-evaluation model is deliberately similar to firewall rule evaluation for anyone already familiar with that mental model: broad rules first, narrower overrides layered on top, and the final effective scope determined only after every rule in the list has been applied in sequence.

Transformation Rules Layered on Top of Selection

Transformation rules operate on the objects that selection rules have already included, applying renames, prefix/suffix additions, or column-level changes. Because transformation rules execute after selection is resolved, an object excluded by a selection rule is never visible to a transformation rule at all, which is a useful mental model for debugging a mapping that appears to silently skip an expected rename — the exclusion, not the transformation rule, is usually the actual cause.

Filters for Row-Level Subsetting

Beyond whole-table selection, source filters can restrict full load and CDC to a subset of rows matching a condition on an indexed column, commonly used to migrate only a specific tenant’s data out of a multi-tenant table, or only data newer than a cutoff date, without needing a separate extraction step outside DMS itself.

flowchart TD
    A[Selection Rules — ordered include/exclude] --> B[Resolved Object Set]
    B --> C[Transformation Rules — rename, prefix, column changes]
    C --> D[Row Filters — optional subsetting]
    D --> E[Final Migration Scope for this Task]
        
FIG 3 — Table mapping evaluation order: selection, then transformation, then row filtering
!
Common Mistake

Assuming rule order does not matter because rules “just get combined.” Two selection rules covering overlapping objects resolve based on which rule appears later in the ordered list, and getting this backwards is a frequent cause of a table unexpectedly appearing or disappearing from a migration’s scope.

13Multi-Task Orchestration for Large Estates

Migrating a large schema or an entire estate is rarely one task — it is usually a deliberately partitioned set of tasks coordinated toward a single cutover.

Partitioning by Table Size and Change Volume, Not Just Alphabetically

Large migrations commonly split tables across tasks by a combination of size and write volume rather than an arbitrary alphabetical or schema-based split — grouping high-volume, high-change tables into their own smaller, carefully-tuned tasks, and batching the long tail of small, low-change reference tables into a handful of bulk tasks where per-task overhead matters less than per-table tuning.

Coordinating Independent Tasks Toward One Cutover Moment

Because each task tracks its own CDC latency and validation state independently, a coordinated cutover across many tasks requires explicitly checking that every task has simultaneously reached near-zero latency and clean validation before pointing the application at the target — a manual or scripted readiness gate that DMS itself does not enforce automatically across tasks, since each task has no inherent awareness of its siblings’ state.

Dependency Ordering Across Tasks

Where foreign-key or application-level dependencies span tables that live in different tasks, cutover readiness needs to consider the full set of dependent tasks together, not each task in isolation — a common failure mode is cutting over a task early because it individually looks healthy, while a dependent task it relies on is still catching up, producing referentially inconsistent data on the target immediately after cutover.

Wave-Based Migration Rollout

A large schema is split into three to five waves of tasks, ordered by dependency and risk, with each wave’s tasks brought to steady-state CDC and validated before the next wave begins full load, keeping the total number of simultaneously active tasks manageable for both monitoring and replication instance capacity planning.

14Design Patterns and Anti-patterns

A handful of recurring patterns separate migrations that finish cleanly from ones that stall or require costly rework.

PATTERN-01 Use
Pattern

Isolate large, high-throughput tables into their own dedicated task, separate from the bulk of small reference tables.

Why It Works

A single slow or failing large table cannot stall progress on hundreds of unrelated small tables, and the large-table task can be tuned (parallel load, larger instance class) independently.

ANTI-PATTERN-01 Avoid
Problem

Defaulting every table to full LOB mode without checking actual LOB size distribution.

Why It’s Harmful

Full LOB mode’s two-step read-then-update path is dramatically slower than limited or inline LOB mode, and applying it uniformly to a schema where most LOBs are actually small wastes throughput for no correctness benefit.

Correct Approach

Profile actual LOB size distribution first, and use inline LOB mode with a threshold set just above the typical size, reserving full LOB mode only for tables genuinely dominated by very large objects.

ANTI-PATTERN-02 Avoid
Problem

Treating cutover as a single big-bang event with no validation pass immediately beforehand.

Why It’s Harmful

Skipping a final validation run before cutover means any silent data discrepancy accumulated over a long-running CDC task is discovered only after the application is already pointed at the target.

Correct Approach

Run DMS’s built-in validation to zero discrepancy immediately before cutover, treating a clean validation pass as a hard gate rather than an optional nicety.

PATTERN-02 Use
Pattern

Run SCT assessment reports before committing to a heterogeneous migration’s timeline.

Why It Works

SCT’s complexity scoring surfaces which stored procedures and objects require manual rework early, letting a realistic project timeline be built instead of discovering conversion blockers midway through.

15Best Practices and Common Mistakes

Most production migration incidents trace back to a short, recurring list of avoidable mistakes.

Best Practice

Size change-log retention generously

Set source binlog/archive-log/WAL retention comfortably beyond the longest expected task pause, not just the expected steady-state lag.

Best Practice

Separate assessment from execution

Run SCT and Fleet Advisor assessments as a distinct, earlier phase from actual task creation, so conversion complexity is known before scheduling cutover dates.

Best Practice

Monitor source and target latency separately

Alarm on CDCLatencySource and CDCLatencyTarget independently to distinguish read-side from apply-side bottlenecks quickly.

Best Practice

Clean up replication artifacts after teardown

Explicitly drop PostgreSQL replication slots and equivalent engine-specific CDC artifacts when a task is deleted, rather than assuming they are cleaned up automatically.

Best Practice

Review default type mapping before full load

Check default source-to-target data type mappings against actual column definitions ahead of time to catch precision-loss risks while they are still a configuration change rather than a production data issue.

!
Common Mistake

Assuming DMS will convert a target schema during a heterogeneous migration. Without a prior SCT pass producing a compatible target schema, a heterogeneous task will fail against a target schema that does not yet structurally match what the source data requires.

!
Common Mistake

Under-sizing the replication instance for a transformation-heavy or LOB-heavy migration based purely on source database size, rather than on the actual in-flight processing load the task engine will carry.

16Real-World and Industry Examples

DMS’s design shows up most clearly in how organizations sequence large, business-critical migrations in practice.

Retail Platform Zero-Downtime Cloud Migration

Large e-commerce platforms migrating an on-premises Oracle order-management database to a cloud-native target commonly run full load plus CDC for days or weeks while validating application behavior against the target in parallel, cutting over only during a low-traffic window once CDC latency and validation have both been steady at effectively zero for an extended observation period.

Financial Services Cross-Engine Modernization

Banks modernizing off legacy SQL Server systems toward open-source-compatible engines lean heavily on SCT’s complexity scoring to sequence which schemas migrate first, tackling low-complexity reporting databases early to validate the pipeline before attempting the highest-complexity core transaction-processing schemas.

Post-Acquisition Estate Consolidation

Companies absorbing an acquired business’s separate AWS account and database estate use cross-account DMS Serverless tasks to consolidate dozens of small databases into the parent organization’s target accounts without manually right-sizing a replication instance for each one individually.

Analytics Offload via Continuous CDC

Organizations that need a production transactional database’s data continuously available in an analytics-optimized target use DMS not as a one-time migration tool but as an ongoing CDC pipeline, keeping a reporting-oriented target perpetually current without placing analytical query load on the production source itself.

Regulated-Industry Phased Cutover with Extended Parallel Run

Healthcare and insurance organizations subject to strict audit requirements often keep a DMS CDC pipeline running for an extended parallel-run period after the application has technically cut over, using the ongoing replication as a rollback safety net and a continuous reconciliation source until the extended audit window closes and the source system is formally decommissioned.

SaaS Multi-Tenant Database Splitting

Software vendors moving from a single large shared multi-tenant database toward a per-tenant or sharded architecture use DMS’s row-filtering capability to selectively extract one tenant’s data at a time into its own target database, running many narrowly-scoped tasks in parallel rather than treating the split as one undifferentiated bulk copy.

17Frequently Asked Questions

Advanced operational questions that come up repeatedly once teams move past a first simple migration.

Q1Why does CDC latency spike right after full load completes for a large table, even though the source workload hasn’t changed?

The cached changes accumulated during that table’s full load are applied in a burst once loading finishes, which temporarily raises apply-side latency until the cached backlog is worked through and the task settles into its normal steady-state CDC rate.

Q2Can a single DMS task migrate to more than one target?

No — a task is bound to exactly one target endpoint; migrating the same source data to multiple destinations requires either multiple tasks against the same source, or a single task to a first target with a separate downstream replication mechanism fanning out from there.

Q3Does enabling data validation slow down the migration itself?

Validation runs as an additional background process on the same replication instance, consuming its own share of CPU and memory, so enabling it during a high-throughput full load phase can measurably compete with load and CDC work — many migrations run with validation disabled during the heaviest phases and enabled specifically ahead of cutover verification.

Q4What happens to DDL changes made on the source during an active CDC task?

Support for DDL propagation is engine- and configuration-dependent and is not universally automatic; most production setups treat schema changes on the source during active migration as an event requiring coordinated, manual handling on the target rather than assuming DMS will transparently replicate structural changes.

Q5Is DMS Serverless always cheaper than a provisioned replication instance?

Not necessarily — Serverless charges for the capacity actually consumed and scales automatically, which tends to be cost-effective for variable or many small workloads, but a sustained, predictably high-throughput single task can sometimes be more cost-efficient on a right-sized provisioned instance running continuously at a known utilization level.

Q6Can transformation rules and row filters be applied differently for full load versus CDC within the same task?

Selection, transformation, and filter rules are defined once per task and apply consistently to both full load and CDC for the objects they cover — achieving genuinely different behavior between the two phases typically requires splitting the work across separate tasks rather than conditioning a single task’s mapping rules on migration phase.

Q7How should cutover be coordinated when a migration is split across many independent tasks?

Readiness needs to be checked across every task together — near-zero latency and clean validation on each individual task, plus attention to cross-task dependencies — since DMS does not provide a built-in multi-task readiness gate, teams typically script this check against the per-task metrics and validation APIs before triggering cutover.

Q8Why would a task show healthy CDC latency but application-level testing still finds missing data on the target?

Latency metrics confirm the task is keeping pace with the change stream, not that every row is byte-for-byte correct — a silent type-mapping precision loss, a table excluded by an overlooked selection rule, or an application writing through a path DMS is not capturing (such as a bulk-load utility that bypasses the logged write path the CDC mechanism relies on) can all produce healthy latency metrics alongside real data discrepancies, which is exactly the gap built-in validation is designed to catch.

Q9Is it safe to modify table mapping rules on a task that is already running CDC?

Most mapping changes require the task to be stopped first, since the engine resolves selection and transformation rules once at task start rather than re-evaluating them continuously against a live stream; adding a newly-included table to an already-running task typically requires a targeted reload of that specific table rather than expecting it to simply appear in the ongoing CDC flow.

18Summary and Key Takeaways

AWS DMS’s real architecture rests on a clean separation of concerns: replication instances that do the compute work, endpoints that describe connections, and tasks that own execution state — with every advanced capability, from the full-load-to-CDC handoff, to LOB mode selection, to engine-specific log-based change capture, built as a variation on that same separation rather than a bolted-on special case. Treating schema conversion as a distinct, prior phase handled by the Schema Conversion Tool, respecting each source engine’s own change-log retention model, and monitoring read-side and apply-side latency as genuinely separate signals are what turn DMS from a black-box migration button into a tool you can reason about, tune, and trust for a business-critical, near-zero-downtime cutover — whether that migration is a single database moved once, or a continuously-running CDC pipeline feeding an analytics platform for years afterward.

Key Takeaways

  • Instances, endpoints, and tasks are independently reusable — the same endpoint can serve multiple tasks, and task state is fully self-contained.
  • The full-load-to-CDC handoff is coordinated, not naive — changes during full load are cached and applied per table only after that table’s bulk load completes.
  • LOB mode is a real performance lever — inline LOB mode with a well-chosen threshold usually beats defaulting every table to full LOB mode.
  • CDC mechanics differ meaningfully by source engine — binlog, logical replication slots, LogMiner, and native CDC each carry their own retention and cleanup responsibilities.
  • Schema conversion is out of scope for DMS itself — heterogeneous migrations depend on the Schema Conversion Tool running as a distinct, earlier phase.
  • Read-side and apply-side latency are different problems — CDCLatencySource and CDCLatencyTarget point to different bottlenecks and different fixes.
  • Validation is a cutover gate, not an afterthought — running it to zero discrepancy immediately before cutover catches silent drift a purely metrics-based view can miss.
  • Table mapping is evaluated in order — selection rules resolve first, then transformations, then row filters, and later rules can override earlier ones for a subset.
  • Large migrations are multi-task by design — partitioning by size and change volume, then coordinating a shared cutover readiness check, beats one monolithic task.