Chapter 1: The Big Picture — Why Systems Need to Be Integrated

No organization of any real size runs on a single piece of software. A typical company might use one system for customer records, another for billing, another for support tickets, another for inventory, another for HR, and another for analytics. Each of these systems is good at its own job — but each one also has its own private copy of data that overlaps with the others. The billing system knows a customer’s name and address. So does the support system. So does the shipping system. When a customer updates their address in one place, the others don’t automatically know.

This is the core problem data integration solves: keeping many independent systems working from a shared, accurate, up-to-date picture of the same underlying facts, without forcing every team to manually copy information between tools, or worse, without letting each system quietly drift out of sync with the others.

Chapter 2: Core Vocabulary You Need First

Chapter 3: What a “System” and a “Silo” Really Mean

It helps to be concrete about what counts as a “system” in this context. A system can be a full application (a CRM, an accounting package), a database that a specific team owns, a third-party service accessed over the internet, or even a spreadsheet that has quietly become someone’s unofficial source of truth. Anything that stores or produces data that other parts of the organization might care about is, functionally, a system worth thinking about.

A data silo forms whenever one of these systems holds data that isn’t reliably shared with the systems that need it. Silos aren’t always a deliberate design choice — they often emerge naturally, as teams pick different tools for good reasons, and nobody plans for how those tools will eventually need to share information. Over time, the cost of silos compounds: reports disagree with each other, customers get asked for the same information twice, and decisions get made on stale or incomplete data.

Chapter 4: The Main Styles of Data Integration

Over the decades, a handful of recurring approaches to integration have emerged. Most real organizations use several of these side by side, chosen based on how fast the data needs to move and how many systems are involved.

• Point-to-point integration — one system is directly connected to another, usually with custom code built specifically for that pair.
• Hub-and-spoke integration — a central piece of middleware connects to every system once, and routes data between them, so no system needs to know about any other directly.
• ETL/ELT pipelines — scheduled jobs extract data from sources, reshape it, and load it into a target, often a data warehouse used for reporting and analytics.
• API-based integration — systems expose defined interfaces that other systems can call on demand to fetch or update data.
• Event-based integration — systems announce meaningful changes as they happen, and interested systems react, similar in spirit to event-driven architecture more broadly.

Each style has a natural home. A nightly sales report might use a straightforward ETL job. A checkout page checking real-time inventory might use an API call. A dozen systems all needing to know “a customer’s address changed” might rely on an event.

Chapter 5: Batch Integration vs. Real-Time Integration

One of the most important decisions in any integration design is timing: does the data need to move the instant something changes, or is it acceptable for it to move on a schedule?

Batch integration collects changes over a period of time — an hour, a day — and moves them all together in one run. It’s simpler to build, easier to reason about, and gentler on system resources, but it means the receiving system is always somewhat behind, working with data that’s minutes or hours old.

Real-time (or near-real-time) integration moves data as soon as it changes, often within seconds. It gives every connected system a much fresher picture, but it demands more infrastructure, more careful error handling, and more attention to what happens when a system is temporarily unavailable mid-flow.

Chapter 6: How Data Actually Moves — A Walkthrough

Consider a common scenario: a customer updates their shipping address on a company’s website, and that change needs to reach the billing system, the shipping system, and the customer support tool.

1. The customer submits the address change on the website, which is handled by the “customer profile” system — the system of record for customer details.
2. The customer profile system saves the change and needs to make sure other systems learn about it.
3. An integration layer picks up this change — either by the profile system calling out via an API, publishing an event, or the change being captured in the next scheduled batch job.
4. The integration layer transforms the data if needed (for example, the billing system might expect the address formatted differently than the profile system stores it).
5. The transformed data is delivered to the billing system, the shipping system, and the support tool.
6. Each receiving system updates its own copy, so that the next person who looks at any of these systems sees the same, current address.

flowchart LR
    A[Customer Profile System
source of truth] -- address changed --> B((Integration Layer))
    B -- transformed data --> C[Billing System]
    B -- transformed data --> D[Shipping System]
    B -- transformed data --> E[Support Tool]
    

sequenceDiagram
    participant Customer
    participant Profile as Customer Profile System
    participant Integ as Integration Layer
    participant Billing as Billing System
    participant Shipping as Shipping System

    Customer->>Profile: Update address
    Profile->>Profile: Save new address
    Profile->>Integ: Notify of change
    Integ->>Integ: Transform / map fields
    Integ->>Billing: Deliver updated address
    Integ->>Shipping: Deliver updated address
    Billing-->>Integ: Acknowledge update
    Shipping-->>Integ: Acknowledge update
    

Chapter 7: Point-to-Point Integration and Why It Breaks Down

The simplest possible integration is a direct connection built specifically between two systems: System A writes a small piece of custom code that talks to System B’s format and interface. For a handful of systems, this is fast to build and easy to understand.

The trouble starts as the number of systems grows. Connecting every system directly to every other system that needs its data produces a number of connections that grows far faster than the number of systems — a classic combinatorial explosion. Ten systems, fully interconnected point-to-point, could mean dozens of separate custom connections, each one built differently, each one a potential point of failure, and each one needing its own maintenance whenever either side changes.

Chapter 8: Hub-and-Spoke and the Rise of the Middle Layer

Instead of every system connecting directly to every other system, hub-and-spoke integration introduces a central layer — sometimes called an integration platform, an enterprise service bus, or simply “the middleware” — that every system connects to just once. When System A has data to share, it sends it to the hub. The hub figures out who needs it, transforms it into the right shape for each recipient, and delivers it onward.

This turns a tangle of dozens of custom connections into a much smaller, more manageable set: each system only needs to know how to talk to the hub, not to every other system individually. Adding a new system means connecting it to the hub once, rather than building new point-to-point links to everything it needs to share data with.

flowchart TD
    subgraph "Point-to-Point (before)"
    A1[CRM] --- A2[Billing]
    A1 --- A3[Support]
    A1 --- A4[Shipping]
    A2 --- A3
    A2 --- A4
    A3 --- A4
    end
    

flowchart TD
    Hub((Integration Hub))
    B1[CRM] --- Hub
    B2[Billing] --- Hub
    B3[Support] --- Hub
    B4[Shipping] --- Hub
    

Chapter 9: ETL, ELT, and Data Pipelines

A huge share of real-world data integration exists to feed reporting and analytics, and the classic tool for this job is the ETL pipeline — Extract, Transform, Load.

Extract means pulling data out of one or more source systems. Transform means reshaping, cleaning, and combining that data into a structure suited for its destination — renaming fields, converting units, filtering out irrelevant rows, joining data from multiple sources together. Load means writing the transformed data into its destination, often a data warehouse built specifically for analysis.

A newer variation, ELT, flips the order: raw data is loaded into the destination first, and the transformation happens afterward, using the destination system’s own processing power. This has become popular as modern data warehouses have grown powerful enough to handle heavy transformation work themselves, rather than needing a separate transformation step beforehand.

Chapter 10: API-Based Integration

Not all integration needs to move large batches of data. Sometimes one system simply needs to ask another system a question, right now, and get an immediate answer — “Is this item in stock?” “What’s this customer’s current balance?” This is the natural home for API-based integration: one system exposes a defined interface, and other systems call it whenever they need something.

APIs are especially well suited to integration scenarios that are interactive, on-demand, and don’t require sending every change ever made — just the current answer to a specific question, whenever it’s asked. They’re a poor fit, however, for keeping many systems continuously synchronized with every change, since that would require constant, repeated polling rather than a single efficient flow.

Chapter 11: Event-Based Integration

Event-based integration borrows the same idea found in event-driven architecture more broadly: instead of systems asking each other for updates, or waiting for a scheduled batch job, systems announce meaningful changes the moment they happen, and any interested system can react.

This approach shines when many systems need to know about the same change, and when freshness matters. A customer updating their email address might need to reach the CRM, the marketing tool, the support tool, and the billing system, all within seconds — event-based integration lets one announcement reach all four without the source system needing to know about any of them individually.

Chapter 12: Data Quality and Transformation Challenges

Moving data between systems is rarely as simple as copying it verbatim. Different systems almost always represent the same real-world facts differently, and integration work spends a large share of its effort reconciling these differences.

• Field mismatches — one system stores a full name as a single field; another splits it into first and last name.
• Format mismatches — dates, currencies, and units may be represented differently across systems.
• Duplicate or conflicting records — the same customer might exist under slightly different details in two systems, and integration needs rules for deciding which is correct, or how to merge them.
• Missing data — a field required by the target system might simply not exist in the source, requiring a default, a lookup, or a decision about how to handle the gap.
• Timing mismatches — if two systems update the same record at nearly the same moment, integration logic needs a clear rule for which change wins.

Good integration design treats these challenges as first-class work, not an afterthought — a pipeline that “just moves the data” without addressing quality issues tends to spread bad data faster and further than it would have spread on its own.

Chapter 13: Where Data Integration Shows Up in Real Organizations

Chapter 14: Trade-Offs and Honest Limitations

Chapter 15: Security and Governance Considerations

• Access control: integration pathways often carry sensitive data (customer details, financial records), so who can configure and access these pathways needs the same scrutiny as access to the underlying systems themselves.
• Data minimization: just because a source system has a field doesn’t mean every target system needs it — sending only what’s necessary limits exposure if something goes wrong downstream.
• Auditability: knowing what data moved, when, and why is essential for compliance, and it also makes debugging integration issues far easier.
• System of record clarity: without a clearly designated authoritative source for each type of data, conflicting updates can create confusion about which system’s version is actually correct.
• Change management: because so many systems can depend on a shared data shape, changes to that shape need coordination, not just a quiet update in one place.

Chapter 16: Design Patterns and Anti-Patterns

Chapter 17: Best Practices for Integrating Systems

• Identify a clear system of record for every important category of data before building integrations around it.
• Choose batch, real-time, or a mix, based on how quickly each specific use case actually needs fresh data — not by default assuming everything needs to be instant.
• Document data mappings and transformation rules as part of the integration itself, not as tribal knowledge in someone’s head.
• Build in monitoring and alerting so integration failures are caught quickly, not discovered weeks later through a confused end user.
• Favor a central integration layer over sprawling point-to-point connections once more than a handful of systems are involved.
• Treat data quality and validation as part of the integration, not a problem to solve after data has already spread everywhere.
• Plan for schema and format changes from the start — build in versioning rather than assuming today’s shape will last forever.

Chapter 18: Frequently Asked Questions

Chapter 19: Summary and Key Takeaways

Data integration is the quiet, essential work that keeps an organization’s many separate systems telling the same story. Without it, every new tool adopted adds another silo, another place where the same facts can drift out of sync, and another source of confusion for the people relying on that data. With it, systems that were never designed to know about each other can still work together as if they were one coherent whole.