Start with the big picture: the idea of publishing and subscribing, and what Google’s service gives you.

Pub/Sub is short for publish and subscribe. It is a fully managed, asynchronous messaging service on Google Cloud. A message is a small package of information, such as “order 4521 was placed”. A publisher is a program that creates messages and sends them. A subscriber is a program that receives and processes them. Between them sits Pub/Sub, which accepts messages, stores them safely and delivers them to everyone who has asked for them.

The word asynchronous means the sender and the receiver do not have to be busy at the same moment. The publisher drops the message off and continues with its work. The subscriber picks the message up whenever it is ready. This is like sending a letter instead of making a phone call: you do not need the other person to be at home when you post it.

“Fully managed” means Google runs all the machinery: the servers, the storage, the copies for safety and the scaling. You do not set up clusters or decide how many machines to use. Pub/Sub is also a global service, which means publishers and subscribers in different parts of the world can use the same topic through Google’s network, while you can still choose where messages are allowed to be stored.

What Pub/Sub is good at

• Decoupling: services do not need to know about each other, only about the shared topic.
• Absorbing bursts: if a thousand messages arrive in one second, Pub/Sub holds them while slower workers catch up.
• Fan-out: one message can be delivered to many different consumers.
• Streaming data in: events from apps, devices and logs can flow into analytics tools continuously.

Every good tool begins with a pain. Here are the pains of connecting systems directly.

Pain 1: Tight coupling

When System A calls System B directly, A must know B’s address, B’s rules and B’s schedule. If B is slow, A is slow. If B is down, A fails. Add a third and fourth system and the web of connections grows quickly. Changing one system risks breaking the others. This is called tight coupling.

Pain 2: Spikes crush slow systems

Traffic is rarely smooth. A flash sale can send fifty times the normal number of orders in a minute. If the order system forwards each order immediately to a billing system that can handle only a steady trickle, the billing system collapses. A buffer in the middle is needed to smooth the bursts.

Pain 3: One event, many interested parties

Many teams care about the same event. Billing, shipping, email, fraud checks and analytics all want to know about a new order. Without a shared channel, the order system must call every one of them. Each time a new team wants the events, the order system must change.

Pain 4: Lost work when things fail

If a receiver crashes while a request is in flight, the request may be lost forever. A reliable middle layer stores each message until the receiver confirms it finished the work, so a crash simply means the message is tried again.

Pain 5: Moving data into analytics

Companies want clicks, sensor readings and logs to flow continuously into data warehouses and processing engines. Building and running that pipe from scratch is costly. A managed streaming entrance saves much effort.

Eight building blocks explain almost everything you will see in Pub/Sub.

Handy words

See how publishers, topics, subscriptions and subscribers connect.

The central design is simple and elegant. Publishers send messages to a topic. The topic does not deliver messages directly to programs. Instead, you attach one or more subscriptions to the topic. Each subscription receives its own full copy of every message published to the topic. Subscribers then read from their subscription. Because each subscription is independent, one slow subscriber does not hold up the others, and a new subscriber can be added without changing any publisher.

flowchart LR
  P1["Web Application Publisher"] --> T["Topic - Orders"]
  P2["Mobile Backend Publisher"] --> T
  P3["Cloud Service Publisher"] --> T
  T --> S1["Subscription - Billing"]
  T --> S2["Subscription - Shipping"]
  T --> S3["Subscription - Analytics"]
  T --> S4["Subscription - Archive"]
  S1 -->|"Pull"| C1["Billing Service"]
  S2 -->|"Push to HTTPS endpoint"| C2["Shipping Service"]
  S3 -->|"BigQuery subscription"| C3["BigQuery Table"]
  S4 -->|"Cloud Storage subscription"| C4["Cloud Storage Files"]
    

Walking through the picture

Three different publishers send “order” messages to one topic. The topic has four subscriptions. The billing team pulls messages at its own pace. The shipping service receives them pushed to its web address. The analytics team has a subscription that writes messages straight into a BigQuery table, and the archive subscription writes them to files in Cloud Storage. Each subscription has its own progress, so billing can be a thousand messages behind while shipping is fully up to date.

The main components

• Topics: named channels that receive published messages.
• Subscriptions: independent delivery queues attached to a topic, each with its own settings such as ack deadline, retry policy, filter and dead-letter rule.
• The Pub/Sub service: Google’s managed system that stores messages durably and tracks which ones each subscription has delivered and acknowledged.
• Client libraries and APIs: tools in many programming languages that make publishing and receiving easy, including batching and flow control.
• Schemas: optional rulebooks attached to topics to check messages.
• Snapshots and seek: features to replay old messages.

Follow a message through its whole life inside the service.

sequenceDiagram
  participant Pub as Publisher
  participant PS as Pub Sub Service
  participant Sub as Subscriber
  Pub->>PS: Publish message to topic
  PS->>PS: Store message durably in several places
  PS-->>Pub: Confirm with message ID
  PS->>Sub: Deliver message from subscription
  Sub->>Sub: Process the message
  Sub->>PS: Acknowledge the message
  PS->>PS: Mark message as done for this subscription
    

Step 1: Publish

The publisher sends one or more messages to a topic. Client libraries usually batch messages, which means they collect several small messages and send them together. Batching is more efficient, but waiting to fill a batch adds a little delay, so libraries let you tune the size and waiting time.

Step 2: Store

Pub/Sub writes the message to durable storage and keeps copies in more than one location within the region before it tells the publisher “success”. This is why a successful publish means the message will not be lost because one machine fails.

Step 3: Fan out to subscriptions

Each subscription attached to the topic now has the new message in its own backlog. From here on, each subscription tracks its own progress independently.

Step 4: Deliver

Messages are delivered to subscribers according to the subscription type. Pull subscribers ask for messages. Push subscriptions send messages to a web address. Export subscriptions write messages into BigQuery or Cloud Storage.

Step 5: Acknowledge or retry

Once a subscriber has finished its work, it acknowledges the message, and Pub/Sub stops delivering it for that subscription. If the ack deadline passes without an acknowledgment, or the subscriber says it failed, Pub/Sub delivers the message again. This is the heart of reliability: a message is never considered done until someone says so.

The “at-least-once” idea

Because Pub/Sub would rather deliver a message twice than lose it, its standard promise is at-least-once delivery. In rare cases, such as when an acknowledgment is lost on the way back, a message may be delivered more than once. Your subscribers should therefore be written so that repeated processing of the same message does not cause harm. That property is called being idempotent.

A message is born, waits, is processed and finally leaves. Here is the full story.

When messages expire

An unacknowledged message does not stay forever. Each subscription has a message retention duration, which is how long Pub/Sub keeps messages that have not been acknowledged. The default is about a week, and it can be extended up to a documented maximum. If a subscriber is down longer than the retention time, the oldest unprocessed messages can be lost, so size this setting with outages in mind.

Choose how messages reach your application.

Pull subscriptions

With pull, your program asks Pub/Sub for messages and receives them in return. The most efficient way is streaming pull, where your program keeps a long-lived connection open and Pub/Sub streams messages over it as they arrive. Pull gives you strong control: you decide how fast to take messages (this is called flow control), you can run several workers on one subscription, and you can place workers anywhere they can reach Google’s services. Pull is a good default for heavy, steady processing.

Push subscriptions

With push, Pub/Sub sends each message as a web request to an HTTPS address that you provide. Your endpoint replies with a success code to acknowledge. Push suits serverless systems, such as Cloud Run services, where you do not want to keep a connection open and where the platform scales with incoming requests. Push endpoints can be protected by requiring Pub/Sub to attach an identity token that your endpoint verifies.

BigQuery subscriptions

A BigQuery subscription writes messages directly into a BigQuery table without any code of your own. It is a very simple way to land streaming events into a data warehouse for analysis, particularly when message formats match the table.

Cloud Storage subscriptions

A Cloud Storage subscription writes messages into files in a storage bucket, in batches by time or size. It is handy for archives, backups and later batch processing.

flowchart TD
  Start["What Should Happen to Each Message?"] --> Q1{"Need custom processing code?"}
  Q1 -->|"No - just store for analytics"| Q2{"Where should data land?"}
  Q2 -->|"Queryable tables"| BQ["Use a BigQuery Subscription"]
  Q2 -->|"Files for archive"| CS["Use a Cloud Storage Subscription"]
  Q1 -->|"Yes"| Q3{"Does your service have a public or private HTTPS endpoint?"}
  Q3 -->|"Yes and traffic suits request-based scaling"| Push["Use a Push Subscription"]
  Q3 -->|"No or need heavy steady throughput"| Pull["Use a Pull Subscription with Streaming Pull"]
    

Comparing the types

The promises Pub/Sub makes, and how to build safely on top of them.

At-least-once delivery

By default, every message is delivered to each subscription at least once. Duplicates are possible, though uncommon. The safe mindset is: “Assume any message may arrive twice.”

Exactly-once delivery

Pub/Sub also offers an exactly-once delivery option for pull subscriptions. When enabled, once a message has been successfully acknowledged, it will not be redelivered, and acknowledgment results can be confirmed back to the subscriber. This reduces duplicates, but it has conditions, such as applying within one region, and it can add some delay. Even then, your overall system might still process an action twice if your own code retries elsewhere, so end-to-end care is still needed. Check current documentation for the exact conditions.

Message ordering

By default, Pub/Sub does not promise that messages arrive in the order they were published, because it spreads work across many machines for speed. If order matters, you can attach an ordering key to messages, such as a customer ID. Messages with the same ordering key, published to the same region, are delivered in order to subscribers who have ordering enabled. Messages with different keys can still flow in parallel. Ordering is useful for things like “account created” before “account updated”, but it can limit how fast a single key can be processed and has special behavior when failures happen.

Making your code idempotent

An action is idempotent if doing it twice has the same effect as doing it once. Here are ways to achieve it:

• Use unique IDs: store the ID of each processed message or business event, and skip repeats.
• Prefer “set” over “add”: “set balance to 50” is safe to repeat. “Add 10 to balance” is not.
• Use database features: unique constraints or “insert if not present” operations stop duplicates.
• Make external calls safe: pass an idempotency key to payment or email services so they ignore a repeated request.

Things go wrong in real systems. Pub/Sub gives you tools to handle that calmly.

The ack deadline

When a message is delivered, a timer starts. If your subscriber does not acknowledge before the ack deadline, Pub/Sub assumes the delivery failed and sends it again. The deadline can be set per subscription, within documented limits, and well-behaved client libraries automatically extend the deadline while your code is still working. Choose a deadline that matches your typical processing time with some margin.

Negative acknowledgment

If your subscriber knows it cannot process a message right now, it can send a negative acknowledgment (often called a nack). This tells Pub/Sub to redeliver sooner, instead of waiting for the deadline.

Retry policy

By default, a failed message may be redelivered quickly. For a struggling downstream system, instant retries can make things worse. A retry policy with exponential backoff makes Pub/Sub wait longer between each attempt, such as a few seconds, then more, up to a maximum, giving the downstream system time to recover.

Dead-letter topics

Some messages can never succeed, for example because they contain bad data that crashes your code. These are called poison messages. If they are retried forever, they waste effort and may hide real problems. A dead-letter topic collects messages that have failed a set number of delivery attempts, within documented limits. You can then inspect them, fix the bug or the data, and replay them later.

flowchart TD
  M["Message Delivered to Subscriber"] --> Work{"Processing Succeeds?"}
  Work -->|"Yes"| Ack["Acknowledge - Message Done"]
  Work -->|"No or Timeout"| Wait["Wait According to Retry Policy"]
  Wait --> Count{"Max Delivery Attempts Reached?"}
  Count -->|"No"| M
  Count -->|"Yes"| DLT["Move to Dead-Letter Topic"]
  DLT --> Review["Engineers Inspect and Fix"]
  Review --> Replay["Republish After the Fix"]
    

Setting up dead-letter handling safely

• Give the dead-letter topic its own subscription, so parked messages are actually kept and visible.
• Grant Pub/Sub the permission it needs to publish to the dead-letter topic and to acknowledge from the original subscription.
• Alert on the dead-letter backlog. A growing pile means something is broken.
• Choose the attempt count with care. Too low sends temporary failures to the dead-letter topic. Too high delays discovery of real problems.

Good messages are small, clear and predictable.

Subscription filters

A filter is a rule on a subscription that lets only matching messages in, based on their attributes. Suppose a topic carries all orders, with an attribute “region”. The Europe warehouse subscription can use a filter so it only receives messages where region is Europe. This saves processing and money, since filtered-out messages are not delivered to that subscription. Filters only look at attributes, so put the fields you may want to filter on into attributes.

Schemas

A schema describes the exact structure of a message, such as “an order has an ID (text), an amount (number) and a time”. Pub/Sub supports common schema formats such as Avro and Protocol Buffers. When a schema is attached to a topic, Pub/Sub rejects messages that do not follow it. This protects subscribers from surprises, because every message that arrives is known to be shaped correctly. Schemas can evolve over time with revisions, but changes should be made carefully so older and newer programs keep understanding each other.

Designing a good message

• Keep it small. Messages can be quite large within documented limits, but small messages are faster, cheaper and easier to retry.
• Describe an event, not a command. “OrderPlaced” is better than “SendEmailNow”. Events let many consumers decide what to do.
• Include a unique ID and timestamp. They help with duplicates and troubleshooting.
• Carry references for large data. Instead of putting a big video in a message, store it in Cloud Storage and send its location.
• Plan for change. Add new fields instead of renaming old ones, and keep a version number.
• Avoid secrets and unneeded personal data. Messages may be stored, logged and copied to several places.

Sometimes you need to go back in time and process messages again.

Normally, once a message is acknowledged on a subscription, it is gone from that subscription. But mistakes happen. A bug might have processed a day of messages wrongly. A new analytics system might want last week’s events. Pub/Sub offers tools for such cases.

Subscription retention of acknowledged messages

You can tell a subscription to retain acknowledged messages for a period of time. They stay available for replay even though they were already processed.

Topic retention

A topic can also keep messages for a chosen period after they are published, up to a documented maximum. This lets new subscriptions, created later, start from older messages, and lets existing ones replay. It acts like a short history of the channel.

Snapshots

A snapshot records the acknowledgment state of a subscription at a moment in time. It is like a bookmark: “everything before this point was handled”. Before a risky deployment, you can create a snapshot, and if something goes wrong, you can rewind to it.

Seek

The seek operation moves a subscription to a snapshot or to a timestamp. After a seek, messages after that point are delivered again. This is a powerful repair tool, but use it carefully, because it causes reprocessing, and your processing code must be ready for repeats.

How Pub/Sub grows with you, and what you still need to tune.

Scaling you do not manage

With a typical message system, you must choose how many machines or partitions to have, and rebalance them as traffic grows. Pub/Sub hides this. It automatically scales its internal capacity to handle more publishers, more subscribers and higher message rates. There are documented quotas, such as limits on throughput per region and on request sizes, but they are large, and many can be raised by request.

Scaling publishers

• Batch messages so that fewer, larger requests are sent.
• Use many publisher instances and keep client objects alive instead of creating new ones for every message.
• Publish to a nearby region using regional endpoints, to cut delay and meet data location rules.

Scaling subscribers

• Add more workers to the same subscription. They share the messages automatically.
• Autoscale on backlog. Watch the number and age of unacknowledged messages, and add workers when they grow.
• Use flow control so each worker takes only as many messages as it can handle at once.
• Keep processing fast. Do slow work asynchronously and extend ack deadlines where needed.

Latency

Pub/Sub is designed for good throughput and reasonably low delay, but it is not an instant function call. End-to-end latency is usually measured in milliseconds to a few hundred milliseconds under normal conditions, and depends on your batching, region choices and subscriber speed. If you need a response from another system immediately, to continue the user’s request, use a direct call instead and use Pub/Sub for follow-up work.

Hot keys and ordering

When ordering is used, all messages with the same key must be handled in sequence. If one key receives a huge share of traffic, such as a single very popular customer, that key becomes a bottleneck. Choose ordering keys that spread load evenly.

How Pub/Sub keeps messages safe, and how you control where they live.

Durability

Pub/Sub stores each message in multiple places within the region before confirming the publish. If a machine or even a whole zone has trouble, the message remains available. Because the service is built on Google’s large infrastructure, you do not have to configure replication yourself.

Availability

Pub/Sub is a global service with regional storage and endpoints. Google publishes an availability commitment for it in its service level agreement, which you should check for exact terms. Clients automatically retry on temporary errors, which smooths over short problems.

Where messages are stored

By default, Pub/Sub chooses storage locations near the publisher. If you have rules about where data may live, such as keeping it inside one country or region, you can set a message storage policy on a topic that lists the allowed regions. Pub/Sub then stores messages only in those places. Publishers and subscribers can use regional endpoints to keep their traffic in the chosen area.

Designing for regional problems

A single region is a single point of risk. For very important systems, teams sometimes publish to topics in more than one region, or maintain a plan to switch publishers and subscribers to another region if the first has trouble. This adds complexity, since messages may need to be de-duplicated, so it should be a deliberate choice based on business need.

Messages often carry valuable data. Protect the whole path.

Who may do what (IAM)

Access is controlled by Google Cloud Identity and Access Management (IAM). There are roles for publishing, for subscribing, for viewing and for administering. A publishing application should receive only the publisher role on its topic. A processing application should receive only the subscriber role on its subscription. This is the principle of least privilege: give only the permissions actually needed. Use a separate service account for each application, so a problem in one does not give access to everything.

Encryption

• In transit: connections to Pub/Sub use encryption.
• At rest: messages are encrypted when stored, by default with keys managed by Google.
• Customer-managed keys: if your rules require, you can use your own keys through Cloud Key Management Service, which lets you control and revoke them.

Securing push endpoints

For push subscriptions, your endpoint must be reachable over HTTPS. To prevent strangers from posting fake messages to it, enable authentication so that Pub/Sub attaches a signed identity token, and have your service verify it. Never rely on a secret web address alone.

Network boundaries

For sensitive environments, Google Cloud provides perimeter controls such as VPC Service Controls, which can restrict where Pub/Sub data may be accessed from and reduce the risk of data leaving a protected boundary. Private connectivity options let workloads reach Google services without public internet exposure.

Auditing

Cloud Audit Logs record administrative actions, such as creating topics or changing permissions, and can record data access if enabled. These logs are valuable for investigations and compliance.

Protecting message content

• Do not put passwords, keys or unnecessary personal data in messages.
• If sensitive fields must travel, consider encrypting them at the application level too.
• Remember that dead-letter topics and retained messages hold copies, so protect those too.
• Validate incoming messages in subscribers. Never trust content blindly just because it came from your own topic.

You cannot fix what you cannot see. Know your dashboards before trouble starts.

Pub/Sub reports metrics to Cloud Monitoring automatically. Metrics are numbers over time, logs are records of events, and alerts notify people when numbers cross a limit.

Practical alerts to set up first

• Oldest unacknowledged message age above the amount of delay your business can tolerate.
• Any messages in the dead-letter subscription.
• Publish error rate above normal.
• Push endpoint failure rate above normal.

Tracing messages end to end

When one user action flows through many services via Pub/Sub, finding where time was lost can be hard. A good habit is to include a trace or correlation ID in message attributes, and log it in each service. Then one search reveals the whole journey.

Pub/Sub bills by data volume. Know what counts and how to control it.

Pub/Sub pricing is usage-based and set by Google, with a monthly free allowance at the time of writing. Exact prices change and vary by use, so always use the official pricing page and calculator. The key is to understand which activities add to the bill.

Ways to keep the bill sensible

• Keep messages lean. Send IDs and references, not huge documents.
• Batch publishing. It improves efficiency and reduces overhead per message.
• Use filters. A filtered-out message is not delivered to that subscription, saving delivery cost.
• Delete unused subscriptions. Forgotten subscriptions keep collecting backlog and cost money until the retention period ends.
• Limit retention. Keep replay windows only as long as you truly need.
• Avoid retry storms. Use backoff so failing messages are not delivered thousands of times.
• Label resources. Make it easy to see which team spends what.

Pub/Sub is the glue of many Google Cloud designs. Here are the most common pictures.

Architecture 1: Event-driven applications

A service publishes events such as “UserSignedUp”. Several other services, perhaps running on Cloud Run, Cloud Run functions or Google Kubernetes Engine, subscribe and react: one sends a welcome email, another creates a profile, another updates analytics. Adding a new reaction means adding a new subscription, with no change to the publisher.

Architecture 2: Streaming analytics

Events from apps, devices and logs flow into a topic. A Dataflow job reads from a subscription, cleans and groups the data, and writes results to BigQuery for dashboards. Dataflow is Google’s managed service for processing streams and batches of data, and it works closely with Pub/Sub.

Architecture 3: Load leveling for a slow system

A front end accepts many requests quickly and publishes jobs to a topic. A group of workers processes them at a steady speed. During a rush, the backlog grows instead of the system collapsing, and later it drains. Users see quick acknowledgments such as “your report is being prepared”.

Architecture 4: Fan-out to many consumers

A single “PaymentCompleted” event reaches billing, loyalty points, receipts and fraud analysis through separate subscriptions. Each team owns its own subscription and failure handling.

Architecture 5: Change data and log pipelines

Changes in databases or application logs are published to topics and flow to search indexes, caches or archives, keeping several systems in sync without tight links.

Architecture 6: Cloud events and automation

Many Google Cloud services can emit events, for example a file being uploaded or a resource changing. Pub/Sub is often the transport. Services such as Eventarc build on messaging to route such events to Cloud Run and other targets, so you can often react to cloud events without writing the plumbing yourself.

Pairing with scheduled and queued work

Cloud Scheduler can publish a message to a topic on a timetable, which is a simple way to trigger regular jobs. For work that must be delivered to one specific target with controlled rates and scheduled delivery times, Cloud Tasks may be a better fit. The comparison table in the next chapter helps with the choice.

No tool is perfect. A good architect weighs both sides before choosing.

How it compares with its neighbors

The main trade-offs, in plain words

• Simplicity versus control: you give up low-level tuning to avoid running infrastructure.
• Decoupling versus visibility: services are free from each other, but flows are harder to follow without good tracing.
• Reliability versus duplicates: never losing a message means occasionally seeing one twice.
• Order versus speed: strict order for a key limits parallel processing for that key.

A pattern is a proven way to solve a common problem. An anti-pattern is a tempting mistake that causes pain later.

Helpful patterns

Anti-patterns to avoid

A checklist you can follow from the first day.

Best practices

1. Create subscriptions before publishing. New subscriptions do not see earlier messages by default.
2. Design for duplicates. Use unique IDs and idempotent operations.
3. Acknowledge after success. Never before the work is safely done.
4. Always configure a dead-letter topic and a retry policy. Alert on the dead-letter backlog.
5. Set the ack deadline to fit your processing time. Let client libraries extend it for long jobs.
6. Use flow control. Prevent workers from taking more than they can handle.
7. Autoscale workers on backlog and message age. These are the truest signals of falling behind.
8. Use schemas and versioning. Protect consumers from bad or surprising messages.
9. Apply least privilege. Separate service accounts and narrow roles for each application.
10. Use filters and small messages. They save money and effort.
11. Add correlation IDs. Make flows traceable across services.
12. Clean up and label resources. Delete unused subscriptions and track ownership.
13. Check the latest documentation. Limits, features and defaults change over time.

Common mistakes

Four illustrative stories show where Pub/Sub fits. The companies below are imaginary examples to teach the ideas.

When Pub/Sub is a strong choice

• You want services to be independent and able to scale on their own.
• Traffic is bursty and you need a buffer.
• Many teams or systems need the same events.
• You want to stream data into analytics with little plumbing.
• You prefer a fully managed service with no cluster to run.

When to think twice

• You need an immediate response to continue a user request, so a direct call is better.
• You need strict global ordering of all messages.
• You depend on Kafka-specific tools and protocols.
• You need precise scheduling and rate control for calls to one target, where Cloud Tasks may suit better.

Quick answers to questions that beginners ask most often.