Start with the big picture before we meet the individual services.

The simple definition

Vision AI is the use of artificial intelligence to understand pictures and videos. In Google Cloud it is delivered as managed services. That means Google has already built, trained, and hosted the models, and you simply call them over the internet. You pay for what you use, and you never run the machines yourself.

Why it exists

Many businesses own mountains of pictures and videos that nobody has time to look at: shop shelf photos, scanned forms, camera footage, product catalogues, user uploads. Humans cannot review them all. Vision AI turns this unreadable pile into searchable, sortable, measurable data.

Where it is used

People use it to read text from receipts and signs, to tag photos so they can be searched, to hide unsafe uploads, to count people or vehicles, to check that safety gear is worn, to help shoppers find products by picture, and to find the exact moment in a long video where something happened.

A few simple ideas make every vision service easier to understand.

The main vision tasks

Pre-trained versus custom models

A pre-trained model has already learned from a huge general collection of images. It is ready to use instantly and works well for common things like dogs, cars, and printed text. A custom model is one that you train on your own images, because your needs are special, such as spotting a specific defect on your factory’s metal parts. Pre-trained is faster and cheaper. Custom is more accurate for unusual jobs.

Google Cloud has several services that deal with sight. Here is a map to help you choose.

flowchart TD
  A["Vision AI on Google Cloud"] --> B["Cloud Vision API for images"]
  A --> C["Video Intelligence API for video"]
  A --> D["Document AI for forms and documents"]
  A --> E["Vertex AI for custom image models"]
  A --> F["Multimodal models such as Gemini"]
  A --> G["Vertex AI Vision style platforms for camera streams"]
  B --> B1["Labels, text, faces, objects, safety, landmarks"]
  C --> C1["Shots, labels, objects, speech, text"]
  D --> D1["Extract fields from invoices and forms"]
  E --> E1["Train and deploy your own models"]
  F --> F1["Ask free-form questions about images and video"]
  G --> G1["Build and run live video analytics apps"]
    

The main services

The most popular entry point. It is simple, fast, and needs no training.

The features

sequenceDiagram
  participant A as Your Application
  participant V as Cloud Vision API
  participant M as Google Vision Models
  A->>V: Send image and the features wanted
  V->>M: Run the selected models
  M-->>V: Labels, boxes, text, and scores
  V-->>A: Structured response
  A->>A: Apply a confidence threshold
  A->>A: Store or act on the results
    

How a request works

Your application sends the image, either as the picture data itself, as a link to a file in Cloud Storage, or sometimes as a public web address. It also lists which features it wants. The service runs the models and replies with a structured answer containing each finding, its position as a box or outline, and a confidence score. Your program then decides what to do with the findings.

Small and large jobs

• Online requests. For quick answers on a few images at a time, such as when a user uploads a photo.
• Batch and asynchronous requests. For thousands of images or large documents, you point the service to files in Cloud Storage and collect the results later in another bucket.
• Regional endpoints. Some services let you pick a region, which helps meet rules about where data may be processed.

Reading the confidence score

The score is not a promise. A label with a score of 0.95 is very likely right, and one with 0.55 is a coin toss. Choose a threshold, a minimum score you accept. A high threshold gives fewer but more reliable results. A low threshold gives more results but more mistakes. The right choice depends on how costly a mistake is.

A video is just many pictures over time, but time adds new questions.

What the Video Intelligence API can find

• Shot changes. Moments when the camera cuts to a new scene.
• Labels over time. What appears in the whole video, in each shot, and in each frame.
• Object tracking. Following an object, such as a car, as it moves through the video, with its position at each moment.
• People and faces. Detecting people, their pose, and clothing details, without naming them.
• Text on screen. Reading captions, signs, and scoreboards that appear.
• Speech. Turning spoken words into text with timings.
• Explicit content. Flagging parts of a video that contain adult material.

Because the answer includes timestamps, you can make a long video searchable. For example, a user searches “goal” and jumps straight to minute twelve and second four.

Live camera streams

Some jobs need answers while the camera is still recording, such as counting people entering a shop or checking that workers wear helmets. Google has offered platforms for building such applications, where you connect cameras, choose or train models, and send results to storage or alerts. Because the products in this area have changed names and availability, check the current documentation before you depend on a particular one.

Video costs and cleverness

Video is large, so processing costs more than for pictures. Reduce cost by analysing only the features you really need, by sampling fewer frames when fine detail is not required, by cutting out boring parts first, and by processing in batches instead of keeping live streams running without need.

A huge amount of business data is trapped in scanned papers and photos of forms.

OCR means optical character recognition: turning pictures of words into real text a computer can search and copy. The Cloud Vision API includes OCR for photos and for dense documents. This is good for reading signs, labels, and pages of text.

Document AI goes one step further. It does not only read the words. It understands the document’s structure and pulls out meaningful fields. On an invoice, it finds the supplier, the date, each line item, the tax, and the total, and gives them as clean data ready for your accounting system.

When the ready-made answers are not specific enough, you can teach a model your own world.

flowchart LR
  A["Collect images"] --> B["Label images"]
  B --> C["Split into train and test sets"]
  C --> D["Train the model"]
  D --> E["Evaluate on the test set"]
  E --> F{"Good enough"}
  F -->|"No"| G["Add data or adjust settings"]
  G --> B
  F -->|"Yes"| H["Deploy to an endpoint"]
  H --> I["Monitor results and drift"]
  I --> G
    

The steps in plain words

Collect. Gather pictures that look like the ones the model will see in real life: same lighting, same cameras, same angles. Label. Mark each picture with the right answer, such as “defective” or “fine”, or draw boxes around the objects of interest. Split. Keep some labelled pictures aside as a test set that the model never sees during learning. Train. Let the platform learn from the training pictures. Evaluate. Check how well the model does on the test set. Deploy. Publish the model so applications can call it. Monitor. Keep checking, because the real world changes.

Two ways to build

• Automated training. The platform tries designs and settings for you, using your labelled images. It needs little machine learning knowledge and is a good start for classification and object detection.
• Custom code training. Your team writes its own training programs using popular frameworks, and uses managed machines, including graphics processors, to run them. This gives full control for advanced work.

Measuring quality

Data is the secret

Most custom model problems come from the data, not the settings. Too few examples, wrong labels, pictures that differ from real use, or classes that are very unbalanced will all hurt accuracy. Spend your time on a good, varied, correctly labelled data set, and the training itself will often look after itself.

A newer way of seeing: ask questions in plain language.

Multimodal models

A multimodal model understands more than one kind of input at once, such as text together with images or video. Instead of choosing from a fixed list of features, you can simply ask: “Describe this photo”, “Is the person wearing a helmet?”, “Summarise what happens in this video”, or “Read the table in this picture and give me the numbers”. Google’s Gemini models, available through Vertex AI, work this way.

Image embeddings and similarity search

An embedding is a list of numbers that captures the meaning of a picture. Pictures that look or mean similar get similar lists. By storing embeddings in a vector search system, you can find similar pictures quickly, or even search pictures with a text description, because text and images can share the same number space. This powers features such as “find products that look like this” and “search my photo library for sunset over water”.

Product Search

The Vision API has offered a feature for shops, where you upload your product catalogue pictures and later a customer sends a photo, and the service returns the closest matching products. Check the current documentation for its status and the best option for new retail search projects.

Real applications combine vision services with storage, events, and databases.

flowchart LR
  A["User uploads photo"] --> B["Cloud Storage bucket"]
  B --> C["Storage event"]
  C --> D["Cloud Run service or function"]
  D --> E["Cloud Vision API"]
  E --> D
  D --> F{"Safe and above threshold"}
  F -->|"Yes"| G["Save labels and text in Firestore or BigQuery"]
  F -->|"No"| H["Move to review bucket and alert moderators"]
  G --> I["Search and reports for users"]
    

How this pipeline works

A user uploads a picture to a Cloud Storage bucket. The upload creates an event that wakes a small piece of code, which asks the Vision API for labels, text, and a safety rating. If the picture is safe and the results are above your threshold, the code saves the findings in a database so users can search by them. If not, the picture goes to a separate bucket and a human moderator is alerted. This design is serverless, scales automatically, and costs almost nothing when nobody uploads.

Key building blocks

• Cloud Storage. Holds the original pictures and videos safely and cheaply.
• Eventarc and Pub/Sub. Carry events that start the processing.
• Cloud Run and Cloud Functions. Run your small pieces of code without servers.
• Firestore and BigQuery. Store the results for apps and for analysis.
• Workflows or Composer. Coordinate larger multi-step jobs.
• Vertex AI. Hosts custom models and multimodal models.

Handle limits and failures

Vision services have limits on image size, request size, and request rate. Resize very large pictures before sending. Retry failed calls with growing waits. Queue work when uploads arrive in bursts, so you do not exceed your quotas. Make your processing safe to repeat, because the same event can occasionally arrive twice.

An honest view of what you gain and what you give up.

The big trade-offs

Ready-made versus custom. Ready-made is fast and cheap but general. Custom is accurate for special tasks but needs data, time, and care.

Fixed features versus multimodal questions. Fixed APIs are consistent, fast, and priced predictably. Multimodal models are flexible but less predictable and often more costly.

Cloud versus edge. Cloud processing is powerful and easy to update. Processing on the camera or device saves bandwidth, reduces delay, and keeps data local, but needs smaller models and more engineering.

Accuracy versus coverage. A strict threshold gives reliable results on fewer pictures. A loose threshold covers more pictures with more mistakes. Pair automation with human review for important decisions.

Pictures can contain faces, addresses, documents, and secrets. Treat them with great care.

Control who can call the services

Access is controlled by IAM, Google’s permission system. Applications should call the services using service accounts with only the roles they need, rather than long-lived keys that may leak. Where simple keys are used, restrict them to specific services and addresses, and rotate them. Separate production and testing into different projects.

Protect the images

• Encrypt storage. Cloud Storage encrypts data at rest by default, and you can use your own keys if your rules require it.
• Limit bucket access. Make buckets private, and use short-lived links when users need to see a picture.
• Choose regions carefully. Keep data in locations allowed by your laws and contracts.
• Use private connectivity and service perimeters. VPC Service Controls can reduce the risk of data leaving your boundary by mistake.
• Remove what you do not need. Delete pictures and results when the purpose has ended, and set retention rules.
• Audit logs. Record who called the services and who accessed pictures.
• Read the data terms. Understand how Google handles data you send, and whether it can be used to improve services, which is governed by the terms and settings of your agreement.

Privacy of the people in the pictures

Faces, number plates, documents, and children’s images deserve special care. Get consent where required, tell people what you are doing, collect only what you need, and consider blurring or masking sensitive parts before storing. Laws about biometric data and surveillance differ greatly between countries and states, so seek legal advice before building anything that identifies or tracks people.

Technology that sees people can also harm them. Build with fairness and humility.

Know the limits

• Face detection is not identification. Finding that a face exists is different from saying whose face it is. Be careful about emotion guesses, since expressions do not reliably reveal inner feelings, and meanings differ across cultures.
• Performance can vary. Models may work worse on certain skin tones, ages, lighting, cameras, languages, or regions, depending on the data they learned from.
• Safety scores are not perfect. Harmless pictures may be flagged, and harmful ones may slip through.
• Context matters. The same picture can be fine in one setting and inappropriate in another.

Practical safeguards

• Keep humans in the loop. Use people to review uncertain, high-impact, or disputed cases.
• Test on your own data. Measure accuracy across different groups and conditions that matter to you, not just overall averages.
• Set thresholds deliberately. Match them to the cost of each kind of mistake.
• Offer appeals. People affected by automatic decisions should have a path to correction.
• Be transparent. Tell users when automated image analysis is used.
• Avoid harmful uses. Do not use vision tools for unlawful surveillance, discrimination, or decisions that affect people’s rights without strong safeguards and legal review.
• Monitor over time. Performance can drift as cameras, seasons, and customers change.

You must watch both the system and the quality of its answers.

What to watch

• Error and throttling rates. A rise means quota limits or bad inputs.
• Latency. Slow calls make user screens feel slow.
• Cost per day. Set budgets and alerts, especially for video and large batches.
• Score distributions. If average confidence suddenly drops, something about the pictures has changed.
• Review queue size. A growing queue of flagged items shows a threshold or data problem.
• Human override rate. If reviewers keep reversing the model, accuracy needs attention.
• Drift. New camera types, lighting, or products can make a custom model slowly worse.

Keep a small, regularly refreshed set of labelled pictures as a golden set. Run it through your system on a schedule and compare the results. If accuracy slips, you will know before your users do.

How to get good answers quickly without a surprise bill.

What affects speed and quality

• Image size and quality. Very small pictures lose detail. Very large ones slow requests. Resize to a sensible size.
• Lighting and focus. Clear, well-lit, sharp pictures give better results.
• Angle and framing. Objects that are tiny, cut off, or partly hidden are harder to find.
• Number of features requested. Each extra feature takes time and costs money.
• Distance to the service. Choose a region close to your application and data.
• Batching. Grouping images can reduce overhead for large jobs.

What you pay for

Pricing differs by service and changes over time, so check the current pricing pages. In general, the main parts are:

Helpful habits and harmful habits seen again and again in real projects.

Helpful patterns

Anti-patterns to avoid

More things to steer clear of

• Requesting every feature on every image. You pay for answers you never use.
• No confidence threshold. Treating every guess as a fact.
• Testing on training pictures. Accuracy looks wonderful but means nothing.
• Ignoring quotas. A sudden upload burst causes errors.
• Storing sensitive pictures forever. Old data is old risk.
• Forgetting to monitor. Models silently get worse as the world changes.

A checklist of habits that experienced teams rely on.

Habits that work

• Define the question clearly. Decide exactly what decision the vision result will support.
• Try the ready-made services first. Measure on real samples.
• Collect realistic data. Match real cameras, light, and angles.
• Keep a separate test set. Never judge a model on pictures it learned from.
• Choose thresholds with the cost of mistakes in mind.
• Keep humans for uncertain and high-impact cases.
• Use least-privilege service accounts and private buckets.
• Mask or remove sensitive details before long-term storage.
• Set budgets and alerts for cost control.
• Resize and clean pictures before sending.
• Monitor accuracy over time with a golden set.
• Document your use and inform the people affected.

Mistakes beginners often make

Common ways teams use vision services. These are typical scenarios, not claims about any particular company.

Choosing the right service

flowchart TD
  A["What are you analysing"] --> B{"Pictures or video"}
  B -->|"Pictures"| C{"Mostly documents and forms"}
  C -->|"Yes"| D["Document AI"]
  C -->|"No"| E{"Do general labels and text meet your need"}
  E -->|"Yes"| F["Cloud Vision API"]
  E -->|"No"| G{"Need flexible questions in plain language"}
  G -->|"Yes"| H["Multimodal model on Vertex AI"]
  G -->|"No"| I["Custom model on Vertex AI"]
  B -->|"Video"| J{"Stored files or live cameras"}
  J -->|"Stored"| K["Video Intelligence API"]
  J -->|"Live"| L["Streaming video analytics platform"]