Vision AI is not one product. It is a family of services that help software understand pictures, scanned pages and video.

Computer vision is the field of teaching computers to understand what is in an image or a video. A human looks at a photo and instantly knows there is a dog on a sofa. A computer sees only a grid of numbers describing the color of each tiny dot, called a pixel. Vision AI services turn those numbers into meaning, such as labels, words, shapes and answers.

Google Cloud offers several services for this. The Cloud Vision API recognizes objects, text, faces, logos and more in single images. The Video Intelligence API does similar work across time in video. Document AI focuses on reading structured business documents. Gemini models, offered through Google Cloud, understand images in a flexible conversational way. And for special needs, you can train custom models on your own pictures.

Why it matters for beginners

Without these services, building even a simple image reader would need a team of researchers, huge training data and expensive hardware. With them, a developer can send an image to an address on the internet and receive a clear answer within seconds. This makes advanced vision available to small teams and students.

Seeing feels effortless to humans and is surprisingly hard for machines.

When you look at a photo of a cat, you do not count pixels. Your brain recognizes shapes, textures and context in a split second. A computer has none of this built in. It receives a table of numbers, and it must learn that certain patterns of numbers usually mean a cat. This learning needs a huge number of example images and a lot of computing power.

Pictures also vary enormously. The same cat can appear in bright light or shadow, close up or far away, upside down or half hidden behind a chair. A good vision system must recognize it in all of these situations. This is why vision services are trained on enormous collections of varied images.

Let us define every word you will meet in the documentation.

The hardest first step is knowing which service fits which job.

A simple way to decide

A short look inside the machine makes every later topic easier.

Modern vision systems use neural networks, which are programs loosely inspired by how brain cells connect. A neural network is made of layers. The first layers look at tiny details such as edges and corners. The middle layers combine those into shapes such as wheels, eyes or letters. The final layers combine shapes into whole concepts such as “car” or “face”.

flowchart LR
  A["Pixels of the image"] --> B["Early layers find edges and colors"]
  B --> C["Middle layers find shapes and parts"]
  C --> D["Late layers recognize whole objects"]
  D --> E["Answer with confidence score"]
        

How a model learns

The Cloud Vision API offers a menu of ready-made image analysis features that you can combine in one request.

You send an image and choose which features you want. Google’s service returns structured results describing what it found. You do not need to train anything, and you do not need to manage servers. The same request format works for a small photo and for thousands of images in a batch.

These two features answer the questions “what is in this picture?” and “where is it?”

Label detection gives a list of descriptive words, each with a confidence score. A photo of a beach might return words such as sea, sand, sky, vacation and coast, with scores showing how sure the model is about each. Labels describe the whole picture in a general way and do not say where things are.

Object localization goes further. It finds separate objects and returns a name, a score and an outline showing where each object sits. A street photo might return two cars, one bicycle and a traffic light, each with its own position. Positions are given as relative coordinates, which means numbers between zero and one that stay correct no matter what size the image is.

OCR turns pictures of words into text that computers can search, copy and analyze.

Optical character recognition is one of the oldest and most useful vision tasks. A photo of a menu, a scanned contract or a picture of a road sign contains words, but to a computer it is just pixels. OCR finds the characters and returns real text, along with positions showing where each word appeared.

The Vision API offers two OCR features for two situations. Text detection is meant for text in everyday photos, such as signs, labels and packaging, where words are sparse. Document text detection is tuned for pages full of dense text, such as scanned books and forms. It keeps the structure of pages, blocks, paragraphs, lines and words, and it handles handwriting in many cases. If both are requested, the document version takes priority.

OCR is only the first step

OCR gives you text, not meaning. If you scan an invoice, OCR returns every word, but it does not know which number is the total or which is the date. Turning raw text into named fields requires extra logic, or a service built for document structure, which we meet in a later chapter.

These features analyze sensitive or special content, so they need extra care and thought.

Face detection

Face detection finds where faces are in an image and describes details, such as the position of eyes and the likelihood of expressions like joy or surprise. It does not tell you who a person is. Detecting that a face exists is very different from recognizing a particular individual, and Google’s service is designed for the first, not the second. Use it for tasks such as counting people or cropping around a face, not for identification.

Landmark detection

Landmark detection recognizes famous places, such as well-known monuments and natural wonders. It returns a name, a confidence score and often the geographic location, which is useful for organizing travel photos.

Logo detection

Logo detection finds popular brand logos. Brands use it to measure how visible their sponsors are in photos, and platforms use it to find unauthorized use of a brand mark.

Safe search detection

Safe search estimates how likely an image is to contain categories of sensitive content, such as adult material, violence or medical imagery. It gives each category a likelihood level, from very unlikely to very likely, not a simple yes or no. Platforms use it to review user uploads before showing them to others.

The remaining features support design, discovery and shopping.

How product search is organized

Every vision request follows the same simple story: send an image, ask for features, receive structured answers.

Your application sends a request that contains two things. First, the image, supplied either as the raw image data or as a location in Cloud Storage, or sometimes a public web address. Second, a list of the features you want. The service analyzes the image and returns a structured response, usually in JSON format, which is a simple text format that programs can read easily.

sequenceDiagram
  participant A as Your application
  participant V as Vision API
  participant M as Pre-trained models
  A->>V: Send image and list of features
  V->>V: Check permissions and request
  V->>M: Run each requested feature
  M->>V: Return labels, text, boxes and scores
  V->>A: Send structured response
        

What the response contains

Several images in one request

You can send more than one image in a single request, and the service returns a separate answer for each. This reduces the number of network round trips and is the basis of efficient batch processing. Each image and feature combination is counted for billing, which we will cover in the pricing chapter.

A result without a sense of its reliability can lead you astray.

Most vision features attach a confidence score to each result. A score near one means the model is quite sure, and a score near zero means it is guessing. But a score is not a promise of correctness. A model can be very confident and still wrong, especially on images unlike its training data. Treat scores as helpful signals, not as proof.

Choosing a threshold

A threshold is the score above which you accept a result. A high threshold gives you fewer results that are mostly correct. A low threshold gives you more results but more mistakes. The right choice depends on the cost of each kind of error in your application.

Two kinds of mistakes

Reducing one kind of mistake usually increases the other. Deciding which matters more is a business decision, not only a technical one.

Video is a long series of images with sound, and understanding it adds the dimension of time.

The Video Intelligence API analyzes video files and reports what appears and when. Instead of a single answer, you get answers tied to time ranges, such as “a dog appears from second 12 to second 20”. This turns a long video into a searchable, structured record.

Typical uses

Reading words is only half the job when documents have structure and meaning.

Vision OCR returns text and positions. Document AI goes further by understanding document types. It can identify that a page is an invoice and extract named fields such as supplier, invoice number, date, line items and total. For business automation, this is often exactly what is needed.

When to choose which

Gemini models treat images as part of a conversation, which opens possibilities that fixed feature lists cannot match.

Gemini is a family of multimodal models, meaning they accept images along with text. Instead of choosing from a menu of features, you write a request in plain language, such as “describe this photo”, “which items on this shelf look damaged?” or “list every price on this menu”. The model answers in text, and you can ask it to answer in a fixed format that your program can read.

What Gemini can do with images

Practical points from the documentation

Images can be supplied by web address, as inline data inside the request, or through a file service for large or reused files. Common formats such as PNG, JPEG, WEBP, HEIC and HEIF are supported. Inline requests have a total size limit, so larger files should use the file service, and a single request can include a large number of images. Each image consumes tokens, which are the units in which model usage is measured, and bigger images use more tokens. Some models allow you to limit the resolution used, trading detail for lower cost and faster replies.

Both read images, but they behave differently, and the best systems often use each where it shines.

Combining them

A common pattern uses the Vision API as a cheap, fast first pass at large scale, for example to check safe content or read text. Only the uncertain or complex images are sent to Gemini for deeper reasoning. This keeps cost low and quality high.

flowchart TD
  A["Incoming images"] --> B["Vision API fast first pass"]
  B --> C{"Clear and simple"}
  C -->|"Yes"| D["Accept result"]
  C -->|"No"| E["Gemini deeper analysis"]
  E --> F{"Still uncertain"}
  F -->|"Yes"| G["Human review"]
  F -->|"No"| D
        

Where each approach is strongest

The Vision API shines when the question is standard and the volume is huge. Reading street names from millions of photos, tagging a photo archive, or screening uploads for sensitive content are tasks where a fixed, predictable, low-cost service wins. Results arrive in the same structure every time, which makes downstream code simple and testing easy.

Gemini shines when the question is unusual or needs reasoning. Asking whether a shelf arrangement follows a store’s layout rules, whether a damaged package shows signs of water, or what steps a diagram describes are tasks no fixed feature list can answer. The price is greater variation in output and the need to design prompts carefully and verify answers.

Whichever you choose, keep a test set of real images with correct answers. When a new model version or a new service appears, rerun the test set and compare. This habit turns product changes from a worry into a routine measurement.

When general services do not know your special products, you can teach a model using your own pictures.

Pre-trained services know common things: cars, cats, food, signs. They do not know your factory’s unique valve, your farm’s rare plant disease, or the subtle scratch pattern that marks a faulty part. For such cases, you train a custom model. Vertex AI is Google Cloud’s platform for building, training and serving machine learning models, and it is where custom vision work lives today.

Older, separate products named AutoML Vision and AutoML Video Intelligence were retired in earlier years, and Google directed customers toward Vertex AI. This is a good reminder that product names move, so always read current documentation before starting a project.

Three ways to get a custom capability

The exact training options on Vertex AI evolve, so treat the list above as the general shape of choices and confirm the current menu in the documentation. The guiding rule stays the same: begin with the cheapest path that could work, measure it, and move to heavier paths only if the numbers demand it.

Data is the real fuel of any vision project, and most project time goes here.

A custom model learns from examples, and every example needs a correct answer attached, called a label. For classification, the label might be “good” or “defective”. For object detection, the label is a box drawn around each object together with its name. People usually create labels by hand using labeling tools, which makes quality and consistency the central challenge.

What good training data looks like

Common data traps

One famous trap is the shortcut. If every photo of defective parts was taken on a blue table and every good part on a gray table, the model may learn to read the table color, not the defect. It will then fail the moment the table changes. Another trap is duplicates leaking between training and test sets, which makes scores look better than reality.

A single accuracy number hides the mistakes that matter most, so we use better measures.

Suppose a model checks 1,000 parts and 20 are truly defective. A model that says “good” for everything is right 980 times, a 98 percent accuracy, yet it finds no defects at all. This shows why accuracy alone can mislead, especially when one outcome is rare.

The seesaw between the two

Precision and recall usually pull against each other. If you make the model stricter, false alarms drop but misses rise. If you make it more eager, misses drop but false alarms rise. The threshold you choose slides you along this seesaw.

Most production systems follow the same event-driven pattern built from a few well-known Google Cloud pieces.

A user or device uploads an image to Cloud Storage. The upload creates an event. A small piece of serverless code reacts to the event, calls a vision service, and stores the results in a database where applications can search them. Because each part does one job, the pipeline is easy to scale, test and replace.

flowchart LR
  U["User or device upload"] --> S["Cloud Storage bucket"]
  S --> E["Upload event"]
  E --> Q["Pub/Sub topic"]
  Q --> F["Serverless worker"]
  F --> V["Vision API or Gemini"]
  V --> F
  F --> D["Results database"]
  F --> R["Review queue for doubtful cases"]
  D --> A["Search and reporting apps"]
        

Why each piece is there

Analyzing a million images is a different problem from analyzing one.

For interactive apps, a synchronous request returns an answer within seconds, and the user waits. For huge archives, waiting on each image one at a time is impractical. The Vision API supports sending several images in one request and also supports asynchronous batch work, where you submit a large set, the service processes it in the background, and the results are written to Cloud Storage for you to pick up later. Documents such as long PDFs and multi-page scans are handled in a similar way, with each page counting as one image for billing.

Staying within limits

Every cloud service has quotas, which are limits on request rate and size. A bulk job that sends requests too quickly will receive rate-limit errors. A well-built worker slows down and retries politely, waiting a little longer after each failure, a method called exponential backoff. It also sets a maximum concurrency so that it never floods the service.

Making retries safe

Retrying can cause duplicates, such as saving the same results twice. A safe design makes each step repeatable without harm. For instance, store results under the image’s unique identifier, so a second write simply overwrites the first.

Vision services are billed by usage, so a clear mental model prevents surprises.

For the Cloud Vision API, the unit of billing is the feature applied to one image. One image with three requested features counts as three units. The first 1,000 units of each feature every month are free, and after that, a price per thousand units applies. Prices differ by feature, with common ones such as label detection and text detection costing about a dollar or two per thousand units at standard volume, while some features, such as web detection and object localization, cost more. Volume discounts apply at very large scale. Each page of a PDF or TIFF counts as one image.

Gemini works differently. It is billed by tokens, the units of input and output text, with images converted into a number of tokens that depends on their size. Video Intelligence is billed by minutes of video analyzed for each feature. Because prices and free allowances change, always read the current pricing pages and use the official pricing calculator for your own estimates.

Practical ways to save

Images can contain faces, documents and secrets, so protect them from the first upload.

Google Cloud controls who may do what through Identity and Access Management, usually called IAM. Instead of sharing passwords, you give each program its own identity, called a service account, and grant it only the permissions it needs. This idea is called least privilege. A worker that only reads images and calls the vision service should not be able to delete buckets or change billing.

Technology that looks at people carries duties beyond accuracy.

Vision systems can hurt people when they are used carelessly. A model trained mostly on images of certain people, places or skin tones may work noticeably worse for others. A system that analyzes faces can intrude on privacy even when it does nothing illegal. Thinking about these matters early is cheaper and kinder than fixing a harmful launch later.

Questions to ask before building

Google has published policies and guidance on acceptable use of its AI services, and some capabilities, such as identifying real people from faces, have been restricted or retired. Read the terms and policies that apply to your use, and involve legal and ethics reviewers for sensitive projects.

The best fix for a poor result is often a better input image.

Models are sensitive to image quality. A little care before sending an image can improve accuracy more than switching services. Preparation steps are cheap and can run in your worker before the vision call.

Launching is only the beginning, because the real world keeps changing.

A vision system that worked on launch day can quietly get worse. Customers buy new phones with different cameras, the packaging changes, the lighting in a warehouse is rearranged, or a new product line appears. When live images drift away from the examples a model knows, accuracy falls. This slow decline is called drift, and you notice it only if you watch.

Build a feedback loop

The best sign of real accuracy is what human reviewers decide on the cases sent to them. Save these decisions with the images. They form a fresh, realistic test set and, later, extra training data. Review a small random sample of automatically accepted cases too, since problems hide among the confident answers.

Concrete stories show how the pieces combine.

Architects record important choices so that future teams understand the reasoning.

Learning from other teams’ mistakes is the cheapest education available.

Five classic traps in detail

When results look wrong, work through causes from simplest to hardest.

A calm method

A short list to review before every launch.

Quick answers to the questions beginners ask most often.

Everything in this guide in one short page.

Vision AI in Google Cloud is a family of services for understanding images, documents and video. Pre-trained APIs give fast, predictable answers for standard tasks. Gemini offers flexible, conversational understanding for open and custom tasks. Document AI structures business paperwork. Custom models on Vertex AI cover special needs when general services fall short. Success comes less from the cleverness of a model than from clean inputs, honest testing, careful thresholds, secure handling and steady monitoring.