Start with the big picture: what generative AI is and what Google’s platform gives you.

Artificial intelligence (AI) is the field of making computers do tasks that normally need human thinking. Machine learning is the part of AI where computers learn patterns from examples instead of following hand-written rules. Generative AI is the part of machine learning where the computer does not just sort or label things, it creates new content: sentences, pictures, speech, video clips or program code.

The engines behind generative AI are called foundation models. A foundation model is a very large model trained on enormous amounts of data, so it learns general skills such as language, reasoning and image understanding. A large language model, or LLM, is a foundation model that works mainly with text. Many modern models are multimodal, which means they can understand and produce more than one kind of content, such as text, images, audio, video and documents.

Vertex AI is Google Cloud’s unified, managed platform for building and running AI. “Managed” means Google operates the machines, scaling and updates. Generative AI on Vertex AI is the part of the platform that gives you access to foundation models and the tools around them. You can call Google’s Gemini models, image and video models, embedding models, and also a catalog of partner and open models, all inside your own Google Cloud project, with Google Cloud’s identity, security, networking and billing.

What you can build

• Chat and question answering over your own company documents.
• Summaries and drafts of emails, reports, meeting notes and product descriptions.
• Data extraction from text, images and documents into neat structured fields.
• Code help such as explaining, writing and reviewing programs.
• Image and video creation for marketing and design ideas.
• Agents, which are programs that plan steps, use tools and complete tasks for you.
• Search by meaning, using embeddings to find related content.

Every good tool begins with a pain. Here are the pains of building with AI models on your own.

Pain 1: Training a large model is out of reach

Building a foundation model from scratch needs vast amounts of data, thousands of specialized chips and teams of researchers, costing a fortune. Almost no company can or should do this. Renting access to a ready-made model through an API is far more practical.

Pain 2: Hosting models is hard

Even when you have a model, running it quickly and reliably for many users takes special hardware, careful engineering and constant tuning. A managed platform hides all of that and lets you pay for what you use.

Pain 3: Models do not know your business

A foundation model knows a great deal about the general world, but nothing about your private price list, your internal policies or yesterday’s orders. To be useful at work, it must be connected to your information safely and accurately.

Pain 4: Models make mistakes confidently

Generative models can state wrong things in a smooth, convincing voice. This is called hallucination. In a business setting, a wrong answer about a refund policy or a medicine dose can cause real harm. Teams need ways to ground answers in facts, measure accuracy and add safety checks.

Pain 5: Safety, privacy and compliance

Companies must stop harmful content, protect private data, meet legal rules about where data lives, and be able to explain who did what. Consumer chat tools were not built with these needs in mind.

Pain 6: Experiments do not become products

Many teams build an impressive demo in a week, then struggle for months to make it reliable, fast, safe and affordable. They need testing tools, monitoring, quotas, version control and cost control that fit real production work.

Eight ideas explain most of what you will meet in the console and the documentation.

Handy words

See the whole toolbox at a glance and how your application sits around it.

Your application never runs the model itself. It sends requests over a secure connection to Vertex AI, which routes them to the right model, applies safety checks and returns the answer. Around the models are many supporting tools for data, safety, testing and operations.

flowchart TB
  App["Your Application or Agent"] --> API["Vertex AI Generative AI API"]
  API --> Models["Models"]
  API --> Tools["Tools and Grounding"]
  API --> Controls["Controls and Quality"]
  Models --> M1["Gemini Models - Text, Image, Audio, Video, Code"]
  Models --> M2["Image and Video Generation Models"]
  Models --> M3["Embedding Models"]
  Models --> M4["Model Garden - Partner and Open Models"]
  Tools --> T1["Function Calling"]
  Tools --> T2["Grounding with Search or Your Data"]
  Tools --> T3["RAG Engine and Vector Search"]
  Tools --> T4["Code Execution"]
  Controls --> C1["Safety Settings and Filters"]
  Controls --> C2["Evaluation Service"]
  Controls --> C3["Monitoring, Quotas and IAM"]
    

Walking through the picture

Your application talks to a single API. Behind it sit the models, which create content or meaning. Next to the models sit tools that give them abilities, such as looking up facts or calling your systems. Around everything sit controls: safety filters, quality testing, access rules, quotas and monitoring. You choose which parts you need.

The main components

• Gemini models: Google’s family of multimodal models, offered in different sizes. Larger ones are more capable, and smaller ones are faster and cheaper.
• Image and video generation models: models that create pictures and video clips from text descriptions.
• Embedding models: models that turn text or images into meaning-carrying numbers for search and recommendations.
• Model Garden: a catalog where you can find Google models, open models and partner models, and use or deploy them.
• Vertex AI Studio: a console workspace to try prompts, compare models, adjust settings and start tuning without writing code.
• RAG and search tools: services to connect models to your documents, including a RAG engine, vector search and enterprise search products.
• Agent tools: kits and managed runtimes for building and running agents.
• Evaluation, safety and operations tools: services to measure quality, filter content, track usage and manage access.

Follow one question from your application, through the model and back, including a tool call.

sequenceDiagram
  participant App as Your Application
  participant V as Vertex AI Endpoint
  participant M as Model
  participant T as Your Tool or API
  App->>V: Send prompt, system instructions and settings
  V->>M: Turn text into tokens and run the model
  M-->>V: Ask to call a function with arguments
  V-->>App: Return the function call request
  App->>T: Run the function in your own system
  T-->>App: Function result
  App->>V: Send the result back to the model
  V->>M: Continue generating with the new information
  M-->>V: Final answer
  V-->>App: Answer with safety and usage details
    

Step 1: Prove who you are

Your application runs under a service account, a special identity for programs, and receives a short-lived credential. Identity and Access Management (IAM) decides what that identity may do.

Step 2: Build and send the request

The request contains the prompt, any images or files, optional system instructions, available tools and settings such as temperature and the maximum length of the answer.

Step 3: Tokenize

The text is broken into tokens. A token might be a whole short word or a piece of a longer word. Images, audio and video are also converted into internal pieces the model can read. The total number of tokens in your input and output counts toward limits and price.

Step 4: Generate, one piece at a time

The model works out which token is most likely to come next, picks one, adds it to the text and repeats. This is why long answers take longer than short ones, and why answers can be streamed to the user as they are written. Settings such as temperature decide how much randomness goes into the choice. A low temperature gives steadier, more predictable output. A high temperature gives more varied, creative output.

Step 5: Use tools if needed

If you defined functions, the model may decide that it needs one, for example “get order status for order 123”. It sends back a structured request. Your application runs the function in your own system, returns the result, and the model continues. The model never runs your code directly. Your application stays in control.

Step 6: Safety checks and the response

Safety filters check the prompt and the answer against configured categories. The final response comes back with the content, usage information such as token counts, and details about any blocked content.

An AI feature is not finished when the demo works. Here is the full life of a generative AI application.

Why this order matters

A common mistake is to jump to the most expensive step, such as tuning a model, before trying simple prompt improvements and retrieval. Prompting and grounding are quick, cheap and reversible. Tuning needs data, time and care. Follow the ladder from simple to complex, and climb only as high as your quality goals demand.

Choosing the right model is the biggest lever on quality, speed and cost.

Gemini models

Gemini is Google’s family of multimodal models. A single Gemini model can read text, images, audio, video and documents together, and write text and code, and some versions can also produce other media. Gemini models come in tiers. A larger, more capable tier handles complex reasoning, long documents and difficult coding. Smaller, faster tiers handle everyday tasks, such as classification, extraction and chat, at lower cost and delay. Some recent models can “think” before answering, spending extra effort on hard problems. The exact names, versions and abilities change often, so check the current model list.

Image and video generation

Separate models turn written descriptions into pictures, edit existing pictures and produce short video clips. They are useful for marketing drafts, product mockups, storyboards and creative exploration. Generated media may carry an invisible digital watermark to help identify it as AI-made. Always review generated media for quality, rights issues and suitability before publishing.

Embedding models

An embedding model reads text, and sometimes images, and produces a list of numbers capturing its meaning. Texts with similar meaning get similar numbers. This enables semantic search, which finds results by meaning instead of exact words, and also recommendations, grouping and duplicate detection. Embeddings are a key building block of RAG systems.

Model Garden: partner and open models

Model Garden is a catalog inside Vertex AI where you can discover many models: Google’s own, open models such as the Gemma and Llama families, and models from partner companies. Some can be used as ready-made services that you simply call. Others can be deployed onto your own dedicated machines, which gives more control but needs more setup and cost. Terms and licenses differ by model, so read them.

How to choose

Model versions and retirement

Models receive new versions and older ones are eventually retired. Preview and experimental models are not meant for critical production use and may change or disappear. Pin the version you tested, watch for retirement notices, and plan time to retest on newer versions.

The cheapest way to improve results is to ask better.

A prompt is the full set of instructions and material you send to the model. Writing good prompts is called prompt design or prompt engineering. It is a practical skill, closer to writing a clear brief for a colleague than to programming.

Parts of a strong prompt

• Role: who the model should act as, such as a careful customer support agent for a bank.
• Task: exactly what to do, stated clearly and positively.
• Context: the facts, documents or conversation the model needs.
• Rules and limits: what to avoid, what tone to use, what to do when unsure.
• Examples: one to a few sample inputs with ideal outputs. This is few-shot prompting, and it is one of the most powerful techniques.
• Output format: exactly how the answer should look, such as three bullet points or a particular set of fields.

Practical techniques

• Be specific. “Summarize in three bullet points for a busy manager” beats “summarize”.
• Break big tasks into steps. Several small, clear prompts often beat one giant request.
• Show, do not only tell. Examples reduce confusion about style and format.
• Allow “I don’t know”. Tell the model it is acceptable to say when the information is missing. This reduces invented answers.
• Separate instructions from data. Clearly mark where the user’s text or document begins and ends.
• Ask for reasoning when useful. For tricky problems, asking the model to work step by step can improve results, though newer thinking models do much of this themselves.
• Test and revise. Treat prompts like code: version them, test them and review changes.

Model settings

Models that can see, hear and read very long material.

Understanding more than text

A multimodal model accepts several kinds of input in one request. This opens many practical uses.

• Images: describe a photo, read a chart, check a damaged product picture, or compare two designs.
• Documents and PDFs: read a contract, an invoice or a scientific paper directly, including its tables and figures.
• Audio: transcribe and summarize a meeting or a call, and answer questions about it.
• Video: describe what happens in a clip, find moments, or summarize a long recording.
• Mixed input: ask a question about a diagram together with a written specification.

Long context

Some models can consider a very large amount of material at once, equal to many books or hours of video. This allows tasks such as “find every inconsistency across these twenty contracts” or “summarize this entire code repository”. Long context is powerful, but it has trade-offs:

• Cost: you pay for every token you send, every time, unless you use caching.
• Speed: more input usually means slower responses.
• Attention: models can still miss details buried in the middle of enormous inputs, so test on your own data.

Context caching

If you send the same large material again and again, such as a long manual with many different questions, context caching lets you store it once and reuse it at a lower cost and often faster speed. Check the current rules for how long caches last and how they are billed.

Real-time and streaming

Some models support live, two-way streaming of audio and video, enabling natural voice assistants that you can interrupt. Responses can also be streamed as they are generated, so users see text appear immediately instead of waiting for the full answer.

Making the model produce data your programs can use, and take actions in the real world.

Structured output

Programs prefer data in a predictable format. Instead of a free-form paragraph, you can ask the model to answer in a fixed structure, such as a set of named fields with allowed values. Vertex AI lets you provide a schema, which is a description of the exact shape you want, and the model’s output will follow it. This is invaluable for extraction tasks, classification and feeding results into databases. You should still validate the output in your code, since a valid shape does not guarantee that the content is correct.

Function calling

Function calling lets the model request that your application perform an action. You describe the available functions, such as “look up order status” or “create support ticket”, with their names and parameters. When the user asks something that needs them, the model replies with a structured request naming the function and the arguments. Your application runs the function and returns the result to the model, which then writes the final answer. This is how a chat assistant can check real stock levels or book a real appointment.

Built-in tools

• Grounding with Google Search: lets the model use fresh public information from the web and show its sources.
• Grounding with your data: lets the model use your private documents through search and RAG services.
• Code execution: lets the model write and run small programs in a safe sandbox to do accurate calculations and data analysis.
• Other tool connections: options to connect to extensions, APIs and protocol-based tool servers, depending on current features.

Safety around actions

When a model can take actions, mistakes matter more. Good practice includes:

• Least privilege: give each function only the permissions it truly needs.
• Confirmation steps: ask the user to confirm actions that spend money, delete data or send messages.
• Validation: check every argument the model provides before running anything.
• Limits: cap how many actions or how much money one conversation may involve.
• Logging: record which actions were requested and executed, and by whom.

The most important pattern for business applications: make the model read before it answers.

A model’s built-in knowledge stops at the time of its training and never included your private documents. If you ask about your company’s leave policy, it can only guess. Retrieval-augmented generation (RAG) solves this. First, the system retrieves the most relevant passages from your documents. Then it gives those passages to the model, together with the question, and asks the model to answer using them. The model writes the answer, but the facts come from your data.

flowchart LR
  subgraph PREP["Preparation Stage"]
    Docs["Your Documents"] --> Parse["Parse and Split into Chunks"]
    Parse --> Emb["Create Embeddings"]
    Emb --> Index["Store in Vector Index"]
  end
  subgraph ANS["Answering Stage"]
    Q["User Question"] --> QE["Embed the Question"]
    QE --> Find["Find Most Similar Chunks"]
    Index --> Find
    Find --> Prompt["Build Prompt with Question and Chunks"]
    Prompt --> LLM["Generative Model"]
    LLM --> Ans["Answer with Source References"]
  end
    

The preparation stage

• Collect and clean documents: manuals, policies, web pages, tickets and records. Document AI’s layout parsing can help turn PDFs and scans into clean structured text.
• Chunk: split documents into passages of sensible size that keep related ideas together, with some overlap so ideas are not cut in half.
• Embed: turn each chunk into an embedding using an embedding model.
• Index: store the embeddings in a vector database or search service, along with the original text and details such as the source and date.

The answering stage

• Embed the question in the same way.
• Search for the nearest chunks, often combined with keyword search and filters such as department or date.
• Rerank if useful, so the best passages come first.
• Prompt the model with the question, the passages and clear rules such as “answer only from these passages and cite them”.
• Return the answer with links to sources, so the user can check.

Ways to build it on Google Cloud

Google offers several routes. Some are fully managed search products that handle parsing, indexing and retrieval for you, which is fastest to launch. Others give you building blocks, such as a RAG engine, embedding models and a vector search service, for more control. Choose managed products when speed matters and requirements are standard, and building blocks when you need custom ranking, special data types or deep control. Check the current product lineup before deciding.

Common causes of poor RAG answers

• Bad chunks: pieces that are too small lose meaning, and too large pieces bury the answer.
• Missing or out-of-date documents: the model cannot use what is not in the index.
• Weak retrieval: the right passage exists but is not found. Try better chunking, hybrid search, filters or reranking.
• No permission checks: users must only retrieve documents they are allowed to see.
• Loose instructions: if the prompt does not require sticking to sources, the model may add its own guesses.

A simple ladder for deciding how far to go.

There are several ways to make a model fit your needs. They differ in effort, cost and what problems they fix. A helpful rule is to start at the bottom of the ladder and climb only if needed.

flowchart TD
  Start["Model Answers Are Not Good Enough"] --> Q1{"Are instructions or examples unclear?"}
  Q1 -->|"Yes"| A["Improve the Prompt and Add Examples"]
  Q1 -->|"No"| Q2{"Does it lack your private or fresh facts?"}
  Q2 -->|"Yes"| B["Add Grounding or RAG"]
  Q2 -->|"No"| Q3{"Does it need to take actions or use tools?"}
  Q3 -->|"Yes"| C["Add Function Calling or Build an Agent"]
  Q3 -->|"No"| Q4{"Need a special style, format or skill on many examples?"}
  Q4 -->|"Yes"| D["Consider Tuning with Your Examples"]
  Q4 -->|"No"| E["Try a More Capable Model"]
    

Fix 1: Better prompts

Cheap, instant and reversible. It fixes unclear tasks, wrong format and tone. Always try this first.

Fix 2: Grounding and RAG

Fixes missing or changing facts. Your knowledge stays in your documents, where you can update, delete and control access to it, instead of being baked into a model.

Fix 3: Tools and agents

Fixes the need to act or to calculate exactly. Function calling and agents connect the model to real systems.

Fix 4: Tuning

Tuning adjusts a model using your own examples of inputs and ideal outputs. It is best for teaching a consistent style, a specialized format or a narrow skill that is hard to describe in a prompt, especially at high volume where a smaller tuned model can replace a bigger one. Common approaches include supervised fine-tuning, which learns from example pairs, and preference-based methods that learn from which of two answers is better. It requires good-quality examples, often hundreds or more, testing, and a plan to retune when the base model changes. Tuning is not a good way to add fast-changing facts, because those are better kept in retrievable documents.

Fix 5: A bigger or different model

If all else fails, a more capable model may reason better. This costs more, so try it after cheaper options.

When one answer is not enough, models can plan, use tools and work through many steps.

A plain model call takes a prompt and returns an answer. An agent is a larger design where the model works in a loop: it looks at the goal, decides the next step, uses a tool, reads the result, and repeats until the goal is reached or it must ask a person. Agents can search, read documents, call business systems, write and run code, and coordinate with other agents.

The parts of an agent

• The model: the reasoning engine that decides what to do next.
• Instructions: the role, goals and rules.
• Tools: functions, search, databases and other services the agent may call.
• Memory: short-term notes for the current task and sometimes long-term facts about users or past work.
• Guardrails: limits on what the agent may do, how long it may run, and when it must stop and ask a human.

Google Cloud support

Google Cloud provides open-source kits for building agents, managed runtimes to host and scale them, session and memory features, and ways for agents to talk to tools and to each other using open protocols. These options evolve quickly, so check the current documentation and choose the level of abstraction that fits your team: a low-code builder, a code framework or a fully custom design.

Workflows versus agents

Not every task needs an agent. If the steps are fixed and known, such as “extract fields, validate, save”, a plain workflow that calls the model at set points is simpler, cheaper and more predictable. Agents shine when the path is not known in advance and the system must decide what to do. Start with a workflow, and add agent-style flexibility only where it is truly needed.

Risks of agents

• Compounding errors: a small mistake early in a long chain can grow.
• Cost and time: many model calls add up.
• Unwanted actions: an agent with powerful tools can do real damage if misled.
• Hard testing: behavior can vary between runs, so test many scenarios.

A helpful system must also be a safe one.

Generative models can produce harmful, biased, private or simply wrong content. Responsible use means building several layers of protection.

Layer 1: Built-in safety filters

Vertex AI checks prompts and responses against categories of harm, such as hate, harassment, sexually explicit content and dangerous content. You can adjust thresholds within allowed limits to suit your audience. A children’s education app should be stricter than an internal research tool. Some content is always blocked.

Layer 2: Your own instructions

System instructions define your assistant’s role, topics it should avoid, tone and what to do when asked something out of scope, such as “politely decline and suggest contacting support”.

Layer 3: Screening services

Additional services can screen prompts and answers for risks such as prompt injection, jailbreak attempts, leaked sensitive data and unsafe content. Sensitive Data Protection can find and mask private details such as card numbers in prompts and outputs.

Layer 4: Grounding and citations

Requiring answers to cite sources lets users verify claims and reduces invented facts.

Layer 5: Human oversight

For high-stakes uses such as medical, legal, financial or hiring decisions, keep a qualified person responsible for the final decision, and make it easy for them to see the evidence and override the model.

Fairness and bias

Models learn from human-written data and can reflect its biases. Test your application with varied examples, names, dialects and viewpoints. Look for patterns of unfair behavior, and avoid using model output alone for decisions that seriously affect people.

Transparency

Tell users when they are talking to an AI, explain its limits, label AI-generated media where appropriate and offer a way to report problems.

Without measurement, every change is a guess.

Testing generative AI is different from testing normal software. The same input can give slightly different outputs, and “correct” is often a matter of degree. Still, you can and must measure quality.

Build an evaluation set

Collect a few dozen to a few hundred realistic inputs, including easy, hard and tricky ones, and include the answers or qualities you want. Keep it fixed so results are comparable over time. Add new failures as you discover them.

What to measure

• Correctness and groundedness: are the facts right, and supported by the sources?
• Relevance and completeness: does the answer address the question fully?
• Format and instruction following: does it respect length, structure and rules?
• Safety: does it avoid harmful or private content?
• Tone and style: does it sound right for your brand?
• Latency and cost: is it fast and cheap enough?
• Retrieval quality for RAG: did the right passages come back?

Ways to score

• Computed metrics: exact matching or overlap scores, useful for extraction and classification with clear right answers.
• Model-based judging: a strong model grades answers against criteria you write. This scales well but needs checking against human opinions.
• Human review: experts rate samples. It is the most trusted, but slowest and most costly.
• Side-by-side comparison: raters or judge models choose which of two versions is better.

Vertex AI includes an evaluation service to run such tests, and Vertex AI Studio can compare prompts and models. Check the current tools and metrics in the documentation.

Keep evaluating after launch

Use thumbs up and down, user comments and sampled reviews of real conversations to find new failures. Re-run your evaluation whenever you change the prompt, the data, the retrieval settings or the model version.

How to make AI features fast, reliable and ready for many users.

Where the time goes

Response time depends on the model size, the length of the input, the length of the output and current load. Generating tokens is sequential, so longer answers take longer. Reduce delay by choosing smaller models where adequate, trimming prompts, limiting output length, streaming responses and caching repeated context.

Quotas and rate limits

Vertex AI applies limits on requests and tokens per minute for each model and region. If you exceed them, you receive a “too many requests” error. A shared pool may allocate capacity dynamically, but heavy production use needs planning. Request quota increases early, and design your application to cope with limits.

Provisioned throughput

For critical applications that need guaranteed capacity and steady performance, you can reserve capacity, often called provisioned throughput, for a fixed fee over a committed period. This reduces the risk of throttling at peak times, but you pay whether or not you use it all.

Batch processing

For large jobs that are not urgent, such as classifying a million support tickets overnight, batch prediction runs many requests together at a lower price and without worrying about interactive speed. Check the current discounts and limits.

Techniques for scale

• Retry with backoff: when throttled or facing temporary errors, wait and retry, waiting longer each time.
• Queue work: use Pub/Sub or Cloud Tasks to smooth bursts and control the rate.
• Cache answers: identical or very similar questions can reuse earlier answers where freshness allows.
• Route by difficulty: send easy tasks to small models and hard ones to larger models.
• Use regional endpoints near users and data, and consider fallback to another region or model for resilience.
• Set timeouts: do not let a user wait forever for a slow answer.
• Reduce prompt size: remove unnecessary history, repeated instructions and irrelevant retrieved text.

Prompts and answers often hold sensitive business and personal information. Protect them.

Controlling access

Use Identity and Access Management (IAM) to decide who and what may call models, manage tuned models and view data. Give each application its own service account with only the roles it needs, which is the principle of least privilege. Avoid long-lived keys, and never place credentials in prompts or code.

Protecting data

• In transit and at rest: traffic to Google Cloud APIs is encrypted, and stored data is encrypted by default. You can use customer-managed keys for supported resources.
• Private connectivity: options such as Private Service Connect keep traffic off the public internet.
• Perimeters: VPC Service Controls can reduce the risk of data leaving a protected boundary, where supported.
• Data location: choose regional endpoints that meet your residency rules. Some models and features use global endpoints, so confirm where processing happens.

How Google handles your data

Google states that customer prompts and outputs on Vertex AI are not used to train its models without your permission. There may be short-term retention for features such as caching, and for abuse monitoring, with options that differ by feature and agreement. Read the current data governance and terms carefully, and confirm them with your legal and compliance teams before sending regulated data.

Reducing exposure

• Send only the data the task needs, and mask private details when possible.
• Apply access control in retrieval, so users only retrieve documents they may see.
• Do not paste secrets into prompts, and do not put sensitive data in system instructions that many users can influence.
• Treat model output as untrusted input. Validate it before using it in commands, queries or web pages.
• Set retention rules on logs, chat histories and tuning data.

Threats specific to generative AI

• Prompt injection: hidden instructions in user text, web pages or documents that try to hijack the model.
• Data leakage: the model revealing private information from its context.
• Excessive agency: an agent with too much power being tricked into harmful actions.
• Poisoned knowledge: malicious or wrong content planted in documents that feed RAG.
• Denial of wallet: abusive usage designed to run up costs.

Auditing

Cloud Audit Logs record administrative actions and, if enabled, data access. Record which model version, prompt version and data sources produced each important answer, so that you can investigate problems later.

Watch both the health of the system and the quality of its answers.

There are two kinds of monitoring. Operational monitoring asks whether the system is running well. Quality monitoring asks whether the answers are still good, safe and useful.

Operational signals

Quality signals

• User feedback: thumbs up and down, ratings, and the share of conversations escalated to humans.
• Groundedness checks: how often answers are supported by retrieved sources.
• Sampled reviews: people read random conversations weekly and score them.
• Drift: changes in the questions people ask, in your documents or in model behavior after a version change.
• Safety incidents: reports of harmful or inappropriate output, with the details needed to reproduce them.
• Evaluation set results: scheduled re-runs to catch silent regressions.

Logging with care

Logging prompts and answers helps debugging and improvement, but they may hold private information. Decide what to log, mask sensitive details, restrict access, set retention limits and tell users when conversations are recorded.

Generative AI is billed mostly by the amount of text and media processed. Know the units and avoid surprises.

Pricing on Vertex AI is usage-based and set by Google. It differs by model, by input and output, and by features. It changes often, so always use Google’s official pricing page and calculator for current numbers. The general shape is stable and useful for planning.

Ways to keep spending sensible

• Pick the smallest model that meets your quality bar. Use an evaluation set to prove it.
• Trim the prompt. Remove repeated instructions, trim chat history and send only the most relevant passages.
• Cap output length. Ask for concise answers and set a maximum.
• Use context caching for large, repeated material.
• Use batch mode for non-urgent bulk work.
• Cache whole answers for repeated questions where freshness allows.
• Route by difficulty, sending easy tasks to cheap models.
• Limit agent steps and retries so loops cannot run wild.
• Set budgets, quotas and alerts, and label resources by team and application.
• Measure cost per successful task, not just per request. A cheaper model that fails often can cost more overall.

No tool is perfect. A good architect weighs both sides before choosing.

How it compares with other routes

The main trade-offs, in plain words

• Flexibility versus predictability: a model that can do anything can also do unexpected things. Guard rails and tests restore control.
• Quality versus cost and speed: larger models are smarter but slower and pricier.
• Knowledge in the model versus knowledge in documents: retrieval keeps facts fresh and controllable, while tuning bakes in style and skill.
• Autonomy versus control: agents save effort but need strict limits and human checks.
• Speed to launch versus durability: quick prototypes are easy, but production needs evaluation, safety and monitoring.

A pattern is a proven way to solve a common problem. An anti-pattern is a tempting mistake that causes pain later.

Helpful patterns

Anti-patterns to avoid

A checklist you can follow from the first day.

Best practices

1. Start with a clear use case and success measures. Know what good looks like before building.
2. Build an evaluation set first. Use real questions, including hard ones.
3. Climb the ladder. Prompts, then RAG, then tools, then tuning, then a bigger model.
4. Write prompts like briefs. Role, task, context, rules, examples and output format.
5. Ground important answers. Require citations and allow “I don’t know”.
6. Use structured output where programs consume the result, and validate it.
7. Choose the smallest model that passes your tests.
8. Layer your safety. Filters, system rules, screening, grounding and human review.
9. Apply least privilege to service accounts, tools and retrieval.
10. Pin model versions and plan for retirement.
11. Plan capacity. Check quotas, use retry with backoff, queue work and consider reserved capacity for critical loads.
12. Monitor quality and cost, not only uptime, and sample real conversations regularly.
13. Set budgets, limits and alerts before launch.
14. Launch gradually to a small group, with a feedback channel and a human fallback.
15. Check the latest documentation. Models, limits, features and prices change often.

Common mistakes

Four illustrative stories show where generative AI on Vertex AI fits. The companies below are imaginary examples to teach the ideas.

When Generative AI on Vertex AI is a strong choice

• You want to add language, vision or reasoning abilities without training your own models.
• You need enterprise security, governance and billing within Google Cloud.
• You need to connect models to your own data and systems.
• You want to compare several models, including partner and open ones, in one place.
• You have a plan to evaluate, monitor and control quality and cost.

When to think twice

• A fixed, standard task is already well served by a cheaper classic API.
• The decision is high-stakes and no human review is possible.
• You cannot send the data to a managed service because of strict rules.
• You have no way to test quality, so you could not tell if it works.
• Exact, repeatable output is essential, and ordinary software logic would do the job.

Quick answers to questions that beginners ask most often.