Multimodal & Agentic Search Search can now see, listen, reason, retrieve and act.
Multimodal search accepts information in more than one form. Text, images, camera input, voice, video, location and other contextual signals – and resolves a task by combining what those modalities reveal. Agentic search extends that process further: instead of stopping at an answer, the system can inspect live state, compare options, call tools, monitor conditions or move the user toward an action.
The frontier is therefore not “visual SEO” plus “AI agents.” It is a continuity problem across perception, entity resolution, retrieval, evidence, state and action. A photographed object must remain the same object when mapped to a product record; a live reservation must remain current when the user reaches checkout; and an agent must preserve constraints when moving from discovery to execution.
Multimodal search expands the query. Agentic search expands the outcome.
A multimodal system can interpret more than text. An agentic system can perform multi-step work using retrieved information, tools and live state. These are related capabilities but they solve different system problems.
Ask with more than words.
Camera input can expose objects and spatial relationships. Voice carries conversational context. Images reveal labels, colors and geometry. Text can add intent and constraints. The system combines them into a richer information need.
- text + image
- voice + camera
- video / scene context
- location / temporal state
Move beyond information retrieval.
An agent can decompose the task, inspect websites or structured data, call tools, compare live options, monitor changes and present a path toward execution while preserving user constraints and confirmation boundaries.
- multi-step planning
- tool use / browsing
- live state inspection
- action + confirmation
Change the input. Watch the evidence pipeline reorganize.
A visual product lookup, live camera conversation, voice request, video diagnosis, shopping task and booking request create different modality, freshness, entity and action requirements.
Each modality contributes a different kind of evidence.
The strongest multimodal query does not simply concatenate signals. It uses each signal for the information it can establish best and preserves uncertainty when one modality is weak.
Text
Expresses intent, constraints, names, comparisons and abstract questions. Text is often the best modality for telling the system what the user wants done with the other signals.
INTENT / CONSTRAINTSImage
Exposes visible objects, shape, color, markings, spatial arrangement and condition. A still image lacks temporal behavior and may hide decisive sides or labels.
OBJECT / APPEARANCECamera / Live
Provides an evolving scene that can be inspected interactively. The system can ask the user to move closer, change angle or show another component.
INTERACTIVE PERCEPTIONVoice
Adds natural conversational instructions and follow-up while the user’s hands or camera remain occupied. Voice does not by itself establish visual state.
CONVERSATIONAL CONTROLVideo
Captures motion, sequence, sound and state transitions. Useful for processes, faults and behavior that cannot be inferred reliably from one frame.
TEMPORAL EVIDENCELocation
Constrains nearby entities, local inventory, jurisdiction, travel time and service availability. Location changes applicability rather than visual identity.
GEOSPATIAL CONTEXTLive Data
Provides current prices, inventory, reservations, sports, finance or other state that may expire rapidly after retrieval.
TIME-SENSITIVE STATETools / APIs
Allow systems to calculate, query structured services, inspect availability or perform actions that cannot be completed through static text retrieval alone.
COMPUTE / ACTIONA photo is not one query. It is a scene containing multiple retrieval branches.
Google described AI Mode’s multimodal behavior as understanding the entire scene, including multiple objects, materials, colors and spatial relationships, then issuing multiple queries to learn more about what is shown.
Kitchen countertop photo
The image includes a coffee machine, milk frother, bean bag, grinder and a visible error code. The user asks, “Why is this not brewing correctly?”
scene = {machine, frother, beans, grinder, display:E05}
intent = diagnose brew failure
MULTI-OBJECT INPUT
Separate objects from evidence responsibilities
The machine model and error code matter for diagnosis. The bean bag may matter for grind/roast conditions. The frother is probably irrelevant unless the fault appears during milk preparation.
branch A → exact machine model
branch B → E05 meaning
branch C → grind / brew constraints
QUERY FAN-OUT FROM VISION
Retrieve model-specific instructions before advising action
Generic espresso-machine troubleshooting should not override exact model documentation. If the fix involves opening electrical components, uncertainty and safety boundaries matter.
retrieve → manual + support docs
verify → model / region
answer → safe troubleshooting only
OBJECT → DOCUMENT → CLAIM
Live search changes retrieval from a snapshot into an interactive perception loop.
Google expanded Search Live globally in March 2026 to the languages and locations where AI Mode is available. Search Live supports interactive conversations using voice and camera, creating a continuously updateable query context rather than a single static image.
Point
User points the camera at the scene or object.
VISUAL INPUTAsk
Voice supplies intent: “What is that connector?”
LANGUAGE INTENTPerceive
The system interprets the current visible scene.
SCENE STATERetrieve
Search gathers relevant web, video, forum or other resources.
EVIDENCERespond
The system answers with links and asks for more information if needed.
SYNTHESISReframe
User moves camera or adds a follow-up, updating the evidence state.
NEW CONTEXTContinue
The conversation repeats until the task is resolved or handed off.
ITERATIVE SEARCHThe same object must remain the same entity across every modality.
Multimodal systems fail when a visual candidate, text label, product record and live inventory state are treated as loosely similar instead of being resolved to the same exact entity or variant.
YES → CONTINUE
Multimodality solves perception. Agenticity solves workflow.
The matrix maps task class to decisive modality, live-data pressure, entity requirements, likely tool use and the failure mode that can corrupt the final action.
An agent must preserve constraints across every step of the task.
Google has described AI Mode agentic capabilities for restaurant reservations, local appointments, event tickets, shopping and travel. The system may search multiple sites, inspect current options and present direct paths to complete the action.
Parse Goal
Extract the user’s actual outcome and non-negotiable constraints.
INTENTPlan
Decompose into searches, comparisons, tools and state checks.
TASK GRAPHDiscover
Find candidate providers, products or information sources.
SEARCHInspect
Read current price, availability, policies, fit or service state.
LIVE STATECompare
Normalize options against the same user constraints.
DECISIONAct
Fill forms, prepare cart, select slot or initiate a supported transaction path.
EXECUTIONConfirm
User reviews important details before commitment when required.
CONTROLRecover
Handle failure, changed availability or invalid constraints without silently changing the task.
RESILIENCEAgentic search is only as reliable as the state it can inspect and the tools it can safely call.
Static web content can explain a product or service. Agentic tasks often require current machine-readable state: availability, price, inventory, booking slots, account state or transaction capability.
Web Navigation
Browser agents can inspect pages, DOM structure and accessibility semantics to locate information and controls.
READ / NAVIGATEPartner Integrations
Direct integrations can expose structured availability, product state or booking inventory more reliably than screen interpretation alone.
STRUCTURED STATEKnowledge / Maps / Shopping
Search can combine web retrieval with specialized Google information systems for entities, geography and commerce.
DOMAIN DATATransaction Protocols
Protocols can make agent-business interaction explicit: capabilities, endpoints, carts, checkout and post-purchase operations.
ACTION INTERFACEPrice
Commercial price can change between discovery and action. The action state should be rechecked at the point of commitment.
REVALIDATEAvailability
Restaurant, ticket, appointment or inventory state is inherently perishable and must be treated as time-bound evidence.
LIVE / EXPIRINGPolicy
Eligibility, cancellation and transaction rules may differ by region, account, provider or effective date.
APPLICABILITYUser Constraint
The agent should not silently relax party size, accessibility needs, budget, dates or other user constraints simply to complete the task.
PRESERVE INTENTAgentic commerce is becoming a protocol problem, not only a browser problem.
Google’s Universal Commerce Protocol documentation describes an open standard for enabling agentic actions in AI Mode and Gemini. UCP lets businesses expose capabilities and transaction endpoints rather than relying only on an agent to interpret a retail website visually.
Agent interprets the website interface
The system discovers the product, reads the page, locates controls and attempts to navigate the site’s existing purchase flow. This can work, but the agent has to infer business capabilities from UI state.
discover page
→ parse DOM / accessibility tree
→ inspect price / inventory
→ locate cart / checkout controls
INTERFACE INTERPRETATION
Business publishes explicit agent capabilities
UCP defines a standardized way for agents and commerce systems to communicate. Google documentation includes public business profiles, versioned capabilities and APIs for actions such as cart or checkout.
/.well-known/ucp
→ capabilities
→ endpoints
→ structured cart / checkout interaction
DECLARED ACTION SURFACE
Seven tasks. Seven different perception-to-action contracts.
These traces show why multimodal input, entity identity, live state and user confirmation must be recomposed differently depending on the task.
Identify a replacement pump
User uploads a photo with an incomplete model label.
Diagnose a garden plant problem
User points camera at leaves and asks why they are yellowing.
Find a laptop under exact constraints
User needs 32 GB RAM, under 1.5 kg, local warranty and a maximum budget.
Find a table for six Friday night
Constraints include time, neighborhood, food type and late closing.
Plan and monitor a flight
User wants a route under a budget and asks for price monitoring.
Monitor apartment listings
User gives a complex set of requirements and wants notification when a match appears.
Build a multi-item cart
User asks an agent to collect compatible items from one merchant.
“Show me where this cable goes.”
User points camera at an unfamiliar device while asking by voice.
Multimodal, visual, live and agentic search are overlapping—not interchangeable.
Separating them prevents a product feature from becoming a vague umbrella claim about how every AI search system works.
Visual Search vs. Multimodal Search
Visual search uses images or camera input. Multimodal search combines visual signals with text, voice or other context to resolve a broader task.
IMAGE ⊂ MULTIMODALMultimodal vs. Live
Multimodal input can be static. Live search adds an interactive temporal loop in which the scene and conversation can change continuously.
STATIC ≠ CONTINUOUSSearch vs. Agent
Search retrieves and synthesizes information. An agent can use that information to perform multi-step work or maintain a future task state.
ANSWER ≠ EXECUTIONBrowser Agent vs. Protocol Agent
A browser agent interprets UI and DOM state. A protocol-integrated agent can call declared capabilities or APIs directly.
UI INTERPRETATION ≠ API CONTRACTMore modalities create more evidence. They also create more ways to be confidently wrong.
These failures cover perception, cross-modal identity, live state, planning, tool use, action and the handoff between generated reasoning and the real-world system.
A visually similar object is mistaken for the exact model or variant.
VERIFY LABEL / IDPartially visible text is read as a complete identifier and drives the wrong entity resolution.
UNCERTAINTY GATEAn irrelevant object in the frame is treated as evidence for the user’s actual target.
TARGET OBJECTImage, text and catalog results refer to different variants while the system treats them as one.
ENTITY CONTINUITYA correct visual identity is combined with stale price, stock, policy or availability.
RECHECK STATEThe agent silently relaxes budget, date, accessibility or product constraints to complete the task.
PRESERVE USER INTENTThe agent reaches execution before confirming the exact item, slot, provider or final price.
CONFIRM BEFORE COMMITThe state retrieved from a partner or website changes before the next tool call.
ATOMIC RECHECKAn agent misreads unlabeled controls, visual-only buttons or unstable client-side state.
ACCESSIBLE UIA monitoring agent reports duplicate or cosmetic updates as meaningful new state.
SEMANTIC DIFFAn agent assumes a business supports a capability or protocol version it has not declared.
NEGOTIATE CAPABILITIESVisual interpretation is converted into physical or transactional action without enough certainty or safety context.
ABSTAIN / ESCALATEDocument what exists. Do not invent a universal agentic ranking layer.
These boundaries separate observable product capabilities from unsupported claims about multimodal ranking weights, agent planner internals or protocol requirements.
Google documents multimodal product capabilities, not a publisher-accessible numeric score for visual or multimodal visibility.
NO PUBLIC SCOREGoogle Lens and AI Mode can interact, but visual-search behavior should not be generalized into every generative search system.
PRODUCT BOUNDARYIts availability depends on where AI Mode and the relevant client experience are offered.
ROLLOUT DEPENDENTImages can hide decisive information and several different entities can look nearly identical.
PERCEPTION UNCERTAINTYGoogle’s documented agentic experiences generally preserve user choice or handoff for final action, depending on the product.
CONFIRMATION LAYERCurrent inventory or reservation state should not be treated as permanently valid after retrieval.
EXPIRING STATEUCP enables agentic commerce actions; it is not documented as a general Search ranking factor.
COMMERCE PROTOCOLAccessible DOM and stable action flows improve usability and agent operability without guaranteeing Search visibility.
PRODUCT READINESSDecorative media does not become useful simply because AI search is multimodal.
UTILITY FIRSTA multimodal result is unsafe if image, text and structured records cannot be resolved to the same underlying thing.
IDENTITY BEFORE ACTIONAgents should inspect declared capabilities and versions instead of assuming every business endpoint supports the same workflow.
NEGOTIATEGood multimodal assets, structured state and agent-ready interfaces do not guarantee appearance in AI Mode, Search Live or any agentic experience.
NO GUARANTEEThe frontier is already shipping. The public documentation still leaves the private planner opaque.
Google has publicly documented multimodal AI Mode, global Search Live expansion, agentic Search workflows, generative-AI optimization guidance and Universal Commerce Protocol integration. These sources define the hard product boundary for this page.
AI Mode can search what the user sees
Google introduced multimodal search in AI Mode by combining Lens visual capabilities with Gemini. Its product description says the system can understand the overall scene, recognize multiple objects and issue multiple queries to learn more about what appears in the image.
Voice + camera enable an interactive multimodal conversation
Google announced in March 2026 that Search Live expanded globally to all languages and locations where AI Mode is available, allowing users to converse with Search using voice and camera.
AI Mode can search across sites and current availability for actions
Google has described agentic AI Mode experiences for reservations, event tickets, local appointments and travel. Its documented approach can combine web browsing, partner integrations, Maps or other Google information systems to locate options that satisfy constraints.
Google’s AI optimization guide now explicitly discusses browser agents
Google Search Central’s 2026 generative-AI optimization guide describes AI agents as systems that can perform tasks on behalf of people. It notes that browser agents may inspect screenshots, DOM structure and the accessibility tree and points site owners toward agent-oriented best practices when relevant.
UCP exposes explicit commerce capabilities to AI surfaces
Google’s UCP documentation describes an open protocol for agentic commerce actions across AI Mode and Gemini. Merchants can publish a UCP profile, declare capabilities and expose supported transaction endpoints.
Search is expanding toward persistent information agents and broader action
Google’s I/O 2026 Search announcements describe information agents that can monitor the web and current data for changes, plus broader agentic booking and task capabilities. This extends Search from a request-response model toward persistent tasks in some product contexts.
Audit perception first. Then audit whether the action still matches the user’s original task.
This checklist covers modality quality, entity continuity, provenance, live state, browser-agent usability, protocol capabilities, action confirmation and failure recovery.
AI Search research. The complete retrieval-to-action system.
This final frontier node closes the AI Search research graph: from query decomposition and retrieval through grounding, sources, passages, optimization and entities into multimodal perception and agentic execution.
What Is AI Search?
Define the retrieval, evidence and synthesis architecture behind AI-oriented search interfaces.
OPEN NODE → AIS / 02 · QUERIESQuery Fan-Out
Decompose complex requests into distinct retrieval branches and evidence responsibilities.
OPEN NODE → AIS / 03 · RETRIEVALInformation Retrieval
Locate candidate documents, passages, records and evidence before downstream synthesis.
OPEN NODE → AIS / 04 · RAGRetrieval-Augmented Generation
Feed retrieved external evidence into generation while preserving context quality and provenance.
OPEN NODE → AIS / 05 · EVIDENCEGrounding
Anchor generated claims to applicable, current and traceable external evidence.
OPEN NODE → AIS / 06 · SOURCESCitations & Source Selection
Choose evidence sources by role, directness, freshness, applicability, independence and provenance.
OPEN NODE → AIS / 07 · PASSAGESPassage Retrieval
Retrieve the smallest semantically complete evidence unit that can resolve a precise information need.
OPEN NODE → AIS / 08 · OPTIMIZATIONAI Search Optimization
Build technically accessible, differentiated, evidence-rich and interaction-ready information systems.
OPEN NODE → AIS / 09 · ENTITIESEntity Visibility in AI Search
Resolve identity, attributes and relationships so machines can distinguish the intended entity from alternatives.
OPEN NODE →Multimodal & Agentic Search
Combine perception, entity continuity, retrieval, live state, tools and action into one auditable search workflow.
CURRENT NODEPERCEIVE / RESOLVE / RETRIEVE / VERIFY / ACT
The frontier of Search is not a better answer box. It is a system that can preserve truth while moving from perception to action.
Multimodal search widens the evidence surface. Agentic search widens the execution surface. The hard engineering problem is continuity: the same user intent, the same entity, the same constraints and the same verified state must survive every transition—from camera frame to product record, from search result to live availability, and from recommendation to action. When that continuity breaks, capability becomes error. When it holds, Search becomes an operational intelligence layer.