On this article, be taught concerning the architectural variations between structured output and performance calls in fashionable language modeling methods.
Matters lined embody:
How structured output and performance calls work beneath the hood. When to make use of every strategy in a real-world machine studying system. Efficiency, value, and reliability are trade-offs between the 2.
Structured output vs. operate calls: which ought to your agent use?
Picture by editor
introduction
The core of a language mannequin (LM) is a textual content enter and textual content output system. That is completely high-quality for human conversations through a chat interface. However for machine studying practitioners constructing autonomous brokers and dependable software program pipelines, parsing, routing, and integrating uncooked unstructured textual content into deterministic methods is a nightmare.
Constructing dependable brokers requires predictable, machine-readable output and the power to seamlessly work together with the exterior atmosphere. To fill this hole, fashionable LM API suppliers (corresponding to OpenAI, Anthropic, and Google Gemini) have launched two principal mechanisms.
Structured output: Forces the mannequin to reply strictly in accordance with a predefined schema, mostly a JSON schema or a Python Pydantic mannequin. Perform calls (utilizing instruments): Equip your mannequin with a library of operate definitions that you would be able to select to name dynamically based mostly on the immediate context.
At first look, these two options are very related. Each usually depend on passing a JSON schema to the API beneath the hood, ensuing within the mannequin outputting structured key-value pairs slightly than conversational prose. Nonetheless, they serve basically completely different architectural functions in agent design.
Complicated the 2 is a standard pitfall. Selecting the fallacious mechanism for a characteristic can result in weak architectures, excessive latency, and unnecessarily excessive API prices. Let’s spotlight the architectural variations between these strategies and supply a decision-making framework for when to make use of every.
Unpacking the mechanism: the way it works beneath the hood
To know when to make use of these options, it’s good to perceive the distinction between the machine degree and the API degree.
How structured output works
Previously, getting a mannequin that outputs uncooked JSON required some fast engineering (“You are a helpful assistant that *solely* speaks in JSON…”). This was error-prone and required in depth retry logic and validation.
Fashionable “structured output” basically modifications this by grammar-constrained decoding. Libraries corresponding to Define, or native options corresponding to OpenAI’s structured output, mathematically constrain the likelihood of a token on the time of era. If the chosen schema specifies that the subsequent token have to be a quote or a sure Boolean worth, the likelihood of all non-compliant tokens is masked (set to zero).
This can be a single-turn era that emphasizes kind. The mannequin responds on to your prompts, however its vocabulary is restricted to the precise buildings you outline, with the purpose of guaranteeing close to 100% schema compliance.
How operate calls work
Perform calls, however, are extremely depending on instruction tuning. Throughout coaching, the mannequin is fine-tuned to acknowledge conditions when it lacks the knowledge wanted to finish a immediate, or when the immediate explicitly asks you to carry out an motion.
Whenever you present an inventory of instruments to the mannequin, you might be telling the mannequin, “If you’d like, you’ll be able to pause textual content era, choose a device from this listing, and generate the mandatory arguments to run it.”
That is primarily a multi-turn interactive move.
The mannequin decides to name the device and prints the device identify and arguments. The mannequin will pause. The code itself can’t be executed. The appliance code executes the chosen operate domestically utilizing the generated arguments. The appliance returns the results of the operate to the mannequin. The mannequin continues to synthesize this new data and generate the ultimate response.
Should you select structured output
For pure information transformation, extraction, or standardization functions, structured output needs to be your default strategy.
Typical use case: The mannequin accommodates all the mandatory data inside the immediate and context window. You simply have to reshape it.
Examples for practitioners:
Information extraction (ETL): Course of uncooked, unstructured textual content, corresponding to buyer assist transcripts, and extract entities. Identify, date, grievance kind, sentiment rating, and so forth. Convert to strict database schema. Question era: Rework messy pure language consumer prompts into rigorous, validated SQL queries or GraphQL payloads. If the schema is damaged, queries will fail, so 100% compliance is vital. Inside agent reasoning: Constructions an agent’s “ideas” earlier than it acts. You’ll be able to power a Pydantic mannequin that requires a thought_process subject, a speculation subject, and eventually a call subject. This forces a sequence of thought course of that’s simply parsed by the backend logging system.
Verdict: In case your “motion” is only a format, use structured output. As a result of there is no such thing as a intermediate era interplay with exterior methods, this strategy ensures excessive reliability, decrease latency, and 0 schema parsing errors.
When selecting a operate name
Perform calls are the engine of agent autonomy. If structured output determines the form of your information, operate calls decide the management move of your utility.
Typical use circumstances: exterior interactions, dynamic determination making, circumstances the place the mannequin must retrieve data it does not presently have.
Examples for practitioners:
Carry out real-world actions: Set off exterior APIs based mostly on dialog intent. If the consumer says, “E-book my common flight to New York,” the mannequin makes use of a operate name to set off the book_flight(vacation spot=”JFK”) device. Search Augmentation Era (RAG): As a substitute of a easy RAG pipeline that always searches the vector database, brokers can use the search_knowledge_base device. The mannequin dynamically decides which search phrases to make use of based mostly on context, or decides to not search in any respect if it already is aware of the reply. Dynamic activity routing: For complicated methods, the router mannequin makes use of operate calls to pick out specialised subagents (e.g., calls to delegate_to_billing_agent and delegate_to_tech_support) which might be greatest suited to deal with a specific question.
Verdict: Select operate calls when your mannequin must work together with the surface world, retrieve hidden information, or conditionally execute software program logic mid-thinking.
Affect on efficiency, latency, and value
When deploying brokers into manufacturing, selecting an structure between these two strategies has a direct influence on unit economics and consumer expertise.
Token consumption: Perform calls usually require a number of spherical journeys. The consumer sends the system immediate, the mannequin sends the device arguments, the consumer sends again the device outcomes, and eventually the mannequin sends the reply. Every step is added to the context window and enter and output token utilization is amassed. Structured outputs are usually resolved in yet another cost-effective flip. Latency overhead: The spherical journeys inherent in operate calls introduce important community and processing latency. The appliance should look ahead to the mannequin, run native code, and look ahead to the mannequin once more. In case your principal purpose is solely to transform information into a selected format, structured output can be considerably sooner. Reliability and retry logic: Carefully structured output (through constrained decoding) supplies practically 100% schema constancy. You’ll be able to belief the output form with out the necessity for complicated evaluation blocks. Nonetheless, operate calls should not statistically predictable. Fashions can hallucinate arguments, select the fallacious instruments, or get caught in diagnostic loops. Manufacturing-grade operate calls require sturdy retry logic, fallback mechanisms, and cautious error dealing with.
Hybrid strategy and greatest practices
Superior agent architectures usually blur the road between these two mechanisms, requiring a hybrid strategy.
Duplicate:
It is price noting that fashionable operate calls truly depend on structured output internally to make sure that the generated arguments match the operate’s signature. Conversely, you’ll be able to design an agent that makes use of solely structured output and returns a JSON object that describes the actions that the deterministic system ought to carry out after the era is full. Successfully disguise device utilization with out incurring multi-turn latencies.
Architectural recommendation:
“Controller” sample: Makes use of operate calls to an orchestrator or “mind” agent. Be happy to name instruments to collect context, question databases, and execute APIs till you might be happy that the required state has been amassed. “Formatter” sample: As soon as the motion is full, cross the uncooked consequence to the ultimate cheap mannequin utilizing solely structured output. This ensures that the ultimate response precisely matches the expectations of the UI element or downstream REST API.
abstract
LM engineering is quickly shifting from creating conversational chatbots to constructing extremely dependable programmatically autonomous brokers. Understanding how you can constrain and direct the mannequin is essential to that transition.
TL;DR
Use structured output to find out the form of your information Use operate calls to find out actions and interactions
Practitioner determination tree
Comply with this straightforward three-step guidelines when constructing new performance:
Do you want exterior information whereas considering, or do it’s good to carry out an motion? ⭢ Utilizing operate calls Are you simply parsing, extracting, or changing unstructured context into structured information? ⭢ Utilizing structured output Do you want absolute and strict adherence to complicated nested objects? ⭢ Utilizing structured output with constrained decoding
closing ideas
The simplest AI engineers ought to deal with operate calls as highly effective however unpredictable capabilities, used sparingly and surrounded by sturdy error dealing with. Conversely, structured output needs to be handled because the dependable foundational glue that holds fashionable AI information pipelines collectively.
