Structured Outputs

Generate structured outputs using FriendliAI’s Structured Outputs feature.

Large language models (LLMs) excel at creative text generation, but we often face a case where we need LLM outputs to be more structured. This is where our exciting new “structured output” feature comes in.

Structured Outputs is also available in Friendli Dedicated Endpoints and Friendli Container.

For more advanced use cases of our Structured Outputs feature, check out our detailed blog post on Structured Output for LLM Agents.

Structured response modes

Type	Description	Name at OpenAI
json_schema	The model returns a JSON object that conforms to the given schema.	Structured Outputs
json_object	The model can return any JSON object.	JSON mode
regex	The model returns a string that conforms to the given regex schema.	N/A

How to use

json_schema

This guide provides a step-by-step example of how to create a structured output response in JSON form.
In this example, we will use Python and the pydantic library to define a schema for the output.

1. Define Object Schema

   Define a schema that contains information about a dish.

   from pydantic import BaseModel
   
   class Result(BaseModel):
       dish: str
       cuisine: str
       calories: int
   

2. Asking the model for structured output

   Call structured output and use schema to structure the response.

   import os
   from openai import OpenAI
   
   client = OpenAI(
       base_url="https://api.friendli.ai/serverless/v1",
       api_key=os.getenv("FRIENDLI_TOKEN"),
   )
   
   completion = client.chat.completions.create(
       model="meta-llama-3.1-8b-instruct",
       messages=[
           {
               "role": "user",
               "content": "Suggest a popular Italian dish in JSON format.",
           },
       ],
       response_format={
           "type": "json_schema",
           "json_schema": {
               "schema": Result.model_json_schema(),
           }
       }
   )
   

3. Using structured output results

   You can use the output in the following way.

   response = completion.choices[0].message.content
   print(response)
   

   The code output result is as follows.

   Result:

   {
       "dish": "Spaghetti Bolognese",
       "cuisine": "Italian",
       "calories": 540
   }
   

Supported JSON schemas

We ensure super-fast schema-guided generation by disabling JSON schema features that cause computation inefficiencies. We support all seven standard JSON schema types (null, boolean, number, integer, string, object, array), and the supported JSON schema keywords are listed below.

Using unsupported or unexpected JSON schema keywords may result in them being ignored, triggering an error, or causing undefined behavior.

Type-specific keywords

• integer
  • exclusiveMinimum, exclusiveMaximum, minimum, maximum (Note: these are not supported in number)
• string
  • pattern
  • format
    • Supported values: uuid, date-time, date, time
• object
  • properties
  • additionalProperties is ignored, and is always set to False.
  • required: We support both required and optional properties, but have these limitations:
    • The sequence of the properties is fixed.
    • The first property should be required. If not, the first required property is moved to the first.
• array
  • items
  • minItems: We support only 0 or 1 for minItems.

Constant values and enumerated values

const and enum only support constant values of null, boolean, number, and string.

Schema composition

We support only anyOf for schema composition.

Referencing subschemas

We only support referencing ($ref) to “internal” subschemas. These subschemas must be defined within $defs, and the value of $ref must be a valid URI pointing to a subschema. Please refer here for more details.

Annotation

JSON schema annotations such as title, $comments or description are accepted but ignored.