Evaluates one text, JSON-compatible, or context state against named questions.
This is a model call, not a test run. Providers handle their own request and answer formats. An evaluation model does not need to support chat generation.
Summary
Functions
Evaluates one state against a map of named questions.
Same as evaluate/4, but raises on error.
Lists model specs that evaluate/4 can call with an installed adapter.
Types
@type state() :: String.t() | map() | list() | ReqLLM.Context.t()
Functions
@spec evaluate(ReqLLM.model_input(), state(), map(), keyword()) :: {:ok, ReqLLM.Response.t()} | {:error, term()}
Evaluates one state against a map of named questions.
questions = %{
department: %{
type: :choice,
instructions: "Which team should handle this?",
criteria: %{billing: "Billing and refunds", support: "Other requests"}
},
urgent: %{type: :boolean, instructions: "Is this urgent?"}
}
{:ok, result} = ReqLLM.evaluate("typesafe:jev-latest", "Please refund me", questions)
result.object["department"]["choice"]
result.object["urgent"]["probability"]Choice and score answers may include probabilities and confidence. Providers
may support more question types. OpenRouter evaluations accept routing preferences
such as provider_options: [openrouter_provider: %{zdr: true}]. Eligible OpenAI
models use the Decisions API. They support boolean, choice, and score questions;
provider_options: [openai: [safety_identifier: "tenant-123"]] adds the optional
safety identifier. OpenAI refusals stay in the result as %{"type" => "refusal"}.
OpenAI Decisions also accepts a ReqLLM.Context state of system, user, and
assistant text, with images in user messages:
context =
ReqLLM.Context.new([
ReqLLM.Context.user([
ReqLLM.Message.ContentPart.text("Inspect the product in this photo."),
ReqLLM.Message.ContentPart.image(png_bytes, "image/png")
])
])
ReqLLM.evaluate("openai:gpt-6-luna", context, questions)Other providers accept only text or JSON state. Azure Microsoft Decision
models use a Foundry resource root as base_url and
provider_options: [azure: [deployment: "my-decision-deployment"]] to select
the deployment. They support boolean, choice, and score questions.
The response stores named answers in object with string keys.
provider_meta.raw_response keeps the original provider data.
@spec evaluate!(ReqLLM.model_input(), state(), map(), keyword()) :: ReqLLM.Response.t() | no_return()
Same as evaluate/4, but raises on error.
@spec models() :: [String.t()]
Lists model specs that evaluate/4 can call with an installed adapter.
Catalog evaluation metadata can also describe models for providers that ReqLLM does not yet support. Those models are excluded.