ReqLLM.Evaluation (ReqLLM v1.30.0)

Copy Markdown View Source

Evaluates one text, JSON-compatible, or context state against named questions.

This is a model call, not a test run. Providers handle their own request and answer formats. An evaluation model does not need to support chat generation.

Summary

Functions

Evaluates one state against a map of named questions.

Lists model specs that evaluate/4 can call with an installed adapter.

Types

state()

@type state() :: String.t() | map() | list() | ReqLLM.Context.t()

Functions

evaluate(model_spec, state, questions, opts \\ [])

@spec evaluate(ReqLLM.model_input(), state(), map(), keyword()) ::
  {:ok, ReqLLM.Response.t()} | {:error, term()}

Evaluates one state against a map of named questions.

questions = %{
  department: %{
    type: :choice,
    instructions: "Which team should handle this?",
    criteria: %{billing: "Billing and refunds", support: "Other requests"}
  },
  urgent: %{type: :boolean, instructions: "Is this urgent?"}
}

{:ok, result} = ReqLLM.evaluate("typesafe:jev-latest", "Please refund me", questions)
result.object["department"]["choice"]
result.object["urgent"]["probability"]

Choice and score answers may include probabilities and confidence. Providers may support more question types. OpenRouter evaluations accept routing preferences such as provider_options: [openrouter_provider: %{zdr: true}]. Eligible OpenAI models use the Decisions API. They support boolean, choice, and score questions; provider_options: [openai: [safety_identifier: "tenant-123"]] adds the optional safety identifier. OpenAI refusals stay in the result as %{"type" => "refusal"}. OpenAI Decisions also accepts a ReqLLM.Context state of system, user, and assistant text, with images in user messages:

context =
  ReqLLM.Context.new([
    ReqLLM.Context.user([
      ReqLLM.Message.ContentPart.text("Inspect the product in this photo."),
      ReqLLM.Message.ContentPart.image(png_bytes, "image/png")
    ])
  ])

ReqLLM.evaluate("openai:gpt-6-luna", context, questions)

Other providers accept only text or JSON state. Azure Microsoft Decision models use a Foundry resource root as base_url and provider_options: [azure: [deployment: "my-decision-deployment"]] to select the deployment. They support boolean, choice, and score questions. The response stores named answers in object with string keys. provider_meta.raw_response keeps the original provider data.

evaluate!(model_spec, state, questions, opts \\ [])

@spec evaluate!(ReqLLM.model_input(), state(), map(), keyword()) ::
  ReqLLM.Response.t() | no_return()

Same as evaluate/4, but raises on error.

models()

@spec models() :: [String.t()]

Lists model specs that evaluate/4 can call with an installed adapter.

Catalog evaluation metadata can also describe models for providers that ReqLLM does not yet support. Those models are excluded.