rtemis (version 1.4.3)

ai_review: Write an assessment of a model review with a language model

Description

Ask a language model to write a summary, an evaluation, next steps and caveats from a review of a trained supervised model. Every statement cites the codes of the review findings it rests on, and the result keeps the review and a record of how the text was produced.

Usage

ai_review(x, llm, context = NULL, temperature = NULL, verbosity = 1L, ...)

Value

AISupervisedReview object.

Arguments

x

SupervisedReview, Supervised or SupervisedRes object: A review, or a trained model to review with default settings.

llm

rtemis.llm LLM or Agent object: The model to write the assessment, for example from rtemis.llm::create_Ollama() or rtemis.llm::create_Anthropic().

context

Optional Character: Domain context for the assessment.

temperature

Optional Numeric [0, Inf): Sampling temperature for this call. NULL uses the model's own setting.

verbosity

Integer: Verbosity level.

...

Not used.

Author

EDG

Details

The model receives the review as JSON and, if given, context: the question the model addresses, the costs of different errors, how its predictions will be used. It never receives the data. Without context, the assessment states that usefulness cannot be judged.

The model answers in a declared structure whose codes are restricted to the review's findings, and the answer is checked: an answer that does not match the structure, or cites a code that is not a finding of the review, is an error. The returned object records the model, the provider, the temperature, the full prompt, a SHA-256 hash of the review sent and the time, so the assessment can be audited and, as far as the model allows, reproduced.

The written assessment is an interpretation of the review, which remains the evidence; read them together. write_text writes both as Markdown.

Requires the rtemis.llm package and access to a model.

Examples

if (FALSE) {
# Requires a running Ollama server with the model pulled.
mod <- train(
  iris,
  hyperparameters = setup_CART(),
  outer_resampling_config = setup_KFold(5L),
  verbosity = 0L
)
llm <- rtemis.llm::create_Ollama(model_name = "gemma4:e4b")
ai_review(
  mod,
  llm = llm,
  context = "Teaching example; no decisions depend on the predictions."
)
}