rlmstudio (version 0.2.2)

lms_score_expected: Calculate Expected Scores and Uncertainty from Logprobs

Description

Takes a logprobs dataframe (from an lms_chat_result) and calculates the weighted average score, normalized probabilities, and uncertainty metrics.

Usage

lms_score_expected(lp_df, scale = 1:5)

Value

A named list containing three numeric elements (expected_value, weighted_sd, entropy) and a data.frame named probabilities with columns label and prob. Returns NULL if the input dataframe is empty or invalid.

Arguments

lp_df

A dataframe of logprobs (e.g., x$logprobs).

scale

Numeric vector. The valid labels (e.g., 1:5).

Examples

# Create a sample logprobs dataframe representing a model's generation step
mock_logprobs <- data.frame(
  step_token = rep("4", 3),
  step_logprob = rep(0, 3),
  candidate_token = c("4", "5", "3"),
  candidate_logprob = c(-0.105, -2.302, -3.506),
  stringsAsFactors = FALSE
)

# Calculate the expected score and uncertainty metrics
lms_score_expected(mock_logprobs, scale = 1:5)