edgemodelr (version 0.1.6)

edge_benchmark: Performance benchmarking for model inference

Description

Test inference speed and throughput with the current model to measure the effectiveness of optimizations.

Usage

edge_benchmark(
  ctx,
  prompt = "The quick brown fox",
  n_predict = 50,
  iterations = 3
)

Value

List with performance metrics

Arguments

ctx

Model context from edge_load_model()

prompt

Test prompt to use for benchmarking (default: standard test)

n_predict

Number of tokens to generate for the test

iterations

Number of test iterations to average results

Examples

if (FALSE) {
setup <- edge_quick_setup("TinyLlama-1.1B")
if (!is.null(setup$context)) {
  ctx <- setup$context
  perf <- edge_benchmark(ctx)
  print(perf)
  edge_free_model(ctx)
}
}