tidyllm (version 0.6.0)

get_stream: A running chat's deltas, as a stream

Description

Returns a coro async generator over the text deltas: the pull form of .on_chunk. It replays whatever has already arrived and then continues live, so it is safe to ask for at any point in a job's life and always tells the same story as get_partial().

Usage

get_stream(.job)

Value

A coro async generator instance yielding character deltas.

Arguments

.job

A streaming tidyllm_chat_job from send_chat().

Details

This is what shinychat::chat_append() consumes directly, because shinychat's whole extensibility contract is inherits(x, "coro_generator_instance"). The generator is asynchronous rather than synchronous on purpose: shinychat's consumer only yields the event loop at an await(), so a synchronous generator would drain in one tick and the reply would appear all at once instead of token by token.

In a plain script this is rarely what you want. An async generator hands a coro::loop() promises rather than text, so consuming it takes coro::async() and coro::await_each(); .on_chunk or a get_partial() loop says the same thing with less ceremony.

Examples

if (FALSE) {
job <- llm_message("Tell me a story") |> send_chat(claude(), .stream = TRUE)
shinychat::chat_append("chat", get_stream(job))
}