3. Sessions and turns¶
A session can make several requests, with tool calls or thinking time between turns. This chapter models the conversation and the context it accumulates.
The program¶
// Chapter 3: a session is many turns.
// A session prefills its context, decodes an answer, calls a tool, and
// comes back with a longer context. It continues with probability p.
use "std/args";
let Lambda = 0.5; // new sessions per second
let p = 0.8; // continue after a turn
let a = 2e-5; // prefill: seconds per token
let d = 2e-4; // decode: seconds per token
fn main() {
let Z = args.number("Z", 3.0); // mean tool time (s)
stage engine : fifo;
workload {
arrive poisson(Lambda);
init { set K = 0; } // context so far
turn {
set n = K == 0 ? ~uniform(1000, 3000) : ~exp(500); // new tokens
set o = ~exp(200); // output tokens
}
session {
turn;
observe context = K + n + o;
while (~bernoulli(p)) {
set K = K + n + o;
tool (~exp(Z));
turn;
observe context = K + n + o;
}
}
}
server {
run engine (cost(engine, a * (K + n) + d * o));
}
}
What is new¶
while and turn¶
session {
turn;
observe context = K + n + o;
while (~bernoulli(p)) {
set K = K + n + o;
tool (~exp(Z));
turn;
observe context = K + n + o;
}
}
turn; draws the next turn's attributes and waits for its response.
while tests its guard before each pass and continues after the loop when
it is zero. Here ~bernoulli(p) draws 1 with probability p = 0.8:
after each response, the client either thinks and sends another turn or
finishes. No explicit server call or final end; is needed.
A guard must be 0 or 1. Write a state condition such as while (more)
for trace continuation, or an explicit draw as above for random continuation.
Attributes carry state across turns¶
init runs once when the session arrives; set assigns a session
attribute, which survives from turn to turn. K is the context the session
has accumulated. It grows with each turn, increasing the memory the session needs.
tool¶
tool (Z) is the client's time between turns: a tool call, a human
reading, a downstream API. It is outside your system, so the program
declares nothing for it. It runs on a delay, an infinite-server station
the workload owns and the report calls tool: every call runs immediately
at rate 1, nobody waits. It is the same as declaring stage tool : delay;
after your stages. A tool server that sessions share, with a
capacity, is part of the deployment: declare it as a stage
(stage sandbox : fifo;) and name it, tool on sandbox (Z);.
Running it¶
run: horizon 20000 end 20000 warmup 2000 seed 1 events 97969 arrivals 9905 ended 8905 turns 43952 mean live 6.295
observe count mean 95% CI cv2 p99
------- ----- --------- -------- ----- ----------
context 43953 4962.0292 ±93.0588 0.460 17100.8941
client count mean 95% CI cv2 p99
-------- ----- ------ ------- ----- ------
response 43953 0.1784 ±0.0049 0.465 0.6027
stage number util done thru wait service iters
------ ------ ----- ----- ------ ------ ------- -----
engine 0.436 0.330 43953 2.4418 0.0434 0.1351 0
tool 5.859 0.999 35049 1.9472 0.0000 3.0091 0
With continuation probability \(p = 0.8\), a complete session has an expected \(1/(1-p) = 5\) turns. The report counts activity within the run's measurement window; it is not a sample of complete session lengths.
The tool stage accounts for 5.859 of the 6.295 live sessions on average. Keeping their context between turns uses memory even while the engine is not serving them. The mean observed context is 4 962 tokens.
What to try¶
Turn the thinking time down and watch the sessions pile back in:
for Z in 0.5 1 3 10; do
serq run docs/tutorial/programs/03-sessions.sq --horizon 20000 --warmup 2000 --seed 1 --set Z=$Z
done
Next: keep the context between turns. → The prefix cache