2. Memory is a resource

A request may need both a server and free memory. A pool models a resource with a fixed capacity, such as KV memory or request slots. A request waits until the pool can supply the units it needs.

The program

docs/tutorial/programs/02-memory.sq

// Chapter 2: memory is a resource.
// Four servers, so compute is not the bottleneck — a pool of 10 memory
// units is. A request holds the units it needs for as long as it runs.
use "std/args";

let Lambda = 0.8;
let S = 1.0;

fn main() {
  let C = args.number("C", 10);           // memory units

  pool mem { cap C; }
  stage svc : fifo(4);

  workload {
    arrive poisson(Lambda);
    turn {
      Size work_size = ~exp(1);
      Size items = ceil(~uniform(1, 5));   // request items
    }
  }

  server {
    Cost processing = {
      svc: S * work_size,
      mem: items
    };
    set t0 = now;
    hold mem (processing.mem) {
      observe admit_wait = now - t0;
      run svc (processing.svc);
    }
  }
}

Four servers now, so compute is not the constraint. Ten memory units are.

Reserve memory with hold

Cost processing = { svc: S * work_size, mem: items };
hold mem (processing.mem) {
  observe admit_wait = now - t0;
  run svc (processing.svc);
}

The workload draws work_size and items. The server converts them to service work and memory demand; here each item needs one memory unit.

hold joins the pool's queue, waits for processing.mem units and reserves them for the body. When the body finishes, the units are released automatically.

When are the units counted?

The declaration computes processing.mem once. The hold reads that stored amount at admission. A header that reads changing cache or budget state must calculate it at admission instead; Chapter 5 writes those expressions in the engine's taking, which is read at admission.

Running it

serq run docs/tutorial/programs/02-memory.sq --horizon 100000 --warmup 5000 --seed 1

run: horizon 100000 end 100000 warmup 5000 seed 1 events 158921 arrivals 79460 ended 75462 turns 75462 mean live 0.870

observe     count    mean   95% CI     cv2     p99
----------  -----  ------  -------  ------  ------
admit_wait  75462  0.0961  ±0.0056  13.711  1.8797

client    count   mean  95% CI    cv2    p99
--------  -----  -----  ------  -----  -----
response  75462  1.096  ±0.014  0.943  4.886

stage  number   util   done    thru    wait  service  iters
-----  ------  -----  -----  ------  ------  -------  -----
svc     0.794  0.559  75462  0.7943  0.0000   0.9997      0

pool  used  cached  queue  holders    wait  admits  evict(n)  evict(u)  preempt  spill  rej  stuck
----  ----  ------  -----  -------  ------  ------  --------  --------  -------  -----  ---  -----
mem    2.8     0.0  0.076    0.794  0.0961   79460         0         0        0      0    0      0

Note wait 0.0000 at the stage: nobody queues for a server. All the waiting has moved to the pool.

The sweep

for C in 4 6 10 20; do
  serq run docs/tutorial/programs/02-memory.sq --horizon 100000 --warmup 5000 --seed 1 --set C=$C
done

cap	admit wait	response	rejected
4	1.2384	2.2379	19,803
6	1.5808	2.5805	0
10	0.0961	1.0958	0
20	0.0006	1.0026	0

At capacity 4, a request needing 5 units is rejected before it waits. The 19,803 rejected sessions explain why mean waiting time can look better than at capacity 6: the means describe different admitted workloads. Compare rejection counts as well as latency.

For this workload, increasing capacity from 6 to 10 sharply reduces admission wait; increasing it to 20 leaves little waiting at either resource.

Head-of-line blocking

A pool's queue is FIFO by default, and only the head can be admitted. If the request at the front needs 5 units and 3 are free, the request behind it needing 2 units waits anyway. A large request can therefore delay smaller requests behind it.

queue by (expr) changes the order if you want to model a priority scheduler instead.

---

Next: real sessions come back. → Sessions and turns