Stopping a generation (host cancel) and ejecting a model

import threading
from abstractcore import create_llm
from abstractcore.exceptions import GenerationCancelledError

llm = create_llm("lmstudio", model="llama-3.2-1b-instruct")
stop = threading.Event()
threading.Timer(3.0, stop.set).start()        # e.g. the user pressed Stop
try:
    llm.generate("Write a very long story.", cancel_event=stop)
except GenerationCancelledError as e:
    print("stopped:", e)                        # never an "Error: ..." answer, never retried

What every provider guarantees

• An event that is already set raises before anything is sent to the model.
• A stop raises GenerationCancelledError (request_local: never retried, never counted against endpoint health). A cancelled answer is never returned as a complete one, and never as an "Error: ..." string with finish_reason="error".
• Streams are checked between chunks for every provider, and the upstream stream is closed.
• A call whose event is set that fails for any reason (for example a severed socket) is reported as the typed stop, not as a transport error.

Per lane

HTTP lanes: the request is severed

Checking "between chunks" is not enough over HTTP: a non-streaming request sits in a blocking socket read for the whole generation, and a streaming request sits there for the whole prefill. Neither the per-chunk check nor an injected exception (the gateway kill switch) can reach a thread blocked in recv.

So when an event is passed, the request runs on its own connection under an HttpCancelGuard: a watcher thread waits on the event and shuts the socket down (shutdown(SHUT_RDWR)), which wakes the blocked read immediately. The server sees the client disconnect and stops decoding. Measured: LM Studio 0.4.20 (MLX engine), Ollama 0.20.2 and llama.cpp's llama-server stop on disconnect, streaming or not. Ollama finishes a prompt evaluation already in progress before it notices; LM Studio and llama-server stop mid-prefill. Uncancellable calls keep using the provider's pooled client, exactly as before.

Not interruptible from Python

• One native call: one transformers prefill forward (chunks of ABSTRACTCORE_TRANSFORMERS_PREFILL_STEP tokens), one llama.cpp n_batch evaluation inside create_chat_completion, one mlx-vlm prefill op. The stop lands when that call returns.
• The Outlines structured-output lane and custom infer() models (for example DeepSeek-OCR) own their loop: they are checked before they start only.

Ejecting a model (unload_model) while it generates

unload_model() first cancels the instance's in-flight generations and waits for them (generation_cancel.InflightGenerations, bounded by DEFAULT_EJECT_DRAIN_TIMEOUT_S, 30 s, logged). In-process providers (MLX, HuggingFace) refuse the unload with a ProviderAPIError if a call is still running at the deadline, so memory is never freed under a running decode (Llama.close() under a running llama.cpp evaluation crashes the process). HuggingFace gives every call a private event when the host passed none, so any call can be stopped.

HTTP providers cancel our requests before asking the server to unload. LM Studio answers an unload under a running request with an in-stream "Model unloaded" error, which the retry layer would resample, reloading the model the operator just ejected.

After an eject:

• load_model() reloads an MLX or HuggingFace instance. It was missing, so a gateway POST /models/load after an unload of the default model could not warm it again.
• A generation on an ejected MLX or HuggingFace instance reloads the model on demand and logs a WARNING. Before, the gateway recorded a completed run whose answer was the string "Error: MLX model not loaded".
• The sync httpx client of LM Studio, Ollama and OpenAI-compatible providers is replaced, not left closed. A closed client made every later request of a pooled instance raise.

Measured through the hermetic gateway routes (POST /api/gateway/models/unload), with the settled process phys_footprint:

The operating system reclaims freed Metal buffers asynchronously, so a footprint read right after the unload can still show the weights for up to about a second. Use the MLX allocator (mx.get_active_memory(), which is 0 immediately) or wait for the reading to settle.

Remote AbstractCore server

RemoteAbstractCoreLLMClient (AbstractRuntime) severs its request to the AbstractCore server when the effect's event is set. The server treats the client disconnect as a cancel: HTTP-backed providers run off the event loop with a disconnect watcher, and their own guard severs the upstream request in turn. There is no abort endpoint and no request id: the wire form of a Stop is the closed connection. Measured chain runtime → server → LM Studio: the client returned in about 0 ms and the LM Studio worker went from 49.5 % CPU to 0 %. In-process providers on the server (transformers, llama.cpp) stay serialized on the event loop and are not cancelled by a disconnect.