inference
Press regenerate and the answer changes. Move the temperature, narrow the token pool, and watch how one small choice sends a reply down another path.
A local LLM does not have one meaningful “tokens per second” number. Before it can write the first token of an answer, it has to process the tokens already in the prompt.
A local LLM being slow does not automatically mean that your GPU is too weak. The model may be spilling into system RAM, the CPU may be doing more work
A local LLM can fit comfortably into your GPU at an 8,000-token context and run out of memory when you increase that context several times over.



