local LLMs
A 16-core CPU can load a quantized local LLM, show activity across many cores, and still generate text much more slowly than expected. Adding threads may help at first.
Q4_K_M, Q5_K_M and Q8_0 can run the same model with very different memory requirements. Here is how to choose without wasting VRAM.
A local LLM can fit comfortably into your GPU at an 8,000-token context and run out of memory when you increase that context several times over.


