Definition
Local inference runs large language models on user-owned hardware (laptop, workstation, edge GPU) rather than cloud APIs—preserving data locality and reducing per-token API cost at the expense of model size and operator complexity.
Key Points
- MoE architectures (mixture-of-experts) and aggressive quantization (NVFP4, GGUF) enable coding agents on consumer GPUs
- flint-coding-model exemplifies laptop-targeted open weights vs datacenter-only frontier models
- Trade-offs: context limits, benchmark gap vs largest closed models, kernel maintenance per platform (Metal, CUDA)