quantization

"When you quantize the K-cache to 4-bit or even 8-bit, you are actively degrading the attention mechanism's ability to perfectly match the exact syntax of a strict schema defined 40,000 tokens ago."

"GGUF (GPT-Generated Unified Format) is a dedicated format for running AI in a local environment, originally developed for the llama.cpp project."

"LLM quantization is a technology that can reduce model size by 50-75% compared to FP16 while keeping perplexity (quality indicator) degradation within 3%."

"Interestingly it has very good perplexity for the size, and *may be* faster than other leading quants especially on Vulkan backend?"

"I tried running Qwen3.5-27B, which was released a few days ago, because I recently bought a 32GB RAM M2 MacBook Pro, and I wanted to try running a local LLM."

"GGUF is not just a "light model file", but a very smart format that packages model architecture information, tokenizers, and quantization parameters into a single file."

"Taalas serves Llama 3.1 8B at 17,000 tokens/second"

"submitted by /u/TKGaming_11 "

"llama.cppとは、一言でいうと「C/C++で書かれた、依存関係ゼロのLLM推論エンジン」である。"

"DGX Sparkで動くコーディング用ローカルモデルの中だと現状一番クオリティが高そう。"

"I just created quant-models for the new RedFire-Image-Edit 1.0"

"In 2026, the turning point was that the performance of NPUs such as Apple Neural Engine, Qualcomm Snapdragon X, and MediaTek Dimensity surpassed entry-class GPUs."

"Edge AI, which executes AI inference directly on the device, offers three major benefits: low latency, privacy protection, and offline operation."

"To address these issues, we propose KBVQ-MoE, a novel VQ framework to enhance extremely low-bit quantization for MoE-based LLMs."

"This is the industry's first implementation of 2bit industrial-grade quantization for an on-device model."

"68% Size Reduction with <0.4pp Accuracy Loss on SNIPS"

"BitNet b1.58[1] is a neural network that quantizes weights to {-1, 0, +1}."

"I found that MXFP4 has lower perplexity than Q4_K_M and Q4_K_XL."

"GGUF is a game-changer, enabling even the RTX 5090 with 32GB of VRAM to run 70B models."

"I have been using GLM 4.7 Flash to perform a few refactoring tasks in some personal web projects and have been quite impressed by how well the model handles Roo Code without breaking apart."

"XQuant's fundamental idea: Instead of directly storing KV, hold the layer's input activation X and create KV during decoding, which saves twice the memory compared to holding KV."

"I was surprised by how usable TQ1_0 turned out to be. In most chat or image‑analysis scenarios it actually feels better than the Qwen3‑VL 30 B model quantised to Q8."

"The key is (1) 1B-class GGUF, (2) quantization (Q4 focused), (3) not increasing the KV cache too much, and configuring llama.cpp (=llama-server) tightly."

"Quantized models can be seamlessly deployed on Amazon SageMaker AI using a few lines of code."

"Quantizing LLMs Step-by-Step: Converting FP16 Models to GGUF"

"So by merging LoRA to full model, it's possible to quantize the merged model and have a Q8_0 GGUF FLUX.2 [dev] Turbo that uses less memory and keeps its high precision."

"HyperNova 60B base architecture is gpt-oss-120b."

"The research is based on an ArXiv publication."

"The research focuses on sensitivity-aware mixed-precision quantization."

"The paper focuses on query-aware mixed-precision KV cache quantization."