quantization

"I'm running qwen3.6-35b-a3b with 8 bit quant and 64k context thru OpenCode on my mbp m5 max 128gb and it's as good as claude"

"I’m giving this one to Qwen 3.6. Opus managed to mess up the bicycle frame!"

"There are ways to avoid these before publishing quants in a rush (like "--validate-quants" to check and show you if you've got "0" blocks in your quant)"

"But at least for my purposes, it seems like Qwen3.5-122B-A10B is still on top for inference speed, code quality, and general quality of life."

"They range from Q1 to BF16. Grab them while they're still hot over at https://huggingface.co/unsloth/MiniMax-M2.7-GGUF"

"Turns out I was able to extract the model weights, but now I need help from the community, especially people who know C++ to help reverse engineer the MTP from the compiled TFLite graph files, back into a usable Pytorch nn.Module."

"Gemma 4: Byte for byte, the most capable open models"

"With the merging of https://github.com/ggml-org/llama.cpp/pull/21534, all of the fixes to known Gemma 4 issues in Llama.cpp have been resolved."

"Quantizing the KV cache to Q4 allowed a 32K context to fit within 8GB — the only thing broken was the math."

"Usually FP16 models require over 16GB, but Bonsai achieves a compression rate of over 14 times with a file size of merely 1.15GB."

"Simplifying parameters eliminates the need for complex multiplication processing and drastically reduces VRAM consumption, making 'inference at sufficient speeds on ordinary CPUs or smartphones possible without a GPU costing hundreds of thousands of yen.'"

"OneCompression is a post-processing quantization (PTQ) framework developed by FKKimura (Mr. Kimura) of Fujitsu Laboratories."

"80% of the benefit of TQ with almost no downsides. Q8 is now ≈ F16"

"tl;dr better quantization -> smarter models"

"That is a gap of only +0.0139 PPL, about 0.19%, on the full wiki.test.raw pass (580 chunks, c=512)."

"TurboQuant rejects that mentality. It treats quantization as a first-class algorithmic problem tied to the geometry of high-dimensional vectors."

"This article introduces the procedure to apply INT8 quantization to Llama-3.1-8B Instruct, reducing Neuron device memory by approximately 24% (for MaxLen=8192) and increasing inference speed by approximately 24%."

"I just quantized and uploaded some Qwen3.5 abliterated models for Comfyui, including a workflow."

"KV cache quantization is a technology that compresses the Attention's Key/Value tensors, which are dynamically generated during Inference."

"The community consensus is, 'quantized larger model wins every time, just don't go below 4bit'."

"The idea is to build a complete pipeline for digit recognition that can run on embedded systems."

"I think this could be great for existing q8 users."

"It gives you a drop‑in replacement for nn.Linear with near‑optimal distortion."

"It gives you a drop‑in replacement for nn.Linear with near‑optimal distortion."

"This article summarizes the conversion procedure and operation verification in LM Studio."

"Google Research described the technology as a novel way to shrink AI’s working memory without impacting performance."

"We introduce a set of advanced theoretically grounded quantization algorithms that enable massive compression for large language models and vector search engines."

"Today, we introduce TurboQuant (to be presented at ICLR 2026), a compression"

"The EU AI Act will be in full effect from August 2026 for high-risk AI, making compliance essential for edge device implementation."

"oQ (oMLX universal dynamic quantization) A new quantization method oQ for MLX has been released. oQ creates mlx‑lm safetensors compatible models that run on Apple Silicon and oMLX, mlx‑lm, and any other inference server."