01 / Abstract
Low-bit quantization breaks the query–key interaction. CASK-Q calibrates for it directly.
Low-bit quantization perturbs the interaction between query and key projections, not just each projection separately. We introduce CASK-Q, a calibration objective for jointly optimizing scalar-quantized query/key weights using randomized sketches of row-centered causal attention-logit error. An exact prefix-sum identity evaluates the sketch in O(Ndr) arithmetic for sequence length N, head dimension d, and sketch width r, without constructing an N×N logit matrix. We characterize the estimator's mean and variance, prove a sharp global attention-divergence bound, and derive an exact O(Nd²) prefix-moment comparator. In a controlled 30-seed synthetic-attention experiment, CASK-Q reduces mean held-out attention-output normalized mean-squared error by 25.20% at nominal 2-bit and 28.13% at nominal 3-bit relative to independent query/key reconstruction, with identical serialized weight sizes and equal update counts.
02 / Results
Same bits. Same bytes. Less attention error.
Held-out attention-output NMSE over 30 synthetic teacher/data seeds (lower is better). CASK-Q matches the accuracy of dense joint calibration while never materializing the full attention-logit matrix.
Nominal 2-bit
Nominal 3-bit
| Track | Bits | Independent | CASK-Q | Reduction | Wins |
|---|---|---|---|---|---|
| Equal-update, 30 seeds | 2 | 0.08600 | 0.06432 | −25.20% | 30/30 |
| Equal-update, 30 seeds | 3 | 0.02601 | 0.01870 | −28.13% | 30/30 |
| Equal-time (2 s budget), 10 seeds | 2 | 0.07616 | 0.05474 | −28.12% | 10/10 |
| Equal-time (2 s budget), 10 seeds | 3 | 0.02411 | 0.01761 | −26.96% | 10/10 |
| Post-export, permuted covariance | 2 | 0.18529 | 0.15389 | −16.95% | 28/30 |
| Post-export, heavy-tailed t₃ | 3 | 0.05545 | 0.04639 | −16.34% | 22/30 |
| Calibration loss, fwd + bwd (CPU, d=64, r=8) | N=128 | N=512 | N=2048 | N=4096 |
|---|---|---|---|---|
| Dense logits | 0.64 ms | 4.90 ms | 112.80 ms | 608.41 ms |
| CASK sketch | 1.66 ms | 5.76 ms | 23.29 ms | 48.08 ms |
03 / Method
How CASK-Q works.
Measure the error softmax can see.
The target is the causal attention-logit error with each row's mean removed — exactly the component softmax is invariant to is dropped.
Probe it with shared random signs.
Teacher and student share fresh Rademacher probes. A prefix-sum identity computes the sketch in O(Ndr) without ever building an N×N matrix.
Train Q and K together.
Gradients flow into both quantizers at once, so a perturbation in one projection can compensate for the other. Export stays plain scalar codes + FP16 scales.
zi⊤ = qi⊤(Ti − viui⊤/i) / √d, Ti = Σj≤i kjRj⊤, L̂R = Σi ‖ẑi − zi‖² · 𝔼[L̂R] = ‖D‖F², KL(p‖q) ≤ ¼‖δc‖²
The paper proves the estimator is unbiased with variance shrinking as 1/r, gives a sharp bound linking the centered error to attention KL divergence, and supplies an exact deterministic prefix-moment comparator. Twenty-six numerical tests check value and gradient agreement against the dense objective.
The implementation, experiment scripts, results and exported weights are open source: github.com/the-decentralized-compute-company/quantization.
04 / Scope
What the paper does — and doesn't — claim.
- 01Experiments use controlled synthetic single-head attention teachers, not pretrained language models.
- 02The baseline is independent Q/K reconstruction on the same quantizer — a controlled ablation, not GPTQ, GSQ or another published system.
- 03Dense joint calibration reaches similar accuracy; CASK-Q's edge is avoiding the N×N matrix at long context.
- 04Downstream perplexity, full-model compression and inference kernels are the next evaluation step.
05 / Cite
Cite this work.
@techreport{tdcc2026caskq,
title = {CASK-Q: Centered Causal Attention Sketches for Joint Query--Key Quantization},
author = {{The Decentralized Compute Company}},
institution = {The Decentralized Compute Company},
year = {2026},
month = oct,
url = {https://decentralizedcompute.company/cask-q.html},
note = {Code: https://github.com/the-decentralized-compute-company/quantization}
}