Definition
Reference Sliding Window Attention (R-SWA) is an attention mechanism where each generated token attends to all reference tokens (visual encodings and prompts) while limiting output self-attention to the preceding n tokens (default 128). This caps KV cache size at a constant during long-sequence decoding.
Key Properties
- Cache size: (C_{R\text{-}SWA}(T) = L_m + \min(n, T) \leq L_m + n) — bounded constant vs. linear growth in standard attention
- Reference tokens are never subjected to recurrent state updates, avoiding image degradation in long OCR runs
- General-purpose beyond OCR: applicable to ASR, translation, and other parsing tasks per Baidu technical report
Unlimited OCR Application
baidu’s unlimited-ocr replaces all decoder attention layers with R-SWA on a DeepSeek OCR baseline. Enables dozens of pages in one 32K forward pass with flat memory and throughput (2026-07-06-baidu-unlimited-ocr-arxiv).