ALHR-A tree based sparse attention system that retains long context accuracy

By vdev-ctrl · 2026-10-09 · 1 points · 0 comments

ALHR - Adaptive Learnable Hierarchical Routing, uses static binary trees and learnable functions to minimize the amount of keys to be read. It does use a dense teacher while phase 1 of training however. MQAR TEST AT 1024 TOKENS - Average keys read per query by dense - 512 Keys A…

Open the full discussion on BetterNews