Authors
M D Tausif Mallick, Saptarshi Banerjee, Ebrahim Abdulla Mattar, Pronaya Bhattacharya, Sujan Pal, Avnish Kumar, Jon Turdiev, Himadri Nath Saha, Amlan Chakrabarti
Published in
Scientific reports. Jul 08, 2026. Epub Jul 08, 2026.
Abstract
Quadratic attention enhances interaction capacity in Transformer models but leads to rapid growth in computational demands as attention rank increases. This paper presents a bounded-rank quadratic attention mechanism where fixed-dimensional feature encodings determine the interaction space and enforce a strict upper bound on attention rank. A Fourier-domain formulation offers a spectral interpretation of the quadratic kernel via unitary transformation while maintaining exact attention computation. The proposed approach achieves bounded-rank attention with computational complexity that scales linearly with sequence length when the encoding dimension remains constant. Experimental validation on a real-world Tea pest image dataset yields 84.5% classification accuracy, surpassing the 82.1% achieved by a standard Vision Transformer under equivalent experimental conditions. Attention processing time per image declines from 14.8 ms to 5.6 ms. Peak memory usage declines from 820 MB to 430 MB. Although memory bandwidth and kernel launch overhead restrict the measured speedup to [Formula: see text], the accuracy loss under additive noise reaches 2.9% compared to 3.1% for the baseline, demonstrating comparable robustness. These findings indicate that explicit rank control in attention mechanisms can be realized through representational design, offering an efficient bounded-rank alternative to conventional full-rank attention.
PMID:
42420357
Bibliographic data and abstract were imported from PubMed on 09 Jul 2026.
Read full publication at:
Please sign in
to see all details.
Advertisement
Stats
- Recommendations n/a n/a positive of 0 vote(s)
- Views 4
- Comments 0