β‘ IVERI-Hyperion: Sub-Quadratic Foundation Model (390M)
Next-Generation Entropy-Gated Selective State-Space Architecture
IVERI-Hyperion is an ultra-efficient state-space foundation model designed to eliminate the quadratic memory bottleneck of traditional Transformers.
π¬ Architectural Innovations
- Sub-Quadratic Sequence Scaling: Replaces quadratic self-attention with O(1) state-space memory retention.
- Entropy-Gated Selective State Units: Selectively filters token representations based on local information density.
- Multi-Domain Pretraining: Trained on the canonical IVERI-ULTRA v1.0 mixture (FineWeb-Edu Score >= 3.0 and FineMath-4+).
π Benchmark Metrics
- Validation Cross-Entropy Loss: 6.8241
- Validation Perplexity: 919.73 (evaluated on unseen educational text)
- TPU Throughput: ~6,324 tokens/sec on Google Cloud TPU v5e
π Quickstart
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('NishantIveri/IVERI-Hyperion-SLM')
π Official Citation
@article{iveri2026hyperion,
title={IVERI-HYPERION: An Entropy-Driven, Byte-Level State-Space Architecture with Associative Memory for Sub-Quadratic Sequence Modeling},
author={Datta, Nishant and Bhoyar, Ishwari and Nandeshwar, Gunjan and Khan, Nazish},
journal={International Journal for Research in Applied Science and Engineering Technology (IJRASET)},
volume={14},
year={2026},
note={Paper ID: IJRASET85419}
}
- Downloads last month
- 38