⚑ IVERI-Hyperion: Sub-Quadratic Foundation Model (390M)

Next-Generation Entropy-Gated Selective State-Space Architecture

Paper Accepted License: Apache 2.0 Hardware

IVERI-Hyperion is an ultra-efficient state-space foundation model designed to eliminate the quadratic memory bottleneck of traditional Transformers.

πŸ”¬ Architectural Innovations

  1. Sub-Quadratic Sequence Scaling: Replaces quadratic self-attention with O(1) state-space memory retention.
  2. Entropy-Gated Selective State Units: Selectively filters token representations based on local information density.
  3. Multi-Domain Pretraining: Trained on the canonical IVERI-ULTRA v1.0 mixture (FineWeb-Edu Score >= 3.0 and FineMath-4+).

πŸ“Š Benchmark Metrics

  • Validation Cross-Entropy Loss: 6.8241
  • Validation Perplexity: 919.73 (evaluated on unseen educational text)
  • TPU Throughput: ~6,324 tokens/sec on Google Cloud TPU v5e

πŸš€ Quickstart

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('NishantIveri/IVERI-Hyperion-SLM')

πŸ“œ Official Citation

@article{iveri2026hyperion,
  title={IVERI-HYPERION: An Entropy-Driven, Byte-Level State-Space Architecture with Associative Memory for Sub-Quadratic Sequence Modeling},
  author={Datta, Nishant and Bhoyar, Ishwari and Nandeshwar, Gunjan and Khan, Nazish},
  journal={International Journal for Research in Applied Science and Engineering Technology (IJRASET)},
  volume={14},
  year={2026},
  note={Paper ID: IJRASET85419}
}
Downloads last month
38
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support