Upload 17 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+banner_fabric.png filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,43 @@
+Fabric AI Open License v1.0
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by this document.
+
+"Licensor" shall mean Fabric AI.
+
+"You" (or "Your") shall mean an individual or legal entity exercising permissions granted by this License.
+
+"Work" shall mean the work of authorship, whether in source or object form, made available under this License.
+
+"Derivative Works" shall mean any work that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship.
+
+2. Grant of Copyright License. Subject to the terms and conditions of this License, Licensor hereby grants You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and Derivative Works in source or object form.
+
+3. Grant of Patent License. Licensor hereby grants You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License shall terminate as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in source or object form, provided that You meet the following conditions:
+
+   (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
+
+   (b) You must cause any modified files to carry prominent notices stating that You changed them; and
+
+   (c) You must retain, in the source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the source form of the Work; and
+
+   (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works You distribute must include a readable copy of the attribution notices contained within such NOTICE file.
+
+5. Attribution. Any use, reproduction, or distribution of the Work or Derivative Works must include the following attribution in a reasonable manner:
+
+   "Based on Fabric 1.5 by Fabric AI."
+
+   You may add Your own attribution to Your modifications, but the above attribution to Fabric AI must remain intact.
+
+6. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including without limitation any warranties of merchantability, fitness for a particular purpose, or non-infringement.
+
+7. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, shall Licensor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages arising as a result of this License or out of the use or inability to use the Work, even if Licensor has been advised of the possibility of such damages.
+
+8. Termination. This License will terminate automatically and immediately if You fail to comply with any of its terms and conditions. Upon termination, You must cease all use of the Work and any Derivative Works.
+
+END OF TERMS AND CONDITIONS

README.md

@@ -0,0 +1,183 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- fabric
+- efficient
+- 0.7b
+- causal-lm
+- chunked-memory
+base_model: FabricAI/Fabric1.5-0.7B-Base
+datasets:
+- HuggingFaceTB/smoltalk2
+- open-r1/Mixture-of-Thoughts
+- nvidia/Nemotron-Post-Training-Dataset-v1
+extra:
+  model_name: Fabric1.5-0.7B-Instruct
+  model_creator: Fabric AI
+  developer: Fabric AI
+  model_type: fabric
+  context_length: 32768
+---
+
+![Fabric 1.5 Banner](banner_fabric.png)
+
+# Fabric 1.5 — 0.7B Instruct
+
+**Fabric 1.5** is a lightweight, high-capability language model developed by **Fabric AI**. It introduces a novel chunked memory architecture and demonstrates that a 0.7B parameter model can achieve meaningful performance on reasoning, knowledge, and instruction-following tasks.
+
+## Model Details
+
+- **Type:** Causal Language Model
+- **Parameters:** 742M (0.7B)
+- **Hidden Dimension:** 1,536
+- **Vocabulary Size:** 65,536
+- **Layers:** 24 (16 LocalBlock + 8 FabricMemoryBlock)
+- **Attention:** GQA (24 query heads, 6 KV heads), RoPE (theta=1M)
+- **Context Length:** 32,768 tokens native
+- **Precision:** FP16
+- **License:** Fabric AI Open License v1.0
+
+## Architecture
+
+Fabric 1.5 employs a **Chunked Fabric Memory** architecture — every 3rd block splits attention into two parallel branches:
+- **Local causal window:** exact attention over the last 2,048 tokens
+- **Chunked memory:** learned summaries (4 per 512-token chunk) of earlier completed chunks
+
+A per-token scalar gate blends the outputs, giving the model 256 summary vectors of long-range memory at full 32K context without quadratic memory growth.
+
+## Benchmark Results
+
+Evaluated on an **NVIDIA A100-SXM4-40GB** using `lm-eval` harness with FP16 weights.
+
+| Benchmark | Accuracy | Type |
+|---|---|---|
+| **ARC Easy** (0-shot) | **54.97%** | Loglikelihood |
+| **ARC Challenge** (0-shot) | **26.88%** | Loglikelihood |
+| **HellaSwag** (0-shot) | **35.00%** | Loglikelihood |
+| **MMLU** (0-shot) | **28.96%** | Loglikelihood |
+| **C-Eval** (0-shot) | **27.12%** | Loglikelihood |
+
+### Comparison vs Qwen3.5-0.8B
+
+| Benchmark | Fabric 1.5 | Qwen3.5-0.8B | Delta |
+|---|---|---|---|
+| **MMLU** (0-shot) | **28.96%** | 29.7% | -0.74% |
+| **C-Eval** (0-shot) | **27.12%** | 46.4% | -19.28% |
+
+Fabric 1.5 performs within **0.74% of Qwen3.5-0.8B** on MMLU despite being 12% smaller (0.7B vs 0.8B) and having no vision encoder.
+
+## Quickstart
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "FabricAI/Fabric1.5-0.7B-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+
+messages = [{"role": "user", "content": "What is gravity?"}]
+text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(text, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    output_ids = model.generate(
+        **inputs,
+        max_new_tokens=256,
+        do_sample=True,
+        temperature=0.65,
+        top_p=0.9,
+        repetition_penalty=1.05,
+        use_cache=True,
+    )
+
+response = tokenizer.decode(output_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()
+print(response)
+```
+
+## Chat Template
+
+```
+<|bos|><|system|>
+You are Fabric, a large language model developed by Fabric AI.<|end|>
+<|user|>
+Hello!<|end|>
+<|assistant|>
+Hi! How can I help you?<|end|>
+```
+
+Special tokens: `<bos>`, `<pad>`, `<|end|>`, `<|system|>`, `<|user|>`, `<|assistant|>`
+
+## Sampling Parameters
+
+| Mode | Temperature | Top-p | Top-k | Repetition Penalty |
+|------|------------|-------|-------|-------------------|
+| General chat | 0.65 | 0.90 | 50 | 1.05 |
+| Creative writing | 0.80 | 0.95 | 50 | 1.00 |
+| Factual/knowledge | 0.40 | 0.85 | 30 | 1.05 |
+| Code generation | 0.30 | 0.90 | 40 | 1.02 |
+
+## Training Data & Process
+
+**Pre-training (18B tokens, 1× NVIDIA DGX H100 8-GPU):**
+
+| Source | Weight | Description |
+|---|---|---|
+| FineWeb-Edu | 60% | High-quality educational web text |
+| DCLM-Baseline | 25% | Deduplicated Common Crawl subset |
+| OpenWebMath | 10% | Math-focused web text |
+| Cosmopedia-v2 | 5% | Instruction-tuned educational data |
+
+- **Optimizer:** AdamW (β1=0.9, β2=0.95, ε=1e-8)
+- **Learning rate:** 3e-4 with cosine decay to 3e-5, 180M token warmup
+- **Weight decay:** 0.1
+- **Gradient clipping:** 1.0
+- **Precision:** FP16
+- **Distributed:** NCCL, 8 ranks, 4 sequences per GPU
+
+**Post-training (SFT):**
+1. Identity tuning — system prompt alignment and persona training
+2. General SFT — diverse instruction-following datasets
+
+## Hardware Notes
+
+Fabric 1.5 was trained on a **single NVIDIA DGX H100** node with 8× H100 80GB GPUs. The model runs efficiently on:
+
+| Hardware | Precision | Batch Size | Context |
+|---|---|---|---|
+| 1× NVIDIA H100 80GB | FP16 | 8 | 32K |
+| 1× NVIDIA A100 40GB | FP16 | 4 | 32K |
+| Apple Silicon (MPS) | FP16 | 1 | 32K |
+| CPU | FP32 | 1 | 32K |
+
+The chunked memory architecture ensures memory scales linearly with context, not quadratically.
+
+## License
+
+**Fabric AI Open License v1.0** — a permissive, attribution-required license. See [LICENSE](LICENSE) for full terms.
+
+Key requirements:
+- You may use, modify, and distribute the model freely
+- You must **retain attribution** to Fabric AI when redistributing
+- Modified files must carry notices of your changes
+- No warranty or liability provisions
+
+## Citation
+
+```bibtex
+@misc{fabric1.5,
+    title  = {{Fabric 1.5}: A Causal Language Model with Chunked Fabric Memory},
+    author = {Fabric AI},
+    year   = 2026,
+}
+```

chat_template.jinja

@@ -0,0 +1,10 @@
+{% set default_system_prompt = "You are Fabric, a large language model developed by Fabric AI." %}{% set ns = namespace(has_system=false) %}{% for message in messages %}{% if message['role'] == 'system' %}{% set ns.has_system = true %}{% endif %}{% endfor %}{% if not ns.has_system %}<|system|>
+{{ default_system_prompt }}<|end|>
+{% endif %}{% for message in messages %}{% if message['role'] == 'system' %}<|system|>
+{{ message['content'] }}<|end|>
+{% elif message['role'] == 'user' %}<|user|>
+{{ message['content'] }}<|end|>
+{% elif message['role'] == 'assistant' %}<|assistant|>
+{{ message['content'] }}<|end|>
+{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>
+{% endif %}

config.json

@@ -0,0 +1,52 @@
+{
+  "activation_checkpointing": true,
+  "architecture": "fabric",
+  "architectures": [
+    "FabricForCausalLM"
+  ],
+  "attention_backend": "auto",
+  "attention_chunk_size": 1024,
+  "auto_map": {
+    "AutoConfig": "configuration_fabric.FabricConfig",
+    "AutoModelForCausalLM": "modeling_fabric.FabricForCausalLM"
+  },
+  "bos_token_id": 2,
+  "checkpoint_dtype": "float16",
+  "checkpoint_note": "Model-only Fabric complete checkpoint; optimizer state is excluded.",
+  "checkpoint_sha256": "a8d12a687fe5eaa05b601ab2803ca2c14b9cc2c14d754110ca4569b061af2bf1",
+  "checkpoint_size_gib": 1.383114,
+  "chunked_cross_entropy": true,
+  "continued_pretraining_tokens": 5000134656,
+  "cumulative_pretraining_tokens": 23000514560,
+  "eos_token_id": 7,
+  "export_kind": "model_only_inference_fp16",
+  "head_dim": 64,
+  "hidden_size": 1536,
+  "intermediate_size": 4096,
+  "license": "fabric-ai-open-1.0",
+  "local_attention_window": 2048,
+  "loss_chunk_size": 1024,
+  "max_position_embeddings": 32768,
+  "memory_chunk_size": 512,
+  "model_name": "Fabric1.5-0.7B-Instruct",
+  "model_type": "fabric",
+  "num_attention_heads": 24,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 6,
+  "num_kv_heads": 6,
+  "num_layers": 24,
+  "num_parameters": 742528520,
+  "num_query_heads": 24,
+  "original_pretraining_tokens": 18000379904,
+  "pad_token_id": 0,
+  "posttraining_supervised_tokens": 318879711,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 1000000.0,
+  "sequence_length": 32768,
+  "summaries_per_chunk": 4,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float16",
+  "transformers_version": "custom",
+  "use_cache": false,
+  "vocab_size": 65536
+}

configuration_fabric.py

@@ -0,0 +1,89 @@
+from __future__ import annotations
+
+from transformers import PretrainedConfig
+
+
+class FabricConfig(PretrainedConfig):
+    model_type = "fabric"
+
+    def __init__(
+        self,
+        model_name: str = "Fabric 1.5",
+        architecture: str = "fabric",
+        vocab_size: int = 65536,
+        hidden_size: int = 1536,
+        intermediate_size: int = 4096,
+        num_layers: int = 24,
+        num_query_heads: int = 24,
+        num_kv_heads: int = 6,
+        head_dim: int = 64,
+        sequence_length: int = 32768,
+        local_attention_window: int = 2048,
+        memory_chunk_size: int = 512,
+        summaries_per_chunk: int = 4,
+        rope_theta: float = 1000000.0,
+        rms_norm_eps: float = 1e-6,
+        tie_word_embeddings: bool = True,
+        attention_backend: str = "auto",
+        attention_chunk_size: int = 1024,
+        activation_checkpointing: bool = False,
+        chunked_cross_entropy: bool = True,
+        loss_chunk_size: int = 1024,
+        **kwargs,
+    ) -> None:
+        kwargs.setdefault("bos_token_id", 2)
+        kwargs.setdefault("eos_token_id", 3)
+        kwargs.setdefault("pad_token_id", 0)
+        kwargs.setdefault("use_cache", False)
+        kwargs.setdefault("tie_word_embeddings", tie_word_embeddings)
+        super().__init__(**kwargs)
+        self.model_name = model_name
+        self.architecture = architecture
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_layers = num_layers
+        self.num_hidden_layers = num_layers
+        self.num_query_heads = num_query_heads
+        self.num_attention_heads = num_query_heads
+        self.num_kv_heads = num_kv_heads
+        self.num_key_value_heads = num_kv_heads
+        self.head_dim = head_dim
+        self.sequence_length = sequence_length
+        self.max_position_embeddings = sequence_length
+        self.local_attention_window = local_attention_window
+        self.memory_chunk_size = memory_chunk_size
+        self.summaries_per_chunk = summaries_per_chunk
+        self.rope_theta = rope_theta
+        self.rms_norm_eps = rms_norm_eps
+        self.tie_word_embeddings = tie_word_embeddings
+        self.attention_backend = attention_backend
+        self.attention_chunk_size = attention_chunk_size
+        self.activation_checkpointing = activation_checkpointing
+        self.chunked_cross_entropy = chunked_cross_entropy
+        self.loss_chunk_size = loss_chunk_size
+
+    def to_fabric_model_config_dict(self) -> dict:
+        return {
+            "model_name": self.model_name,
+            "architecture": self.architecture,
+            "vocab_size": self.vocab_size,
+            "hidden_size": self.hidden_size,
+            "intermediate_size": self.intermediate_size,
+            "num_layers": self.num_layers,
+            "num_query_heads": self.num_query_heads,
+            "num_kv_heads": self.num_kv_heads,
+            "head_dim": self.head_dim,
+            "sequence_length": self.sequence_length,
+            "local_attention_window": self.local_attention_window,
+            "memory_chunk_size": self.memory_chunk_size,
+            "summaries_per_chunk": self.summaries_per_chunk,
+            "rope_theta": self.rope_theta,
+            "rms_norm_eps": self.rms_norm_eps,
+            "tie_word_embeddings": self.tie_word_embeddings,
+            "attention_backend": self.attention_backend,
+            "attention_chunk_size": self.attention_chunk_size,
+            "activation_checkpointing": self.activation_checkpointing,
+            "chunked_cross_entropy": self.chunked_cross_entropy,
+            "loss_chunk_size": self.loss_chunk_size,
+        }

fabric_config.json

@@ -0,0 +1,35 @@
+{
+  "activation_checkpointing": true,
+  "architecture": "fabric",
+  "attention_backend": "auto",
+  "attention_chunk_size": 1024,
+  "checkpoint_dtype": "float16",
+  "checkpoint_note": "Model-only Fabric complete checkpoint; optimizer state is excluded.",
+  "checkpoint_sha256": "a8d12a687fe5eaa05b601ab2803ca2c14b9cc2c14d754110ca4569b061af2bf1",
+  "checkpoint_size_gib": 1.383114,
+  "chunked_cross_entropy": true,
+  "continued_pretraining_tokens": 5000134656,
+  "cumulative_pretraining_tokens": 23000514560,
+  "export_kind": "model_only_inference_fp16",
+  "head_dim": 64,
+  "hidden_size": 1536,
+  "intermediate_size": 4096,
+  "license": "fabric-ai-open-1.0",
+  "local_attention_window": 2048,
+  "loss_chunk_size": 1024,
+  "memory_chunk_size": 512,
+  "model_name": "Fabric1.5-0.7B-Instruct",
+  "num_kv_heads": 6,
+  "num_layers": 24,
+  "num_parameters": 742528520,
+  "num_query_heads": 24,
+  "original_pretraining_tokens": 18000379904,
+  "posttraining_supervised_tokens": 318879711,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 1000000.0,
+  "sequence_length": 32768,
+  "summaries_per_chunk": 4,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float16",
+  "vocab_size": 65536
+}

fabric_runtime.py

@@ -0,0 +1,339 @@
+from __future__ import annotations
+
+import base64
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import torch
+from safetensors import safe_open
+from torch import nn
+from torch.nn import functional as F
+
+
+@dataclass
+class ModelConfig:
+    model_name: str = "Fabric 1.5"
+    architecture: str = "fabric"
+    vocab_size: int = 65536
+    hidden_size: int = 1536
+    intermediate_size: int = 4096
+    num_layers: int = 24
+    num_query_heads: int = 24
+    num_kv_heads: int = 6
+    head_dim: int = 64
+    sequence_length: int = 32768
+    local_attention_window: int = 2048
+    memory_chunk_size: int = 512
+    summaries_per_chunk: int = 4
+    rope_theta: float = 1000000.0
+    rms_norm_eps: float = 1e-6
+    tie_word_embeddings: bool = True
+    attention_backend: str = "auto"
+    attention_chunk_size: int = 1024
+    activation_checkpointing: bool = False
+    chunked_cross_entropy: bool = True
+    loss_chunk_size: int = 1024
+
+
+def _decode_structure(value: Any, tensors: dict[str, torch.Tensor]) -> Any:
+    if not isinstance(value, dict) or "__kind__" not in value:
+        return value
+    kind = value["__kind__"]
+    if kind == "tensor":
+        return tensors[value["key"]]
+    if kind == "dict":
+        return {
+            _decode_structure(key, tensors): _decode_structure(item, tensors)
+            for key, item in value["items"]
+        }
+    if kind == "tuple":
+        return tuple(_decode_structure(item, tensors) for item in value["items"])
+    if kind == "list":
+        return [_decode_structure(item, tensors) for item in value["items"]]
+    if kind == "ndarray":
+        return np.asarray(value["items"], dtype=np.dtype(value["dtype"])).reshape(value["shape"])
+    if kind == "path":
+        return Path(value["value"])
+    if kind == "bytes":
+        return base64.b64decode(value["value"])
+    raise ValueError(f"unknown checkpoint structure kind: {kind}")
+
+
+def load_checkpoint(path: str | Path, map_location: str | torch.device = "cpu") -> dict[str, Any]:
+    with safe_open(path, framework="pt", device=str(map_location)) as handle:
+        metadata = handle.metadata()
+        if metadata.get("format") != "fabric_complete_checkpoint":
+            raise ValueError("file is not a Fabric complete checkpoint")
+        tensors = {key: handle.get_tensor(key) for key in handle.keys()}
+        structure = json.loads(metadata["structure"])
+    state = _decode_structure(structure, tensors)
+    if not isinstance(state, dict):
+        raise ValueError("checkpoint root must be a dictionary")
+    return state
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, hidden_size: int, eps: float = 1e-6) -> None:
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.eps = eps
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        dtype = x.dtype
+        variance = x.float().pow(2).mean(dim=-1, keepdim=True)
+        return (x.float() * torch.rsqrt(variance + self.eps)).to(dtype) * self.weight
+
+
+class SwiGLU(nn.Module):
+    def __init__(self, hidden_size: int, intermediate_size: int) -> None:
+        super().__init__()
+        self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
+        self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
+        self.down_proj = nn.Linear(intermediate_size, hidden_size, bias=False)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
+
+
+def rotate_half(x: torch.Tensor) -> torch.Tensor:
+    x1, x2 = x.chunk(2, dim=-1)
+    return torch.cat((-x2, x1), dim=-1)
+
+
+class RotaryEmbedding(nn.Module):
+    def __init__(self, head_dim: int, theta: float = 10000.0) -> None:
+        super().__init__()
+        inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+    def forward(self, q: torch.Tensor, k: torch.Tensor, position_ids: torch.Tensor):
+        angles = position_ids.float().unsqueeze(-1) * self.inv_freq.float()
+        emb = torch.cat((angles, angles), dim=-1)
+        cos = emb.cos().to(q.dtype).unsqueeze(1)
+        sin = emb.sin().to(q.dtype).unsqueeze(1)
+        return q * cos + rotate_half(q) * sin, k * cos + rotate_half(k) * sin
+
+
+def build_local_causal_mask(query_length: int, key_length: int, window: int, device, query_offset: int = 0):
+    query_positions = torch.arange(query_offset, query_offset + query_length, device=device)
+    key_positions = torch.arange(key_length, device=device)
+    return (key_positions[None, :] <= query_positions[:, None]) & (
+        key_positions[None, :] > query_positions[:, None] - window
+    )
+
+
+def repeat_kv(x: torch.Tensor, groups: int) -> torch.Tensor:
+    if groups == 1:
+        return x
+    batch, kv_heads, length, dim = x.shape
+    return x[:, :, None, :, :].expand(batch, kv_heads, groups, length, dim).reshape(
+        batch, kv_heads * groups, length, dim
+    )
+
+
+def reference_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, allowed_mask: torch.Tensor):
+    scores = torch.matmul(q.float(), k.float().transpose(-1, -2)) / math.sqrt(q.shape[-1])
+    scores = scores.masked_fill(~allowed_mask, torch.finfo(scores.dtype).min)
+    probabilities = torch.softmax(scores, dim=-1)
+    probabilities = torch.where(allowed_mask.any(dim=-1, keepdim=True), probabilities, 0.0)
+    return torch.matmul(probabilities.to(v.dtype), v)
+
+
+class GQAAttention(nn.Module):
+    def __init__(self, config: ModelConfig, window: int | None = None) -> None:
+        super().__init__()
+        self.num_query_heads = config.num_query_heads
+        self.num_kv_heads = config.num_kv_heads
+        self.head_dim = config.head_dim
+        self.groups = config.num_query_heads // config.num_kv_heads
+        self.window = window or config.sequence_length
+        self.backend = "sdpa" if config.attention_backend == "flash_attn" else config.attention_backend
+        self.attention_chunk_size = config.attention_chunk_size
+        self.q_proj = nn.Linear(config.hidden_size, config.num_query_heads * config.head_dim, bias=False)
+        self.k_proj = nn.Linear(config.hidden_size, config.num_kv_heads * config.head_dim, bias=False)
+        self.v_proj = nn.Linear(config.hidden_size, config.num_kv_heads * config.head_dim, bias=False)
+        self.o_proj = nn.Linear(config.num_query_heads * config.head_dim, config.hidden_size, bias=False)
+        self.rope = RotaryEmbedding(config.head_dim, config.rope_theta)
+
+    def forward(self, x: torch.Tensor, position_ids: torch.Tensor | None = None) -> torch.Tensor:
+        batch, length, _ = x.shape
+        if position_ids is None:
+            position_ids = torch.arange(length, device=x.device).expand(batch, -1)
+        q = self.q_proj(x).view(batch, length, self.num_query_heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(batch, length, self.num_kv_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(batch, length, self.num_kv_heads, self.head_dim).transpose(1, 2)
+        q, k = self.rope(q, k, position_ids)
+        k = repeat_kv(k, self.groups)
+        v = repeat_kv(v, self.groups)
+        use_sdpa = self.backend in {"auto", "sdpa"} and hasattr(F, "scaled_dot_product_attention")
+        if use_sdpa:
+            outputs = []
+            for start in range(0, length, self.attention_chunk_size):
+                end = min(start + self.attention_chunk_size, length)
+                key_start = max(0, start - self.window + 1)
+                key_end = end
+                allowed = build_local_causal_mask(
+                    end - start,
+                    key_end - key_start,
+                    self.window,
+                    x.device,
+                    query_offset=start - key_start,
+                )[None, None]
+                outputs.append(
+                    F.scaled_dot_product_attention(
+                        q[:, :, start:end],
+                        k[:, :, key_start:key_end],
+                        v[:, :, key_start:key_end],
+                        attn_mask=allowed,
+                        dropout_p=0.0,
+                    )
+                )
+            output = torch.cat(outputs, dim=2)
+        else:
+            allowed = build_local_causal_mask(length, length, self.window, x.device)[None, None]
+            output = reference_attention(q, k, v, allowed)
+        output = output.transpose(1, 2).contiguous().view(batch, length, -1)
+        return self.o_proj(output)
+
+
+def build_completed_chunk_mask(sequence_length: int, num_chunks: int, summaries_per_chunk: int, chunk_size: int, device):
+    query_chunk = torch.arange(sequence_length, device=device) // chunk_size
+    summary_chunk = torch.arange(num_chunks, device=device).repeat_interleave(summaries_per_chunk)
+    return summary_chunk[None, :] < query_chunk[:, None]
+
+
+class ChunkSummarizer(nn.Module):
+    def __init__(self, config: ModelConfig) -> None:
+        super().__init__()
+        self.chunk_size = config.memory_chunk_size
+        self.num_summaries = config.summaries_per_chunk
+        self.hidden_size = config.hidden_size
+        self.queries = nn.Parameter(torch.empty(self.num_summaries, self.hidden_size))
+        nn.init.normal_(self.queries, std=0.02)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        batch, length, hidden = x.shape
+        num_chunks = (length + self.chunk_size - 1) // self.chunk_size
+        padded_length = num_chunks * self.chunk_size
+        if padded_length != length:
+            x = torch.cat((x, x.new_zeros(batch, padded_length - length, hidden)), dim=1)
+        chunks = x.view(batch, num_chunks, self.chunk_size, hidden)
+        scores = torch.einsum("mh,bnch->bnmc", self.queries.float(), chunks.float()) / math.sqrt(hidden)
+        if padded_length != length:
+            valid = torch.arange(padded_length, device=x.device).view(num_chunks, self.chunk_size) < length
+            scores = scores.masked_fill(~valid[None, :, None, :], torch.finfo(scores.dtype).min)
+        weights = torch.softmax(scores, dim=-1).to(chunks.dtype)
+        return torch.einsum("bnmc,bnch->bnmh", weights, chunks)
+
+
+class MemoryAttention(nn.Module):
+    def __init__(self, config: ModelConfig) -> None:
+        super().__init__()
+        self.num_query_heads = config.num_query_heads
+        self.num_kv_heads = config.num_kv_heads
+        self.head_dim = config.head_dim
+        self.groups = config.num_query_heads // config.num_kv_heads
+        self.chunk_size = config.memory_chunk_size
+        self.num_summaries = config.summaries_per_chunk
+        self.q_proj = nn.Linear(config.hidden_size, config.num_query_heads * config.head_dim, bias=False)
+        self.k_proj = nn.Linear(config.hidden_size, config.num_kv_heads * config.head_dim, bias=False)
+        self.v_proj = nn.Linear(config.hidden_size, config.num_kv_heads * config.head_dim, bias=False)
+        self.o_proj = nn.Linear(config.num_query_heads * config.head_dim, config.hidden_size, bias=False)
+
+    def forward(self, x: torch.Tensor, summaries: torch.Tensor) -> torch.Tensor:
+        batch, length, _ = x.shape
+        num_chunks = summaries.shape[1]
+        flat = summaries.reshape(batch, num_chunks * self.num_summaries, -1)
+        q = self.q_proj(x).view(batch, length, self.num_query_heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(flat).view(batch, -1, self.num_kv_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(flat).view(batch, -1, self.num_kv_heads, self.head_dim).transpose(1, 2)
+        k, v = repeat_kv(k, self.groups), repeat_kv(v, self.groups)
+        scores = torch.matmul(q.float(), k.float().transpose(-1, -2)) / math.sqrt(self.head_dim)
+        allowed = build_completed_chunk_mask(length, num_chunks, self.num_summaries, self.chunk_size, x.device)[None, None]
+        scores = scores.masked_fill(~allowed, torch.finfo(scores.dtype).min)
+        probabilities = torch.softmax(scores, dim=-1)
+        probabilities = torch.where(allowed.any(dim=-1, keepdim=True), probabilities, 0.0)
+        output = torch.matmul(probabilities.to(v.dtype), v)
+        output = output.transpose(1, 2).contiguous().view(batch, length, -1)
+        return self.o_proj(output)
+
+
+class LocalBlock(nn.Module):
+    def __init__(self, config: ModelConfig, window: int | None = None) -> None:
+        super().__init__()
+        self.attention_norm = RMSNorm(config.hidden_size, config.rms_norm_eps)
+        self.attention = GQAAttention(config, window or config.local_attention_window)
+        self.mlp_norm = RMSNorm(config.hidden_size, config.rms_norm_eps)
+        self.mlp = SwiGLU(config.hidden_size, config.intermediate_size)
+
+    def forward(self, x: torch.Tensor, position_ids: torch.Tensor | None = None) -> torch.Tensor:
+        x = x + self.attention(self.attention_norm(x), position_ids)
+        return x + self.mlp(self.mlp_norm(x))
+
+
+class FabricMemoryBlock(nn.Module):
+    def __init__(self, config: ModelConfig) -> None:
+        super().__init__()
+        self.attention_norm = RMSNorm(config.hidden_size, config.rms_norm_eps)
+        self.local_attention = GQAAttention(config, config.local_attention_window)
+        self.summarizer = ChunkSummarizer(config)
+        self.memory_attention = MemoryAttention(config)
+        self.gate = nn.Linear(config.hidden_size, 1, bias=True)
+        self.mlp_norm = RMSNorm(config.hidden_size, config.rms_norm_eps)
+        self.mlp = SwiGLU(config.hidden_size, config.intermediate_size)
+
+    def forward(self, x: torch.Tensor, position_ids: torch.Tensor | None = None) -> torch.Tensor:
+        normalized = self.attention_norm(x)
+        local = self.local_attention(normalized, position_ids)
+        summaries = self.summarizer(normalized)
+        memory = self.memory_attention(normalized, summaries)
+        gate = torch.sigmoid(self.gate(normalized))
+        x = x + gate * local + (1.0 - gate) * memory
+        return x + self.mlp(self.mlp_norm(x))
+
+
+@dataclass
+class CausalLMOutput:
+    logits: torch.Tensor | None
+    loss: torch.Tensor | None = None
+
+
+class FabricCoreForCausalLM(nn.Module):
+    def __init__(self, config: ModelConfig) -> None:
+        super().__init__()
+        self.config = config
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size)
+        if config.architecture == "fabric":
+            layers = [FabricMemoryBlock(config) if i % 3 == 2 else LocalBlock(config) for i in range(config.num_layers)]
+        else:
+            window = config.sequence_length if config.architecture == "full" else config.local_attention_window
+            layers = [LocalBlock(config, window) for _ in range(config.num_layers)]
+        self.layers = nn.ModuleList(layers)
+        self.norm = RMSNorm(config.hidden_size, config.rms_norm_eps)
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+        if config.tie_word_embeddings:
+            self.lm_head.weight = self.embed_tokens.weight
+
+    def forward(self, input_ids: torch.Tensor, labels: torch.Tensor | None = None) -> CausalLMOutput:
+        if input_ids.ndim != 2:
+            raise ValueError("input_ids must have shape [batch, sequence]")
+        if input_ids.shape[1] > self.config.sequence_length:
+            raise ValueError("input sequence exceeds configured sequence_length")
+        position_ids = torch.arange(input_ids.shape[1], device=input_ids.device).expand(input_ids.shape[0], -1)
+        hidden = self.embed_tokens(input_ids)
+        for layer in self.layers:
+            hidden = layer(hidden, position_ids)
+        normalized = self.norm(hidden)
+        logits = self.lm_head(normalized).float()
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(
+                logits[:, :-1].reshape(-1, logits.shape[-1]),
+                labels[:, 1:].reshape(-1),
+                ignore_index=-100,
+            )
+        return CausalLMOutput(logits=logits, loss=loss)

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "bos_token_id": 2,
+  "do_sample": false,
+  "eos_token_id": 7,
+  "max_new_tokens": 128,
+  "pad_token_id": 0,
+  "temperature": 1.0,
+  "top_p": 1.0,
+  "use_cache": false
+}

inference.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+import sys, time, torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+P = "/Users/tudor/Documents/Fabric AI/Fabric_1.5_NVDA/HF_Final"
+def get_device():
+    if torch.backends.mps.is_available(): return "mps"
+    if torch.cuda.is_available(): return "cuda"
+    return "cpu"
+def load_model():
+    dev = get_device(); dt = torch.float16 if dev != "cpu" else torch.float32
+    print(f"Loading Fabric 1.5 on {dev}...")
+    m = AutoModelForCausalLM.from_pretrained(P, trust_remote_code=True, torch_dtype=dt)
+    m.to(dev); m.eval()
+    tok = AutoTokenizer.from_pretrained(P, trust_remote_code=True)
+    print(f"Loaded! {sum(p.numel() for p in m.parameters()):,} params")
+    return m, tok, dev
+def generate(m, tok, prompt, dev, max_new=512):
+    text = tok.apply_chat_template([{"role":"user","content":prompt}], tokenize=False, add_generation_prompt=True)
+    inp = tok(text, return_tensors="pt").to(dev)
+    with torch.no_grad():
+        out = m.generate(**inp, max_new_tokens=max_new, do_sample=True, temperature=0.65, top_p=0.9, top_k=50, repetition_penalty=1.05, use_cache=True)
+    return tok.decode(out[0,inp["input_ids"].shape[1]:], skip_special_tokens=True).strip()
+m, tok, dev = load_model()
+if len(sys.argv) > 1:
+    q = " ".join(sys.argv[1:]); t0 = time.time(); r = generate(m, tok, q, dev)
+    print(f"\nYou: {q}\nFabric: {r}\n[{time.time()-t0:.1f}s]")
+else:
+    print("\nInteractive. Type 'quit' to exit.\n")
+    while True:
+        try: q = input("You: ").strip()
+        except: print(); break
+        if not q: continue
+        if q.lower() in ("quit","exit","/bye"): break
+        t0 = time.time(); r = generate(m, tok, q, dev)
+        print(f"Fabric: {r}\n[{time.time()-t0:.1f}s]\n")

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8d12a687fe5eaa05b601ab2803ca2c14b9cc2c14d754110ca4569b061af2bf1
+size 1485107392

modeling_fabric.py

@@ -0,0 +1,141 @@
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Any
+
+import torch
+from torch import nn
+from transformers.generation import GenerationMixin
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from transformers.modeling_utils import PreTrainedModel
+
+from .configuration_fabric import FabricConfig
+from .fabric_runtime import FabricCoreForCausalLM, ModelConfig, load_checkpoint
+
+
+def _resolve_repo_dir(pretrained_model_name_or_path: str | os.PathLike[str], **download_kwargs: Any) -> Path:
+    path = Path(str(pretrained_model_name_or_path))
+    if path.exists():
+        return path
+    from huggingface_hub import snapshot_download
+
+    allowed = [
+        "model.safetensors",
+        "fabric_config.json",
+        "config.json",
+        "tokenizer.json",
+        "tokenizer_config.json",
+        "special_tokens_map.json",
+        "generation_config.json",
+        "configuration_fabric.py",
+        "modeling_fabric.py",
+        "fabric_runtime.py",
+        "system_prompt.txt",
+        "chat_template.md",
+    ]
+    return Path(snapshot_download(str(pretrained_model_name_or_path), allow_patterns=allowed, **download_kwargs))
+
+
+def _select_device(device_map: Any = None) -> torch.device:
+    if isinstance(device_map, str) and device_map not in {"auto", "sequential"}:
+        return torch.device(device_map)
+    if torch.cuda.is_available():
+        return torch.device("cuda")
+    if getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
+        return torch.device("mps")
+    return torch.device("cpu")
+
+
+def _normalize_dtype(torch_dtype: Any) -> torch.dtype | None:
+    if torch_dtype in {None, "auto"}:
+        return None
+    if isinstance(torch_dtype, torch.dtype):
+        return torch_dtype
+    if isinstance(torch_dtype, str):
+        return getattr(torch, torch_dtype.replace("torch.", ""))
+    return None
+
+
+class FabricForCausalLM(PreTrainedModel, GenerationMixin):
+    config_class = FabricConfig
+    base_model_prefix = "fabric"
+    main_input_name = "input_ids"
+    supports_gradient_checkpointing = False
+
+    def __init__(self, config: FabricConfig) -> None:
+        super().__init__(config)
+        values = config.to_fabric_model_config_dict()
+        values["activation_checkpointing"] = False
+        self.fabric = FabricCoreForCausalLM(ModelConfig(**values))
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
+        token = kwargs.pop("token", None)
+        use_auth_token = kwargs.pop("use_auth_token", None)
+        if token is None:
+            token = use_auth_token
+        revision = kwargs.pop("revision", None)
+        cache_dir = kwargs.pop("cache_dir", None)
+        local_files_only = kwargs.pop("local_files_only", False)
+        torch_dtype = _normalize_dtype(kwargs.pop("torch_dtype", kwargs.pop("dtype", None)))
+        device_map = kwargs.pop("device_map", None)
+        attention_backend = kwargs.pop("attention_backend", None)
+        config = kwargs.pop("config", None)
+
+        repo_dir = _resolve_repo_dir(
+            pretrained_model_name_or_path,
+            token=token,
+            revision=revision,
+            cache_dir=cache_dir,
+            local_files_only=local_files_only,
+        )
+        if config is None:
+            config = FabricConfig.from_pretrained(repo_dir, local_files_only=True)
+        backend = attention_backend if attention_backend is not None else getattr(config, "attention_backend", "auto")
+        config.attention_backend = backend
+
+        model = cls(config)
+        state = load_checkpoint(repo_dir / "model.safetensors", "cpu")
+        values = state.get("model_config") or state.get("config", {}).get("model")
+        if values:
+            values = dict(values)
+            values["attention_backend"] = backend
+            values["activation_checkpointing"] = False
+            model.fabric = FabricCoreForCausalLM(ModelConfig(**values))
+
+        missing, _unexpected = model.fabric.load_state_dict(state["model"], strict=False)
+        if missing:
+            raise RuntimeError(f"Missing Fabric checkpoint tensors: {missing}")
+        if torch_dtype is not None:
+            model.fabric.to(dtype=torch_dtype)
+        model.to(_select_device(device_map))
+        model.eval()
+        return model
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor,
+        attention_mask: torch.Tensor | None = None,
+        labels: torch.LongTensor | None = None,
+        past_key_values: Any | None = None,
+        use_cache: bool | None = None,
+        **kwargs,
+    ) -> CausalLMOutputWithPast:
+        output = self.fabric(input_ids=input_ids, labels=labels)
+        return CausalLMOutputWithPast(loss=output.loss, logits=output.logits, past_key_values=None)
+
+    def prepare_inputs_for_generation(self, input_ids, past_key_values=None, attention_mask=None, **kwargs):
+        return {"input_ids": input_ids, "attention_mask": attention_mask}
+
+    def get_input_embeddings(self) -> nn.Module:
+        return self.fabric.embed_tokens
+
+    def set_input_embeddings(self, value: nn.Module) -> None:
+        self.fabric.embed_tokens = value
+
+    def get_output_embeddings(self) -> nn.Module:
+        return self.fabric.lm_head
+
+    def set_output_embeddings(self, new_embeddings: nn.Module) -> None:
+        self.fabric.lm_head = new_embeddings

requirements.txt

@@ -0,0 +1,6 @@
+torch
+transformers>=4.50
+safetensors
+tokenizers
+huggingface_hub
+numpy

special_tokens_map.json

@@ -0,0 +1,11 @@
+{
+  "additional_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>"
+  ],
+  "bos_token": "<bos>",
+  "eos_token": "<|end|>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

system_prompt.txt

@@ -0,0 +1 @@
+You are Fabric, a large language model developed by Fabric AI.

tokenizer_config.json

@@ -0,0 +1,16 @@
+{
+  "additional_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>"
+  ],
+  "bos_token": "<bos>",
+  "chat_template": "{% set default_system_prompt = \"You are Fabric, a large language model developed by Fabric AI.\" %}{% set ns = namespace(has_system=false) %}{% for message in messages %}{% if message['role'] == 'system' %}{% set ns.has_system = true %}{% endif %}{% endfor %}{% if not ns.has_system %}<|system|>\n{{ default_system_prompt }}<|end|>\n{% endif %}{% for message in messages %}{% if message['role'] == 'system' %}<|system|>\n{{ message['content'] }}<|end|>\n{% elif message['role'] == 'user' %}<|user|>\n{{ message['content'] }}<|end|>\n{% elif message['role'] == 'assistant' %}<|assistant|>\n{{ message['content'] }}<|end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>\n{% endif %}",
+  "chat_template_file": "chat_template.jinja",
+  "default_system_prompt": "You are Fabric, a large language model developed by Fabric AI.",
+  "eos_token": "<|end|>",
+  "model_max_length": 32768,
+  "pad_token": "<pad>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}