README.md

---
license: apache-2.0
---

# NeuroBLAST‑1.9B Instruct - Early Preview (Research‑Only)

**Status:** 🚧 *Early preview checkpoint* - **not** production‑ready.

**Core message:** This model is released to showcase the **NeuroBLAST (Brain‑Like Architecture with Stacked Transformers)** design and to catalyze community research. Do **not** rely on its outputs for clinical, legal, financial, or safety‑critical decisions.

> **CRITICAL MEDICAL DISCLAIMER:** This model is **NOT** for medical diagnosis, treatment, triage, or as a source of medical knowledge/reference. Do not use its outputs to inform clinical decisions, guidelines, or patient education. Always consult qualified healthcare professionals and verified medical resources.

---

## TL;DR

* **Parameters:** 1.9B
* **Objective:** Instruction‑tuned conversational model demonstrating the NeuroBLAST architecture.
* **Training tokens:** 100B tokens of permissively licensed, open‑source text.
* **Context length**: trained on 16k-token sequences; RoPE configured to 132k positions (effective max).
* **Sources:** Mix of general web text (**FineWeb**, **FineWeb‑Edu**) and permissive/open medical corpora (e.g., PMC Open Access subset, PubMedQA, MedMCQA).
* **License:** Apache‑2.0 for model weights & code. Downstream users must also respect underlying data licenses.
* **Use scope:** Research, benchmarking, and architecture exploration only. **No medical advice, patient care use, or use as a source of medical knowledge/reference.**

---

## 1. Model Details

### 1.1 Architecture

NeuroBLAST (Brain‑Like Architecture with Stacked Transformers) arranges transformer modules in stacked, semi‑independent “blocks” with explicit interfaces intended to mimic hierarchical processing pathways. This checkpoint keeps the stack compact (1.9B params) to ease experimentation and ablation.

![Architecture](arch.jpg)

#### Diagramy summary:

Input tokens are embedded with a token embedding layer plus Rotary Positional Embeddings (RoPE). 
The stream first enters an Association Cortex block (× N layers) composed of standard self‑attention followed by a SwiGLU MLP. 
From here, two pooler layers branch out (Assoc → Sensory Pooler and Assoc → Motor Pooler) that compress/route state to the downstream modules.

A **Sensory Cortex stack** (× M layers, green) receives:

- Its own self‑attention.

- Gated cross‑attention to the Association Cortex (σ gating).

- A SwiGLU MLP.

It can also cross‑attend to the Motor Cortex.

A **Motor Cortex stack** (× K layers) includes:

- Self‑attention.

- Gated cross‑attention to the Sensory Cortex (σ gate) and ungated cross‑attention to the Association Cortex.

- A SwiGLU MLP.

A third pooler (Sensory → Motor Pooler) provides a condensed sensory representation to the motor block. 
After these specialized cortex modules, activations pass through a LayerNorm and a Linear head for next‑token prediction.

**Memory Layer**: *WIP*

### 1.2 Checkpoint Type

* **Pretraining:** Next‑token prediction on mixed‑domain corpora.
* **Instruction tuning:** Lightweight SFT/DPO pass for chat usability.
* **Precision:** BF16 weights.

### 1.3 Files & Formats

* Weights in `safetensors` format.
* `trust_remote_code=True` is required due to custom generation utilities.

---

## 2. Intended Use

### 2.1 Primary Intended Use

* Research on model architectures (e.g., stacked transformers), data curation, and instruction‑tuning techniques.
* Benchmarking against similarly sized open models.
* Educational demos of training/inference pipelines.

### 2.2 Out‑of‑Scope / Prohibited Uses

* **Medical, diagnostic, or therapeutic decision support.**
* **Any clinical workflow or patient‑facing chatbot.**
* Automated decision‑making in safety‑critical or regulated contexts (finance, law, aviation, etc.).
* Generation of disallowed content per applicable laws/policies (e.g., hate speech, harassment, illegal acts) without appropriate safeguards.

---

## 3. Safety & Disclaimer (Read This!)

* **NOT for medical diagnosis, treatment, triage, or as a source of medical knowledge/reference.** Outputs may be wrong, outdated, or dangerously misleading; never use them to make or inform clinical decisions. Always consult qualified healthcare professionals and authoritative medical resources.
* **Hallucinations happen.** The model can produce confident but incorrect statements and fabricated citations.
* **No warranty.** The model is provided “AS IS,” without guarantees of accuracy or fitness for purpose.
* **Handle sensitive data carefully.** Do not input Protected Health Information (PHI) or other confidential data unless you have legal permission and proper safeguards.
* **Regulatory compliance is your responsibility.** If you fine‑tune, deploy, or integrate this model, ensure compliance with relevant laws (e.g., HIPAA, GDPR, FDA/EMA rules for clinical decision support).

**Emergency disclaimer:** If you or someone else may be experiencing a medical emergency, call your local emergency number immediately. Do not rely on this model.

---

## 4. Training Data

### 4.1 High‑Level Composition (\~100B tokens)

* **General web corpus:** Curated slices of **FineWeb** and **FineWeb‑Edu**.
* **Medical/biomedical corpora (permissive licenses):** Examples include the PMC Open Access subset (Creative Commons family), PubMedQA (MIT), MedMCQA (MIT/CC0 variants), and similar open QA/MCQ datasets.
* **Curated SFT data:** Created by medical doctors and authors of the [Medcases Application](https://medcases.io).
* **Synthethic SFT data**
  
### 4.2 Licensing Summary

| Source                 | License (examples)                                | Notes                                                                          |
| ---------------------- | ------------------------------------------------- | -------------------------------------------------------------------------------|
| FineWeb / FineWeb‑Edu  | ODC‑By 1.0                                        | Subject to CommonCrawl ToS                                                     |
| PMC Open Access subset | CC‑BY / CC0 / CC‑BY‑NC, etc. (varies per article) | Only OA commercially allowed articles with machine‑readable licenses were used |
| PubMedQA               | MIT                                               | QA over biomedical abstracts                                                   |
| MedMCQA                | MIT/CC0 (check source)                            | Multiple‑choice medical exam questions                                         |

**Downstream users must respect these licenses when redistributing data derivatives.**

### 4.3 Cleaning & Filtering

* Deduplication, language/id filters.
* Basic quality filters (minimum length, low quality content & false information removal).

---

## 5. Evaluation (Preliminary)

This early preview has *not* undergone a comprehensive evaluation. Known issues:

* Inconsistent factuality on long, multi‑step questions.
* Tendency to over‑generalize or fabricate references in medical domains.
* Benchmarks: *WIP*

We encourage the community to:

* Run transparent, reproducible evaluations.
* Share red‑team findings via the HF “Community” tab.

---

## 6. Limitations & Known Failure Modes

* **Factual errors / hallucinations:** Especially in niche biomedical topics or when pressed for citations.
* **Temporal staleness:** Knowledge cutoff at pretraining time; no guarantee of recency.
* **Biases & harmful content:** Can reflect biases present in web/medical text (gender, race, socioeconomic, geographic, etc.).
* **Prompt sensitivity:** Small phrasing changes can yield drastically different answers.

---

## 7. How to Use

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

device = "cuda" if torch.cuda.is_available() else "cpu"
model_name = "meditsolutions/NeuroBLAST-1.9B-Instruct-Early-Preview"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map=device,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_name)

messages = [
    {"role": "user", "content": "Explain enalapril in simple terms."}
]

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

out = pipe(messages, max_new_tokens=500, return_full_text=False, temperature=0.7)
print(out[0]["generated_text"])
```

**Prompting tips:**

* Provide structured instructions (role, constraints, format).
* Use lower temperatures and max token limits for factual tasks.

---

## 8. Citation

If you use this model, please don't hesitate to cite:

```
@misc{meditsolutions2025neuroblast,
  title        = {NeuroBLAST-1.9B Instruct: Early Preview},
  author       = {MedIT Solutions},
  year         = {2025},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/meditsolutions/NeuroBLAST-1.9B-Instruct-Early-Preview}
}
```

---

## 9. Changelog

* **2025‑07‑23:** Initial public release (early preview).

---

## 10. Contact & Feedback

* Open an issue/discussion on the HF repo
* Email: `mariusz.kurman@meditsolutions.pl`
* X: `@mkurman88`

---

### Acknowledgements

Thanks to the maintainers of FineWeb/FineWeb-Edu and all open medical datasets, as well as to the broader open-source ML community.

---

**Reminder:** \*This checkpoint is a research artifact. Treat every answer with skepticism and always verify critical information. \*