Instructions to use Ostixe360/MoMv3-mixed-precision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Ostixe360/MoMv3-mixed-precision with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Ostixe360/MoMv3-mixed-precision")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Ostixe360/MoMv3-mixed-precision", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Ostixe360/MoMv3-mixed-precision with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Ostixe360/MoMv3-mixed-precision"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ostixe360/MoMv3-mixed-precision",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Ostixe360/MoMv3-mixed-precision

SGLang

How to use Ostixe360/MoMv3-mixed-precision with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Ostixe360/MoMv3-mixed-precision" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ostixe360/MoMv3-mixed-precision",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Ostixe360/MoMv3-mixed-precision" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ostixe360/MoMv3-mixed-precision",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Ostixe360/MoMv3-mixed-precision with Docker Model Runner:
```
docker model run hf.co/Ostixe360/MoMv3-mixed-precision
```

Model Card for Model ID

Model Details

Model Description

MoM: Mixture of Mixture

This Model is a test to combine Jamba architecture with 1.58 bits linear layers excpted for attention layer, mixture of attention head and mixture of depth.

The goal is to developpe and test if this kind of architectures have not too much quality loss for a fast inference.

Only 17.8M parameter over 1025 is in bf16 precision wich is ~ 1.7% of the total number of parameters

Model type: Mixture of attention head mixture of depth and mixture of expert 1.58bit linear layers excepted for attention layer
License: Apache licence 2.0

Model Sources [optional]

Repository: https://github.com/ostix360/optimized-LLM

How to Get Started with the Model

If you want to test this model please look at this repo at this commit

Training Details

wandb: training detail

Training Data

We use the first 100k data of Locutusque/UltraTextbooks to train this model

Training Procedure

We use adam-8 bits with default betas and epsilon values

Preprocessing [optional]

The data fit the model max length i.e. 512 tokens

Training Hyperparameters

Please look at the wandb metadata file or the train.py file in the repo to see the hyperparameters

Technical Specifications [optional]

Compute Infrastructure

Hardware

one 4070 ti GPU

Software

pytorch, transformers etc

Downloads last month: 10

Safetensors

Model size

1B params

Tensor type

BF16

Ostixe360
/

MoMv3-mixed-precision