adapt model card to neat
Browse files
README.md
CHANGED
|
@@ -34,33 +34,25 @@ This repository contains the **Phi-3.5-mini-instruct-a16w4** model, optimized an
|
|
| 34 |
| Phi-3.5-mini-instruct | A16W8/A16W4 | Modalix | 512 | 20.84 tokens/sec | 0.31 sec |
|
| 35 |
| Phi-3.5-mini-instruct | A16W8/A16W4 | Modalix | 1024 | 17.11 tokens/sec | 0.82 sec |
|
| 36 |
|
| 37 |
-
|
| 38 |
## Prerequisites
|
| 39 |
|
| 40 |
To run this model, you need:
|
| 41 |
|
| 42 |
1. **SiMa.ai Modalix Device**
|
| 43 |
-
2. **SiMa.ai CLI**:
|
| 44 |
-
3. **
|
|
|
|
| 45 |
|
| 46 |
## Installation & Deployment
|
| 47 |
|
| 48 |
Follow these steps to deploy the model to your Modalix device.
|
| 49 |
|
| 50 |
-
### 1. Install
|
| 51 |
-
> **Note:** This is a **one-time setup**. If
|
| 52 |
|
| 53 |
-
|
| 54 |
|
| 55 |
-
```
|
| 56 |
-
# Create a directory for LLiMa
|
| 57 |
-
cd /media/nvme
|
| 58 |
-
mkdir llima
|
| 59 |
-
cd llima
|
| 60 |
-
# Install the LLiMa runtime code
|
| 61 |
-
sima-cli install -v 2.1.0 tools/llima -t selection
|
| 62 |
-
```
|
| 63 |
-
> **Note:** To only download the LLiMa runtime code, select **Demo Web App** when prompted.
|
| 64 |
|
| 65 |
### 2. Download the Model
|
| 66 |
|
|
@@ -83,65 +75,50 @@ scp -r Phi-3.5-mini-instruct-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/
|
|
| 83 |
|
| 84 |
```text
|
| 85 |
/media/nvme/llima/
|
| 86 |
-
βββ run.sh
|
| 87 |
βββ models/
|
| 88 |
βββ Phi-3.5-mini-instruct-a16w4/ # The compiled model
|
| 89 |
```
|
| 90 |
|
| 91 |
## Usage
|
| 92 |
|
| 93 |
-
###
|
| 94 |
|
| 95 |
-
|
| 96 |
|
| 97 |
```bash
|
| 98 |
-
|
| 99 |
-
./run.sh
|
| 100 |
```
|
| 101 |
|
| 102 |
-
|
| 103 |
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
https://<modalix-ip>:5000/
|
| 107 |
```
|
| 108 |
-
*Replace \<modalix-ip\> with the IP address of your Modalix device.*
|
| 109 |
|
| 110 |
-
###
|
| 111 |
|
| 112 |
-
|
| 113 |
-
```bash
|
| 114 |
-
llima run Phi-3.5-mini-instruct-a16w4 --mode web
|
| 115 |
-
```
|
| 116 |
|
| 117 |
-
|
| 118 |
|
| 119 |
-
|
| 120 |
|
| 121 |
-
|
| 122 |
-
curl -N -k -X POST "https://<modalix-ip>:9998/v1/chat/completions" \
|
| 123 |
-
-H "Content-Type: application/json" \
|
| 124 |
-
-d '{
|
| 125 |
-
"messages": [
|
| 126 |
-
{ "role": "user", "content": "Why is the sky blue?" }
|
| 127 |
-
],
|
| 128 |
-
"stream": true
|
| 129 |
-
}'
|
| 130 |
-
```
|
| 131 |
-
*Replace \<modalix-ip\> with the IP address of your Modalix device.*
|
| 132 |
|
| 133 |
## Limitations
|
| 134 |
|
| 135 |
- **Quantization**: This model is quantized (A16W4/A16W8) for optimal performance on embedded devices. While this maintains high accuracy, minor deviations from the full-precision model may occur.
|
| 136 |
|
| 137 |
-
|
| 138 |
## Troubleshooting
|
| 139 |
|
| 140 |
-
- **`sima-cli` not found**: Ensure that sima-cli is installed on your Modalix device.
|
|
|
|
| 141 |
- **Model can't be run**: Verify the model directory is exactly inside `/media/nvme/llima/models/` and not nested (e.g., `/media/nvme/llima/models/Phi-3.5-mini-instruct-a16w4/Phi-3.5-mini-instruct-a16w4`).
|
| 142 |
- **Permission Denied**: Ensure you have read/write permissions for the `/media/nvme` directory.
|
| 143 |
|
| 144 |
## Resources
|
| 145 |
|
| 146 |
-
- [
|
| 147 |
-
- [
|
|
|
|
|
|
|
|
|
| 34 |
| Phi-3.5-mini-instruct | A16W8/A16W4 | Modalix | 512 | 20.84 tokens/sec | 0.31 sec |
|
| 35 |
| Phi-3.5-mini-instruct | A16W8/A16W4 | Modalix | 1024 | 17.11 tokens/sec | 0.82 sec |
|
| 36 |
|
|
|
|
| 37 |
## Prerequisites
|
| 38 |
|
| 39 |
To run this model, you need:
|
| 40 |
|
| 41 |
1. **SiMa.ai Modalix Device**
|
| 42 |
+
2. **SiMa.ai CLI**: Installed on your Modalix device.
|
| 43 |
+
3. **SiMa.ai Neat Runtime**: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime.
|
| 44 |
+
4. **Hugging Face CLI**: Optional, for downloading the model on a host before copying it to Modalix.
|
| 45 |
|
| 46 |
## Installation & Deployment
|
| 47 |
|
| 48 |
Follow these steps to deploy the model to your Modalix device.
|
| 49 |
|
| 50 |
+
### 1. Install or Update Neat Runtime
|
| 51 |
+
> **Note:** This is a **one-time setup**. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download.
|
| 52 |
|
| 53 |
+
Follow the [SiMa.ai Neat getting started guide](https://developer.sima.ai/software/getting-started/) to install or update the Neat Library on your Modalix device.
|
| 54 |
|
| 55 |
+
The `llima` CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under `/media/nvme/llima/models` by default. Set `LLIMA_MODELS_PATH` to use a different model directory.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
|
| 57 |
### 2. Download the Model
|
| 58 |
|
|
|
|
| 75 |
|
| 76 |
```text
|
| 77 |
/media/nvme/llima/
|
|
|
|
| 78 |
βββ models/
|
| 79 |
βββ Phi-3.5-mini-instruct-a16w4/ # The compiled model
|
| 80 |
```
|
| 81 |
|
| 82 |
## Usage
|
| 83 |
|
| 84 |
+
### Validate with LLiMa CLI
|
| 85 |
|
| 86 |
+
Run the model directly on Modalix:
|
| 87 |
|
| 88 |
```bash
|
| 89 |
+
llima run Phi-3.5-mini-instruct-a16w4
|
|
|
|
| 90 |
```
|
| 91 |
|
| 92 |
+
For all runtime options, run:
|
| 93 |
|
| 94 |
+
```bash
|
| 95 |
+
llima run -h
|
|
|
|
| 96 |
```
|
|
|
|
| 97 |
|
| 98 |
+
### GenAI Demo Application
|
| 99 |
|
| 100 |
+
The GenAI demo application is separate from LLiMa installation. Use the [GenAI Multimodal Assistant](https://developer.sima.ai/examples/app/genai%2Fmultimodal-assistant) page to install and run the demo app. Once installed, the demo app can use precompiled models such as this one.
|
|
|
|
|
|
|
|
|
|
| 101 |
|
| 102 |
+
### API Usage
|
| 103 |
|
| 104 |
+
To serve this model with OpenAI- or Ollama-compatible APIs and send requests to it, use the GenAI server workflow in [Serve GenAI Models](https://developer.sima.ai/software/tutorials/serve-genai-models).
|
| 105 |
|
| 106 |
+
For direct LLM calls without setting up a server, see [Run an LLM](https://developer.sima.ai/software/tutorials/run-an-llm).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
|
| 108 |
## Limitations
|
| 109 |
|
| 110 |
- **Quantization**: This model is quantized (A16W4/A16W8) for optimal performance on embedded devices. While this maintains high accuracy, minor deviations from the full-precision model may occur.
|
| 111 |
|
|
|
|
| 112 |
## Troubleshooting
|
| 113 |
|
| 114 |
+
- **`sima-cli` not found**: Ensure that `sima-cli` is installed on your Modalix device.
|
| 115 |
+
- **`llima` not found**: Install or update the Neat Library. See [Getting Started](https://developer.sima.ai/software/getting-started/).
|
| 116 |
- **Model can't be run**: Verify the model directory is exactly inside `/media/nvme/llima/models/` and not nested (e.g., `/media/nvme/llima/models/Phi-3.5-mini-instruct-a16w4/Phi-3.5-mini-instruct-a16w4`).
|
| 117 |
- **Permission Denied**: Ensure you have read/write permissions for the `/media/nvme` directory.
|
| 118 |
|
| 119 |
## Resources
|
| 120 |
|
| 121 |
+
- [GenAI with LLiMa](https://developer.sima.ai/software/genai-llima/)
|
| 122 |
+
- [Serve GenAI Models](https://developer.sima.ai/software/tutorials/serve-genai-models)
|
| 123 |
+
- [Run an LLM](https://developer.sima.ai/software/tutorials/run-an-llm)
|
| 124 |
+
- [GenAI Multimodal Assistant](https://developer.sima.ai/examples/app/genai%2Fmultimodal-assistant)
|