florianvoss commited on
Commit
229e4fe
Β·
verified Β·
1 Parent(s): 1b44283

adapt model card to neat

Browse files
Files changed (1) hide show
  1. README.md +24 -47
README.md CHANGED
@@ -34,33 +34,25 @@ This repository contains the **Phi-3.5-mini-instruct-a16w4** model, optimized an
34
  | Phi-3.5-mini-instruct | A16W8/A16W4 | Modalix | 512 | 20.84 tokens/sec | 0.31 sec |
35
  | Phi-3.5-mini-instruct | A16W8/A16W4 | Modalix | 1024 | 17.11 tokens/sec | 0.82 sec |
36
 
37
-
38
  ## Prerequisites
39
 
40
  To run this model, you need:
41
 
42
  1. **SiMa.ai Modalix Device**
43
- 2. **SiMa.ai CLI**: [Installed](https://docs.sima.ai/pages/sima_cli/main.html#installation) on your Modalix device.
44
- 3. **Hugging Face CLI**: For downloading the model.
 
45
 
46
  ## Installation & Deployment
47
 
48
  Follow these steps to deploy the model to your Modalix device.
49
 
50
- ### 1. Install LLiMa Demo Application
51
- > **Note:** This is a **one-time setup**. If you have already installed the LLiMa demo application (e.g. for another model), you can skip this step and continue with model download.
52
 
53
- On your Modalix device, install the LLiMa demo application using the `sima-cli`:
54
 
55
- ```bash
56
- # Create a directory for LLiMa
57
- cd /media/nvme
58
- mkdir llima
59
- cd llima
60
- # Install the LLiMa runtime code
61
- sima-cli install -v 2.1.0 tools/llima -t selection
62
- ```
63
- > **Note:** To only download the LLiMa runtime code, select **Demo Web App** when prompted.
64
 
65
  ### 2. Download the Model
66
 
@@ -83,65 +75,50 @@ scp -r Phi-3.5-mini-instruct-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/
83
 
84
  ```text
85
  /media/nvme/llima/
86
- β”œβ”€β”€ run.sh
87
  └── models/
88
  └── Phi-3.5-mini-instruct-a16w4/ # The compiled model
89
  ```
90
 
91
  ## Usage
92
 
93
- ### Run the Application
94
 
95
- Navigate to the demo directory and start the application:
96
 
97
  ```bash
98
- cd /media/nvme/llima/
99
- ./run.sh
100
  ```
101
 
102
- The script will detect the installed model(s) and prompt you to select one.
103
 
104
- Once the application is running, open a browser and navigate to:
105
- ```text
106
- https://<modalix-ip>:5000/
107
  ```
108
- *Replace \<modalix-ip\> with the IP address of your Modalix device.*
109
 
110
- ### API Usage
111
 
112
- To use OpenAI-compatible API, run the model in API mode:
113
- ```bash
114
- llima run Phi-3.5-mini-instruct-a16w4 --mode web
115
- ```
116
 
117
- You can interact with it using `curl` or Python.
118
 
119
- **Example: Chat Completion**
120
 
121
- ```bash
122
- curl -N -k -X POST "https://<modalix-ip>:9998/v1/chat/completions" \
123
- -H "Content-Type: application/json" \
124
- -d '{
125
- "messages": [
126
- { "role": "user", "content": "Why is the sky blue?" }
127
- ],
128
- "stream": true
129
- }'
130
- ```
131
- *Replace \<modalix-ip\> with the IP address of your Modalix device.*
132
 
133
  ## Limitations
134
 
135
  - **Quantization**: This model is quantized (A16W4/A16W8) for optimal performance on embedded devices. While this maintains high accuracy, minor deviations from the full-precision model may occur.
136
 
137
-
138
  ## Troubleshooting
139
 
140
- - **`sima-cli` not found**: Ensure that sima-cli is installed on your Modalix device.
 
141
  - **Model can't be run**: Verify the model directory is exactly inside `/media/nvme/llima/models/` and not nested (e.g., `/media/nvme/llima/models/Phi-3.5-mini-instruct-a16w4/Phi-3.5-mini-instruct-a16w4`).
142
  - **Permission Denied**: Ensure you have read/write permissions for the `/media/nvme` directory.
143
 
144
  ## Resources
145
 
146
- - [SiMa.ai Documentation](https://docs.sima.ai)
147
- - [SiMa.ai Hugging Face Organization](https://huggingface.co/simaai)
 
 
 
34
  | Phi-3.5-mini-instruct | A16W8/A16W4 | Modalix | 512 | 20.84 tokens/sec | 0.31 sec |
35
  | Phi-3.5-mini-instruct | A16W8/A16W4 | Modalix | 1024 | 17.11 tokens/sec | 0.82 sec |
36
 
 
37
  ## Prerequisites
38
 
39
  To run this model, you need:
40
 
41
  1. **SiMa.ai Modalix Device**
42
+ 2. **SiMa.ai CLI**: Installed on your Modalix device.
43
+ 3. **SiMa.ai Neat Runtime**: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime.
44
+ 4. **Hugging Face CLI**: Optional, for downloading the model on a host before copying it to Modalix.
45
 
46
  ## Installation & Deployment
47
 
48
  Follow these steps to deploy the model to your Modalix device.
49
 
50
+ ### 1. Install or Update Neat Runtime
51
+ > **Note:** This is a **one-time setup**. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download.
52
 
53
+ Follow the [SiMa.ai Neat getting started guide](https://developer.sima.ai/software/getting-started/) to install or update the Neat Library on your Modalix device.
54
 
55
+ The `llima` CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under `/media/nvme/llima/models` by default. Set `LLIMA_MODELS_PATH` to use a different model directory.
 
 
 
 
 
 
 
 
56
 
57
  ### 2. Download the Model
58
 
 
75
 
76
  ```text
77
  /media/nvme/llima/
 
78
  └── models/
79
  └── Phi-3.5-mini-instruct-a16w4/ # The compiled model
80
  ```
81
 
82
  ## Usage
83
 
84
+ ### Validate with LLiMa CLI
85
 
86
+ Run the model directly on Modalix:
87
 
88
  ```bash
89
+ llima run Phi-3.5-mini-instruct-a16w4
 
90
  ```
91
 
92
+ For all runtime options, run:
93
 
94
+ ```bash
95
+ llima run -h
 
96
  ```
 
97
 
98
+ ### GenAI Demo Application
99
 
100
+ The GenAI demo application is separate from LLiMa installation. Use the [GenAI Multimodal Assistant](https://developer.sima.ai/examples/app/genai%2Fmultimodal-assistant) page to install and run the demo app. Once installed, the demo app can use precompiled models such as this one.
 
 
 
101
 
102
+ ### API Usage
103
 
104
+ To serve this model with OpenAI- or Ollama-compatible APIs and send requests to it, use the GenAI server workflow in [Serve GenAI Models](https://developer.sima.ai/software/tutorials/serve-genai-models).
105
 
106
+ For direct LLM calls without setting up a server, see [Run an LLM](https://developer.sima.ai/software/tutorials/run-an-llm).
 
 
 
 
 
 
 
 
 
 
107
 
108
  ## Limitations
109
 
110
  - **Quantization**: This model is quantized (A16W4/A16W8) for optimal performance on embedded devices. While this maintains high accuracy, minor deviations from the full-precision model may occur.
111
 
 
112
  ## Troubleshooting
113
 
114
+ - **`sima-cli` not found**: Ensure that `sima-cli` is installed on your Modalix device.
115
+ - **`llima` not found**: Install or update the Neat Library. See [Getting Started](https://developer.sima.ai/software/getting-started/).
116
  - **Model can't be run**: Verify the model directory is exactly inside `/media/nvme/llima/models/` and not nested (e.g., `/media/nvme/llima/models/Phi-3.5-mini-instruct-a16w4/Phi-3.5-mini-instruct-a16w4`).
117
  - **Permission Denied**: Ensure you have read/write permissions for the `/media/nvme` directory.
118
 
119
  ## Resources
120
 
121
+ - [GenAI with LLiMa](https://developer.sima.ai/software/genai-llima/)
122
+ - [Serve GenAI Models](https://developer.sima.ai/software/tutorials/serve-genai-models)
123
+ - [Run an LLM](https://developer.sima.ai/software/tutorials/run-an-llm)
124
+ - [GenAI Multimodal Assistant](https://developer.sima.ai/examples/app/genai%2Fmultimodal-assistant)