Instructions to use h2oai/h2o-lightning-4b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use h2oai/h2o-lightning-4b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="h2oai/h2o-lightning-4b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("h2oai/h2o-lightning-4b")
model = AutoModelForMultimodalLM.from_pretrained("h2oai/h2o-lightning-4b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use h2oai/h2o-lightning-4b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "h2oai/h2o-lightning-4b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-4b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/h2oai/h2o-lightning-4b

SGLang

How to use h2oai/h2o-lightning-4b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "h2oai/h2o-lightning-4b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-4b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "h2oai/h2o-lightning-4b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-4b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use h2oai/h2o-lightning-4b with Docker Model Runner:
```
docker model run hf.co/h2oai/h2o-lightning-4b
```

Shim: do not answer a client that already hung up

by mjvk - opened 8 days ago

base: refs/heads/main

←

from: refs/pr/5

Discussion Files changed

8 days ago

When a client gives up before its answer is ready (for example, its own deadline passed), the shim still finishes the decision and writes the answer to the closed connection. The write raises BrokenPipeError or ConnectionResetError. do_POST's catch-all treats that as a failed decision: it prints a traceback and then tries to send a 422 on the same closed connection. That second write fails too, and the standard library's server prints a second traceback.

Behind a caller with short deadlines this produces two or three tracebacks on stderr per abandoned request. Cloud logging platforms read those as errors, so they hide real failures.

The change: _send catches BrokenPipeError and ConnectionResetError, marks the connection to close, and returns. Nobody is left to answer, so no traceback is printed and no 422 follows. Every other error path stays as it is.

test_a_client_that_left_is_not_answered reproduces it. A client sends a request, waits until the decision starts, and closes the connection with a reset. The test checks that the shim prints no traceback. It fails on the current main with the two tracebacks above, and passes with the change. python3 test_shim.py passes in full.

Unrelated: ImageCalibrationTest.test_text_path_byte_identical fails now and then on the current main as well. It compares seen[0] and seen[1], which can arrive in either order.

Shim: do not answer a client that already hung upcdcc8362

Remove PR.mddc8cf70e

mjvk changed pull request status to open 8 days ago

pseudotensor

H2O.ai org 8 days ago

Thank you for the clear report and the fix. We've taken your change as written into v1.4.2: _send now treats BrokenPipeError / ConnectionResetError as a client that has already left, closes the connection, and prints nothing; every other error path is unchanged. Your test test_a_client_that_left_is_not_answered ships in test_shim.py: it fails on the v1.4.1 shim with the two tracebacks you describe and passes with the change, and clients that stay connected are answered byte for byte as before. The ordering flake in ImageCalibrationTest.test_text_path_byte_identical is fixed in v1.4.2 as well (it now compares the prompts sorted). The change lands with the v1.4.2 release commit, credited to you in the release notes; we'll post the link here and close this PR once v1.4.2 is tagged.

pseudotensor changed pull request status to closed 7 days ago

pseudotensor

H2O.ai org 7 days ago

Thank you, mjvk: your fix shipped in v1.4.2 (https://huggingface.co/h2oai/h2o-lightning-4b/tree/v1.4.2). The shim no longer answers a client that has already hung up, and your test test_a_client_that_left_is_not_answered ships in test_shim.py, together with the fix for the ImageCalibrationTest ordering flake you reported. Closing this PR, since the change is now on main.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment