Instructions to use QuantFactory/Llama3.1-8B-Fireplace2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantFactory/Llama3.1-8B-Fireplace2-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="QuantFactory/Llama3.1-8B-Fireplace2-GGUF",
	filename="Llama3.1-8B-Fireplace2.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use QuantFactory/Llama3.1-8B-Fireplace2-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M

Use Docker

docker model run hf.co/QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use QuantFactory/Llama3.1-8B-Fireplace2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantFactory/Llama3.1-8B-Fireplace2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Llama3.1-8B-Fireplace2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M

Ollama
How to use QuantFactory/Llama3.1-8B-Fireplace2-GGUF with Ollama:
```
ollama run hf.co/QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M
```

Unsloth Studio new

How to use QuantFactory/Llama3.1-8B-Fireplace2-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Llama3.1-8B-Fireplace2-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Llama3.1-8B-Fireplace2-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for QuantFactory/Llama3.1-8B-Fireplace2-GGUF to start chatting

Docker Model Runner
How to use QuantFactory/Llama3.1-8B-Fireplace2-GGUF with Docker Model Runner:
```
docker model run hf.co/QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M
```

Lemonade

How to use QuantFactory/Llama3.1-8B-Fireplace2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QuantFactory/Llama3.1-8B-Fireplace2-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Llama3.1-8B-Fireplace2-GGUF-Q4_K_M

List all available models

lemonade list

QuantFactory/Llama3.1-8B-Fireplace2-GGUF

This is quantized version of ValiantLabs/Llama3.1-8B-Fireplace2 created using llama.cpp

Original Model Card

Fireplace 2 is a chat model, adding helpful structured outputs to Llama 3.1 8b Instruct.

an expansion pack of supplementary outputs - request them at will within your chat:
- Inline function calls
- SQL queries
- JSON objects
- Data visualization with matplotlib
Mix normal chat and structured outputs within the same conversation.
Fireplace 2 supplements the existing strengths of Llama 3.1, providing inline capabilities within the Llama 3 Instruct format.

Version

This is the 2024-07-23 release of Fireplace 2 for Llama 3.1 8b.

We're excited to bring further upgrades and releases to Fireplace 2 in the future.

Help us and recommend Fireplace 2 to your friends!

Prompting Guide

Fireplace uses the Llama 3.1 Instruct prompt format. The example script below can be used as a starting point for general chat with Llama 3.1 and also includes the different special tokens used for Fireplace 2's added features:

import transformers import torch

model_id = "ValiantLabs/Llama3.1-8B-Fireplace2"

pipeline = transformers.pipeline( "text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto", )

messages = [ {"role": "system", "content": "You are Fireplace, an expert technical assistant."}, {"role": "user", "content": "Hi, can you explain local area networking to me?"}, #general Llama 3.1 chat #{"role": "user", "content": "I have the following SQL table: employees (job_id VARCHAR, salary INTEGER)\n\nCan you find all employees with a salary above $75000?<|request_sql|>"}, #for SQL query #{"role": "user", "content": "{""name"": ""get_news_headlines"",""description"": ""Get the latest news headlines"",""parameters"": {""type"": ""object"",""properties"": {""country"": {""type"": ""string"",""description"": ""The country for which news headlines are to be retrieved""}},""required"": [""country""]}}\n\nHi, can you get me the latest news headlines for the United States?<|request_function_call|>"}, # for function call #{"role": "user", "content": "Show me an example of a histogram with a fixed bin size. Use attractive colors.<|request_matplotlib|>"}, #for data visualization #{"role": "user", "content": "Can you define the word 'presence' for me, thanks!<|request_json|>"}, #for JSON output ]

outputs = pipeline( messages, max_new_tokens=512, ) print(outputs[0]["generated_text"][-1])

While Fireplace 2 is trained to minimize incorrect structured outputs, they can still occur occasionally. Production uses of Fireplace 2 should verify the structure of all model outputs and remove any unneeded components of the output.

For handling of function call responses, use the Llama 3.1 Instruct tool response style.

Special Tokens

Fireplace 2 utilizes special tokens applied to the Llama 3.1 tokenizer:

<|request_json|>
<|start_json|>
<|end_json|>
<|request_sql|>
<|start_sql|>
<|end_sql|>
<|request_matplotlib|>
<|start_matplotlib|>
<|end_matplotlib|>
<|request_function_call|>
<|start_function_call|>
<|end_function_call|>

These are supplemental to the existing special tokens used by Llama 3.1, such as <|python_tag|> and <|start_header_id|>. Fireplace 2 has been trained using the Llama 3.1 Instruct chat structure, with new special tokens added within the conversation.

The 'request' tokens are used by the user to request a specific type of structured output. They should be appended to the end of the user's message and can be alternated with normal chat responses throughout the conversation.

The Model

Fireplace 2 is built on top of Llama 3.1 8b Instruct.

This version of Fireplace 2 uses data from the following datasets:

Additional capabilities will be added to future releases.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	18.31
IFEval (0-Shot)	54.83
BBH (3-Shot)	24.07
MATH Lvl 5 (4-Shot)	5.82
GPQA (0-shot)	5.15
MuSR (0-shot)	4.38
MMLU-PRO (5-shot)	15.63

Fireplace 2 is created by Valiant Labs.

Check out our HuggingFace page for Shining Valiant 2 and our other models!

We care about open source. For everyone to use.

We encourage others to finetune further from our models.

Downloads last month: 131

GGUF

Model size

8B params

Architecture

llama

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Evaluation results

strict accuracy on IFEval (0-Shot)
Open LLM Leaderboard

54.830
normalized accuracy on BBH (3-Shot)
Open LLM Leaderboard

24.070
exact match on MATH Lvl 5 (4-Shot)
Open LLM Leaderboard

5.820
acc_norm on GPQA (0-shot)
Open LLM Leaderboard

5.150
acc_norm on MuSR (0-shot)
Open LLM Leaderboard

4.380
accuracy on MMLU-PRO (5-shot)
test set Open LLM Leaderboard

15.630