Instructions to use bharatgenai/Param2-17B-A2.4B-Thinking with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use bharatgenai/Param2-17B-A2.4B-Thinking with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="bharatgenai/Param2-17B-A2.4B-Thinking", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("bharatgenai/Param2-17B-A2.4B-Thinking", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use bharatgenai/Param2-17B-A2.4B-Thinking with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "bharatgenai/Param2-17B-A2.4B-Thinking" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bharatgenai/Param2-17B-A2.4B-Thinking", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/bharatgenai/Param2-17B-A2.4B-Thinking
- SGLang
How to use bharatgenai/Param2-17B-A2.4B-Thinking with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "bharatgenai/Param2-17B-A2.4B-Thinking" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bharatgenai/Param2-17B-A2.4B-Thinking", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "bharatgenai/Param2-17B-A2.4B-Thinking" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bharatgenai/Param2-17B-A2.4B-Thinking", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use bharatgenai/Param2-17B-A2.4B-Thinking with Docker Model Runner:
docker model run hf.co/bharatgenai/Param2-17B-A2.4B-Thinking
Kaggle: ImportError: cannot import name 'is_torch_fx_available' from 'transformers.utils.import_utils'
Have you used the Transformers version specified in the README?
Also, could you please share the complete code snippet you’re using where you’re encountering this error?
Back then, I was trying on Kaggle.
I have shifted to a local GPU, and there has been no issue.
Here is the code snippet I used in Kaggle:
from transformers import AutoConfig, pipeline
import transformers.utils.import_utils as import_utils
1. First, apply the previous hotfix for the Import Error
if not hasattr(import_utils, "is_torch_fx_available"):
import_utils.is_torch_fx_available = lambda: False
2. Load the config and fix the RoPE type mismatch
model_id = "bharatgenai/Param2-17B-A2.4B-Thinking"
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
The script expects a specific key; 'llama' or 'linear' are common defaults
for these architectures if 'default' is failing.
if not hasattr(config, "rope_type") or config.rope_type == "default":
config.rope_type = "linear" # Or "llama", depending on the base architecture
3. Load the pipeline with the corrected config
pipe = pipeline(
"text-generation",
model=model_id,
config=config,
trust_remote_code=True,
device_map="auto" # Recommended for a 17B model
)
messages = [{"role": "user", "content": "Who are you?"}]
print(pipe(messages))
Python: 3.12
Library: transformers (latest) - In the Readme no specific version was mentioned
Please use transformers version 4.52.3