Seed-0.5B

• 0.5B decoder only dense model trained from scratch with 6B tokens.
• Model and tokenizer is trained on fineweb-edu sample-10BT for achieving good results with low token size.
• As it isn't instruction finetuned, the model performs document completion, not conversational generation.
• While 6B tokens are enough to learn reasonable grammar and structure, this amount is insufficient for strong factual grounding, and model can generate inaccurate or imprecise informations.
• Model is released primarily for educational, research, and experimental purposes.

GitHub: merterbak/llm-from-scratch

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("merterbak/Seed-0.5B")
model = AutoModelForCausalLM.from_pretrained(
    "merterbak/Seed-0.5B",
    trust_remote_code=True,
    dtype="auto"
)
prompt = "Climate change can affect"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    temperature=0.3,
    top_k=40,
    do_sample=True,
    repetition_penalty=1.2,
    no_repeat_ngram_size=3,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))