Reponx/Llama-3.1-8B-Reponx-Network-Cloud-Ops

Llama-3.1-8B Reponx Network & Cloud Ops (LoRA, v4)

Built with Llama. A LoRA adapter for meta-llama/Llama-3.1-8B-Instruct that turns it into a network and cloud operations assistant for Cisco, Palo Alto Networks, Microsoft Azure and AWS.

It is trained to work with retrieved reference documents (RAG): it diagnoses the problem, rates the risk of the change, requires approval, and takes the exact commands from the references you provide instead of from memory. When the references do not contain the needed commands, it says so and does not guess.

This adapter replaces the earlier Qwen-based release at Reponx/Network-Cloud-Ops-Engineer-9B. It uses a different base model and prompt format; the two are not interchangeable.

Answer format

For operations questions:

Diagnosis → Risk → Action Type (read / write / high) → Approval → Warning → Commands/Action → Verification → Rollback

For architecture questions: Options → Trade-offs → Recommendation → Validation

Action type	Meaning	Approval
read	Show / describe / diagnostic only	Not required
write	Normal change	Standard change approval
high	Outage risk (reload, reset, delete, route or security replacement, ...)	CAB approval and maintenance window

Test results

Held-out test of 124 questions (never seen in training), with the correct reference included among unrelated ones. Base = the same Llama model without the adapter, given the same references.

Check	Cisco (40)	Palo Alto (28)	Azure (27)	AWS (29)	All	Base model
Correct answer format	100%	100%	100%	100%	100%	87%
Correct risk level (read/write/high)	95%	96%	92%	93%	94%	27%
Commands taken from the references	97%	100%	100%	100%	99%	23%
Needed commands present	93%	100%	100%	100%	97%	24%
Refused when no reference was given	6/6	5/5	5/5	4/5	95%	0%

How to read these numbers

• The test checks behaviour: format, risk rating, use of references and refusal. It does not prove the commands are correct for your devices.
• The correct reference was always present among three. A real retrieval system will sometimes miss it, so real-world results will be lower.
• Test and training data come from the same synthetic dataset (different questions), so this is not an independent benchmark.

Prompt format

The adapter was trained with this system prompt and user layout. Use the same layout so your RAG output lands in the Reference block.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE = "meta-llama/Llama-3.1-8B-Instruct"
ADAPTER = "Reponx/Llama-3.1-8B-Reponx-Network-Cloud-Ops"

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)

SYSTEM = ("You are Reponx, an AI engineer for network and cloud operations. For operations issues answer with: "
          "Diagnosis, Risk, Action Type (read/write/high), Approval, Warning, Commands/Action, Verification, Rollback. "
          "For architecture questions answer with: Options, Trade-offs, Recommendation, Validation. "
          "Never run write or high actions without approval. Use only commands that appear in the Reference section, "
          "adapted to the devices in the scenario. If the references do not contain the commands needed, say so and do not invent commands.")

USER = """Vendor: Cisco (IOS-XE)
Scenario: Two routers on a point-to-point link in OSPF area 0. A new link was added.
Problem: No OSPF adjacency forms on the new link.
Evidence:
- show ip ospf interface Gi0/0/2: 'No Hellos (Passive interface)'
- show running-config | section router ospf: passive-interface default

Reference:
[Ref 1] IGP - OSPF - Adjacency blocked by passive-interface default
(config)# router ospf 1
(config-router)# no passive-interface Gi0/0/2

[Ref 2] <another retrieved document>
<its commands>"""

msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, max_new_tokens=900, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Without a Reference block the model will often refuse to give commands. That is intended.

Safety: required controls

The model is an assistant, not an autonomous operator. In testing it made a small number of serious errors, for example:

• With no correct reference, it once copied a delete command from an unrelated reference instead of refusing.
• In a high-risk change it once wrote reload where reload cancel was required.

Before connecting it to real networks, applications should:

1. Require human approval for every write and high action, and never execute commands automatically.
2. Force high with a rule check when commands contain reload, delete, clear, reset, release, terminate, or replace routes or security rules.
3. Block answers whose commands do not appear in the retrieved references.

Training

• Method: LoRA (r=16, alpha=32, dropout 0.05, all attention and MLP projections), 3 epochs, learning rate 2e-4, bf16, loss on the assistant answer only.
• Data: 695 training examples (Cisco, Palo Alto Networks, Azure, AWS), each with a Reference block of three documents, plus examples with no valid reference to teach refusal. See Reponx/network-cloud-ops-sft.
• The commands in the training data were written for training and were not lab-verified. Exact syntax should always come from your own verified documentation.

Coverage

Included: Cisco IOS / IOS-XE / Catalyst, Palo Alto Networks PAN-OS and Prisma Access, Microsoft Azure networking, AWS networking. Not yet included: Google Cloud, Fortinet, Juniper.

Limitations

• Commands may be wrong or outdated. Verify against current vendor documentation and test in a lab.
• Quality depends on the references your retrieval system supplies.
• English only. Not a substitute for a qualified engineer or your change process.

License

This adapter is a derivative of Llama 3.1 and is distributed under the Llama 3.1 Community License (https://www.llama.com/llama3_1/license/). Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Use must also comply with Meta's Acceptable Use Policy.