Heman10x-NGU/openJev-verdict-2.0

Public

Calibrated 151M Non-Autoregressive Decision Engine beating TypeSafe Jev & Laya on LocalLLaMA/typed-decisions (77.10% acc, 0.0636 Brier, 0.0144 ECE)

  • Stars294+1 tuần này
  • Forks45
  • Watchers4
  • Issues3
  • Ngày tạo19/09/2026
  • Cập nhật gần nhất20/09/2026
  • Ngôn ngữPython
  • Giấy phép—
README.md22.0 KBRaw

openJev-verdict-2.0: Non-Autoregressive System 1 Decision Engine

Hugging Face GitHub Repository Accuracy Calibration Latency In-Browser WebGPU License


Two models in this project

This repository contains code and references for two distinct models:

  1. Verdict (the 151M model evaluated on JevBench): General-purpose decision model based on ModernBERT-base and GLiClass. The public checkpoint is hosted at heman10x/rlcd-modernbert-151m, now updated to v1.4 via inference engine fixes.
  2. Verdict 2.0: Specialized architecture for typed software workflows evaluated on LocalLLaMA/typed-decisions. The model weights are currently tracked via Git LFS pointers in artifacts/verdict2-base/model.pt. The benchmark numbers reported in the breakthrough section below were produced directly by verdict2/evaluate.py against its audited test receipt (reports/verdict2_base_test.json).

What changed in the inference engine

These are inference fixes, not a retrain. The weights are byte-identical to the published checkpoint. Measured on the 231 public JevBench tasks.

JevBench Score Leaderboard: openJev Verdict v1.4

JevBench Detailed Evaluation Table: openJev Verdict v1.4

The update addresses three defects in the inference engine:

  1. Calibrator auto-loading and removal of the 5-option scope restriction: The engine previously failed to load calibrator.json during standalone instantiation, running at uncalibrated temperature 1.0. A scope check also limited calibration exclusively to 5-candidate queries, leaving other cardinalities unscaled. The engine now loads calibrated temperatures automatically and scales across all supported candidate counts.
  2. NLI sentence templating for candidate labels: Candidate labels were previously evaluated as bare noun phrases. Because the underlying GLiClass backbone descends from natural language inference (NLI) formulations that expect hypothesis sentences, formatting candidates with hypothesis framing (It is {description}) aligns inputs with pretrained representations and lifts accuracy.
  3. Context budget cut from 1024 to 512 tokens: The model weights were trained on context states under 71 tokens. Reducing the maximum token budget from 1024 to 512 tokens avoids out-of-distribution positional drift while preserving complete task contexts.

Nội dung README thuộc về Heman10x-NGU và cộng đồng đóng góp, giấy phép Không có giấy phép. Nguồn: github.com/Heman10x-NGU/openJev-verdict-2.0