bespokelabsai/nimble

Public

Local typed decisions, contrastive data curation, and model evaluation.

  • Stars2.2k+66 tuần này
  • Forks168
  • Watchers9
  • Issues3
  • Ngày tạo18/09/2026
  • Cập nhật gần nhất14/10/2026
  • Ngôn ngữPython
  • Giấy phép—
README.md57.4 KBRaw

Bespoke Nimble

Data, Model, Recipe for an open Jev

Model · Updates · Capabilities · Quickstart · Methodology · Documentation and development · Citation

Introducing Bespoke Nimble. Serving reads the prompt once and then scores one answer token per question. Data curation changes one fact so that the correct answer flips. Training fine-tunes Qwen3.5-9B with LoRA on the answer tokens only. On 324 held-out examples, Bespoke-Nimble-9B matches 90.1% of the reference labels, compared with 66.4% for its base model and 93.2% for Jev 1.13.0.

Nimble takes some text and a schema, and makes typed decisions about the text. The schema is the list of questions to answer. Each question is either a choice from a list that you give or a true or false question. For each question, Nimble returns the answer it picked and the probability of each allowed answer.

Nimble makes each decision in one step and does not write out any reasoning first, so it is fast (blazing fast!). Nimble is inspired by the System One approach of TypeSafe's Jev. In this repository, we share our recipe for training such a model.

Note that we did not distill from Jev. The point of the repository is to show how to curate data, how to train, and to serve such a model, and encourage more research!

You can run Bespoke-Nimble-9B on a Mac with Apple Silicon or on a machine with an NVIDIA GPU.

Updates

  • On September 22, 2026, we fitted a temperature for Bespoke-Nimble-9B. With this temperature, the probabilities better match how often the answers are right. The model picks the same answers as before. Noul probabilities and Score values do change, so if you compare them with a threshold, test the threshold again. See Probability temperature and PR #7.
  • On September 20, 2026, we published the 2,676 training examples and the 324 held-out examples for Bespoke-Nimble-9B. We had left them out of the first release by mistake. See the dataset guide and PR #5.
  • On September 19, 2026, we raised the prompt limit of the hosted API to 8,192 tokens for each question. The model was trained on prompts of up to 2,048 tokens, so shorter prompts are better tested. See the SGLang deployment guide and PR #4.
  • On September 18, 2026, Edgar Dyck added a public benchmark suite. With it, you can run Bespoke-Nimble-9B and Jev on the same records from 13 public subsets with human labels. The public benchmarks guide has the steps and the results. See

Nội dung README thuộc về bespokelabsai và cộng đồng đóng góp, giấy phép Không có giấy phép. Nguồn: github.com/bespokelabsai/nimble