{
  "id": 21057008,
  "title": "A Japanese NLI checker for the space between generation and trust (sealed 0.961, zero-shot baselines 0.79–0.81)",
  "url": "https://urgent.news/2026/11/21/a-japanese-nli-checker-for-the-space-between-generation-and-trust",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-21T14:31:55.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/japanopendatalab/a-japanese-nli-checker-for-the-space-between-generation-and-trust-sealed-0961-zero-shot-5h52"
  },
  "original_language": "en",
  "account": "We released ja-nli-v1, a small Japanese natural language inference (NLI) model that assesses relationships between a premise and hypothesis. On a sealed test of 5,000 pairs, the model achieved an accuracy of 0.9612 and a Macro F1 score of 0.9612. This outperformed zero-shot baselines of Japanese models, which scored between 0.79 and 0.81 on the same test set. In a human-verified set of 600 pairs, the model achieved 0.9567 accuracy with entailment/contradiction confusion at 3 (0.5%).\n\nThe model relies on synthetic data built from public-domain licensed government documents and Japanese OSS documentation under MIT/Apache-2.0 licenses. It was trained without using existing NLI datasets. The model is licensed under Apache 2.0 and available for use via Transformers. It achieves ~109 ms per pair on a CPU.\n\nThe model includes diagnostics to ensure it is actually reading the premise, such as accuracy dropping to ~0.38 when the premise is shuffled to a completely unrelated text. It also performs well on a hypothesis-only probe, scoring ~0.63 on the human-verified set. Long premises are truncated after 1024 tokens, so for longer texts, the premise should be chunked. The model was not trained on other Japanese NLI benchmarks, so performance on those should be considered separately.",
  "summary": "TL;DR — We released ja-nli-v1, a small Japanese natural language inference model: given a premise and a hypothesis, it returns probabilities for entailment / neutral / contradiction. On a sealed test of 5,000 pairs it scores accuracy and Macro F1 of 0.9612; on a 600-pair human-verified set, 0.9567, with entailment↔contradiction confusions at 3 (0.5%). Two multilingual NLI models used zero-shot…",
  "key_points": [
    "ja-nli-v1 Japanese NLI model achieves 0.9612 accuracy on sealed test",
    "Outperforms zero-shot baselines with 0.79–0.81 accuracy",
    "Model trained on synthetic data from public-domain government documents"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}