A Japanese NLI checker for the space between generation and trust (sealed 0.961, zero-shot baselines 0.79–0.81)
TL;DR — We released ja-nli-v1, a small Japanese natural language inference model: given a premise and a hypothesis, it returns probabilities for entailment / neutral / contradiction. On a sealed test of 5,000 pairs it scores accuracy and Macro F1 of 0.9612; on a 600-pair human-verified set, 0.9567, with entailment↔contradiction confusions at 3 (0.5%). Two multilingual NLI models used zero-shot…
We released ja-nli-v1, a small Japanese natural language inference (NLI) model that assesses relationships between a premise and hypothesis. On a sealed test of 5,000 pairs, the model achieved an accuracy of 0.9612 and a Macro F1 score of 0.9612. This outperformed zero-shot baselines of Japanese models, which scored between 0.79 and 0.81 on the same test set. In a human-verified set of 600 pairs, the model achieved 0.9567 accuracy with entailment/contradiction confusion at 3 (0.5%).
The model relies on synthetic data built from public-domain licensed government documents and Japanese OSS documentation under MIT/Apache-2.0 licenses. It was trained without using existing NLI datasets. The model is licensed under Apache 2.0 and available for use via Transformers. It achieves ~109 ms per pair on a CPU.
The model includes diagnostics to ensure it is actually reading the premise, such as accuracy dropping to ~0.38 when the premise is shuffled to a completely unrelated text. It also performs well on a hypothesis-only probe, scoring ~0.63 on the human-verified set. Long premises are truncated after 1024 tokens, so for longer texts, the premise should be chunked. The model was not trained on other Japanese NLI benchmarks, so performance on those should be considered separately.
Rewritten by urgent.news; may contain errors — check the original.