DKnown-Jev: Fine-Tuning and Evaluation of a JEV-Style Model for Tasks Involving Normative Knowledge

Abstract:Tasks involving normative knowledge include concept interpretation, eligibility determination, procedure explanation, material requirements, and service-outcome assessment, all of which require rules and facts published, maintained, or recorded by authorized entities. Whether a candidate paragraph can answer a question therefore depends on applicability and answer completeness as well as topical relevance. We develop DKnown-Jev to classify the normative usability of candidate paragraphs into two classes. The 2B and 27B configurations are fine-tuned directly from the native Qwen3.5-2B and Qwen3.8-27B base models, respectively. The models combine question conditions, article metadata, and paragraph content, adopt the classes Answer Evidence (AE) and Non-answer Evidence (NE), and undergo domain adaptation using human-verified training data. Five configurations are evaluated on the same test set of 1,120 human-verified examples. DKnown-Jev 27B trained for one epoch achieves the highest accuracy of 98.75% and macro-F1 of 98.60%, exceeding Jev by 6.07 and 6.76 percentage points, respectively. DKnown-Jev 2B trained for one epoch achieves 92.86% accuracy and 92.18% macro-F1, the best aggregate results among the tested 2B configurations. Its gains over Jev are 0.18 and 0.33 percentage points on this test set, corresponding to only two additional correct predictions, which is insufficient to establish a stable advantage. The 27B one-epoch, 27B two-epoch, and 2B one-epoch configurations all exceed Jev on both metrics on this test set. Class-level analysis shows that the 27B two-epoch configuration has the highest AE recall, while the 27B one-epoch configuration has the highest NE recall. This study provides practical and empirical evidence for developing and applying candidate-scoring models to tasks involving normative knowledge.

Artificial Intelligence (cs.AI)

arXiv:2610.24358 [cs.AI]

(or arXiv:2610.24358v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2610.24358 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.