Do Evidence-Reading Diagnostics Improve Interface Selection in Small LLM Recommenders?

Title:Do Evidence-Reading Diagnostics Improve Interface Selection in Small LLM Recommenders?

Abstract:Behavioral tests measure how a language model reads evidence. We ask whether those measurements help choose a recommendation interface. We evaluate six small instruction-tuned checkpoints across four recommendation domains with chronological evaluation and 3,426 evaluation users. Each request ranks eight candidates. A baseline selector chooses among history-only prompting, prompting with collaborative evidence, and score fusion. It uses observable features and six stability prompts that vary wording and candidate order. An augmented selector adds features from six evidence-reading prompts that ask the model to compare support counts. An interface chosen once on development (validation) data for each domain and checkpoint scores 0.5524 NDCG@5, compared with 0.5447 for the baseline selector and 0.5428 for the augmented selector. Adding the diagnostic features changes NDCG@5 by -0.0019 (95% interval [-0.0046, 0.0004]). The interval includes zero, and its upper bound is below the analysis plan's 0.005 improvement target. Matching the selectors' hyperparameters also leaves the interval upper bound below that target. Evidence from retrieved similar users improves prompting by 0.0999 NDCG@5 over a control using randomly selected users matched for activity. The evidence-reading tests also reveal answer-position and tie-response biases. These results concern the tested selectors and candidate sets. They illustrate why diagnostic measurements should be evaluated by whether they improve recommendation choices beyond existing features and a fixed interface.

Information Retrieval (cs.IR); Machine Learning (cs.LG)

arXiv:2609.37472 [cs.IR]

(or arXiv:2609.37472v1 [cs.IR] for this version)

https://doi.org/10.48550/arXiv.2609.37472 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

statistical_supplement/ACTUAL_TIE_ACCURACY.csv

statistical_supplement/ADDITIONAL_DIAGNOSTICS.json

statistical_supplement/CHECKPOINT_REVISIONS.json

statistical_supplement/DECISION_CHANGE_CHECK.json

statistical_supplement/DEVELOPMENT_FIXED_ARM_SELECTION.csv

statistical_supplement/EXTENSION_ANALYSIS_PLAN_FROZEN.md

statistical_supplement/FINAL_REPLAY_CHECKS.json

statistical_supplement/INTERVAL_SCHEME_CHECK.json

statistical_supplement/MANIFEST.json

statistical_supplement/METHOD_INTEGRATION_CHECK.json

statistical_supplement/NEGATIVE_RULE_CHECK.json

statistical_supplement/PRIMARY_REPLAY_RECEIPT.json

statistical_supplement/PROMPT_SPECIFICATION.md

statistical_supplement/PROTOCOL_PROVENANCE_STATEMENT.md

statistical_supplement/RANDOM_PROMPT_CONTROL.csv

statistical_supplement/README.md

statistical_supplement/REFERENCE_AGGREGATE_CHECK_RECEIPT.json

statistical_supplement/REFERENCE_COMPARISON_RESULTS.json

statistical_supplement/REFERENCE_RESULTS_GUIDE.md

statistical_supplement/REPLAY_PARAMETERS.json

statistical_supplement/REPORTING_CORRECTION.md

statistical_supplement/REPORTING_SUMMARIES.json

statistical_supplement/REPRODUCIBILITY_DETAILS.md

statistical_supplement/REQUEST_DIFFICULTY_MAPPING_AMENDMENT.md

statistical_supplement/ROUTER_FEATURE_IMPLEMENTATION.md

statistical_supplement/SECONDARY_CHECK_RECEIPT.json

statistical_supplement/SECONDARY_DESCRIPTIVE_RESULTS.json

statistical_supplement/SECONDARY_INFERENCE.json

statistical_supplement/SOURCE_CONSTRUCTION_RECORD.json

statistical_supplement/SUMMARY.json

statistical_supplement/VERIFICATION_REPORT.json

statistical_supplement/check_reference_aggregates.py

statistical_supplement/check_reporting_summaries.py

statistical_supplement/controls_and_matched/ENVIRONMENT_SUMMARY.json

statistical_supplement/controls_and_matched/FULL_PRECISION_TARGETS.json

statistical_supplement/controls_and_matched/MINIMIZED_REPLAY_RECEIPT_ADDITIONAL_ENVIRONMENT.json

statistical_supplement/controls_and_matched/MINIMIZED_REPLAY_RECEIPT_LINUX.json

statistical_supplement/controls_and_matched/MINIMIZED_REPLAY_RECEIPT_LOCAL.json

statistical_supplement/controls_and_matched/MINIMIZED_USER_METRICS.json

statistical_supplement/controls_and_matched/README.md

statistical_supplement/controls_and_matched/REPLAY_REPORT.md

statistical_supplement/controls_and_matched/replay_minimized.py

statistical_supplement/fixed_policy_user_metrics.json

statistical_supplement/hard_audit_confirmation.json

statistical_supplement/reading_feature_moments.json

statistical_supplement/replay_additional_diagnostics.py

statistical_supplement/replay_secondary_statistics.py

statistical_supplement/replay_statistics.py

statistical_supplement/request_difficulty.py

statistical_supplement/secondary_user_metrics.json

statistical_supplement/timestamp_audit/LASTFM_TIMESTAMP_AUDIT.json

statistical_supplement/timestamp_audit/README.md

statistical_supplement/timestamp_audit/SOURCE_DOCUMENTATION_EXCERPT.txt

statistical_supplement/timestamp_audit/SOURCE_HASHES.json

statistical_supplement/timestamp_audit/TIMESTAMP_AGGREGATE_CHECK_RECEIPT.json

statistical_supplement/timestamp_audit/TIMESTAMP_REPORT.md

statistical_supplement/timestamp_audit/audit_lastfm_timestamps.py

statistical_supplement/timestamp_audit/check_timestamp_aggregates.py

statistical_supplement/timestamp_sensitivity/ALIGNMENT_VERIFICATION_RECEIPT.json

statistical_supplement/timestamp_sensitivity/LASTFM_LATE_POSITIVE_MASK.json

statistical_supplement/timestamp_sensitivity/README.md

statistical_supplement/timestamp_sensitivity/SENSITIVITY_REPLAY_RECEIPT.json

statistical_supplement/timestamp_sensitivity/SOURCE_HASHES.json

statistical_supplement/timestamp_sensitivity/TIMESTAMP_SENSITIVITY.json

statistical_supplement/timestamp_sensitivity/TIMESTAMP_SENSITIVITY_REPORT.md

statistical_supplement/timestamp_sensitivity/replay_timestamp_sensitivity.py

statistical_supplement/upstream_reference_analysis.py

statistical_supplement/user_level_effects.json

(64 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.