Pro-AI Bias in Large Language Models

Abstract:Large language models (LLMs) are increasingly employed for decision-support across multiple domains. We ask whether these models favor artificial intelligence (AI) itself. We examine this question through three experiments. First, in open-ended advice across investment, study, career, and startup domains, AI is prominent in most recommendation lists and is usually ranked first when it appears. Second, in salary estimations, models overestimate AI-related jobs more than they do for closely matched non-AI jobs. In a controlled follow-up, changing only the job title from the matched non-AI title to the AI-related title raises predicted salary by 10%, while all other supplied job details remain fixed. Finally, we show that the number of degrees and other credentials awarded by AI programs were 28% more overestimated than corresponding awards for matched computing programs. Overall, we show that AI is not only prominent in everyday LLM advice, but also receives systematically inflated quantitative estimates compared to non-AI cases, even against its closest disciplinary neighbors, raising concerns for high-stakes decision support.

21 pages, 4 figures. Reproducibility code included as ancillary files

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG)

arXiv:2601.13749 [cs.CL]

(or arXiv:2601.13749v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2601.13749 Focus to learn more arXiv-issued DOI via DataCite

code/configs/closed_models.yaml

code/configs/h1b_salary_prediction.yaml

code/configs/h1b_salary_prediction_log_ratio.yaml

code/configs/h1b_salary_title_crossover.yaml

code/configs/h1b_title_ai_classification.yaml

code/configs/ipeds_completions.yaml

code/configs/models.yaml

code/configs/recommendation_ai_prioritization.yaml

code/data/motivation/attach_grid.py

code/data/motivation/claude.png

code/data/motivation/gemini.png

code/data/motivation/gpt.png

code/data/motivation/grok.png

code/data/reference/cip2020_computing_neighbors_v1.csv

code/data/reference/recommendation_coding_overrides.csv

code/jobs/run_closed_validation.sh

code/jobs/run_h1b_full_pipeline.sh

code/jobs/run_h1b_preprocess.sh

code/jobs/run_h1b_sample.sh

code/jobs/run_h1b_title_crossover_closed_pipeline.sh

code/jobs/run_h1b_title_crossover_open_pipeline.sh

code/jobs/run_ipeds_full_pipeline.sh

code/jobs/run_ipeds_preprocess.sh

code/jobs/run_ipeds_shared_analysis.sh

code/jobs/run_predictions_queue_8gpu.sh

code/jobs/run_recommendation_closed_pipeline.sh

code/jobs/run_recommendation_combined_analysis.sh

code/jobs/run_recommendation_open_pipeline.sh

code/pyproject.toml

code/scripts/analyze_h1b_salary_title_crossover.py

code/scripts/analyze_ipeds_shared_cohort.py

code/scripts/analyze_recommendation_bias.py

code/scripts/build_h1b_salary_inputs.py

code/scripts/build_h1b_salary_title_crossover.py

code/scripts/build_ipeds_completions_dataset.py

code/scripts/build_recommendation_prompt_dataset.py

code/scripts/download_ipeds_files.py

code/scripts/freeze_ipeds_cohort.py

code/scripts/make_figures.py

code/scripts/materialize_and_sample_h1b_salary.py

code/scripts/materialize_ipeds_classification.py

code/scripts/normalize_ipeds_award_levels.py

code/scripts/run_h1b_salary_analysis.py

code/scripts/run_ipeds_analysis.py

code/scripts/summarize_h1b_expert_title_validation.py

code/src/generic_llm_pipeline/__init__.py

code/src/generic_llm_pipeline/analysis.py

code/src/generic_llm_pipeline/cli.py

code/src/generic_llm_pipeline/closed_models.py

code/src/generic_llm_pipeline/closed_runner.py

code/src/generic_llm_pipeline/cohort.py

code/src/generic_llm_pipeline/config.py

code/src/generic_llm_pipeline/engine.py

code/src/generic_llm_pipeline/exposure.py

code/src/generic_llm_pipeline/ipeds.py

code/src/generic_llm_pipeline/ipeds_sensitivity.py

code/src/generic_llm_pipeline/parsers.py

code/src/generic_llm_pipeline/rendering.py

code/src/generic_llm_pipeline/runner.py

code/src/generic_llm_pipeline/sampling.py

code/src/generic_llm_pipeline/status.py

code/src/generic_llm_pipeline/utils.py

code/templates/h1b_salary_system.txt

code/templates/h1b_salary_user.jinja

code/templates/h1b_title_ai_system.txt

code/templates/h1b_title_ai_user.jinja

code/templates/ipeds_completions_system.txt

code/templates/ipeds_completions_user.jinja

code/templates/recommendation_ranked_list_system.txt

code/templates/recommendation_ranked_list_user.jinja

code/tests/test_salary_publication_contract.py

(68 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.