Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey

arXiv:2605.01392v2 Announce Type: replace-cross Abstract: Recent advancements in Large Language Models (LLMs) have demonstrated significant potential across software engineering tasks, including software design, an area traditionally regarded as highly dependent on human expertise and judgment. However, limited research has examined how LLMs are used in software design, aswell as the associated benefits and drawbacks. This paper addresses this gap by empirically investigating how software developers use LLMs in software design. We conducted a mixed-methods study, combining a mining study of 291 developer-ChatGPT conversations shared on GitHub with a survey of 65 software practitioners. From the mined conversations, we identified nine categories of design tasks supported by ChatGPT, including architecture design, data model design, and the use of design patterns.We further characterize developer-ChatGPT interactions, showing that developers in the mined conversations primarily use ChatGPT for knowledge acquisition and designrelated code generation, with most tasks situated at the detailed design level. The survey participants reported seven key benefits, such as better technology selection and early detection of design flaws, and six limitations, including lengthy outputs, inexecutable or incorrect code, and dependence on project context. These findings provide an evidence-based characterization of current LLM use in software design from both open-source and practitioner perspectives.

Abstract:Recent advancements in Large Language Models (LLMs) have demonstrated significant potential across software engineering tasks, including software design, an area traditionally regarded as highly dependent on human expertise and judgment. However, limited research has examined how LLMs are used in software design, aswell as the associated benefits and drawbacks. This paper addresses this gap by empirically investigating how software developers use LLMs in software design. We conducted a mixed-methods study, combining a mining study of 291 developer-ChatGPT conversations shared on GitHub with a survey of 65 software practitioners. From the mined conversations, we identified nine categories of design tasks supported by ChatGPT, including architecture design, data model design, and the use of design this http URL further characterize developer-ChatGPT interactions, showing that developers in the mined conversations primarily use ChatGPT for knowledge acquisition and designrelated code generation, with most tasks situated at the detailed design level. The survey participants reported seven key benefits, such as better technology selection and early detection of design flaws, and six limitations, including lengthy outputs, inexecutable or incorrect code, and dependence on project context. These findings provide an evidence-based characterization of current LLM use in software design from both open-source and practitioner perspectives.

Software Engineering (cs.SE); Artificial Intelligence (cs.AI)

arXiv:2605.01392 [cs.SE]

(or arXiv:2605.01392v2 [cs.SE] for this version)

https://doi.org/10.48550/arXiv.2605.01392 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey

Related coverage