Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges

Abstract:The Segment Anything Model (SAM) has transformed image segmentation by introducing a prompt-based paradigm that enables strong zero-shot generalization. In this framework, prompts serve as a semantic interface between human intent and machine perception, making prompt engineering a central factor in model performance. Despite its importance, prompt engineering within SAM and its variants has not yet been systematically reviewed in the literature. This survey addresses that gap by providing a structured and comprehensive overview of prompt engineering techniques developed for SAM and its rapidly growing ecosystem. We introduce a hierarchical taxonomy that organizes methods into geometric prompts, textual semantic prompts, and multimodal fusion prompts, and analyze how these categories reflect different design principles and application goals. In addition, we examine the transition from manually crafted prompts to more advanced, automated approaches based on detector outputs, prototype learning, reinforcement learning, and vision-language models. Beyond categorizing existing work, we trace how prompt engineering has enabled SAM to generalize across domains such as medical imaging, remote sensing, industrial inspection, and anomaly detection. We further identify key challenges---including prompt sensitivity, cross-modal misalignment, and computational inefficiency---and highlight promising research directions such as causal prompt reasoning, collaborative multi-agent prompting, and diffusion-based progressive refinement. By consolidating these developments into a unified perspective, our survey provides a timely reference for understanding the role of prompt engineering in segmentation foundation models and lays the groundwork for future advances in this evolving field.

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

arXiv:2507.09562 [cs.CV]

(or arXiv:2507.09562v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2507.09562 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.