Developing NVIDIA Holoscan Applications with CLI, Skills, and AI Coding Agents

NVIDIA Holoscan is a platform for building real-time AI applications at the edge, from medical imaging to robotics. HoloHub is its companion repository: a growing collection of reference applications and components that demonstrate what’s possible.

We wanted to explore how a general-purpose coding agent could use the same examples, documentation, and development tools available to an engineer in an actual development task.

In this post, we walk through building a real-time endoscopic tool segmentation application using an AI coding agent. HoloHub examples and documentation provide implementation patterns, and the development skills guide the agent through the HoloHub development process.

The Holoscan CLI, invoked via ./holohub wrapper, provides the shared execution interface. The agent can discover development operations through the CLI, while the engineer can inspect and repeat the same commands.

The workflow is agent-agnostic; in this example, we used Codex with GPT-5.6 sol max mode, and the agent processing times mentioned were approximate.

Figure 1. AI-assisted development and review workflow

The overall objective is an end-to-end endoscopic tool segmentation application: real-time inference with a live visualization of the segmentation masks, along with statistical analysis rendering.

We reused the existing MONAI endoscopic tool segmentation model and a Holoscan sample video, and confirmed that the existing application monai_endoscopic_tool_seg worked locally. We focused on a new application reusing the deep learning segmentation pipeline, and adding comprehensive visualization, runtime telemetry and repeatable benchmarking.

HoloHub repository with documentation in a progressive disclosure pattern via the agents.md

The next sections show how the pieces work together in an engineer-guided, agentic development workflow.

Instead of trying to build the entire application with a single prompt, developers should decompose the ultimate objective into smaller, verifiable engineering iterations guided by uncertainty and evidence. This approach ensures that design choices are reviewed in a timely manner.

Is the development environment configured correctly to run a similar existing application locally?

Can the existing model and video run in a separate end-to-end application?

Does the visualization present meaningful information?

Can latency be measured repeatedly?

Can rendering throughput be improved without feature regressions?

Each iteration produces reviewable code, outputs, and tests, which inform the prompts and design choices for the next iteration.

The first prompt defined the outcome while constraining model and data reuse.

That left implementation choices to the agent while keeping model reuse, visual evidence, and weight integrity explicit.

The agents collected information from the designated sources: read the app lifecycle skill, nearby HoloHub examples, project metadata, and CLI documentation.

Dry-ran and invoked ./holohub create to generate and register the standard scaffold

Built and ran the application via ./holohub run with the application metadata defined by the CLI

The resulting application connected video replay, preprocessing, TensorRT inference, the SDK segmentation postprocessor, telemetry, and HoloViz. Inference and mask postprocessing ran for every replayed frame. The overlay reported frame-derived measurements.

Figure 2. Endoscopic tool segmentation dashboard

The agentic processing time was 40 minutes. The developer could review the live app through the same CLI used by the agent:

After reviewing the implementation, the visual outputs, and the test cases, we confirmed that the reused model and sample video worked in the new application. Visual review also showed that the overlay needed clearer measurements, so we proceed to define the next iteration.

In response, the agent revised the dynamic measurements and screenshot readability, then made visual review and benchmarking explicit application modes. The app was implemented to have three named application modes:

Table 1. The three execution modes available via ./holohub run

Benchmark mode used Holoscan Data Flow Tracking for the configured path from the video replayer through preprocessing, inference, telemetry, offscreen HoloViz, and the rendered-frame sink. It effectively reused the ideas presented in the existing holoscan flow benchmarking module. The agentic processing time was 20 minutes.

Figure 3. Performance benchmark of the endoscopic application, comparing application-path latency over 300 tracked frames

The revised benchmark provided more repeatable measurements. With that baseline available, the next iteration could investigate performance without relying on visual checks alone.

In response, the agent confirmed that inference still ran on every frame. It considered reusing masks across neighboring frames, which could avoid some inference work but would require a policy for deciding when stale output is acceptable. For this engineering iteration, it kept inference on every frame and first removed lower-risk dashboard overhead:

Reused HoloViz input specifications and static coordinate tensors while refreshing text and dynamic geometry for every frame.

Queued the current 10-value GPU-to-host telemetry copy asynchronously in two pinned buffers and rendered with the previous completed values.

We compared the first iteration’s implementation with the final one on the same test system. The optimized version was faster in all five measured trials. The agentic processing time was 30 minutes.

Table 2. Performance comparison between the baseline and optimized endoscopy dashboard implementations

Figure 4. Final performance comparison between the baseline and optimized versions of the endoscopy dashboard

After the three engineering iterations, the developer gave the agent a separate handoff prompt.

To finalize the progress, we reran the application and the tests, verified the output figures and headless test results. The implementation and benchmark evidence were retained along with the commit hash and dependency versions via ./holohub env-check and ./holohub env-info.

The development iterations show what the workflow produces. To better understand the impact of the CLI, skills and documentation, we next compare the same development task under different combinations of these resources.

We compared resource usage using the same coding agent and sandbox environment across identical settings (note that the study was last conducted on Aug. 1, using Codex 0.146.0 with GPT-5.6 Sol at max reasoning effort). All evaluations were based on the single prompt used in the first iteration (with the skill name removed if unavailable) with the goal of creating a new application.

The agentic processing time was 40 minutes, with a total cost of 11M tokens.

The agents were given agents.md including references to CLI usage guide and docs, as well as the HoloHub codebase. No HoloHub development skills were provided.

The agentic processing time was 65 minutes, with a total cost of 20M tokens. While the workflow resulted in a complete application that achieved the implementation goal, the process was less efficient. The agents correctly located similar apps within the HoloHub repository, but tended to use generic Bash tools that required more trial-and-error probing of the development environment. For instance, they often applied generic linting tools before CLI-based linting, or attempted to install Python dependencies and run inference scripts directly on the host (should have consistently worked in a container).

The agents were given the HoloHub codebase but without the agents.md and explicit guidance on the CLI usage. No HoloHub development skills were provided.

The agentic processing time was 40min, with a total cost of 15M tokens.

No CLI guidance was provided, the coding agents still gained understanding of the CLI (from the overall codebase examples) and used it as the main dev tooling. However the coding quality is suboptimal compared to the other two settings:

Existing optimized Holoscan operators such as TensorRT inference and format converter were ignored in the implementation, as a result this version is 2.6x slower then the other two settings

While follow-up prompts could potentially address these issues, the combination of CLI, skills, and documentation/examples provided the best developer experience with the lowest resource overhead.

Table 3. Agentic processing time and token usage across three scaffolding configurations

This post follows a small, verifiable development loop. The result is an engineering prototype built on an existing model, sample video, and Holoscan components. It runs end to end, provides several application modes and automated tests, preserves the model weights, and records reproducible benchmark evidence.

The main takeaway is the development loop shared by the developer and the agents: ./holohub supplies consistent operations, the skill encodes project-specific sequences and checks, and examples and documentation provide engineering context. The agent and the developer use the same CLI commands. The developer can focus on defining the objectives, setting the constraints, evaluating trade-offs, and deciding whether the evidence is sufficient.

HoloHub: https://github.com/nvidia-holoscan/holohub/tree/holoscan-sdk-4.5.0

Holoscan CLI: https://github.com/nvidia-holoscan/holoscan-cli/tree/v4.5.0

HoloHub skills source: https://github.com/nvidia-holoscan/holohub/tree/main/skills

HoloHub skills at Nvidia catalog: https://build.nvidia.com/skills?filters=library%3Alibrary_holoscan&q=holohub