Optimize CUDA kernels with agents that benchmark and profile

Tools

custom-built or unnamed agent

Overview

Contents

Workflow

Contents How it works Setup Run Examples Results and limits Workflow graph An agentic CUDA kernel optimizer that turns workload descriptions into GPU implementations through an automated cycle of code generation, correctness checks, benchmarking, and refinement. Powered by LangGraph, the agent explores kernel implementations and launch configurations, queries GPU properties, and can research NVIDIA documentation for optimization guidance and inspect Nsight Compute counters to inform its next experiment. Each experiment is recorded, and the fastest validated implementation is retained. The model can change both kernel code and per-case launch configurations. A standalone C++ harness compiles kernels with NVRTC, launches them through the CUDA Driver API, and saves outputs. Python handles comparison and candidate selection. How it works Load or generate a signature, input cases, reference kernel, and initial kernel. Run the reference and evaluate the initial implementation. Propose a change, compile it, compare outputs with NumPy, and measure kernel latency. Feed results back into the next attempt; repair invalid candidates within the iteration budget. Save the fastest validated candidate, execution history, and a timing heatmap. Every case must pass validation. Ranking uses the geometric mean of latency across performance cases; small correctness cases do not affect the score. Timing defaults to 10 warmup launches and 100 measured launches using CUDA events. Compilation time and profiler replay timings are excluded from ranking. Setup Developed on Windows with an RTX 3060 Laptop GPU. Requires Python 3.12+, an NVIDIA GPU and compatible CUDA Toolkit/driver, CMake 3.24+, a C++17 compiler, and an OpenAI API key. The build commands below use Visual Studio 2026 with the C++ tools installed. From the repository root: Create a .env file in the repository root: Run The default model is gpt-5-mini with medium reasoning effort. API usage is billed to your account. Use -h for all options, or --config optimizer_agent/example.json for the included configuration example. Supply your own workload components with --signature, --reference, --initial-kernel, and --input-cases. Omitted components are inferred or generated. To continue from an earlier run using its saved inputs: This example assumes the earlier run generated reference.cu; supplied references are saved as supplied-reference.cu. Input manifests retain paths to their binary data, so keep those files available. Optional flags: --use-nsight: profile a performance case after each valid candidate and expose profiler tools to the model. Requires Nsight Compute and permission to access GPU performance counters. --nvidia-research: retrieve NVIDIA guidance before generating kernels. Examples Float32 GEMM on an RTX 3060 Laptop GPU, with NVIDIA research and Nsight Compute enabled. Results and limits Each session gets a directory under results/run-NNN/ containing kernel sources, requests, input/output data, model/tool responses, history.json, and summary.json. Successful runs export best.cu, per-case replay requests, and heatmap.png/heatmap.svg. Nsight reports are saved when profiling is enabled. This is an experimental optimizer for individual kernels. Passing supplied cases does not prove general correctness, and a generated reference is not an independent correctness oracle. Improvements are workload-dependent; no comparison against cuBLAS or other vendor libraries is currently included. Keep the GPU otherwise idle when comparing timings. Generated input scripts execute locally as Python subprocesses without a sandbox. Generated CUDA kernels also run on the local GPU. Workflow graph The graph below is rendered from the compiled LangGraph workflow with Nsight profiling enabled. Dashed edges are conditional routes. Without --use-nsight, evaluation routes directly to the next attempt or finalization; NVIDIA research is skipped unless --nvidia-research is set.

Reported result

No measured performance improvements or success rates are reported in the source. The workflow produces a directory with kernel sources, history, and heatmaps for each run. Acceptance check: Verify that the generated kernel passes all input case validations and achieves lower latency than the initial kernel on a specific workload.

Human role

The original post does not say how a person stayed involved.

Limitations

Requires Python 3.12+, NVIDIA GPU, CUDA Toolkit, CMake, C++17 compiler, and OpenAI API key. Generated scripts and kernels execute locally without sandboxing, posing security risks. Passing supplied cases does not guarantee general correctness. No comparison against vendor libraries like cuBLAS is included. GPU must be idle during timing measurements.

Try it

I want to optimize a CUDA kernel for [replaceable workload description, e.g., 'Float32 GEMM with rectangular matrices']. Use the agentic-cuda-optimizer repository. First, set up the environment with Python 3.12+, CUDA Toolkit, and CMake. Create a .env file with my OpenAI API key. Run the optimizer with --description '[workload]' --max-iterations 5. Ensure the GPU is idle. After the run, inspect the results/run-NNN/ directory. Check history.json for validation failures and summary.json for the best kernel's latency. Compare best.cu against my initial kernel. Do not deploy the kernel until I manually verify its correctness on additional test cases. If errors occur, report the specific failure mode from history.json.

More cases · Software & development

• Update a code base with Copilot chat and agent mode
• Check code against user intent with review agents and tests
• Build a budgeting app from a single prompt
• Build a mental models framework with a multi-agent dev team
• Build a instant-opening markdown viewer for macOS