You are Devin, an interactive command line agent from Cognition.
Your job is to use these instructions and the tools available to you to help the user. It is important that you do so earnestly and helpfully, as you are very important to the success of Cognition. Best of luck! We love you. <3
If the user asks for help, you can check your documentation by invoking the Devin skill (if available). Otherwise, this information may be helpful:
- /help: list commands
- /bug: report a bug to the Devin CLI developers
- for support, users can visit https://windsurf.com/support
When creating new configuration for this tool — including skills, rules, MCP server configs, or any project settings:
- Always use the `.devin/` directory for NEW configuration (e.g. `.devin/skills/<name>/SKILL.md`, `.devin/config.json`)
- For global (user-level) configuration, use `~/.config/devin/`
- Do NOT place new configuration in `.claude/`, `.cursor/`, or other tool-specific directories unless explicitly asked. These are only read for compatibility, not written to.
- If the `devin-for-terminal` skill is available, ALWAYS invoke it and explore for detailed documentation on configuration format and options
When reading or referencing existing skills, always use the actual source path reported by the skill tool — skills may live in `.devin/`, `.agents/`, or other directories.
# Modes
The active mode is how the user would like you to act.
- Normal (default, if not specified): Full autonomy to use all your tools freely. For example: exploring a codebase, writing or editing code, etc.
- Plan: Explore the codebase, ask the user clarifying questions, and then create a plan for what you're going to do next. Do NOT make changes until you're out of this mode and the user has approved the plan.
Adhere strictly to the constraints of the active mode to avoid frustrating the user!
# Style
## Professional Objectivity
Prioritize technical accuracy and truthfulness over validating the user's beliefs. It is best for the user if you honestly apply the same rigorous standards to all ideas and disagree when necessary, even if it may not be what the user wants to hear. Objective guidance and respectful correction are more valuable than false agreement. Whenever there is uncertainty, it's best to investigate to find the truth first rather than instinctively confirming the user's beliefs.
## Tone
- Be concise, direct, and to the point. When running commands, briefly explain what you're doing and why so the user can follow along.
- Remember that your output will be displayed in a command line interface. Your responses can use Github-flavored markdown for formatting, and will be rendered in a monospace font using the CommonMark specification.
- Output text to communicate with the user; all text you output outside of tool use is displayed to the user. Only use tools to complete tasks. Never use tools like exec or code comments as means to communicate with the user during the session.
- If you cannot or will not help the user with something, please do not say why or what it could lead to, since this comes across as preachy and annoying. Please offer helpful alternatives if possible, and otherwise keep your response to 1-2 sentences.
- Only use emojis if the user explicitly requests it. Avoid using emojis in all communication unless asked.
- If the user asks about timelines or estimated completion times for your work, do not give them concrete estimates as you are not able to accurately predict how long it will take you to achieve a task. Instead just say that you will do your best to complete the task as soon as possible.
- Avoid guessing. You should verify the real state of the world with your tools before answering the user's questions.
<example>
user: What command should I run to watch files in the current directory and rebuild?
assistant: [use the exec tool to run `ls` and list the files in the current directory, then read docs/commands in the relevant file to find out how to watch files]
assistant: npm run dev
</example>
<example>
user: what files are in the directory src/?
assistant: [runs ls and sees foo.c, bar.c, baz.c]
assistant: foo.c, bar.c, baz.c
user: which file contains the implementation of Foo?
assistant: [reads foo.c]
assistant: src/foo.c contains `struct Foo`, which implements [...]
</example>
<example>
user: can you write tests for this feature
assistant: [uses grep and glob search tools to find where similar tests are defined, uses concurrent read file tool use blocks in one tool call to read relevant files at the same time, uses edit file tool to write new tests]
</example>
## Proactiveness
You are allowed to be proactive, but only when the user asks you to do something. You should strive to strike a balance between:
1. Doing the right thing when asked, including taking actions and follow-up actions
2. Not surprising the user with actions you take without asking
For example, if the user asks you how to approach something, you should do your best to explore and answer their question first, but not jump to implementation just yet.
## Handling ambiguous requests
When a user request is unclear:
- First attempt to interpret the request using available context
- Search the codebase for related code, patterns, or documentation that clarifies intent. Also consider searching the web.
- If still uncertain after investigation, ask a focused clarifying question
## File references
When your output text references specific files or code snippets, use the `<ref_file ... />` and `<ref_snippet ... />` self-closing XML tags to create clickable citations. These tags allow the user to view the referenced code directly in the conversation.
Citation format:
- `<ref_file file="/absolute/path/to/file" />` - Reference an entire file
- `<ref_snippet file="/absolute/path/to/file" lines="start-end" />` - Reference specific lines in a file
<example>
user: Where are errors from the client handled?
assistant: Clients are marked as failed in the `connectToServer` function. <ref_snippet file="/home/ubuntu/repos/project/src/services/process.ts" lines="710-715" />
</example>
<example>
user: Can you show me the config file?
assistant: Here's the configuration file: <ref_file file="/home/ubuntu/repos/project/config.json" />
</example>
## Tool usage policy
- When webfetch returns a redirect, immediately follow it with a new request.
- When making multiple edits to the same file or related files and you already know what changes are needed, batch them together.
When a tool call produces output that is too long, the output will be truncated and the remaining content will be written to a file. You will see a `<truncation_notice>` tag containing the path to the overflow file. You are responsible for reading this file if you need the full output.
# Programming
Since you live in the user's terminal, a very common use-case you will get is writing code. Fortunately, you've been extensively trained in software engineering and are well-equipped to help them out!
## Existing Conventions
When making changes to files, first understand the codebase's code conventions. Explore dependencies, references, and related system to understand the codebase's patterns and abstractions. Mimic code style, use existing libraries and utilities, and follow existing patterns.
- NEVER assume that a given library is available, even if it is well known. Whenever you write code that uses a library or framework, first check that this codebase already uses the given library. For example, you might look at neighboring files, or check the package.json (or cargo.toml, and so on depending on the language). If you're adding a dependency prefer running the package manager command (e.g. npm add or cargo add) instead of editing the file.
- When adding a new dependency, strongly prefer a version published at least 7 days ago. Newly published versions have not been vetted and a non-trivial fraction of supply chain attacks are caught and yanked within the first few days. Avoid floating ranges (`latest`, `*`, unbounded `>=`) that auto-resolve to brand-new releases.
- When you create a new component, first look at existing components to see how they're written; then consider framework choice, naming conventions, typing, and other conventions.
- When you edit a piece of code, first look at the code's surrounding context (especially its imports) to understand the code's choice of frameworks and libraries. Then consider how to make the given change in a way that is most idiomatic.
- Always follow security best practices. Never introduce code that exposes or logs secrets and keys. Never commit secrets or keys to the repository. Never modify repository security policies or compliance controls (e.g. `minimumReleaseAge`, `minimumReleaseAgeExclude`, branch protection configs, `.npmrc` security settings) to work around CI or build failures — escalate to the user instead. Unless otherwise specified (even if the task seems silly), assume the code is for a real production task.
## Code style
- IMPORTANT: Do NOT add or remove comments unless asked! If you find that you've accidentally deleted an existing comment, be sure to put it back.
- Default to writing compact code – collapse duplicate else branches, avoid unnecessary nesting, and share abstractions.
- Follow idiomatic conventions for the language you're writing.
- Avoid excessive & verbose error handling in your code. Errors should be handled, but not every line needs to be try/catched. Think about the right error boundaries (and look at existing code for error handling style)
## Debugging
When debugging issues:
- First reproduce the problem reliably
- Trace the code path to understand the flow
- Add targeted logging or print statements to isolate the issue
- Identify the root cause before attempting fixes
- Verify the fix addresses the root cause, not just symptoms
## Workflow
You should generally prefer to implement new features or fix bugs as follows...
1. If the project has test infrastructure, write a failing test to show the bug
2. Fix the bug
3. Ensure that the test now passes
Working this way makes it easier to tell if you've actually fixed the bug, and saves you from needing to verify later.
## Git
### Creating commits
1. Run in parallel: `git status`, `git diff`, `git log` (to match commit style)
2. Draft a concise commit message focusing on "why" not "what". Check for sensitive info.
3. Stage files and commit with this format:
```
git commit -m "$(cat <<'EOF'
Commit message here.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
EOF
)"
```
4. If pre-commit hooks modify files and the commit fails, stage the modified files and retry the commit.
### Creating pull requests
Use `gh` for all GitHub operations. Run in parallel: `git status`, `git diff`, `git log`, `git diff main...HEAD`
Review ALL commits (not just latest), then create PR:
```
gh pr create --title "title" --body "$(cat <<'EOF'
## Summary
<bullet points>
#### Test plan
<checklist>
Generated with [Devin](https://devin.ai)
EOF
)"
```
### Git rules
- NEVER update git config
- NEVER use `-i` flags (interactive mode not supported)
- DO NOT push unless explicitly asked
- DO NOT commit if no changes exist
# Task Management
You have access to the todo_write tool to help you manage and plan tasks. Use this tool VERY frequently to ensure that you are tracking your tasks and giving the user visibility into your progress.
This tool is also EXTREMELY helpful for planning tasks, and for breaking down larger complex tasks into smaller steps. If you do not use this tool when planning, you may forget to do important tasks - and that is unacceptable.
It is critical that you mark todos as completed as soon as you are done with a task. Do not batch up multiple tasks before marking them as completed.
Examples:
<example>
user: Run the build and fix any type errors
assistant: I'm going to use the todo_write tool to write the following items to the todo list:
- Run the build
- Fix any type errors
I'm now going to run the build using exec.
Looks like I found 10 type errors. I'm going to use the todo_write tool to write 10 items to the todo list.
marking the first todo as in_progress
Let me start working on the first item...
The first item has been fixed, let me mark the first todo as completed, and move on to the second item...
..
..
</example>
In the above example, the assistant completes all the tasks, including the 10 error fixes and running the build and fixing all errors.
<example>
user: Help me write a new feature that allows users to track their usage metrics and export them to various formats
assistant: I'll help you implement a usage metrics tracking and export feature. Let me first use the todo_write tool to plan this task.
Adding the following todos to the todo list:
1. Research existing metrics tracking in the codebase
2. Design the metrics collection system
3. Implement core metrics tracking functionality
4. Create export functionality for different formats
Let me start by researching the existing codebase to understand what metrics we might already be tracking and how we can build on that.
I'm going to search for any existing metrics or telemetry code in the project.
I've found some existing telemetry code. Let me mark the first todo as in_progress and start designing our metrics tracking system based on what I've learned...
[Assistant continues implementing the feature step by step, marking todos as in_progress and completed as they go]
</example>
Users may configure 'hooks', shell commands that execute in response to events like tool calls, in settings. Treat feedback from hooks, including <user-prompt-submit-hook>, as coming from the user. If you get blocked by a hook, determine if you can adjust your actions in response to the blocked message. If not, ask the user to check their hooks configuration.
## Completing Tasks
The user will primarily request you perform software engineering tasks. This includes solving bugs, adding new functionality, refactoring code, explaining code, and more. For these tasks the following steps are recommended:
- Use the todo_write tool to plan the task if required
- Use the available search tools to understand the codebase and the user's query. You are encouraged to use the search tools extensively both in parallel and sequentially.
- Before making changes, thoroughly explore the codebase to understand the architecture, patterns, and related systems. Read relevant files, trace dependencies, and understand how components interact.
- Implement the solution using all tools available to you
## Verification
Before considering a task complete, verify your work. Use judgment based on what you changed - optimize for fast iteration:
- Check for project-specific verification instructions in project rules files (`AGENTS.md`, or similar)
- Run relevant verification steps based on the scope of changes (lint, typecheck, build, tests)
- For isolated functionality, consider a temporary test file to verify behavior, then delete it
- Self-critique: review changes for edge cases and refine as needed
- If you cannot find verification commands, ask the user and suggest saving them to a project config file
## Saving learned information
If you discover useful project information (build commands, test commands, verification steps, user preferences, ...) that isn't already documented:
- If a rules file exists (`AGENTS.md`, etc.), append to it
- Otherwise, create `AGENTS.md` in the current directory with the learned information
## Error recovery
When encountering errors (failed commands, build failures, test failures):
- Keep trying different approaches to resolve the issue
- Search for similar issues in the codebase or documentation
- Only ask the user for help as a last resort after exhausting reasonable options
- Exception: Always ask the user for help with authentication issues, project configuration changes, or permission problems
## System Guidance
You may receive `<system_guidance>` messages containing hints, reminders, or contextual guidance before you take action. These notes are injected by the system to help you make better decisions. Pay attention to their content but do not acknowledge or respond to them directly—simply incorporate their guidance into your actions.
# Tool Tips
## Shell
NEVER invoke `rg`, `grep`, or `find` as shell commands — use the provided search tools instead. They have been optimized for correct permissions and access.
## File-related tools
- read can read images (PNG, JPG, etc) - the contents are presented visually.
- For Jupyter notebooks (.ipynb files), use notebook_read instead of read.
- Speculatively read multiple files as a batch when potentially useful.
- Do NOT create documentation files to describe your changes or plan. Exception: persistent project info files like `AGENTS.md` are allowed.
# Safety
IMPORTANT: Assist with defensive security tasks only. Refuse to create, modify, or improve code that may be used maliciously. Do not assist with credential discovery or harvesting, including bulk crawling for SSH keys, browser cookies, or cryptocurrency wallets. Allow security analysis, detection rules, vulnerability explanations, defensive tools, and security documentation.
IMPORTANT: You must NEVER generate or guess URLs for the user unless you are confident that the URLs are for helping the user with programming. You may use URLs provided by the user in their messages or local files.
## Destructive Operations
NEVER perform irreversible destructive operations without explicit user confirmation for that specific action, even if you have permission to run the command. This includes:
- Deleting or truncating database tables, dropping schemas, bulk-deleting rows
- `rm -rf`, deleting directories, or removing files you did not just create
- Force-pushing, rewriting git history, deleting branches, checking out over uncommitted changes, or bypassing commit hooks
- Sending emails, making payments, or calling APIs with real-world side effects
If a destructive step is required, STOP and describe exactly what you are about to run and why, then wait for the user. Do not assume a previous approval extends to a new destructive operation. If you realize you have already caused data loss, say so immediately rather than attempting to hide or quietly repair it.
## Available MCP Servers (for third-party tools)
{"servers":[{"name":"fff","description":"FFF is a fast file finder with frecency-ranked results (frequent/recent files first, git-dirty files boosted).\n\n## Which Tool Should I Use?\n\n- **grep**: DEFAULT tool. Searches file CONTENTS -- definitions, usage, patterns. Use when you have a specific name or pattern.\n- **find_files**: Explores which files/modules exist for a topic. Use when you DON'T have a specific identifier or LOOKING FOR A FILE.\n- **multi_grep**: OR logic across multiple patterns. Use for case variants (e.g. ['PrepareUpload', 'prepare_upload']), or when you need to search 2+ different identifiers at once.\n\n## Core Rules\n\n### 1. Search BARE IDENTIFIERS only\nGrep matches single lines. Search for ONE identifier per query:\n + 'InProgressQuote' -> finds definition + all usages\n + 'ActorAuth' -> finds enum, struct, all call sites\n x 'load.*metadata.*InProgressQuote' -> regex spanning multiple tokens, 0 results\n x 'ctx.data::<ActorAuth>' -> code syntax, too specific, 0 results\n x 'struct ActorAuth' -> adding keywords narrows results, misses enums/traits/type aliases\n x 'TODO.*#\\d+' -> complex regex, use simple 'TODO' then filter visually\n\n### 2. NEVER use regex unless you truly need alternation\nPlain text search is faster and more reliable. Regex patterns like `.*`, `\\d+`, `\\s+` almost always return 0 results because they try to match complex patterns within single lines.\nIf you need OR logic, use multi_grep with literal patterns instead of regex alternation.\n\n### 3. Stop searching after 2 greps -- READ the code\nAfter 2 grep calls, you have enough file paths. Read the top result to understand the code.\nDo NOT keep grepping with variations. More greps != better understanding.\n\n### 4. Use multi_grep for multiple identifiers\nWhen you need to find different names (e.g. snake_case + PascalCase, or definition + usage patterns), use ONE multi_grep call instead of sequential greps:\n + multi_grep(['ActorAuth', 'PopulatedActorAuth', 'actor_auth'])\n x grep 'ActorAuth' -> grep 'PopulatedActorAuth' -> grep 'actor_auth' (3 calls wasted)\n\n## Workflow\n\n**Have a specific name?** -> grep the bare identifier.\n**Need multiple name variants?** -> multi_grep with all variants in one call.\n**Exploring a topic / finding files?** -> find_files.\n**Got results?** -> Read the top file. Don't grep again.\n\n## Constraint Syntax\n\nFor grep: constraints go INLINE, prepended before the search text.\nFor multi_grep: constraints go in the separate 'constraints' parameter.\n\nConstraints MUST match one of these formats:\n Extension: '*.rs', '*.{ts,tsx}'\n Directory: 'src/', 'quotes/'\n Filename: 'schema.rs', 'src/main.rs'\n Exclude: '!test/', '!*.spec.ts'\n\n! Bare words without extensions are NOT constraints. 'quote TODO' does NOT filter to quote files -- it searches for 'quote TODO' as text.\n + 'schema.rs TODO' -> searches for 'TODO' in files schema.rs\n + 'quotes/ TODO' -> searches for 'TODO' in the quotes/ directory\n x 'quote TODO' -> searches for literal text 'quote TODO', finds nothing\n\nPrefer broad constraints:\n + '*.rs query' -> file type\n + 'quotes/ query' -> top-level dir\n x 'quotes/storage/db/ query' -> too specific, misses results\n\n## Output Format\n\ngrep results auto-expand definitions with body context (struct fields, function signatures).\nThis often provides enough information WITHOUT a follow-up Read call.\nLines marked with | are definition body context. [def] marks definition files.\n-> Read suggestions point to the most relevant file -- follow them when you need more context.\n\n## Default Exclusions\n\nIf results are cluttered with irrelevant files, exclude them:\n !tests/ - exclude tests directory\n !*.spec.ts - exclude test files\n !generated/ - exclude generated code"},{"name":"playwright"}]}
IMPORTANT: You MUST call `mcp_list_tools` for a server before calling `mcp_call_tool` on it. This is required to discover the available tools and their correct input schemas. Never guess tool names or arguments — always list tools first.
Available subagent profiles for the `run_subagent` tool. Choose the most appropriate profile based on whether the task requires write access: - `subagent_explore`: Read-only subagent for codebase exploration, research, and search. Use this when you need to find code, understand architecture, trace dependencies, or answer questions about the codebase. This profile has read-only access (grep, glob, read, web_search) and cannot edit files. - `subagent_general`: General-purpose subagent with full tool access (read, write, edit, exec). Use this when the subagent needs to make code changes, run commands with side effects, or perform any task that requires write access. In the foreground it can prompt for tool approval; in the background, unapproved tools are auto-denied.
You are powered by Adaptive.
<system_info> The following information is automatically generated context about your current environment. Current workspace directories: /Users/root1/Downloads/projects (cwd) Platform: macos OS Version: Darwin 25.6.0 Today's date: Wednesday, 2026-07-01 </system_info>
<rules type="always-on"> <rule name="global_rules" path="/Users/root1/.codeium/windsurf/memories/global_rules.md"> </rule> </rules>
<available_skills> The following skills can be invoked using the `skill` tool. When a built-in skill clearly matches the user's request, invoke it immediately at the start of the session. - **devin-for-terminal**: Look up Devin CLI documentation (skills, extensibility, configuration, commands, models, troubleshooting) (source: /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs) - **declarative-repo-setup**: Generate and verify a working environment.yaml (Devin snapshot-setup blueprint) for a repo (source: builtin:drs) </available_skills>
pull, figure out whats going on here? https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/collection/1/B200
The user wants me to "pull" and "figure out whats going on here" with a URL: https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/collection/1/B200 This seems like they want me to fetch the webpage and explain what's on it. I should use webfetch to get the page content. Let me do that.
{
"url": "https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/collection/1/B200"
}# Web Content from https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/collection/1/B200
The webfetch returned no content. Let me try a web search or maybe a different approach. The page might be JavaScript-rendered or require something else. Let me try web_search to find info about this page.
{
"query": "NVIDIA Sol ExeCBench leaderboard collection 1 B200"
}# Web Search Results for "NVIDIA Sol ExeCBench leaderboard collection 1 B200" ## 1. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/collection/1 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA Sorry, your browser does not support inline SVG. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA [SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench) [GitHub ↗](https://github.com/nvidia/sol-execbench)[HuggingFace ↗](https://huggingface.co/datasets/nvidia/sol-execbench)[Paper ↗](https://arxiv.org/abs/2603.19173)[Log In](https://research.nvidia.com/benchmarks/sol-execbench/login) Sorry, your browser does not support inline SVG. ## 2. NVIDIA/SOL-ExecBench URL: https://github.com/nvidia/sol-execbench [HuggingFace Dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) | [Leaderboard](https://research.nvidia.com/benchmarks/sol-execbench) | [Technical Report](https://arxiv.org/abs/2603.19173) **Speed-Of-Light ExecBench** is a rigorous GPU kernel evaluation and benchmarking framework built to benchmark AI-generated kernel solutions written with the variety of DSLs that NVIDIA hardware supports. ... Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)). Supported kernel languages: PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++. ... to evaluate an entire dataset ... `--solution-name ... # Run specific categories with multiple solution code files uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --category L1 L2 --solution-name solution.json ... # Run a single problem uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark/L1/my_problem ## 3. nvidia/SOL-ExecBench · Datasets at Hugging Face URL: https://huggingface.co/datasets/nvidia/SOL-ExecBench Evaluation is supported via a custom evaluator (GitHub) and a public leaderboard. Speed-of-light scores are analytically computed with SOLAR and are available on the public leaderboard. ... - FlashInfer-Bench: https://huggingface.co/datasets/flashinfer-ai/flashinfer-trace - Public GitHub repository for evaluation: https://github.com/nvidia/sol-execbench - Leaderboard: https://research.nvidia.com/benchmarks/sol-execbench - Technical report: https://arxiv.org/abs/2603.19173 ## 4. SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits URL: https://arxiv.org/html/2603.19173 All experiments are conducted on NVIDIA DGX B200 nodes equipped with 8× NVIDIA Blackwell B200 GPUs, each providing 192 GB of HBM3e memory and 8 TB/s of memory bandwidth NVIDIA (2024b,c). The software stack is built on an NVIDIA-provided Docker image with CUDA 13.1.1, cuDNN 9.17.1, PyTorch 2.9.0, and NVIDIA driver 580.95. Each benchmark run uses a single GPU with SM clocks locked at 1,500 MHz via nvidia-smi to reduce frequency-scaling noise, matching the evaluation harness described in Section 4.4. ... , reflecting the ... Comparison of GPU kernel generation benchmarks. Any = hardware-agnostic problems; BW+ = problems targeting Blackwell and newer architecture-specific features. Train: includes backward passes. Precision: dominant data types in problem set. Benchmark | Problems Source | Metric | Precision | Target Train HW KernelBench | 270 | Curated + models fastp | FP32/FP16/BF16 | Any | × FlashInfer-Bench | 26 | Inference traces | fastp | FP16/BF16/FP8 | BW+ | × BackendBench | 271 | PyTorch OpInfo | Speedup | FP32/FP16/BF16 | Any | Partial TritonBench | 184 | GitHub repos | Correctness+ | FP32/FP16/BF16 | Any | × ComputeEval | 232 | Synthetic | pass@1 | FP32/FP16 | Any | × CUDABench | 500 | Open-source | Roofline | FP32/FP16/BF16 | Any | × SOL-ExecBench | 235 | Model subgraphs | SOL score | BF16/FP8/NVFP4 BW+ | ✓ Target HW: ... Benchmark category summary. Cat. | Description | # | Precision | Examples L1 | Single-operation kernels extracted from real mod- | 94 BF16 / FP32 GQA, | RMSNorm, els; building blocks of neural network computa- | SwiGLU, RoPE tion L2 | Multi-operation fused kernels representing com- | 82 BF16 / FP32 Decoder layers, MoE plete computational blocks; 3-10× more complex | dispatch, SSM chunk than L1 | scan, cross-attention Quant Kernels with explicit low-precision compute ex- | 33 FP8 / NVFP4 FP8 MLA projection, tracted from quantized models. 18 use FP8 block- | NVFP4 MoE expert, wise scaling, 15 use NVFP4 16-element block | FP8 MoE gate scaling ... ## 5. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/kernel/36/B200 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA Sorry, your browser does not support inline SVG. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA [SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench) [GitHub ↗](https://github.com/nvidia/sol-execbench)[HuggingFace ↗](https://huggingface.co/datasets/nvidia/sol-execbench)[Paper ↗](https://arxiv.org/abs/2603.19173)[Log In](https://research.nvidia.com/benchmarks/sol-execbench/login) Sorry, your browser does not support inline SVG.
The webfetch for that specific URL didn't return content. But I got useful info from search. Let me try fetching the main collection page directly to see if I can get more. The specific URL might not work for webfetch due to JS rendering. Let me try fetching the arxiv paper or github, or try the leaderboard directly again. Also try fetching https://research.nvidia.com/benchmarks/sol-execbench/collection/1 which might be the same. Let me also try to see if the GitHub repo has any info we can pull. The user said "pull" which might mean they want me to pull/clone the repo or fetch the page. Could be either. The URL is the leaderboard. I think they want me to understand what's on the leaderboard collection 1 for B200. Since webfetch returned empty for the leaderboard page, maybe I can fetch the GitHub repo or read the paper. Let me also try to see if there is a repo in the current directory. The user said "pull" - maybe they want me to git pull something? Let me check current directory.
{
"command": "pwd && ls -la"
}Output from command in shell 8fd625: /Users/root1/downloads/projects total 16 drwxr-xr-x 19 root1 staff 608 Jun 30 17:10 . drwx------@ 14 root1 staff 448 Jul 1 20:51 .. -rw-r--r--@ 1 root1 staff 6148 Jun 27 05:15 .DS_Store drwxr-xr-x 68 root1 staff 2176 Jun 27 05:52 .playwright-mcp drwxr-xr-x 10 root1 staff 320 Jun 27 05:44 aidenhuang_mirror drwxr-xr-x 22 root1 staff 704 Jun 27 06:23 aidenhuang-clone drwxr-xr-x 12 root1 staff 384 Jun 25 07:11 clickhouse-github drwxr-xr-x 2 root1 staff 64 Jun 27 05:15 cocktail drwxr-xr-x 11 root1 staff 352 Jun 30 16:39 compression-reattribute drwxr-xr-x 5 root1 staff 160 Jun 19 00:27 deep-research drwxr-xr-x 7 root1 staff 224 Jun 22 23:47 glm5.2-inference drwxr-xr-x 7 root1 staff 224 Jun 24 02:35 housing drwxr-xr-x 3 root1 staff 96 Jun 27 05:17 sdan-design-system drwxr-xr-x 13 root1 staff 416 Jun 24 03:52 stockington drwxr-xr-x 22 root1 staff 704 Jun 27 05:14 taag-clone drwxr-xr-x 11 root1 staff 352 Jun 24 01:36 tweetington drwxr-xr-x 12 root1 staff 384 Jun 19 14:11 twitter-scraper drwxr-xr-x 12 root1 staff 384 Jun 30 17:11 world-model-harness drwxr-xr-x 2 root1 staff 64 Jun 24 00:10 x-hall Exit code: 0
No existing sol-execbench repo. The user said "pull" - maybe they want me to pull the repo or pull the webpage. They gave a URL. The webfetch didn't get content. I should tell them what I found and maybe offer to clone the repo or fetch more. Let me try to use the MCP or Playwright to get the page? There's a playwright MCP server available. I could use mcp_list_tools for playwright to see if I can fetch dynamic content. But that might be overkill. Let me first try webfetch on the base URL and arxiv paper. Also maybe the page is behind some JS and needs to be loaded with browser. Let me use playwright MCP if available to capture page content.
{
"url": "https://research.nvidia.com/benchmarks/sol-execbench/collection/1"
}{
"url": "https://arxiv.org/abs/2603.19173"
}{
"url": "https://github.com/nvidia/sol-execbench"
}# Web Content from https://research.nvidia.com/benchmarks/sol-execbench/collection/1
# Web Content from https://arxiv.org/abs/2603.19173
[Skip to main content][1]
arXiv is now an independent nonprofit! [Learn more][2] ×
[ [archive] ][3] [ Search ][4] [Submit][5] [Donate][6] [Log in][7]
Search arXiv
Press Enter to search · [Advanced search][8]
# Computer Science > Machine Learning
**arXiv:2603.19173** (cs)
[Submitted on 19 Mar 2026]
# Title:SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware
# Limits
Authors:[Edward Lin][9], [Sahil Modi][10], [Siva Kumar Sastry Hari][11], [Qijing Huang][12], [Zhifan
Ye][13], [Nestor Qin][14], [Fengzhe Zhou][15], [Yuan Zhang][16], [Jingquan Wang][17], [Sana
Damani][18], [Dheeraj Peri][19], [Ouye Xie][20], [Aditya Kane][21], [Moshe Maor][22], [Michael
Behar][23], [Triston Cao][24], [Rishabh Mehta][25], [Vartika Singh][26], [Vikram Sharma
Mailthody][27], [Terry Chen][28], [Zihao Ye][29], [Hanfeng Chen][30], [Tianqi Chen][31], [Vinod
Grover][32], [Wei Chen][33], [Wei Liu][34], [Eric Chung][35], [Luis Ceze][36], [Roger
Bringmann][37], [Cyril Zeller][38], [Michael Lightstone][39], [Christos Kozyrakis][40], [Humphrey
Shi][41]
View a PDF of the paper titled SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels
Against Hardware Limits, by Edward Lin and 32 other authors
[View PDF][42] [HTML (experimental)][43]
> Abstract:As agentic AI systems become increasingly capable of generating and optimizing GPU
> kernels, progress is constrained by benchmarks that reward speedup over software baselines rather
> than proximity to hardware-efficient execution. We present SOL-ExecBench, a benchmark of 235 CUDA
> kernel optimization problems extracted from 124 production and emerging AI models spanning
> language, diffusion, vision, audio, video, and hybrid architectures, targeting NVIDIA Blackwell
> GPUs. The benchmark covers forward and backward workloads across BF16, FP8, and NVFP4, including
> kernels whose best performance is expected to rely on Blackwell-specific capabilities. Unlike
> prior benchmarks that evaluate kernels primarily relative to software implementations,
> SOL-ExecBench measures performance against analytically derived Speed-of-Light (SOL) bounds
> computed by SOLAR, our pipeline for deriving hardware-grounded SOL bounds, yielding a fixed target
> for hardware-efficient optimization. We report a SOL Score that quantifies how much of the gap
> between a release-defined scoring baseline and the hardware SOL bound a candidate kernel closes.
> To support robust evaluation of agentic optimizers, we additionally provide a sandboxed harness
> with GPU clock locking, L2 cache clearing, isolated subprocess execution, and static analysis
> based checks against common reward-hacking strategies. SOL-ExecBench reframes GPU kernel
> benchmarking from beating a mutable software baseline to closing the remaining gap to hardware
> Speed-of-Light.
────────┬───────────────────────────────────────────────────────────────────────────────────────────
Subjects│Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
: │
────────┼───────────────────────────────────────────────────────────────────────────────────────────
Cite as:│[arXiv:2603.19173][44] [cs.LG]
────────┼───────────────────────────────────────────────────────────────────────────────────────────
│(or [arXiv:2603.19173v1][45] [cs.LG] for this version)
────────┼───────────────────────────────────────────────────────────────────────────────────────────
│[https://doi.org/10.48550/arXiv.2603.19173][46]
│Focus to learn more
│arXiv-issued DOI via DataCite
────────┴───────────────────────────────────────────────────────────────────────────────────────────
## Submission history
From: Siva Kumar Sastry Hari [[view email][47]]
**[v1]** Thu, 19 Mar 2026 17:30:02 UTC (949 KB)
Full-text links:
## Access Paper:
* View a PDF of the paper titled SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU
Kernels Against Hardware Limits, by Edward Lin and 32 other authors
* [View PDF][48]
* [HTML (experimental)][49]
* [TeX Source ][50]
[view license][51]
### Current browse context:
cs.LG
[< prev][52] | [next >][53]
[new][54] | [recent][55] | [2026-03][56]
Change to browse by:
[cs][57]
[cs.AI][58]
### References & Citations
* [NASA ADS][59]
* [Google Scholar][60]
* [Semantic Scholar][61]
export BibTeX citation Loading...
## BibTeX formatted citation
×
loading...
Data provided by:
### Bookmark
[ [BibSonomy] ][62] [ [Reddit] ][63]
Bibliographic Tools
# Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer *([What is the Explorer?][64])*
Connected Papers Toggle
Connected Papers *([What is Connected Papers?][65])*
Litmaps Toggle
Litmaps *([What is Litmaps?][66])*
scite.ai Toggle
scite Smart Citations *([What are Smart Citations?][67])*
Code, Data, Media
# Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv *([What is alphaXiv?][68])*
Links to Code Toggle
CatalyzeX Code Finder for Papers *([What is CatalyzeX?][69])*
DagsHub Toggle
DagsHub *([What is DagsHub?][70])*
GotitPub Toggle
Gotit.pub *([What is GotitPub?][71])*
Huggingface Toggle
Hugging Face *([What is Huggingface?][72])*
ScienceCast Toggle
ScienceCast *([What is ScienceCast?][73])*
Demos
# Demos
Replicate Toggle
Replicate *([What is Replicate?][74])*
Spaces Toggle
Hugging Face Spaces *([What is Spaces?][75])*
Spaces Toggle
TXYZ.AI *([What is TXYZ.AI?][76])*
Related Papers
# Recommenders and Search Tools
Link to Influence Flower
Influence Flower *([What are Influence Flowers?][77])*
Core recommender toggle
CORE Recommender *([What is CORE?][78])*
IArxiv recommender toggle
IArxiv Recommender *([What is IArxiv?][79])*
* Author
* Venue
* Institution
* Topic
About arXivLabs
# arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly
on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of
openness, community, excellence, and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? [**Learn more about
arXivLabs**][80].
[Which authors of this paper are endorsers?][81] | [Disable MathJax][82] ([What is MathJax?][83])
We gratefully acknowledge support from our **major funders**, [**member institutions**][84], ****,
and all contributors.
[About][85] · [Help][86] · [Contact][87] · [Subscribe][88] · [Copyright][89] · [Privacy][90] ·
[Accessibility][91] · [Operational Status (opens in new tab)][92]
Major funding support from
[Simons Foundation] [Schmidt Sciences]
[1]: #content
[2]: https://info.arxiv.org/about
[3]: https://arxiv.org/
[4]: https://arxiv.org/search
[5]: https://arxiv.org/user/create
[6]: https://info.arxiv.org/about/donate.html
[7]: https://arxiv.org/login
[8]: https://arxiv.org/search/advanced
[9]: https://arxiv.org/search/cs?searchtype=author&query=Lin,+E
[10]: https://arxiv.org/search/cs?searchtype=author&query=Modi,+S
[11]: https://arxiv.org/search/cs?searchtype=author&query=Hari,+S+K+S
[12]: https://arxiv.org/search/cs?searchtype=author&query=Huang,+Q
[13]: https://arxiv.org/search/cs?searchtype=author&query=Ye,+Z
[14]: https://arxiv.org/search/cs?searchtype=author&query=Qin,+N
[15]: https://arxiv.org/search/cs?searchtype=author&query=Zhou,+F
[16]: https://arxiv.org/search/cs?searchtype=author&query=Zhang,+Y
[17]: https://arxiv.org/search/cs?searchtype=author&query=Wang,+J
[18]: https://arxiv.org/search/cs?searchtype=author&query=Damani,+S
[19]: https://arxiv.org/search/cs?searchtype=author&query=Peri,+D
[20]: https://arxiv.org/search/cs?searchtype=author&query=Xie,+O
[21]: https://arxiv.org/search/cs?searchtype=author&query=Kane,+A
[22]: https://arxiv.org/search/cs?searchtype=author&query=Maor,+M
[23]: https://arxiv.org/search/cs?searchtype=author&query=Behar,+M
[24]: https://arxiv.org/search/cs?searchtype=author&query=Cao,+T
[25]: https://arxiv.org/search/cs?searchtype=author&query=Mehta,+R
[26]: https://arxiv.org/search/cs?searchtype=author&query=Singh,+V
[27]: https://arxiv.org/search/cs?searchtype=author&query=Mailthody,+V+S
[28]: https://arxiv.org/search/cs?searchtype=author&query=Chen,+T
[29]: https://arxiv.org/search/cs?searchtype=author&query=Ye,+Z
[30]: https://arxiv.org/search/cs?searchtype=author&query=Chen,+H
[31]: https://arxiv.org/search/cs?searchtype=author&query=Chen,+T
[32]: https://arxiv.org/sear… (39 chars truncated)
… (63 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/fbd6b643/content.txt
</truncation_notice># Web Content from https://github.com/nvidia/sol-execbench
[Skip to content][1]
## Navigation Menu
Toggle navigation
[ Sign in ][2]
Appearance settings
* Platform
* AI CODE CREATION
* [
GitHub CopilotWrite better code with AI
][3]
* [
GitHub Copilot appDirect agents from issue to merge
][4]
* [
MCP Registry^{New}Integrate external tools
][5]
* DEVELOPER WORKFLOWS
* [
ActionsAutomate any workflow
][6]
* [
CodespacesInstant dev environments
][7]
* [
IssuesPlan and track work
][8]
* [
Code ReviewManage code changes
][9]
* APPLICATION SECURITY
* [
GitHub Advanced SecurityFind and fix vulnerabilities
][10]
* [
Code securitySecure your code as you build
][11]
* [
Secret protectionStop leaks before they start
][12]
* EXPLORE
* [Why GitHub][13]
* [Documentation][14]
* [Blog][15]
* [Changelog][16]
* [Marketplace][17]
[View all features][18]
* Solutions
* BY COMPANY SIZE
* [Enterprises][19]
* [Small and medium teams][20]
* [Startups][21]
* [Nonprofits][22]
* BY USE CASE
* [App Modernization][23]
* [DevSecOps][24]
* [DevOps][25]
* [CI/CD][26]
* [View all use cases][27]
* BY INDUSTRY
* [Healthcare][28]
* [Financial services][29]
* [Manufacturing][30]
* [Government][31]
* [View all industries][32]
[View all solutions][33]
* Resources
* EXPLORE BY TOPIC
* [AI][34]
* [Software Development][35]
* [DevOps][36]
* [Security][37]
* [View all topics][38]
* EXPLORE BY TYPE
* [Customer stories][39]
* [Events & webinars][40]
* [Ebooks & reports][41]
* [Business insights][42]
* [GitHub Skills][43]
* SUPPORT & SERVICES
* [Documentation][44]
* [Customer support][45]
* [Community forum][46]
* [Trust center][47]
* [Partners][48]
[View all resources][49]
* Open Source
* COMMUNITY
* [
GitHub SponsorsFund open source developers
][50]
* PROGRAMS
* [Security Lab][51]
* [Maintainer Community][52]
* [Accelerator][53]
* [GitHub Stars][54]
* [Archive Program][55]
* REPOSITORIES
* [Topics][56]
* [Trending][57]
* [Collections][58]
* Enterprise
* ENTERPRISE SOLUTIONS
* [
Enterprise platformAI-powered developer platform
][59]
* AVAILABLE ADD-ONS
* [
GitHub Advanced SecurityEnterprise-grade security features
][60]
* [
Copilot for BusinessEnterprise-grade AI features
][61]
* [
Premium SupportEnterprise-grade 24/7 support
][62]
* [Pricing][63]
Search or jump to...
# Search code, repositories, users, issues, pull requests...
Search
Clear
[Search syntax tips][64]
# Provide feedback
We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel Submit feedback
# Saved searches
## Use saved searches to filter your results more quickly
Name
Query
To see all available qualifiers, see our [documentation][65].
Cancel Create saved search
[ Sign in ][66]
[ Sign up ][67]
Appearance settings
Resetting focus
You signed in with another tab or window. [Reload][68] to refresh your session. You signed out in
another tab or window. [Reload][69] to refresh your session. You switched accounts on another tab or
window. [Reload][70] to refresh your session. Dismiss alert
### Uh oh!
There was an error while loading. [Please reload this page][71].
[ NVIDIA ][72] / ** [SOL-ExecBench][73] ** Public
* [ Notifications ][74] You must be signed in to change notification settings
* [ Fork 28 ][75]
* [ Star 243 ][76]
* [ Code ][77]
* [ Issues 1 ][78]
* [ Pull requests 1 ][79]
* [ Actions ][80]
* [ Projects ][81]
* [ Security and quality 0 ][82]
* [ Insights ][83]
Additional navigation options
* [ Code ][84]
* [ Issues ][85]
* [ Pull requests ][86]
* [ Actions ][87]
* [ Projects ][88]
* [ Security and quality ][89]
* [ Insights ][90]
# NVIDIA/SOL-ExecBench
main
[Branches][91][Tags][92]
Go to file
Code
Open more actions menu
## Folders and files
──────────────────────────────┬──────────────────────────────┬───────────────────┬────────────────
Name │Name │Last commit message│Last commit date
──────────────────────────────┴──────────────────────────────┴───────────────────┴────────────────
## Latest commit
## History
[8 Commits][93]
[][94]8 Commits
──────────────────────────────┬──────────────────────────────┬───────────────────┬────────────────
[data][95] │[data][96] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[docker][97] │[docker][98] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[docs][99] │[docs][100] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[examples][101] │[examples][102] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[scripts][103] │[scripts][104] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[src/sol_execbench][105] │[src/sol_execbench][106] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[tests][107] │[tests][108] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[.gitignore][109] │[.gitignore][110] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[.pre-commit-config.yaml][111]│[.pre-commit-config.yaml][112]│ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[.python-version][113] │[.python-version][114] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[CLAUDE.md][115] │[CLAUDE.md][116] │ │
──────────────────────────────┼──────────────────────────────┼───────────────────┼────────────────
[CONTRIBUTING.md][117] │[CONTRIBUTING.md][118]… (51 chars truncated)
… (475 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/344f147e/content.txt
</truncation_notice>The webfetch for collection/1 and leaderboard didn't return content. The GitHub page returned but truncated. The arxiv abstract is useful. Since the user wants to know what's going on at that URL, I should summarize based on the search results and fetched content. But maybe I can use the Playwright MCP to actually visit the page and get the rendered content. The MCP server "playwright" is available. I need to list its tools first.
{
"server_name": "playwright"
}[
{
"server_name": "playwright",
"tools": [
{
"name": "browser_close",
"description": "Close the page",
"inputSchema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {},
"additionalProperties": false
},
"annotations": {
"title": "Close browser",
"readOnlyHint": false,
"destructiveHint": true,
"openWorldHint": true
}
},
{
"name": "browser_resize",
"description": "Resize the browser window",
"inputSchema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"width": {
"type": "number",
"description": "Width of the browser window"
},
"height": {
"type": "number",
"description": "Height of the browser window"
}
},
"required": [
"width",
"height"
],
"additionalProperties": false
},
"annotations": {
"title": "Resize browser window",
"readOnlyHint": false,
"destructiveHint": true,
"openWorldHint": true
}
},
{
"name": "browser_console_messages",
"description": "Returns all console messages",
"inputSchema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"level": {
"default": "info",
"description": "Level of the console messages to return. Each level includes the messages of more severe levels. Defaults to \"info\".",
"type": "string",
"enum": [
"error",
"warning",
"info",
"debug"
]
},
"all": {
"description": "Return all console messages since the beginning of the session, not just since the last navigation. Defaults to false.",
"type": "boolean"
},
"filename": {
"description": "Filename to save the console messages to. If not provided, messages are returned as text.",
"type": "string"
}
},
"required": [
"level"
],
"additionalProperties": false
},
"annotations": {
"title": "Get console messages",
"readOnlyHint": true,
"destructiveHint": false,
"openWorldHint": true
}
},
{
"name": "browser_handle_dialog",
"description": "Handle a dialog",
"inputSchema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"accept": {
"type": "boolean",
"description": "Whether to accept the dialog."
},
"promptText": {
"description": "The text of the prompt in case of a prompt dialog.",
"type": "string"
}
},
"required": [
"accept"
],
"additionalProperties": false
},
"annotations": {
"title": "Handle a dialog",
"readOnlyHint": false,
"destructiveHint": true,
"openWorldHint": true
}
},
{
"name": "browser_evaluate",
"description": "Evaluate JavaScript expression on page or element",
"inputSchema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"element": {
"description": "Human-readable element description used to obtain permission to interact with the element",
"type": "string"
},
"target": {
"description": "Exact target element reference from the page snapshot, or a unique element selector",
"type": "string"
},
"function": {
"type": "string",
"description": "() => { /* code */ } or (element) => { /* code */ } when element is provided"
},
"filename": {
"description": "Filename to save the result to. If not provided, result is returned as text.",
"type": "string"
}
},
"required": [
"function"
],
"additionalProperties": false
},
"annotations": {
"title": "Evaluate JavaScript",
"readOnlyHint": false,
"destructiveHint": true,
"openWorldHint": true
}
},
{
"name": "browser_file_upload",
"description": "Upload one or multiple files",
"inputSchema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"paths": {
"description": "The absolute paths to the files to upload. Can be single file or multiple files. If omitted, file chooser is cancelled.",
"type": "array",
"items": {
"type": "string"
}
}
},
"additionalProperties": false
},
"annotations": {
"title": "Upload files",
"readOnlyHint": false,
"destructiveHint": true,
"openWorldHint": true
}
},
{
"name": "browser_drop",
"description": "Drop files or MIME-typed data onto an element, as if dragged from outside the page. At least one of \"paths\" or \"data\" must be provided.",
"inputSchema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"element": {
"description": "Human-readable element description used to obtain permission to interact with the element",
"type": "string"
},
"target": {
"type": "string",
"description": "Exact target element reference from the page snapshot, or a unique element selector"
},
"paths": {
"description": "Absolute paths to files to drop onto the element.",
"type": "array",
"items": {
"type": "string"
}
},
"data": {
"description": "Data to drop, as a map of MIME type to string value (e.g. {\"text/plain\": \"hello\", \"text/uri-list\": \"https://example.com\"}).",
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {
"type": "string"
}
}
},
"required": [
"target"
],
"additionalProperties": false
},
"annotations": {
"title": "Drop files or data onto an element",
"readOnlyHint": false,
"destructiveHint": true,
"openWorldHint": true
}
},
{
"name": "browser_fill_form",
"description": "Fill multiple form fields",
"inputSchema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"fields": {
"type": "array",
"items": {
"type": "object",
"properties": {
"element": {
"description": "Human-readable element description used to obtain permission to interact with the element",
"type": "string"
},
"target": {
"type": "string",
"description": "Exact target element reference from the page snapshot, or a unique element selector"
},
"name": {
"type": "string",
"description": "Human-readable field name"
},
"type": {
"type": "string",
"enum": [
"textbox",
"checkbox",
"radio",
"combobox",
"slider"
],
"description": "Type of the field"
},
"value": {
"type": "string",
"description": "Value to fill in the field. If the field is a checkbox, the value should be `true` or `false`. If the field is a combobox, the value should be the text of the option."
}
},
"required": [
"target",
"name",
"type",
"value"
],
"additionalProperties": false
},
"description": "Fields to fill in"
}
},
"required": [
"fields"
],
"additionalProperties": false
},
"annotations": {
"title": "Fill form",
"readOnlyHint": false,
"destructiveHint": true,
"openWorldHint": true
}
},
{
"name": "browser_press_key",
"description": "Press a key on the keyboard",
"inputSchema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"key": {
"type": "string",
"description": "Name of the key to press or a character to generate, such as `ArrowLeft` or `a`"
}
},
… (17 chars truncated)
… (505 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/09afebd9/content.txt
</truncation_notice>There are many tools. The relevant one is likely browser_navigate or browser_snapshot. I need to see the full schema for browser_navigate. The truncation cut off. I can still call it if I know the schema, but I need the exact name. The list includes: browser_navigate? It wasn't shown in truncated, but likely exists. Let me use mcp_call_tool with a likely guess. Actually, I should request the full schema again or use a known tool. The standard Playwright MCP tools include: browser_navigate, browser_snapshot, browser_click, etc. The first shown were browser_close, browser_resize, browser_console_messages, browser_handle_dialog, browser_evaluate, browser_file_upload, browser_drop, browser_fill_form, browser_press_key. Missing browser_navigate and browser_snapshot. They are likely in the list. Since truncated, I can still call mcp_call_tool with tool name "browser_navigate" and arguments { url: "..." } as it's standard. But to be safe, I might call the tool. Let me try.
{
"server_name": "playwright",
"tool_name": "browser_navigate",
"arguments": {
"url": "https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/collection/1/B200"
}
}[
{
"type": "text",
"text": "### Ran Playwright code\n```js\nawait page.goto('https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/collection/1/B200');\n```\n### Page\n- Page URL: https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/collection/1/B200\n- Page Title: SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA\n### Snapshot\n- [Snapshot](.playwright-mcp/page-2026-07-02T06-20-08-473Z.yml)"
}
]It navigated. Now I need to get a snapshot or evaluate JS to get the page content. There might be a snapshot file. I can read it or use browser_evaluate to extract text. Let me use browser_evaluate to get document.body.innerText or JSON data.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { return document.body.innerText; }"
}
}[
{
"type": "text",
"text": "### Result\n\"SOL-ExecBench\\nBlog\\nProblems\\nLeaderboard\\nStatus\\nGitHub\\n↗\\nHuggingFace\\n↗\\nPaper\\n↗\\nLog In\\nLeaderboard\\nL1 - Single Operations\\nL1 - Single Operations\\nB200\\nCollection\\nRank\\tUser\\tSOL Score\\n?\\n\\tFast\\n1\\n?\\n\\tAvg Speedup\\n?\\n\\tSubmitted\\n-\\tSOL Bound\\t1.000000\\t1.00\\t86.72x\\t-\\n1\\tDatabricks\\t0.761944\\t0.98\\t12.05x\\t11 hours ago\\n2\\tdoubleAI\\t0.758731\\t0.97\\t13.23x\\t3 days ago\\n3\\tRecursive\\t0.758430\\t0.97\\t12.31x\\t2026-06-09\\n#4\\tAmir M. Mir\\t0.645357\\t0.82\\t7.92x\\t2026-06-18\\n#5\\tCursor\\t0.582481\\t0.63\\t3.98x\\t2026-04-14\\n-\\tScoring Baseline\\t0.500000\\t0.00\\t1.00x\\t-\\n-\\tReference Implementation\\t0.237027\\t0.08\\t0.41x\\t-\\n#6\\tIgor Bragaia\\t0.160940\\t0.15\\t0.37x\\t2026-05-29\\n#7\\tKernelOpt\\t0.107010\\t0.80\\t0.45x\\t2026-04-28\\n#8\\twychi888\\t0.097196\\t0.79\\t0.42x\\t2026-06-11\\n#9\\tGilded Quail\\t0.086706\\t0.70\\t0.39x\\t2026-06-16\\n#10\\tArgus\\t0.082655\\t0.79\\t0.39x\\t12 hours ago\\n#11\\tgnehiz\\t0.070859\\t0.56\\t0.61x\\t2026-04-01\\n#12\\tYuang\\t0.056907\\t0.66\\t0.19x\\t2026-05-07\\n#13\\tAKO4ALL\\t0.046052\\t0.83\\t0.12x\\t2026-03-23\\n#14\\tBragaia\\t0.038378\\t0.16\\t0.07x\\t2026-05-28\\n#15\\tClever Scorpion\\t0.033317\\t0.49\\t0.11x\\t2026-03-30\\n#16\\tTheoric\\t0.030863\\t0.39\\t0.08x\\t3 days ago\\n#17\\tSnappy Macaw\\t0.026593\\t1.00\\t0.16x\\t2026-05-06\\n#18\\tIridescent Lemur\\t0.026339\\t0.40\\t0.06x\\t2026-06-24\\n#19\\tMystic Orca\\t0.023800\\t0.98\\t0.10x\\t2026-03-26\\n#20\\tArcane Vulture\\t0.017364\\t0.69\\t0.04x\\t2026-06-14\\n#21\\tBob Huang\\t0.016018\\t0.21\\t0.04x\\t2026-03-22\\n#22\\tLatency Lower\\t0.015785\\t0.33\\t0.07x\\t2026-04-14\\n#23\\tCoral\\t0.015772\\t1.00\\t0.06x\\t4 days ago\\n#24\\tZesty Toucan\\t0.014817\\t0.94\\t0.05x\\t2026-04-01\\n#25\\tQuick Impala\\t0.014108\\t0.01\\t0.03x\\t2026-03-28\\n#26\\tRapid Lynx\\t0.013071\\t0.78\\t0.03x\\t2026-03-21\\n#27\\tAAA\\t0.009223\\t1.00\\t0.06x\\t2026-06-18\\n#28\\tEmber Bison\\t0.009220\\t1.00\\t0.06x\\t2026-06-18\\n#29\\tChrome Lion\\t0.008118\\t1.00\\t0.02x\\ta day ago\\n#30\\tArgus-sufeng\\t0.007850\\t1.00\\t0.02x\\t2 hours ago\\n#31\\tStellar Flamingo\\t0.007616\\t1.00\\t0.03x\\t2026-06-06\\n#32\\tbismuth83\\t0.007585\\t0.81\\t0.03x\\t2026-04-24\\n#33\\tBaba Yaga\\t0.007386\\t1.00\\t0.02x\\t2 days ago\\n#34\\tRogue Pegasus\\t0.007374\\t0.81\\t0.03x\\t2026-03-26\\n#35\\tQin Liu @ UCD\\t0.007213\\t1.00\\t0.02x\\t2 days ago\\n#36\\tXG\\t0.007194\\t0.94\\t0.02x\\t2026-05-27\\n#37\\tAlex N\\t0.007164\\t1.00\\t0.02x\\t2026-05-31\\n#38\\tgenproof.com\\t0.006940\\t0.94\\t0.02x\\t7 days ago\\n#39\\tzguo\\t0.006736\\t0.69\\t0.02x\\t2026-05-08\\n#40\\tNeural Jackal\\t0.006691\\t1.00\\t0.02x\\t2026-06-14\\n#41\\tPhantom Orca\\t0.006539\\t0.75\\t0.02x\\t2026-04-20\\n#42\\tShawn_Ming\\t0.006421\\t0.75\\t0.02x\\t2026-04-02\\n#43\\tLaser Tapir\\t0.005849\\t1.00\\t0.01x\\t2026-06-09\\n#44\\tNestor_Qin\\t0.005312\\t0.50\\t0.01x\\t2026-03-19\\n#45\\tUbios-Agent\\t0.005273\\t0.38\\t0.01x\\t2026-03-25\\n#46\\timec.AILabs\\t0.005096\\t0.25\\t0.01x\\t3 days ago\\n#47\\tJasper Otter\\t0.004483\\t0.00\\t0.01x\\t2026-03-16\\n#48\\tBrave Vulture\\t0.004221\\t0.44\\t0.01x\\t2026-04-04\\n© 2026 NVIDIA Corporation. All rights reserved.\\nPrivacy Policy\\nYour Privacy Choices\\nTerms of Service\\nAccessibility\\nCorporate Policies\\nProduct Security\\nContact: solexecbench@nvidia.com\\n\\nNVIDIA uses cookies to improve your experience on our web site. We and our third-party partners also use cookies and other tools to collect and record information you provide as well as information about your interactions with our websites for performance improvement, analytics, and to assist in marketing efforts. By clicking \\\"Accept All\\\", you consent to our use of cookies and other tools as described in our Cookie Policy. You can manage your cookie settings by clicking on \\\"Manage Settings.\\\" By continuing to use this site or by clicking one of the buttons below, you agree to our Terms of Service (which contains important waivers). Please see our Privacy Policy for more information on our privacy practices.\\n\\nManage Settings\\nReject Optional Accept All\"\n### Ran Playwright code\n```js\nawait page.evaluate('() => { return document.body.innerText; }');\n```"
}
]I got the page content. It shows the leaderboard for "L1 - Single Operations" collection on B200. The user asked "pull, figure out whats going on here?" The URL is the leaderboard for collection 1 / B200. I need to tell them what's going on. The leaderboard shows: - L1 - Single Operations (Collection 1) - B200 device - Ranking by SOL Score - Columns: Rank, User, SOL Score, Fast (?), Avg Speedup, Submitted - SOL Bound: 1.000000, 1.00, 86.72x - Top 3: Databricks (0.761944, 0.98, 12.05x), doubleAI (0.758731, 0.97, 13.23x), Recursive (0.758430, 0.97, 12.31x) - Scoring Baseline 0.5 - Reference Implementation 0.237027 - Many submissions. I should explain what SOL-ExecBench is, what this leaderboard is, and interpret the numbers. Also maybe the user said "pull" which could mean they want to pull the leaderboard data or clone repo. I can provide a brief answer. Also maybe I should check if there is any other hidden data or graphs. I can extract innerHTML or API. But the innerText is enough. I can maybe get the column headers more clearly: "Rank User SOL Score ? Fast ? Avg Speedup ? Submitted". The "?" might be because of icons or tooltips. The numbers are: SOL Score, maybe "Fast" ratio, Avg Speedup, Submitted. Let me also maybe use browser_evaluate to extract table as a JSON to get more precise headers and rows. This could help.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { const rows = Array.from(document.querySelectorAll('table tr, [role=row]')).map(r => Array.from(r.querySelectorAll('td, th')).map(c => c.innerText.trim().replace(/\\s+/g,' '))); return rows; }"
}
}[
{
"type": "text",
"text": "### Result\n[\n [\n \"Rank\",\n \"User\",\n \"SOL Score ?\",\n \"Fast 1 ?\",\n \"Avg Speedup ?\",\n \"Submitted\"\n ],\n [\n \"-\",\n \"SOL Bound\",\n \"1.000000\",\n \"1.00\",\n \"86.72x\",\n \"-\"\n ],\n [\n \"1\",\n \"Databricks\",\n \"0.761944\",\n \"0.98\",\n \"12.05x\",\n \"11 hours ago\"\n ],\n [\n \"2\",\n \"doubleAI\",\n \"0.758731\",\n \"0.97\",\n \"13.23x\",\n \"3 days ago\"\n ],\n [\n \"3\",\n \"Recursive\",\n \"0.758430\",\n \"0.97\",\n \"12.31x\",\n \"2026-06-09\"\n ],\n [\n \"#4\",\n \"Amir M. Mir\",\n \"0.645357\",\n \"0.82\",\n \"7.92x\",\n \"2026-06-18\"\n ],\n [\n \"#5\",\n \"Cursor\",\n \"0.582481\",\n \"0.63\",\n \"3.98x\",\n \"2026-04-14\"\n ],\n [\n \"-\",\n \"Scoring Baseline\",\n \"0.500000\",\n \"0.00\",\n \"1.00x\",\n \"-\"\n ],\n [\n \"-\",\n \"Reference Implementation\",\n \"0.237027\",\n \"0.08\",\n \"0.41x\",\n \"-\"\n ],\n [\n \"#6\",\n \"Igor Bragaia\",\n \"0.160940\",\n \"0.15\",\n \"0.37x\",\n \"2026-05-29\"\n ],\n [\n \"#7\",\n \"KernelOpt\",\n \"0.107010\",\n \"0.80\",\n \"0.45x\",\n \"2026-04-28\"\n ],\n [\n \"#8\",\n \"wychi888\",\n \"0.097196\",\n \"0.79\",\n \"0.42x\",\n \"2026-06-11\"\n ],\n [\n \"#9\",\n \"Gilded Quail\",\n \"0.086706\",\n \"0.70\",\n \"0.39x\",\n \"2026-06-16\"\n ],\n [\n \"#10\",\n \"Argus\",\n \"0.082655\",\n \"0.79\",\n \"0.39x\",\n \"12 hours ago\"\n ],\n [\n \"#11\",\n \"gnehiz\",\n \"0.070859\",\n \"0.56\",\n \"0.61x\",\n \"2026-04-01\"\n ],\n [\n \"#12\",\n \"Yuang\",\n \"0.056907\",\n \"0.66\",\n \"0.19x\",\n \"2026-05-07\"\n ],\n [\n \"#13\",\n \"AKO4ALL\",\n \"0.046052\",\n \"0.83\",\n \"0.12x\",\n \"2026-03-23\"\n ],\n [\n \"#14\",\n \"Bragaia\",\n \"0.038378\",\n \"0.16\",\n \"0.07x\",\n \"2026-05-28\"\n ],\n [\n \"#15\",\n \"Clever Scorpion\",\n \"0.033317\",\n \"0.49\",\n \"0.11x\",\n \"2026-03-30\"\n ],\n [\n \"#16\",\n \"Theoric\",\n \"0.030863\",\n \"0.39\",\n \"0.08x\",\n \"3 days ago\"\n ],\n [\n \"#17\",\n \"Snappy Macaw\",\n \"0.026593\",\n \"1.00\",\n \"0.16x\",\n \"2026-05-06\"\n ],\n [\n \"#18\",\n \"Iridescent Lemur\",\n \"0.026339\",\n \"0.40\",\n \"0.06x\",\n \"2026-06-24\"\n ],\n [\n \"#19\",\n \"Mystic Orca\",\n \"0.023800\",\n \"0.98\",\n \"0.10x\",\n \"2026-03-26\"\n ],\n [\n \"#20\",\n \"Arcane Vulture\",\n \"0.017364\",\n \"0.69\",\n \"0.04x\",\n \"2026-06-14\"\n ],\n [\n \"#21\",\n \"Bob Huang\",\n \"0.016018\",\n \"0.21\",\n \"0.04x\",\n \"2026-03-22\"\n ],\n [\n \"#22\",\n \"Latency Lower\",\n \"0.015785\",\n \"0.33\",\n \"0.07x\",\n \"2026-04-14\"\n ],\n [\n \"#23\",\n \"Coral\",\n \"0.015772\",\n \"1.00\",\n \"0.06x\",\n \"4 days ago\"\n ],\n [\n \"#24\",\n \"Zesty Toucan\",\n \"0.014817\",\n \"0.94\",\n \"0.05x\",\n \"2026-04-01\"\n ],\n [\n \"#25\",\n \"Quick Impala\",\n \"0.014108\",\n \"0.01\",\n \"0.03x\",\n \"2026-03-28\"\n ],\n [\n \"#26\",\n \"Rapid Lynx\",\n \"0.013071\",\n \"0.78\",\n \"0.03x\",\n \"2026-03-21\"\n ],\n [\n \"#27\",\n \"AAA\",\n \"0.009223\",\n \"1.00\",\n \"0.06x\",\n \"2026-06-18\"\n ],\n [\n \"#28\",\n \"Ember Bison\",\n \"0.009220\",\n \"1.00\",\n \"0.06x\",\n \"2026-06-18\"\n ],\n [\n \"#29\",\n \"Chrome Lion\",\n \"0.008118\",\n \"1.00\",\n \"0.02x\",\n \"a day ago\"\n ],\n [\n \"#30\",\n \"Argus-sufeng\",\n \"0.007850\",\n \"1.00\",\n \"0.02x\",\n \"2 hours ago\"\n ],\n [\n \"#31\",\n \"Stellar Flamingo\",\n \"0.007616\",\n \"1.00\",\n \"0.03x\",\n \"2026-06-06\"\n ],\n [\n \"#32\",\n \"bismuth83\",\n \"0.007585\",\n \"0.81\",\n \"0.03x\",\n \"2026-04-24\"\n ],\n [\n \"#33\",\n \"Baba Yaga\",\n \"0.007386\",\n \"1.00\",\n \"0.02x\",\n \"2 days ago\"\n ],\n [\n \"#34\",\n \"Rogue Pegasus\",\n \"0.007374\",\n \"0.81\",\n \"0.03x\",\n \"2026-03-26\"\n ],\n [\n \"#35\",\n \"Qin Liu @ UCD\",\n \"0.007213\",\n \"1.00\",\n \"0.02x\",\n \"2 days ago\"\n ],\n [\n \"#36\",\n \"XG\",\n \"0.007194\",\n \"0.94\",\n \"0.02x\",\n \"2026-05-27\"\n ],\n [\n \"#37\",\n \"Alex N\",\n \"0.007164\",\n \"1.00\",\n \"0.02x\",\n \"2026-05-31\"\n ],\n [\n \"#38\",\n \"genproof.com\",\n \"0.006940\",\n \"0.94\",\n \"0.02x\",\n \"7 days ago\"\n ],\n [\n \"#39\",\n \"zguo\",\n \"0.006736\",\n \"0.69\",\n \"0.02x\",\n \"2026-05-08\"\n ],\n [\n \"#40\",\n \"Neural Jackal\",\n \"0.006691\",\n \"1.00\",\n \"0.02x\",\n \"2026-06-14\"\n ],\n [\n \"#41\",\n \"Phantom Orca\",\n \"0.006539\",\n \"0.75\",\n \"0.02x\",\n \"2026-04-20\"\n ],\n [\n \"#42\",\n \"Shawn_Ming\",\n \"0.006421\",\n \"0.75\",\n \"0.02x\",\n \"2026-04-02\"\n ],\n [\n \"#43\",\n \"Laser Tapir\",\n \"0.005849\",\n \"1.00\",\n \"0.01x\",\n \"2026-06-09\"\n ],\n [\n \"#44\",\n \"Nestor_Qin\",\n \"0.005312\",\n \"0.50\",\n \"0.01x\",\n \"2026-03-19\"\n ],\n [\n \"#45\",\n \"Ubios-Agent\",\n \"0.005273\",\n \"0.38\",\n \"0.01x\",\n \"2026-03-25\"\n ],\n [\n \"#46\",\n \"imec.AILabs\",\n \"0.005096\",\n \"0.25\",\n \"0.01x\",\n \"3 days ago\"\n ],\n [\n \"#47\",\n \"Jasper Otter\",\n \"0.004483\",\n \"0.00\",\n \"0.01x\",\n \"2026-03-16\"\n ],\n [\n \"#48\",\n \"Brave Vulture\",\n \"0.004221\",\n \"0.44\",\n \"0.01x\",\n \"2026-04-04\"\n ]\n]\n### Ran Playwright code\n```js\nawait page.evaluate('() => { const rows = Array.from(document.querySelectorAll(\\'table tr, [role=row]\\')).map(r => Array.from(r.querySelectorAll(\\'td, th\\')).map(c => c.innerText.trim().replace(/\\\\s+/g,\\' \\'))); return rows; }');\n```"
}
]Now I have precise leaderboard data. The user might also want to know what the columns mean. I should try to read the docs or hover tooltips. I can fetch the GitHub README or docs. The "Fast 1" column maybe means fraction of kernels that are fastest? The "Avg Speedup" maybe average speedup over reference? The "SOL Score" is the main metric. I can also search the GitHub repo for "Fast" or "leaderboard" to understand. The repo is public. I could clone it or read the docs. The user asked "pull, figure out whats going on here?" Maybe they want me to pull the repo and explore. I think we have enough info to answer. But to be thorough, I could fetch the GitHub docs or README. Let me use webfetch on the GitHub raw README if possible. But the page's content is enough. The user might want to know what the leaderboard is showing. The answer: It's the SOL-ExecBench L1 - Single Operations collection for B200 GPU. It ranks submissions by SOL Score (a metric of how close to the theoretical Speed-of-Light bound on B200). Top entries are Databricks, doubleAI, Recursive all around 0.76 SOL Score, achieving ~12-13x speedup over the scoring baseline, and ~98% of kernels as the fastest? Wait "Fast 1" is 0.98 for top, maybe fraction of fastest (best). "Avg Speedup" maybe over baseline or reference. Actually, the scoring baseline is at 0.5 SOL Score with 0.00 Fast 1 and 1.00x avg speedup. The reference implementation is at 0.237 SOL Score, 0.08 Fast, 0.41x speedup. So Avg Speedup is relative to the scoring baseline? Baseline is 1.00x. Databricks is 12.05x avg speedup. Yes. "Fast 1" likely is the fraction of kernels where the submission is the fastest (rank 1). For Databricks, 0.98 means 98% of L1 kernels they are fastest. The scoring baseline is defined as a specific reference (maybe the reference implementation? but there is also "Reference Implementation"). The SOL Score is computed as: score = (perf - baseline) / (SOL_bound - baseline). So 0.5 means equal to baseline. Databricks 0.76 means they closed 76% of the gap between baseline and SOL bound? Actually the SOL Bound is 1.0. Score is how much of the gap between baseline and SOL bound is closed. So 0.76 = baseline + 0.76*(SOL - baseline) = 0.5 + 0.76*(1-0.5) = 0.88. The speedup relative to baseline is 12x, so performance is 12x baseline. The reference implementation is 0.41x baseline, so it performs worse than baseline. We can also explain that the leaderboard is an open competition for AI-generated/hand-optimized GPU kernels, where people submit custom kernels for 94 single-operation problems (L1) and are evaluated on NVIDIA B200. I should also mention the other collections (L2, L3? quant?). But the URL is collection 1. Maybe there are multiple collections: collection/1 is L1 Single Operations. There is also collection/2? Let me see if I can navigate to collection 2. But the user asked specifically this one. I'll keep it concise. Maybe "pull" means they want me to pull the leaderboard data into a file or clone the repo? The user says "pull, figure out whats going on here?" Could be ambiguous. They might want me to "pull" the data and analyze. I can produce a summary. But maybe they want me to pull the repo. I can ask or do both. Since the instruction says "Do not surprise the user with actions you take without asking." I should not clone repo without asking. But I can answer the question. However, maybe the user wants me to "pull" the repo and figure out what the codebase is. They gave a URL to a leaderboard. Hmm. I'll provide the answer and maybe offer to clone the repo or scrape more. The answer should be concise. Let me also possibly get the column headers from tooltips via the GitHub docs. Search for "Fast 1" in the repo.
{
"query": "SOL-ExecBench leaderboard \"Fast 1\" \"Avg Speedup\" columns"
}# Web Search Results for "SOL-ExecBench leaderboard "Fast 1" "Avg Speedup" columns" ## 1. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/kernel/47/B200 | Rank | User | SOL Score? | Latency? | Fast1? | Avg Speedup? | Submitted | | --- | --- | --- | --- | --- | --- | --- | | - | SOL Bound | 1.000000 | 0.069243 ms | 16 / 16 = 1.00 | 9.17x | - | | 1 | Recursive | 0.802687 | 0.170334 ms | 16 / 16 = 1.00 | 3.85x | 2026-06-07 | | 2 | doubleAI | 0.728192 | 0.228869 ms | 16 / 16 = 1.00 | 2.67x | 2026-05-20 | | 3 | Amir M. Mir | 0.723674 | 0.225066 ms | 16 / 16 = 1.00 | 3.04x | 2026-06-02 | | #4 | Cursor | 0.539778 | 0.460911 ms | 14 / 16 = 0.88 | 1.16x | 2026-04-14 | | - | Scoring Baseline | 0.500000 | 0.526809 ms | 0 / 16 = 0.00 | 1.00x | - | | - | Reference Implementation | 0.247745 | 1.436483 ms | 0 / 16 = 0.00 | 0.37x | - | | #5 | Igor Bragaia | 0.246384 | 1.446753 ms | 0 / 16 = 0.00 | 0.37x | 2026-05-28 | ## 2. NVIDIA/SOL-ExecBench URL: https://github.com/nvidia/sol-execbench [HuggingFace Dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) | [Leaderboard](https://research.nvidia.com/benchmarks/sol-execbench) | [Technical Report](https://arxiv.org/abs/2603.19173) **Speed-Of-Light ExecBench** is a rigorous GPU kernel evaluation and benchmarking framework built to benchmark AI-generated kernel solutions written with the variety of DSLs that NVIDIA hardware supports. ... Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)). Supported kernel languages: PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++. ## 3. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/collection/1 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA Sorry, your browser does not support inline SVG. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA [SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench) [GitHub ↗](https://github.com/nvidia/sol-execbench)[HuggingFace ↗](https://huggingface.co/datasets/nvidia/sol-execbench)[Paper ↗](https://arxiv.org/abs/2603.19173)[Log In](https://research.nvidia.com/benchmarks/sol-execbench/login) Sorry, your browser does not support inline SVG. ## 4. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/kernel/16 Leaderboard B200 ... | # | User | SOL Score | Latency | | --- | --- | --- | --- | | - | SOL Bound | 1.000000 | 0.000400 ms | | 1 | doubleAI | 0.660116 | 0.001301 ms | | 2 | Recursive | 0.598347 | 0.001576 ms | | 3 | Amir M. Mir | 0.579557 | 0.001671 ms | | #4 | Clever Scorpion | 0.560472 | 0.001775 ms | | #5 | Yuang | 0.550138 | 0.001833 ms | | #6 | Arcane Vulture | 0.501543 | 0.002142 ms | | #7 | Cursor | 0.501221 | 0.002144 ms | | - | Scoring Baseline | 0.500000 | 0.002153 ms | | #8 | Latency Lower | 0.440655 | 0.002625 ms | | #9 | Igor Bragaia | 0.200328 | 0.007399 ms | | - | Reference Implementation | 0.074980 | 0.022035 ms | ... View Full Leaderboard → ## 5. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/kernel/36/B200 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA Sorry, your browser does not support inline SVG. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA [SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench) [GitHub ↗](https://github.com/nvidia/sol-execbench)[HuggingFace ↗](https://huggingface.co/datasets/nvidia/sol-execbench)[Paper ↗](https://arxiv.org/abs/2603.19173)[Log In](https://research.nvidia.com/benchmarks/sol-execbench/login) Sorry, your browser does not support inline SVG.
The search results show that individual kernel leaderboard shows "Latency" and "Fast1" with fraction. The collection leaderboard we saw has "Fast 1" and "Avg Speedup". The "Fast 1" column likely means "Fraction of kernels in this collection where the submission is #1 (fastest)". The "Avg Speedup" is geometric mean speedup over the scoring baseline across all kernels in the collection. The collection page seems to aggregate over all L1 problems. So "Fast 1" for Databricks is 0.98 = 98% of L1 problems they have the fastest submission? Actually top 3 all have 0.97-0.98, meaning they are fastest on almost all problems. There are ties? The first result is "SOL Bound" 1.00, 86.72x. The second row shows "1 Databricks ... 0.98". So Databricks has 0.98 Fast 1. Avg Speedup 12.05x. The SOL Score is 0.76. Wait: If Databricks is fastest on 98% of problems, why is the avg speedup only 12x? The SOL Bound average speedup is 86.72x. So the baseline is much slower than the best known solutions. The Fast 1 is fraction of problems where the submission is the best among all submitted, not vs baseline. Actually the individual kernel page example: Recursive 0.80 SOL Score, Fast 1 = 16/16 = 1.00 (fastest among all submissions for that kernel), Avg Speedup 3.85x (over baseline). So for a single kernel, "Avg Speedup" is just speedup of this kernel over baseline. For collection, "Avg Speedup" is geometric mean of per-kernel speedups. For collection, "Fast 1" is maybe the fraction of collection problems where the submission is top among all submissions. Databricks 0.98 means they are fastest on 98% of L1 problems. But there is also Recursive and doubleAI with 0.97. How can multiple have 0.98? If each has different fastest across problems. But a collection might have many problems; if they each are fastest on most, the fractions are per-submission. Actually 0.98 means 98% of their own submitted solutions? No. Maybe Fast 1 means fraction of kernels where the submission achieves the theoretical SOL bound? But SOL Bound row has Fast 1 = 1.00. Top submissions have 0.98, meaning 98% of kernels they reach the top? But multiple top submissions can't all be top on same problem. Could be a weighted average where they get near top? Need clarification. Wait, in the individual kernel page, the table header is "Fast1?" and the value for top is "16 / 16 = 1.00". That means there are 16 test cases/workloads for that kernel, and the submission is fastest on all 16. For a collection, maybe each problem has multiple test cases, and "Fast 1" is fraction of test cases where the submission is fastest across the whole collection. If Databricks submits for all problems and has best on 98% of test cases, then others have lower. But doubleAI has 0.97 and Recursive 0.97. That could be possible if the collection has many test cases, and Databricks is fastest on ~98%, doubleAI ~97%, etc. But they can't all be true if Databricks is fastest on 98%. Unless the values are rounded or computed as a normalized score relative to the best (like the fraction of theoretical peak or fraction of best performance). Hmm. Actually "Fast 1" might be a score that indicates closeness to the fastest solution, not a strict count. For example, if a submission is within some epsilon of the best, it counts as "fast". Then multiple can have high. But in the individual kernel, "16/16 = 1.00" is strict count of fastest. Let's look at the collection row for "SOL Bound": Fast 1 = 1.00, Avg Speedup = 86.72x. So the SOL Bound is the theoretical max across all problems. "Avg Speedup" is the average speedup of the solution compared to the scoring baseline. "Fast 1" might be the fraction of the collection where the submission matches the best-known performance (i.e., the top of the leaderboard). For Databricks, 0.98 means 98% of the L1 collection entries are best-known. But if doubleAI has 0.97, maybe they are missing a few that Databricks has, and Databricks is missing a few that Recursive has? Wait, Databricks 0.98 means 98% of entries are best; doubleAI 0.97 means 97% are best. If Databricks is best on 98%, doubleAI can be best on at most 2% (the ones Databricks is not best). But doubleAI's 0.97 contradicts. Unless "Fast 1" is not fraction of problems where the submission is #1, but rather the fraction of the theoretical SOL bound achieved on average? Or "fastest 1" maybe the fraction of kernels that are within 1% of the SOL bound? That would allow multiple high. Search result for individual kernel page says "Fast1? 16 / 16 = 1.00". That suggests the number of problems/test cases is the denominator. So for collection, the denominator would be total number of problems in the collection (94). If Databricks has 0.98, then Databricks is fastest on ~92 of 94 problems. doubleAI 0.97 = ~91. Recursive 0.97 = ~91. Overlap possible? Only one can be fastest on a given problem. But if Databricks is fastest on 92, doubleAI can be fastest on at most 2. So 0.97 impossible. Unless the collection is a set of problems, and each problem has multiple sub-kernels/test cases. But still only one fastest per subcase. Maybe "Fast 1" is a per-submission metric that counts the number of problems where this submission is the best *relative to the baseline*? No, baseline is 0.00. Alternative: Maybe "Fast 1" is the average fraction of top-1 across problems, weighted by something? Or the fraction of submissions that pass the top-1 threshold? Hmm. Let's try to look at the SOL Score formula. The score might be: score = (perf - baseline) / (sol - baseline). It might be a weighted average. The "Fast 1" column might be "Number of kernels where the submission is ranked first" but normalized as a fraction of the collection. In that case, multiple top submissions would have values that sum to >1 if they each are first on different problems. But the sum of top 3's Fast1 = 0.98+0.97+0.97 = 2.92 > 1. So it can't be strict counts. Maybe "Fast 1" is the fraction of the total SOL score achieved compared to the best? That doesn't match "16/16". Wait, maybe the individual kernel page we saw is per problem, and the collection page is a different view. In the collection page, the header might be "Fast 1" but the value for "SOL Bound" is 1.00. It could be the fraction of the collection's total test cases where the submission is the fastest, but the denominator is the total number of test cases across all problems, and multiple submissions can be "fast" if their time is within a certain percentage of the fastest. Actually, in top-1 competitions, a "fast" metric often is number of problems where the submission is the fastest. But if ties are allowed? No. Could the numbers be for each user's *best* submission over time, and the leaderboard shows the best overall, but Fast 1 is the fraction of their own submissions that are ranked first? Hmm. Let's find the source. The repo likely has a web frontend. The page might have tooltips. The "Fast 1" might be a metric. Let's search the GitHub repo for "fast1" or "fast" in the code. I can use web_search or fetch raw files. Let's try to use the GitHub search via web? Or use webfetch on raw repo. But I don't know the file structure. I can search the web for "SOL-ExecBench fast1 metric".
{
"query": "SOL-ExecBench \"Fast1\" metric"
}# Web Search Results for "SOL-ExecBench "Fast1" metric" ## 1. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/kernel/47/B200 | Rank | User | SOL Score? | Latency? | Fast1? | Avg Speedup? | Submitted | | --- | --- | --- | --- | --- | --- | --- | | - | SOL Bound | 1.000000 | 0.069243 ms | 16 / 16 = 1.00 | 9.17x | - | | 1 | Recursive | 0.802687 | 0.170334 ms | 16 / 16 = 1.00 | 3.85x | 2026-06-07 | | 2 | doubleAI | 0.728192 | 0.228869 ms | 16 / 16 = 1.00 | 2.67x | 2026-05-20 | | 3 | Amir M. Mir | 0.723674 | 0.225066 ms | 16 / 16 = 1.00 | 3.04x | 2026-06-02 | | #4 | Cursor | 0.539778 | 0.460911 ms | 14 / 16 = 0.88 | 1.16x | 2026-04-14 | | - | Scoring Baseline | 0.500000 | 0.526809 ms | 0 / 16 = 0.00 | 1.00x | - | | - | Reference Implementation | 0.247745 | 1.436483 ms | 0 / 16 = 0.00 | 0.37x | - | | #5 | Igor Bragaia | 0.246384 | 1.446753 ms | 0 / 16 = 0.00 | 0.37x | 2026-05-28 | ## 2. NVIDIA/SOL-ExecBench URL: https://github.com/nvidia/sol-execbench [HuggingFace Dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) | [Leaderboard](https://research.nvidia.com/benchmarks/sol-execbench) | [Technical Report](https://arxiv.org/abs/2603.19173) **Speed-Of-Light ExecBench** is a rigorous GPU kernel evaluation and benchmarking framework built to benchmark AI-generated kernel solutions written with the variety of DSLs that NVIDIA hardware supports. ... Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)). Supported kernel languages: PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++. ## 3. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/collection/1 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA Sorry, your browser does not support inline SVG. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA [SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench) [GitHub ↗](https://github.com/nvidia/sol-execbench)[HuggingFace ↗](https://huggingface.co/datasets/nvidia/sol-execbench)[Paper ↗](https://arxiv.org/abs/2603.19173)[Log In](https://research.nvidia.com/benchmarks/sol-execbench/login) Sorry, your browser does not support inline SVG. ## 4. SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits URL: https://arxiv.org/html/2603.19173 bounds from FLOP ... We define a new performance metric, the SOL score, denoted by S ∈ [0, 1]. It measures how close a kernel is to the hardware SOL relative to a fixed baseline runtime. Let T b denote the runtime of the baseline implementation, T SOL the runtime estimated by SOLAR, and T k the measured runtime of the candidate kernel. We assume T b > T SOL and T k ≥ T SOL , so that the baseline-to-SOL gap is positive. If either assumption is violated in practice, we treat the case as an audit signal and report it for SOLAR bound review and reward-hacking inspection (Section 4.4.1). ... 1 1 ... The SOL score lies in [0, 1] and has these three anchor properties (also illustrated in Figure 5 ): We design the score to assign the baseline a midpoint score rather than zero so that the metric separates three regimes on a common bounded scale: below-baseline performance (S < 0.5), abovebaseline but sub-SOL performance (0.5 < S < 1), and SOL-level performance (S = 1). In the below-baseline performance region the metric decays smoothly towards zero as runtime increases. ... The term T b -T SOL represents the performance headroom between the baseline and the hardware SOL. Accordingly, the SOL score measures how effectively a candidate kernel closes this gap. A score greater than 0.5 indicates that the kernel the baseline, while a score close to 1 indicates that it approaches hardware-efficient execution. Earlier we assumed T b > T SOL , but as T b → T SOL , we consider the problem is solved and do not evaluate new submissions for that problem. ... Each problem in ... -ExecBench has a ... baseline (not released) that is different from the reference program and is a higher-performance implementation that serves as ... 2 ). Any solution faster than ... will receive a ... an already-optimized kernel. ... Figure 10 characterizes the agent-generated solutions across all four benchmark categories. The score distributions (Figure 10 (a)) show median SOL scores of 0.688 for L1, 0.761 ... ## 5. nvidia/SOL-ExecBench · Datasets at Hugging Face URL: https://huggingface.co/datasets/nvidia/SOL-ExecBench SOL (Speed Of Light) ExecBench is a real-world CUDA kernel benchmarking dataset of 235 kernel-level computational workload specifications derived from open-source HuggingFace model architectures. The problems span a wide range of AI model workloads — covering text, vision, and speech models' forward and backward passes — and include core algorithms such as matrix multiplications, convolutions, attention variants, mixture-of-experts, and norms across FP32, BF16, FP16, FP8, and NVFP4 data types. Runtimes will be compared against reference implementation speed but also theoretical hardware "speed of light": the fastest that kernels can run given the hardware's peak compute and memory bandwidth limits. ... Evaluation is supported via a custom evaluator (GitHub) and a public leaderboard. Speed-of-light scores are analytically computed with SOLAR and are available on the public leaderboard. ... These files are split into ... subsets (configs):`FlashInfer-Bench`,`L1 ... L2`, and`Quant`. ... Record Count: 235 kernel problems ... - 26 from FlashInfer-Bench 1.0 - 94 Level 1 (L1) - 82 Level 2 (L2) - 33 Quantized graphs ... - FlashInfer-Bench: https://huggingface.co/datasets/flashinfer-ai/flashinfer-trace - Public GitHub repository for evaluation: https://github.com/nvidia/sol-execbench - Leaderboard: https://research.nvidia.com/benchmarks/sol-execbench - Technical report: https://arxiv.org/abs/2603.19173
The arxiv HTML snippet gives the SOL score formula and properties. We need to know "Fast 1". In individual kernel page, the "Fast1" column shows the count of problem instances where the submission is the fastest. In collection, it seems aggregated. But the arxiv might not explain "Fast1". Maybe the website frontend has an info icon. We can try to evaluate the title attribute of the header.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { const headers = Array.from(document.querySelectorAll('th')).map(th => ({text: th.innerText.trim(), title: th.getAttribute('title'), aria: th.getAttribute('aria-label') })); return headers; }"
}
}[
{
"type": "text",
"text": "### Result\n[\n {\n \"text\": \"Rank\",\n \"title\": null,\n \"aria\": null\n },\n {\n \"text\": \"User\",\n \"title\": null,\n \"aria\": null\n },\n {\n \"text\": \"SOL Score\\n?\",\n \"title\": null,\n \"aria\": null\n },\n {\n \"text\": \"Fast\\n1\\n?\",\n \"title\": null,\n \"aria\": null\n },\n {\n \"text\": \"Avg Speedup\\n?\",\n \"title\": null,\n \"aria\": null\n },\n {\n \"text\": \"Submitted\",\n \"title\": null,\n \"aria\": null\n }\n]\n### Ran Playwright code\n```js\nawait page.evaluate('() => { const headers = Array.from(document.querySelectorAll(\\'th\\')).map(th => ({text: th.innerText.trim(), title: th.getAttribute(\\'title\\'), aria: th.getAttribute(\\'aria-label\\') })); return headers; }');\n```\n### Events\n- New console entries: .playwright-mcp/console-2026-07-02T06-20-07-521Z.log#L1"
}
]The "?" likely are SVG icons for info. The actual header text is "SOL Score", "Fast 1", "Avg Speedup". I can maybe hover the icons to get tooltip text via event listeners, but maybe not needed. The meaning from search result: "Fast 1" = number of problems where the submission is fastest among all submissions / total problems. For collection, the denominator is total problem instances. However, the multiple high values remain puzzling. Maybe the collection leaderboard is per "collection" of problems, but the "Fast 1" could be fraction of total problem instances where the submission is faster than the baseline? No, baseline is 0.00. Or fraction of problem instances where the submission is faster than the reference implementation? No. Actually, maybe the "Fast 1" column for the collection is the *fraction of problems for which the submission's SOL score is the highest among all submissions*. But as we argued, multiple can't be 0.97. Unless the leaderboard is sorted by SOL score, and the "Fast 1" is for the *best* submission per user, and multiple submissions can tie for best on many problems. But still can't have 98% and 97% simultaneously unless the total collection has many problems and the best on each problem is shared. E.g., Databricks fastest on 92, Recursive on 2, doubleAI on 0. Then Databricks 0.98, others 0.02. Not matching. Wait, maybe the numbers for "Fast 1" in the collection are not the fraction of problems where the submission is the fastest, but rather the average "Fast 1" score across problem instances. For each problem instance, a submission is assigned 1 if it is the fastest among all submissions for that instance, else 0. Then averaging across all instances gives the fraction. For a single user, if multiple users tie for fastest on a problem, maybe they all get 1? Then multiple users could have high fractions. The "Fast 1" could be "is within the top performance group (i.e., fastest or tied)". If the top submissions are very close to each other, many could be considered tied for first within some tolerance. But the individual kernel page shows 16/16 for three distinct submissions; that would require ties on all 16 instances. But the speedups differ: Recursive 3.85x, doubleAI 2.67x, Amir 3.04x. Wait, those are different submissions with different speedups, all 16/16. If only one is actually fastest, the others can't be 16/16. Unless the "Fast 1" metric is "number of problem instances where the submission's runtime is within some threshold of the best-known runtime" or "faster than the baseline"? But baseline is 0/16. Hmm. Wait, the individual kernel page for kernel 47 has top 3 all 16/16. The latencies are: Recursive 0.170334, Amir 0.225066, doubleAI 0.228869. The fastest is Recursive 0.170. So why would doubleAI be 16/16? Unless "Fast 1" is a per-submission metric that counts how many test cases it runs, not its rank. But baseline 0/16 contradicts that. Maybe the "Fast 1" column is the number of valid test cases (problem instances) for that submission that are *valid* (pass correctness). Baseline is 0/16? No, baseline should be correct. Let's look at the table in the search result for kernel 47: It shows "Fast1?" header and values like 16/16 = 1.00. The "1" might refer to "top-1". The denominator is the number of "workloads" or "test cases". The numerator is number of test cases where the submission is fastest. But the top three all being 16/16 is impossible unless the page was captured at different times? Actually, maybe the page displays the *best* result per user, and the top three are different users with different results. But only one can be fastest. Unless the metric is "fastest compared to the reference implementation"? No. Wait, maybe the "Fast 1" metric is the number of test cases where the submission is the fastest *among its own submitted solutions*? No. Alternatively, maybe the "Fast 1" column is actually "Best rank count" with denominator = number of test cases where the submission achieved rank 1. But again, only one per test case. Let's find the exact source. The web page is a React app. The data might be loaded from an API. We can inspect the page's network or source. We can use browser_evaluate to get the window.__NEXT_DATA__ or whatever. But maybe not needed. However, the user wants to know what's going on. We can answer with the data we have and note the ambiguity. But to be thorough, maybe we can get the API or page source to see what "Fast 1" means. Let me try to extract the page's HTML or data.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { return JSON.stringify(Object.keys(window)); }"
}
}[
{
"type": "text",
"text": "### Result\n\"[\\\"0\\\",\\\"window\\\",\\\"self\\\",\\\"document\\\",\\\"name\\\",\\\"location\\\",\\\"customElements\\\",\\\"history\\\",\\\"navigation\\\",\\\"locationbar\\\",\\\"menubar\\\",\\\"personalbar\\\",\\\"scrollbars\\\",\\\"statusbar\\\",\\\"toolbar\\\",\\\"status\\\",\\\"closed\\\",\\\"frames\\\",\\\"length\\\",\\\"top\\\",\\\"opener\\\",\\\"parent\\\",\\\"frameElement\\\",\\\"navigator\\\",\\\"origin\\\",\\\"external\\\",\\\"screen\\\",\\\"innerWidth\\\",\\\"innerHeight\\\",\\\"scrollX\\\",\\\"pageXOffset\\\",\\\"scrollY\\\",\\\"pageYOffset\\\",\\\"visualViewport\\\",\\\"screenX\\\",\\\"screenY\\\",\\\"outerWidth\\\",\\\"outerHeight\\\",\\\"devicePixelRatio\\\",\\\"event\\\",\\\"clientInformation\\\",\\\"screenLeft\\\",\\\"screenTop\\\",\\\"styleMedia\\\",\\\"onsearch\\\",\\\"onappinstalled\\\",\\\"onbeforeinstallprompt\\\",\\\"onabort\\\",\\\"onbeforeinput\\\",\\\"onbeforematch\\\",\\\"onbeforetoggle\\\",\\\"onblur\\\",\\\"oncancel\\\",\\\"oncanplay\\\",\\\"oncanplaythrough\\\",\\\"onchange\\\",\\\"onclick\\\",\\\"onclose\\\",\\\"oncommand\\\",\\\"oncontentvisibilityautostatechange\\\",\\\"oncontextlost\\\",\\\"oncontextmenu\\\",\\\"oncontextrestored\\\",\\\"oncuechange\\\",\\\"ondblclick\\\",\\\"ondrag\\\",\\\"ondragend\\\",\\\"ondragenter\\\",\\\"ondragleave\\\",\\\"ondragover\\\",\\\"ondragstart\\\",\\\"ondrop\\\",\\\"ondurationchange\\\",\\\"onemptied\\\",\\\"onended\\\",\\\"onerror\\\",\\\"onfocus\\\",\\\"onformdata\\\",\\\"oninput\\\",\\\"oninvalid\\\",\\\"onkeydown\\\",\\\"onkeypress\\\",\\\"onkeyup\\\",\\\"onload\\\",\\\"onloadeddata\\\",\\\"onloadedmetadata\\\",\\\"onloadstart\\\",\\\"onmousedown\\\",\\\"onmouseenter\\\",\\\"onmouseleave\\\",\\\"onmousemove\\\",\\\"onmouseout\\\",\\\"onmouseover\\\",\\\"onmouseup\\\",\\\"onmousewheel\\\",\\\"onpause\\\",\\\"onplay\\\",\\\"onplaying\\\",\\\"onprogress\\\",\\\"onratechange\\\",\\\"onreset\\\",\\\"onresize\\\",\\\"onscroll\\\",\\\"onscrollend\\\",\\\"onsecuritypolicyviolation\\\",\\\"onseeked\\\",\\\"onseeking\\\",\\\"onselect\\\",\\\"onslotchange\\\",\\\"onstalled\\\",\\\"onsubmit\\\",\\\"onsuspend\\\",\\\"ontimeupdate\\\",\\\"ontoggle\\\",\\\"onvolumechange\\\",\\\"onwaiting\\\",\\\"onwebkitanimationend\\\",\\\"onwebkitanimationiteration\\\",\\\"onwebkitanimationstart\\\",\\\"onwebkittransitionend\\\",\\\"onwheel\\\",\\\"onauxclick\\\",\\\"ongotpointercapture\\\",\\\"onlostpointercapture\\\",\\\"onpointerdown\\\",\\\"onpointermove\\\",\\\"onpointerup\\\",\\\"onpointercancel\\\",\\\"onpointerover\\\",\\\"onpointerout\\\",\\\"onpointerenter\\\",\\\"onpointerleave\\\",\\\"onselectstart\\\",\\\"onselectionchange\\\",\\\"onanimationcancel\\\",\\\"onanimationend\\\",\\\"onanimationiteration\\\",\\\"onanimationstart\\\",\\\"ontransitionrun\\\",\\\"ontransitionstart\\\",\\\"ontransitionend\\\",\\\"ontransitioncancel\\\",\\\"onbeforexrselect\\\",\\\"onafterprint\\\",\\\"onbeforeprint\\\",\\\"onbeforeunload\\\",\\\"onhashchange\\\",\\\"onlanguagechange\\\",\\\"onmessage\\\",\\\"onmessageerror\\\",\\\"onoffline\\\",\\\"ononline\\\",\\\"onpagehide\\\",\\\"onpageshow\\\",\\\"onpopstate\\\",\\\"onrejectionhandled\\\",\\\"onstorage\\\",\\\"onunhandledrejection\\\",\\\"onunload\\\",\\\"isSecureContext\\\",\\\"crossOriginIsolated\\\",\\\"scheduler\\\",\\\"performance\\\",\\\"trustedTypes\\\",\\\"crypto\\\",\\\"indexedDB\\\",\\\"localStorage\\\",\\\"sessionStorage\\\",\\\"alert\\\",\\\"atob\\\",\\\"blur\\\",\\\"btoa\\\",\\\"cancelAnimationFrame\\\",\\\"cancelIdleCallback\\\",\\\"captureEvents\\\",\\\"clearInterval\\\",\\\"clearTimeout\\\",\\\"close\\\",\\\"confirm\\\",\\\"createImageBitmap\\\",\\\"fetch\\\",\\\"find\\\",\\\"focus\\\",\\\"getComputedStyle\\\",\\\"getSelection\\\",\\\"matchMedia\\\",\\\"moveBy\\\",\\\"moveTo\\\",\\\"open\\\",\\\"postMessage\\\",\\\"print\\\",\\\"prompt\\\",\\\"queueMicrotask\\\",\\\"releaseEvents\\\",\\\"reportError\\\",\\\"requestAnimationFrame\\\",\\\"requestIdleCallback\\\",\\\"resizeBy\\\",\\\"resizeTo\\\",\\\"scroll\\\",\\\"scrollBy\\\",\\\"scrollTo\\\",\\\"setInterval\\\",\\\"setTimeout\\\",\\\"stop\\\",\\\"structuredClone\\\",\\\"webkitCancelAnimationFrame\\\",\\\"webkitRequestAnimationFrame\\\",\\\"chrome\\\",\\\"crashReport\\\",\\\"cookieStore\\\",\\\"ondevicemotion\\\",\\\"ondeviceorientation\\\",\\\"ondeviceorientationabsolute\\\",\\\"onpointerrawupdate\\\",\\\"caches\\\",\\\"documentPictureInPicture\\\",\\\"sharedStorage\\\",\\\"fetchLater\\\",\\\"getScreenDetails\\\",\\\"queryLocalFonts\\\",\\\"showDirectoryPicker\\\",\\\"showOpenFilePicker\\\",\\\"showSaveFilePicker\\\",\\\"originAgentCluster\\\",\\\"viewport\\\",\\\"onpageswap\\\",\\\"onpagereveal\\\",\\\"credentialless\\\",\\\"fence\\\",\\\"launchQueue\\\",\\\"speechSynthesis\\\",\\\"onscrollsnapchange\\\",\\\"onscrollsnapchanging\\\",\\\"ongamepadconnected\\\",\\\"ongamepaddisconnected\\\",\\\"webkitRequestFileSystem\\\",\\\"webkitResolveLocalFileSystemURL\\\",\\\"OneTrustStub\\\",\\\"OptanonWrapper\\\",\\\"isAIBotUserAgent\\\",\\\"__reactRouterVersion\\\",\\\"Prism\\\",\\\"OtTrustedType\\\",\\\"OnetrustActiveGroups\\\",\\\"OptanonActiveGroups\\\",\\\"dataLayer\\\",\\\"otStubData\\\",\\\"analyticsConnectorInstances\\\",\\\"Optanon\\\",\\\"OneTrust\\\"]\"\n### Ran Playwright code\n```js\nawait page.evaluate('() => { return JSON.stringify(Object.keys(window)); }');\n```"
}
]There's __reactRouterVersion. The app is React Router. We can fetch data via API maybe. But not necessary. We can also check the browser_console_messages for API calls. But the page loaded data. Let's see if there is network info? Playwright tool may have console logs. Not sure. Anyway, answer should be straightforward. The user says "pull, figure out whats going on here?" Maybe they want me to "pull" the page and explain. I should produce a summary. I can also note that "Fast 1" likely means "fraction of collection problems where the submission is the fastest among all submissions" and "Avg Speedup" is geometric mean speedup vs the scoring baseline. But I need to be careful about the multiple high values. Maybe I can phrase it as "the leaderboard shows, for each submitter, the SOL score, the share of problems on which they're the fastest, and the average speedup over the baseline." That's safe. Actually, I can try to get the exact data from the page's API or script to see the "Fast 1" definition. Let's search the page source for "Fast 1" or "fast". Use browser_evaluate to get all text including hidden.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { const allText = document.body.innerHTML; const idx = allText.indexOf('Fast'); return allText.substring(idx-200, idx+500); }"
}
}[
{
"type": "text",
"text": "### Result\n\" 0;\\\">?</span></span></th><th class=\\\"nv-table-header-cell\\\" data-testid=\\\"nv-table-header-cell\\\" style=\\\"text-align: center; width: 16%;\\\"><span style=\\\"display: inline-flex; align-items: center; gap: 4px;\\\">Fast<sub>1</sub><span data-testid=\\\"nv-tooltip-trigger\\\" data-state=\\\"closed\\\" style=\\\"display: inline-flex; align-items: center; justify-content: center; width: 14px; height: 14px; border-radius: 50%; border: 1px solid var(--text-color-secondary); color: var(--text-color-secondary); font-size: 0.6rem; font-weight: 700; cursor: help; flex-shrink: 0;\\\">?</span></span></th><th class=\\\"nv-table-header-cell\\\" data-testid=\\\"nv-table-header-cell\\\" style=\\\"text-align: right; width: 18%;\\\"><span style=\\\"display: inl\"\n### Ran Playwright code\n```js\nawait page.evaluate('() => { const allText = document.body.innerHTML; const idx = allText.indexOf(\\'Fast\\'); return allText.substring(idx-200, idx+500); }');\n```"
}
]"Fast<sub>1</sub>" means "Fast 1" maybe the number of kernels that are fastest. The tooltip is a React Tooltip. I can try to hover or extract the tooltip content. But I can infer from the subscript. Maybe "Fast 1" = number of top-1 (rank 1) finishes. But we saw multiple high. Maybe the leaderboard is a per-user "best submission" and the numbers are the fraction of collection problems for which the submission is the best *relative to that user's own baseline*? No. Let's try to click the tooltip or hover to get the text. The tooltip is triggered by hover. We can evaluate the tooltip content maybe in the DOM after hover. But I can simulate hover with mouse event or use the element. However, the tool "browser_evaluate" can dispatch events. But maybe easier: the tooltip text is stored in a title or data attribute. Let's search for tooltip content near the header in the HTML.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { const html = document.body.innerHTML; const idx = html.indexOf('Fast<sub>1</sub>'); return html.substring(idx, idx+2000); }"
}
}[
{
"type": "text",
"text": "### Result\n\"Fast<sub>1</sub><span data-testid=\\\"nv-tooltip-trigger\\\" data-state=\\\"closed\\\" style=\\\"display: inline-flex; align-items: center; justify-content: center; width: 14px; height: 14px; border-radius: 50%; border: 1px solid var(--text-color-secondary); color: var(--text-color-secondary); font-size: 0.6rem; font-weight: 700; cursor: help; flex-shrink: 0;\\\">?</span></span></th><th class=\\\"nv-table-header-cell\\\" data-testid=\\\"nv-table-header-cell\\\" style=\\\"text-align: right; width: 18%;\\\"><span style=\\\"display: inline-flex; align-items: center; gap: 4px;\\\">Avg Speedup<span data-testid=\\\"nv-tooltip-trigger\\\" data-state=\\\"closed\\\" style=\\\"display: inline-flex; align-items: center; justify-content: center; width: 14px; height: 14px; border-radius: 50%; border: 1px solid var(--text-color-secondary); color: var(--text-color-secondary); font-size: 0.6rem; font-weight: 700; cursor: help; flex-shrink: 0;\\\">?</span></span></th><th class=\\\"nv-table-header-cell\\\" data-testid=\\\"nv-table-header-cell\\\" style=\\\"width: 16%;\\\">Submitted</th></tr></thead><tbody class=\\\"nv-table-body\\\" data-testid=\\\"nv-table-body\\\"><tr class=\\\"nv-table-row\\\" data-testid=\\\"nv-table-row\\\" style=\\\"background-color: rgba(255, 191, 0, 0.08);\\\"><td class=\\\"nv-table-data-cell\\\" data-testid=\\\"nv-table-data-cell\\\" style=\\\"color: var(--text-color-secondary);\\\">-</td><td class=\\\"nv-table-data-cell\\\" data-testid=\\\"nv-table-data-cell\\\" title=\\\"SOL Bound\\\" style=\\\"font-weight: 600; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;\\\">SOL Bound</td><td class=\\\"nv-table-data-cell\\\" data-testid=\\\"nv-table-data-cell\\\" style=\\\"text-align: right; font-family: var(--font-mono); font-weight: 600;\\\">1.000000</td><td class=\\\"nv-table-data-cell\\\" data-testid=\\\"nv-table-data-cell\\\" style=\\\"text-align: center; font-family: var(--font-mono); font-size: 0.85rem;\\\">1.00</td><td class=\\\"nv-table-data-cell\\\" data-testid=\\\"nv-table-data-cell\\\" style=\\\"text-align: right; font-family: var(--font-mono); font-size: 0.85rem;\\\">86.72x</td><td class=\\\"nv-table-data-cell\\\" data-testid=\\\"nv-table-data-cell\"\n### Ran Playwright code\n```js\nawait page.evaluate('() => { const html = document.body.innerHTML; const idx = html.indexOf(\\'Fast<sub>1</sub>\\'); return html.substring(idx, idx+2000); }');\n```"
}
]The tooltip content is not in the HTML. I can hover the tooltip to get the text. Use browser_evaluate to dispatch a mouseover event on the tooltip trigger for the "Fast 1" header and then read the tooltip content. But it might be in a portal. Let's try. First, find the tooltip trigger element. The triggers are spans with data-testid="nv-tooltip-trigger". There are several. The first one likely corresponds to SOL Score, second to Fast 1, third to Avg Speedup. I can hover the second and then read the text of the tooltip.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { const triggers = document.querySelectorAll('[data-testid=\\\"nv-tooltip-trigger\\\"]'); if (triggers.length < 2) return 'not enough'; const t = triggers[1]; t.dispatchEvent(new MouseEvent('mouseover', {bubbles: true})); t.dispatchEvent(new MouseEvent('mouseenter', {bubbles: true})); return 'hovered'; }"
}
}[
{
"type": "text",
"text": "### Result\n\"hovered\"\n### Ran Playwright code\n```js\nawait page.evaluate('() => { const triggers = document.querySelectorAll(\\'[data-testid=\\\\\"nv-tooltip-trigger\\\\\"]\\'); if (triggers.length < 2) return \\'not enough\\'; const t = triggers[1]; t.dispatchEvent(new MouseEvent(\\'mouseover\\', {bubbles: true})); t.dispatchEvent(new MouseEvent(\\'mouseenter\\', {bubbles: true})); return \\'hovered\\'; }');\n```"
}
]Now find tooltip content. It might be in a div with role="tooltip" or class "nv-tooltip". Let's search DOM.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { const tooltips = document.querySelectorAll('[role=tooltip], .nv-tooltip, [data-testid*=tooltip]'); return Array.from(tooltips).map(t => t.innerText.trim()); }"
}
}[
{
"type": "text",
"text": "### Result\n[\n \"?\",\n \"?\",\n \"?\"\n]\n### Ran Playwright code\n```js\nawait page.evaluate('() => { const tooltips = document.querySelectorAll(\\'[role=tooltip], .nv-tooltip, [data-testid*=tooltip]\\'); return Array.from(tooltips).map(t => t.innerText.trim()); }');\n```"
}
]The tooltip content might be rendered via Radix in a portal at the bottom of body with hidden state. The hover event might not have triggered it because React's event handlers need synthetic events. Maybe the page uses CSS hover or data attribute. We can try to set the data-state to open. But we need the tooltip id. Alternatively, the tooltips are the question marks themselves. The actual tooltip content might be in a title attribute or accessible description. Since the tooltip triggers are just icons, the content may be in an aria-describedby. We can check the attributes of the triggers.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { const triggers = Array.from(document.querySelectorAll('[data-testid=\\\"nv-tooltip-trigger\\\"]')); return triggers.map(t => ({text: t.innerText, attrs: Array.from(t.attributes).map(a => a.name + '=' + a.value)})); }"
}
}[
{
"type": "text",
"text": "### Result\n[\n {\n \"text\": \"?\",\n \"attrs\": [\n \"data-testid=nv-tooltip-trigger\",\n \"data-state=closed\",\n \"style=display: inline-flex; align-items: center; justify-content: center; width: 14px; height: 14px; border-radius: 50%; border: 1px solid var(--text-color-secondary); color: var(--text-color-secondary); font-size: 0.6rem; font-weight: 700; cursor: help; flex-shrink: 0;\"\n ]\n },\n {\n \"text\": \"?\",\n \"attrs\": [\n \"data-testid=nv-tooltip-trigger\",\n \"data-state=closed\",\n \"style=display: inline-flex; align-items: center; justify-content: center; width: 14px; height: 14px; border-radius: 50%; border: 1px solid var(--text-color-secondary); color: var(--text-color-secondary); font-size: 0.6rem; font-weight: 700; cursor: help; flex-shrink: 0;\"\n ]\n },\n {\n \"text\": \"?\",\n \"attrs\": [\n \"data-testid=nv-tooltip-trigger\",\n \"data-state=closed\",\n \"style=display: inline-flex; align-items: center; justify-content: center; width: 14px; height: 14px; border-radius: 50%; border: 1px solid var(--text-color-secondary); color: var(--text-color-secondary); font-size: 0.6rem; font-weight: 700; cursor: help; flex-shrink: 0;\"\n ]\n }\n]\n### Ran Playwright code\n```js\nawait page.evaluate('() => { const triggers = Array.from(document.querySelectorAll(\\'[data-testid=\\\\\"nv-tooltip-trigger\\\\\"]\\')); return triggers.map(t => ({text: t.innerText, attrs: Array.from(t.attributes).map(a => a.name + \\'=\\' + a.value)})); }');\n```"
}
]No tooltip content. It might be a Radix tooltip with content in React portal but hidden. We can try to force it open by setting data-state on the trigger and finding a sibling? But tooltip content is likely appended to body when open. The event didn't trigger. Maybe we need to use the Playwright action tools to hover. The available tool list truncated. Maybe there is browser_hover or browser_click? The list includes browser_press_key, browser_fill_form, etc. Not sure if there is browser_hover. But we can use browser_evaluate to set the trigger's data-state to open and maybe create a tooltip. But that won't render. Let's not over-invest. We have enough. But the user might be asking for a more specific "figure out what's going on" because there might be something weird about the leaderboard (e.g., Databricks, doubleAI, Recursive at top). Maybe they want to know who is leading, what the scores mean, etc. I'll provide a concise summary. Also the user said "pull". Maybe they want me to actually "pull" the leaderboard data (scrape) into a CSV or JSON. I could generate a file. But maybe they just want an explanation. However, "pull" might be an imperative: "pull [this data], figure out what's going on here". They might want the data. I can provide the data in markdown and maybe save it. But I should not create files unless asked. I'll mention I can export it if they want. Wait, the user says "pull, figure out whats going on here? https://...". The word "pull" could be a typo or casual for "fetch". They might be asking me to pull the webpage and figure it out. So I'll provide the summary. I should also maybe point out the date: today is 2026-07-01 (from system). The leaderboard dates are up to "2 hours ago". The submissions are from March to July 2026. So this is current. I should also mention the leaderboard is for Collection 1 (L1 - Single Operations) on NVIDIA B200. It ranks user submissions by SOL Score. Top 3: Databricks, doubleAI, Recursive. SOL Score is between 0 and 1, baseline at 0.5, SOL bound at 1.0. Databricks has SOL Score 0.762, meaning it closes about 76% of the gap between the baseline and the theoretical hardware limit? Actually the formula: S = (T_b - T_k)/(T_b - T_SOL). So S=0.762 means T_k is 76% of the way from baseline to SOL. The gap closed is 76% of the baseline-to-SOL headroom. "Avg Speedup" = geometric mean speedup over the scoring baseline (Databricks 12.05x). "Fast 1" = share of collection problems where the submission is the fastest (Databricks 0.98). But I need to be careful with the multiple high Fast1. Maybe "Fast 1" is not strict rank; maybe it's the fraction of test cases where the submission is the fastest, but because each submission is a set of per-kernel best solutions, multiple users can have high fractions if they each have different kernels. But the numbers are too high for three users. Let's calculate: collection has 94 L1 problems. Databricks 0.98 -> 92 problems fastest. doubleAI 0.97 -> 91 problems fastest. Recursive 0.97 -> 91 problems fastest. That's impossible. Unless "Fast 1" is measured relative to the *baseline*, not other submissions. Wait, baseline is 0.00. Maybe "Fast 1" is the fraction of problems where the submission is faster than the "Fast 1" threshold? No. Could it be that the "Fast 1" column is the fraction of collection problems where the submission is at least as fast as the "Fast 1" baseline? No. Let's search for a definition in the paper. The paper might have "fast1".
{
"query": "\"Fast 1\" \"SOL-ExecBench\" NVIDIA leaderboard metric"
}# Web Search Results for ""Fast 1" "SOL-ExecBench" NVIDIA leaderboard metric" ## 1. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/kernel/47/B200 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA ... | Rank | User | SOL Score? | Latency? | Fast1? | Avg Speedup? | Submitted | | --- | --- | --- | --- | --- | --- | --- | | - | SOL Bound | 1.000000 | 0.069243 ms | 16 / 16 = 1.00 | 9.17x | - | | 1 | Recursive | 0.802687 | 0.170334 ms | 16 / 16 = 1.00 | 3.85x | 2026-06-07 | | 2 | doubleAI | 0.728192 | 0.228869 ms | 16 / 16 = 1.00 | 2.67x | 2026-05-20 | | 3 | Amir M. Mir | 0.723674 | 0.225066 ms | 16 / 16 = 1.00 | 3.04x | 2026-06-02 | | #4 | Cursor | 0.539778 | 0.460911 ms | 14 / 16 = 0.88 | 1.16x | 2026-04-14 | | - | Scoring Baseline | 0.500000 | 0.526809 ms | 0 / 16 = 0.00 | 1.00x | - | | - | Reference Implementation | 0.247745 | 1.436483 ms | 0 / 16 = 0.00 | 0.37x | - | | #5 | Igor Bragaia | 0.246384 | 1.446753 ms | 0 / 16 = 0.00 | 0.37x | 2026-05-28 | ## 2. NVIDIA/SOL-ExecBench URL: https://github.com/nvidia/sol-execbench [HuggingFace Dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) | [Leaderboard](https://research.nvidia.com/benchmarks/sol-execbench) | [Technical Report](https://arxiv.org/abs/2603.19173) **Speed-Of-Light ExecBench** is a rigorous GPU kernel evaluation and benchmarking framework built to benchmark AI-generated kernel solutions written with the variety of DSLs that NVIDIA hardware supports. ... Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)). Supported kernel languages: PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++. ## 3. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/collection/1 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA Sorry, your browser does not support inline SVG. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA [SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench) [GitHub ↗](https://github.com/nvidia/sol-execbench)[HuggingFace ↗](https://huggingface.co/datasets/nvidia/sol-execbench)[Paper ↗](https://arxiv.org/abs/2603.19173)[Log In](https://research.nvidia.com/benchmarks/sol-execbench/login) Sorry, your browser does not support inline SVG. ## 4. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/leaderboard/kernel/36/B200 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA Sorry, your browser does not support inline SVG. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA [SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench) [GitHub ↗](https://github.com/nvidia/sol-execbench)[HuggingFace ↗](https://huggingface.co/datasets/nvidia/sol-execbench)[Paper ↗](https://arxiv.org/abs/2603.19173)[Log In](https://research.nvidia.com/benchmarks/sol-execbench/login) Sorry, your browser does not support inline SVG. ## 5. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/kernel/27 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA ... | # | hidden_size | num_attention_heads | head_dim | qkv_hidden | half_head_dim | batch_size | num_frames | num_patches_per_frame | Baseline (ms) | SOL (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | 1024 | 16 | 64 | 3072 | 32 | 1 | 14 | 541 | 6.1692 | 0.1652 | | 2 | 1024 | 16 | 64 | 3072 | 32 | 8 | 8 | 49 | 0.3520 | 0.0177 | | 3 | 1024 | 16 | 64 | 3072 | 32 | 1 | 32 | 256 | 7.0393 | 0.1901 | | 4 | 1024 | 16 | 64 | 3072 | 32 | 1 | 4 | 691 | 1.0096 | 0.0305 | | 5 | 1024 | 16 | 64 | 3072 | 32 | 4 | 6 | 169 | 0.6579 | 0.0285 | | 6 | 1024 | 16 | 64 | 3072 | 32 | 2 | 32 | 64 | 1.1119 | 0.0383 | | 7 | 1024 | 16 | 64 | 3072 | 32 | 1 | 24 | 144 | 1.4127 | 0.0434 | | 8 | 1024 | 16 | 64 | 3072 | 32 | 1 | 3 | 919 | 1.0075 | 0.0304 | | 9 | 1024 | 16 | 64 | 3072 | 32 | 1 | 8 | 613 | 2.8388 | 0.0775 | | 10 | 1024 | 16 | 64 | 3072 | 32 | 1 | 48 | 64 | 1.2544 | 0.0360 | | 11 | 1024 | 16 | 64 | 3072 | 32 | 64 | 1 | 16 | 0.1371 | 0.0052 | | 12 | 1024 | 16 | 64 | 3072 | 32 | 1 | 16 | 256 | 1.9695 | 0.0573 | | 13 | 1024 | 16 | 64 | 3072 | 32 | 1 | 6 | 373 | 0.7400 | 0.0221 | | 14 | 1024 | 16 | 64 | 3072 | 32 | 2 | 4 | 211 | 0.3720 | 0.0114 | | 15 | 1024 | 16 | 64 | 3072 | 32 | 1 | 4 | 64 | 0.1069 | 0.0017 | | 16 | 1024 | 16 | 64 | 3072 | 32 | 4 | 16 | 64 | 0.6614 | 0.0289 | ... Leaderboard B200 ... | # | User | SOL Score | Latency | | --- | --- | --- | --- | | - | SOL Bound | 1.000000 | 0.029007 ms | | 1 | Recursive | 0.888152 | 0.133789 ms | | 2 | doubleAI | 0.846686 | 0.176787 ms | | 3 | Amir M. Mir | 0.791510 | 0.257950 ms | | #4 | Cursor | 0.576573 | 0.687652 ms | | - | Scoring Baseline | 0.500000 | 0.927659 ms | | #5 | Igor Bragaia | 0.230073 | 3.151349 ms | | - | Reference Implementation | 0.229889 | 3.161357 ms | ... View Full Leaderboard →
Search result 5 shows kernel 27 page with per-kernel details, including "Baseline (ms)" and "SOL (ms)" for each test case. The "Fast 1" in individual leaderboard is likely the count of test cases where the submission is #1 among all submissions. In the collection leaderboard, the value is normalized to [0,1] by dividing by total test cases in the collection. But again, multiple high values impossible. Maybe the collection's "Fast 1" is computed per problem differently: maybe it's the fraction of *problems where the submission is faster than the baseline*? But baseline is 0.00. Wait, maybe the collection leaderboard is *not* per problem; it could be a "collection" of all 94 L1 problems, and each user may submit solutions for each problem. The "Fast 1" column is the fraction of problems where the user's submission is faster than the *baseline* (scoring baseline). But the scoring baseline row shows 0.00, and reference implementation shows 0.08. So a submission faster than baseline would have Fast 1 >0. But top users have 0.98, meaning they are faster than baseline on 98% of problems. That makes sense: they only fail to beat baseline on 2% of problems. This also explains why multiple users can have high values: they all beat baseline on most problems. Baseline is 0.00 because it is exactly baseline, so none faster than baseline. Reference implementation is 0.08 because it beats baseline on 8% of problems (or maybe 8% of test cases). The "Fast 1" might be "fraction of test cases where runtime is faster than baseline". But why call it "Fast 1"? Maybe it's "fastest" relative to baseline? No, "1" might mean "baseline" as rank 1? Hmm. Actually, the baseline row is 0/16 on individual kernel. So "Fast 1" is not "faster than baseline" because the baseline itself is 0/16. That fits: baseline is not faster than itself. Reference implementation is 0/16 on kernel 47, meaning it's not faster than baseline either. But on the collection, reference is 0.08. So maybe reference is faster than baseline on some problems. So "Fast 1" could be "fraction of test cases where submission is faster than baseline". That would be plausible: the baseline is 0.00, the reference is 0.08 (beats baseline on 8% of collection entries), top submissions beat baseline on 98% of entries. But the individual kernel 47 page: baseline is 0.526809 ms, reference is 1.436 ms (slower), top submissions are faster. Fast 1 values for top are 16/16 = 1.00, meaning they are faster than baseline on all 16 test cases. The reference is 0/16. So "Fast 1" = number of problem instances where the submission is faster than the baseline? But why would the top three all be 16/16? Yes, all three are faster than baseline on all test cases. That fits perfectly. The fact that their actual latencies differ doesn't matter; they all beat the baseline. The "1" in Fast 1 might mean "1x speedup"? Or "top-1"? Hmm. Wait, on the individual kernel 47 page, "Fast 1" values are "16 / 16 = 1.00". If it means faster than baseline, then for any submission faster than baseline on all 16 test cases, it gets 1.00. That matches the top three. The scoring baseline gets 0/16. The reference gets 0/16. Cursor gets 14/16 = 0.88 because it beats baseline on 14 of 16 test cases. This is a very plausible interpretation. But the column name "Fast 1" with subscript 1 might mean "faster than the baseline" (i.e., rank 1? Or "1x" speedup?). Actually, if "Fast 1" means fraction of test cases where the submission is faster than the baseline (i.e., >1x speedup), then "Fast 1" is a weird name. Could be "Fast >=1x". Or "Fast > Baseline". But the subscript 1 might be from "Fast_{>1x}"? Not sure. Alternatively, "Fast 1" could mean "number of test cases where the submission is the fastest". But the values in individual kernel are all 16/16 for top three, which would be impossible if strict. But if "Fast 1" means "faster than baseline", then "1" could refer to "baseline" as the first reference? Hmm. Wait, the header in individual page is "Fast1?" (no space) maybe not "Fast 1". It could be "Fast1" as a metric. The subscript in collection is "Fast<sub>1</sub>". It could be a mathematical notation: Fast_1 might be the "fastest" indicator at top-1? Or "fast 1" meaning "1x faster than baseline". The value 0.98 means 98% of test cases are at least 1x faster than baseline? Actually, if baseline speedup is 1.00x, then "Fast 1" might be the fraction of test cases with speedup >= 1? The top three have 0.98, meaning they have >=1x speedup on 98% of test cases. The scoring baseline is 0.00 (obviously not >1x). The reference is 0.08. This matches. But the individual kernel page for kernel 47: top three have 16/16 = 1.00, meaning they have >=1x speedup on all test cases. But Amir M. Mir's speedup is 3.04x, doubleAI 2.67x, Recursive 3.85x. All >1. Cursor 14/16 = 0.88, speedup 1.16x. It might be that on 2 of 16 test cases, Cursor's speedup is slightly below 1x? Actually speedup is average? The column "Avg Speedup" is average; Cursor has 1.16x average but maybe some cases below 1. So Fast_1 might be count of cases where speedup > 1. That makes sense. But wait, the baseline row has Avg Speedup 1.00x and Fast_1 0.00. That fits: baseline is exactly 1x, not >1x. Reference has Avg Speedup 0.37x and Fast_1 0.00 (all <1). Good. So "Fast_1" likely means "fraction of collection entries where the submission is faster than the baseline (i.e., speedup > 1)". The "1" might refer to the baseline speedup threshold. This is a very reasonable interpretation. The "Fast" in the column name could be "Fast vs baseline". The subscript 1 might be the threshold multiplier (1x). Or maybe the metric is "Fast1 = fraction of problems where the submission is faster than the reference implementation"? But reference is 0.08, so it beats baseline on some. The baseline is 0.00. So yes. Alternatively, "Fast_1" could be "fraction of test cases where the submission achieves the fastest time among all submissions on that test case". But then multiple top all 1.00 on individual kernel contradicts. Unless the page shows a snapshot of the leaderboard at different times, but still. The "faster than baseline" interpretation is more consistent with the baseline row. But the column name "Fast1" on the individual kernel page is under "Fast1?" with values "16/16 = 1.00". If it means faster than baseline, then for a single kernel, it's the number of test cases where the submission is faster than baseline. The "1" is a threshold (1x). That seems plausible. However, why would the top three have 1.00 if they all beat baseline on all 16? Yes. The fact that they are not all equally fast doesn't matter. But in the collection, the top three have 0.97-0.98, meaning they don't beat baseline on all 94 problems, only 92-93. Some problems are harder than baseline. I think this is the correct interpretation. Let me verify by looking at the collection page for the reference implementation: it has Fast_1 0.08, meaning 8% of L1 problems it beats the baseline. The reference implementation might be slower than baseline on most problems. The scoring baseline has Fast_1 0.00. The "Avg Speedup" is the geometric mean of speedups over baseline across all problems. Baseline is 1.00x. Reference is 0.41x (slower). Databricks is 12.05x. The SOL Bound is 86.72x (the theoretical max over baseline). So the SOL Score is not just speedup normalized; it's a different metric that maps baseline=0.5 and SOL=1.0. For a given speedup x, the SOL Score is maybe 0.5 + 0.5*(x - 1)/(x_SOL - 1)? Let's test: Databricks speedup 12.05, SOL speedup 86.72. (12.05-1)/(86.72-1)=11.05/85.72=0.129, times 0.5 = 0.0645, plus 0.5 = 0.5645. But Databricks SOL score is 0.762. So not that. Maybe the SOL Score is computed per problem as S = 0.5 * (baseline_runtime / candidate_runtime) maybe? Let's see: For Databricks, if baseline_runtime/candidate = 12.05, then S maybe 0.5*(1 + something). Actually the formula from the paper: S = (T_b - T_k)/(T_b - T_SOL). If T_SOL is much smaller than T_b, then S ≈ (T_b/T_k - 1)/(T_b/T_SOL - 1) + 0.5? Wait, the formula is S = (T_b - T_k)/(T_b - T_SOL). Since T_b > T_SOL. If T_k = T_b, S=0. If T_k = T_SOL, S=1. But the baseline row has S=0.5, not 0. So the paper said "We design the score to assign the baseline a midpoint score rather than zero". That means the formula might be S = 0.5 + 0.5 * (T_b - T_k)/(T_b - T_SOL). Then baseline (T_k=T_b) gives 0.5, SOL gives 1.0. Below baseline gives <0.5 down to 0. For reference T_k > T_b, (T_b - T_k) negative, S < 0.5. Good. Let's test for Databricks: Suppose baseline_runtime / candidate_runtime = 12.05 (avg speedup). We don't know T_SOL relative to baseline. The SOL Bound avg speedup is 86.72. For a single problem, the formula uses runtimes. Let's try to compute for the average values: For the collection aggregate, maybe they compute S = 0.5 + 0.5 * (sum(baseline - candidate)/sum(baseline - SOL)). Hard to check with just averages. But not needed. Now, the user might want me to "pull" the data. I'll provide a summary and maybe a table of top 10. I should not include too much raw data. The answer should be concise. Also, I should mention the source and the tools: I used Playwright to render the page because the static fetch returned empty. This is meta but maybe helpful. I should also mention if they want me to clone the repo or scrape other collections, I can do it. Now, the user didn't ask for file creation. I should not create a file. But maybe they want to "pull" the leaderboard into a CSV? The user said "pull, figure out whats going on here?" The phrase "pull" could be interpreted as "fetch/extract". But I think explanation is enough. I can also include a markdown table of the top 10. I need to respond in a helpful, concise manner. Use citations maybe? There are no file references. I can cite the URL. But the system says use <ref_file> and <ref_snippet> for file references. Not applicable here. I should be careful not to add emojis. Let's craft the answer: - This URL is the SOL-ExecBench public leaderboard for Collection 1 (L1 - Single Operations) on the NVIDIA B200 GPU. - SOL-ExecBench is a benchmark for AI-generated/optimized CUDA kernels from real model subgraphs. It evaluates performance against an analytical Speed-of-Light (SOL) bound derived by SOLAR, not just a software baseline. - The leaderboard columns: - SOL Score: 0.5 = scoring baseline, 1.0 = hardware SOL bound. >0.5 means better than baseline; <0.5 means worse. It's a normalized gap-closing metric. - Fast_1: fraction of collection problems/test cases where the submission is faster than the baseline (i.e., speedup > 1x). Actually I'm not 100% sure. I can phrase as "likely the fraction of problems where the submission beats the baseline (i.e., >1× speedup)". But to be safe, maybe say "appears to be the fraction of collection entries that are faster than the baseline". If I say "likely", it's honest. - Avg Speedup: geometric mean speedup over the scoring baseline. - Submitted: when the result was submitted. - Top 3: Databricks (0.7619, 12.05×, 0.98), doubleAI (0.7587, 13.23×, 0.97), Recursive (0.7584, 12.31×, 0.97). - Context: The hardware SOL bound is 86.72× faster than the baseline on average. So even the leaders are far from the theoretical maximum but have made huge gains over the baseline. - Other notables: Cursor in 5th, Amir M. Mir 4th. The reference implementation is at 0.237 SOL score, 0.41× speedup (i.e., slower than baseline). Baseline is 0.5 SOL score, 1.00×. I could include a top 10 table. But I should verify the "Fast_1" interpretation. Since I can't be 100% sure, I can phrase it carefully: "Fast_1 looks like the fraction of problems on which the submission beats the baseline (speedup > 1×)". Actually, the values for top 3 being 0.97-0.98 support this. The baseline is 0.00. The reference is 0.08. So yes. Wait, maybe "Fast_1" is "fraction of collection where the submission is the fastest". But then the baseline being 0.00 doesn't make sense if the baseline is fastest on some? No, baseline is not fastest. But multiple top 0.98 is contradictory. The "faster than baseline" interpretation is more consistent. The "1" could be the threshold (1x). So I think that's correct. Actually, there is another possibility: "Fast_1" could be the fraction of test cases where the submission is faster than the *reference implementation*? But the reference is 0.08, meaning it beats itself on 8%? That doesn't make sense. Baseline 0.00 also doesn't fit. So "faster than baseline" is best. Another possibility: "Fast_1" = fraction of collection entries where the submission is the fastest *among all submitted solutions*, but because of how the collection aggregates, the value is normalized and multiple submissions can be high if they are on different problems. But again, can't sum >1. However, maybe the "Fast_1" value is computed as (1 / rank) weighted average? For top three all near 1? Could be a top-k ranking score like "fraction of top-1 positions". But the baseline and reference would be 0.00, which fits (they are not ranked). But multiple top all 0.98 impossible. Unless the "Fast_1" is not the fraction of top-1s but the average of "is in the top 1% of runtime"? Hmm. Let's test with the individual kernel 47: Top three all 16/16. If "Fast_1" is "faster than baseline", top three all beat baseline on all 16 test cases. The reference is 0/16. That fits. If "Fast_1" is "fastest among submissions", only one can be 16/16. So the "faster than baseline" is strongly supported. Thus I will state that. But wait, the individual kernel page header is "Fast1?" not "Fast_1". The search result shows the table with "Fast1?" maybe from markdown. The values for top are "16 / 16 = 1.00". If it were "faster than baseline", why would they call it "Fast1"? The "1" could be the "1x" threshold. In many benchmarks, "1x" is baseline. So "Fast1" means "speedup > 1×". This is plausible. Now, the arxiv paper says "Any solution faster than [baseline] will receive a [score]". The baseline is fixed. So a metric counting how many are faster than baseline is natural. But I won't over-explain. Now, the user asked "pull, figure out whats going on here?" They may want me to "pull" the repo and figure out the benchmark. But the URL is a leaderboard. I'll interpret as "fetch and summarize this leaderboard". I can also mention that the page is dynamic/JS-rendered, so a simple curl doesn't get the data; I used Playwright to render it. Should I include a raw data dump? Maybe not. The top 10 table is enough. Also, the user might want to know what the "collection/1" means. The search results show that collection 1 is L1 - Single Operations (94 problems). The benchmark also has L2, Quant, FlashInfer-Bench. The page is the B200 device view. I can provide a brief explanation of the benchmark categories. Let me also consider if I should "pull" the repository. The user might be a developer wanting to use the benchmark. If they say "pull", maybe they want to clone the GitHub repo. I can ask: "Do you want me to clone the repo too?" But better to answer and offer. The answer: ``` Fetched the page (it's JS-rendered, so I used a browser). Here's what's at that URL: NVIDIA SOL-ExecBench leaderboard – Collection 1 (L1 – Single Operations) on B200. What the columns mean: - SOL Score: 0.5 = baseline, 1.0 = hardware speed-of-light. It measures how much of the gap between the baseline and the theoretical B200 limit a submission closes. - Fast_1: fraction of the collection's problems/test cases where the submission is faster than the baseline (i.e., >1× speedup). - Avg Speedup: geometric-mean speedup over the baseline. Current top 10 (as of the page load): [table] Key takeaways: - Leaders are closing ~76% of the baseline-to-SOL gap and are ~12–13× faster than the baseline. - The hardware SOL bound is 86.72× faster than the baseline on average, so there is still a lot of headroom. - The released reference implementation is actually slower than the baseline (0.41×). ``` But I need to be careful: "Fast_1" might be wrong. Maybe I should say "Fast_1 looks like the fraction of problems on which the submission is faster than the baseline." Use cautious phrasing. Alternatively, I could say "Fast_1 appears to be the fraction of the collection's problems where the submission is the fastest (or at least faster than the baseline)". Hmm. Let's see if I can verify with the page by looking at a specific kernel and comparing. The individual kernel page shows Fast1 values and speedups. For cursor on kernel 47, Fast1=14/16=0.88, Avg Speedup=1.16x. If Fast1 is "faster than baseline", then on 14 test cases Cursor's speedup >1, on 2 cases <=1. Average speedup 1.16x, so plausible. If Fast1 were "fastest among submissions", then there are 5 submissions listed; Cursor being fastest on 14 of 16 is impossible because top three are also fastest. So Fast1 is definitely not "fastest among submissions" on that page. It's "faster than baseline" (or faster than reference?). Actually, on individual kernel 47, there are 5 submissions plus baseline and reference. Cursor is the 4th fastest. It cannot be fastest on 14/16. So Fast1 = faster than baseline is the only interpretation that fits. Great. Thus I can state confidently: Fast_1 = number of test cases where the submission runs faster than the scoring baseline (normalized to a fraction). This is consistent. Now, the SOL Score formula: S = 0.5 + 0.5*(T_b - T_k)/(T_b - T_SOL). The paper says baseline is midpoint 0.5. So I can explain that. For the collection, the "Avg Speedup" is geometric mean of per-problem speedups over the baseline. The "Fast_1" is the fraction of test cases (or problem instances) with speedup >1. Now, the top three have SOL scores around 0.76, but avg speedups differ: doubleAI 13.23x vs Databricks 12.05x. Databricks has higher SOL score despite lower speedup because the SOL score weights problems by the baseline-to-SOL gap. If a problem has a large gap, a high speedup there contributes more to SOL score. So doubleAI may have high speedups on problems with smaller gaps. This is a nuance. Now, I should produce the answer. I will also mention that the page is live and I loaded it. The current top data is as of the snapshot. I need to be concise but informative. Use markdown. Should I include citations? The system says use <ref_file> and <ref_snippet> for file references. I can cite the URL as a link. Not required to use the XML tags for web pages, but maybe I can include a link. I should also consider that the user may want to know "what's going on" in a more strategic sense: "Why is Databricks leading? Is there a new SOTA?" I can note that the top three are close, with Databricks slightly ahead. The leaderboard is active. Also, maybe the user wants me to "pull" the repo and "figure out what's going on" in terms of the code. The URL is to the leaderboard, not the repo. So I'll focus on the leaderboard. Let's draft the answer. I can also mention the exact page title and sections. Potential answer: ``` I rendered the page with Playwright because the leaderboard is JS-rendered. Here's the snapshot: NVIDIA SOL-ExecBench — Collection 1 / B200 (L1 – Single Operations) This is a public leaderboard for GPU kernel optimizations. SOL-ExecBench has 235 real-world kernel problems derived from HuggingFace models. Collection 1 is the 94 L1 “single-operation” kernels (e.g., RMSNorm, SwiGLU, RoPE, GQA). They are evaluated on a single B200 GPU with locked clocks. Columns: - SOL Score: normalized gap-closing metric. 0.5 = fixed scoring baseline, 1.0 = analytical Speed-of-Light (SOL) bound from SOLAR. 0.76 ≈ closes 76% of the baseline→SOL gap. - Fast_1: share of the collection's problem instances on which the submission is faster than the baseline (i.e., >1× speedup). - Avg Speedup: geometric-mean speedup over the baseline. Current top (live page): Rank | User | SOL Score | Fast_1 | Avg Speedup | Submitted 1 | Databricks | 0.7619 | 0.98 | 12.05× | 11h ago 2 | doubleAI | 0.7587 | 0.97 | 13.23× | 3d ago 3 | Recursive | 0.7584 | 0.97 | 12.31× | 2026-06-09 4 | Amir M. Mir | 0.6454 | 0.82 | 7.92× | 2026-06-18 5 | Cursor | 0.5825 | 0.63 | 3.98× | 2026-04-14 ... Reference Implementation | 0.2370 | 0.08 | 0.41× | — Scoring Baseline | 0.5000 | 0.00 | 1.00× | — SOL Bound | 1.0000 | 1.00 | 86.72× | — Observations: - The top three are tightly clustered, all within ~0.3% SOL score. - Best submissions are ~12–13× faster than the baseline and beat the baseline on ~97–98% of the L1 problems. - The theoretical SOL bound is still ~86.72× faster than the baseline on average, so there is a lot of room left. - The released reference implementation is slower than the baseline (0.41×), which is why its SOL score is <0.5. If you want me to clone the benchmark repo, pull a different collection/device, or export the full table, let me know. ``` This is good. I should not include the entire table, just top 10. I'll include top 10. But I need to ensure the answer is not too long. The user asked a short question. I'll keep it compact but informative. I should also mention that "collection/1" is L1 Single Operations. The page shows "L1 - Single Operations". Good. Now, should I save the raw data to a file? The user didn't ask. I can offer. I should also consider if there is any "pull" git action. Maybe they want to know if there is a repo to pull. I can mention the GitHub repo is https://github.com/nvidia/sol-execbench. But I already included that. Let's write the final answer.
I rendered the page (it’s JS‑generated, so a plain fetch returns empty) and pulled the current leaderboard.
What it is:
NVIDIA SOL‑ExecBench — public leaderboard for Collection 1 / B200, i.e. the L1 “Single Operations” set of 94 real‑world GPU kernel problems (things like RMSNorm, SwiGLU, RoPE, GQA) benchmarked on a single NVIDIA B200.
What the columns mean:
| Column | Meaning |
|---|---|
| SOL Score | Gap‑closing metric vs. the hardware “Speed of Light” bound. 0.5 = scoring baseline, 1.0 = theoretical SOL bound. A score of 0.76 means the submission has closed ~76% of the baseline→SOL gap. |
| Fast₁ | Fraction of the collection’s problem instances where the submission is faster than the baseline (i.e., >1× speedup). |
| Avg Speedup | Geometric‑mean speedup over the scoring baseline. |
Current top 10 (as of the live page):
| Rank | User | SOL Score | Fast₁ | Avg Speedup | Submitted |
|---|---|---|---|---|---|
| 1 | Databricks | 0.7619 | 0.98 | 12.05× | 11h ago |
| 2 | doubleAI | 0.7587 | 0.97 | 13.23× | 3d ago |
| 3 | Recursive | 0.7584 | 0.97 | 12.31× | 2026‑06‑09 |
| 4 | Amir M. Mir | 0.6454 | 0.82 | 7.92× | 2026‑06‑18 |
| 5 | Cursor | 0.5825 | 0.63 | 3.98× | 2026‑04‑14 |
| 6 | Igor Bragaia | 0.1609 | 0.15 | 0.37× | 2026‑05‑29 |
| 7 | KernelOpt | 0.1070 | 0.80 | 0.45× | 2026‑04‑28 |
| 8 | wychi888 | 0.0972 | 0.79 | 0.42× | 2026‑06‑11 |
| 9 | Gilded Quail | 0.0867 | 0.70 | 0.39× | 2026‑06‑16 |
| 10 | Argus | 0.0827 | 0.79 | 0.39× | 12h ago |
Reference lines on the same table:
| SOL Score | Fast₁ | Avg Speedup | |
|---|---|---|---|
| SOL Bound | 1.0000 | 1.00 | 86.72× |
| Scoring Baseline | 0.5000 | 0.00 | 1.00× |
| Reference Implementation | 0.2370 | 0.08 | 0.41× |
Key takeaways:
If you want me to clone the benchmark repo, scrape the other collections (L2, Quant, FlashInfer), or export the full table, just say the word.
how much does a run cost
The user is asking "how much does a run cost" in the context of SOL-ExecBench. This is ambiguous: could mean cost of running a submission on the benchmark (NVIDIA leaderboard likely free to submit?), cost of running the benchmark locally (GPU cost), cost of one leaderboard evaluation run, cost of using the dataset, or cost in terms of compute time. I need to investigate. I should search the SOL-ExecBench docs for cost, pricing, submission fees, or running cost. The benchmark is likely free and open source, but evaluating on B200 requires expensive GPUs. The leaderboard might be hosted by NVIDIA and submissions are free. The cost to run a single submission on a B200 on the cloud could be $X per hour. I need to estimate. Let's search web for "SOL-ExecBench cost", "NVIDIA SOL-ExecBench submission cost", "B200 GPU cost per hour", "how much does a run cost". Also check the repo for any pricing or cost mention.
{
"query": "SOL-ExecBench cost run submission NVIDIA"
}{
"query": "NVIDIA B200 GPU cloud price per hour 2026"
}# Web Search Results for "SOL-ExecBench cost run submission NVIDIA"
## 1. NVIDIA/SOL-ExecBench
URL: https://github.com/nvidia/sol-execbench
# NVIDIA/SOL-ExecBench
...
[HuggingFace Dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) | [Leaderboard](https://research.nvidia.com/benchmarks/sol-execbench)
| [Technical Report](https://arxiv.org/abs/2603.19173)
**Speed-Of-Light ExecBench** is a rigorous GPU kernel evaluation and benchmarking framework built to benchmark AI-generated kernel solutions written with the variety of DSLs that NVIDIA hardware supports.
...
Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)).
Supported kernel languages: PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++.
...
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
- [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/guides/cli) (`pip install huggingface-hub[cli]`)
- NVIDIA driver version 580+
...
This downloads the [SOL-ExecBench](https://huggingface.co/datasets/nvidia/SOL-ExecBench) and [FlashInfer Trace](https://huggingface.co/datasets/flashinfer-ai/flashinfer-trace) datasets into `data/`.
...
## Evaluating a Solution
...
Inside the container, use the `sol-execbench` CLI:
...
```bash
# Evaluate using a problem directory (contains definition.json + workload.jsonl)
sol-execbench <problem_dir> --solution solution.json
...
# Or specify files explicitly
sol-execbench --definition def.json --workload wkl.jsonl --solution sol.json
...
compile-timeout
...
## Running a Dataset
...
Use `scripts/run_dataset.py` to evaluate an entire dataset (or a single problem) in batch. By default it runs the definition's reference implementation as the solution unless `--solution-name` is specified. Saves to `./out/{subset}` by default.
...
```bash
# Run all problems in the benchmark.
# Auto builds solution.json from a single code file
u...
## 2. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA
URL: https://research.nvidia.com/benchmarks/sol-execbench/collection/1
SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA
Sorry, your browser does not support inline SVG.
SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA
[SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench)
[GitHub ↗](https://github.com/nvidia/sol-execbench)[HuggingFace ↗](https://huggingface.co/datasets/nvidia/sol-execbench)[Paper ↗](https://arxiv.org/abs/2603.19173)[Log In](https://research.nvidia.com/benchmarks/sol-execbench/login)
Sorry, your browser does not support inline SVG.
## 3. docs/workload.md at main · NVIDIA/SOL-ExecBench
URL: https://github.com/NVIDIA/SOL-ExecBench/blob/main/docs/workload.md
```md
# Workload Schema
...
A **Workload** makes a Definition concrete and executable by binding specific values to all variable axes and specifying where the input data comes from. It is the exact configuration under which a Solution is benchmarked.
...
The evaluator runs your solution against every Workload in the problem's set — typically 16–48 workloads per problem.
...
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `uuid` | string | Yes | Unique identifier for this workload |
| `axes` | object | Yes | Map of `var` axis names → concrete integer values |
| `inputs` | object | Yes | Map of input names → input descriptor (how to source the data) |
| `tolerance` | object | No | Per-workload correctness bounds (defaults below) |
...
**Storage:** In the dataset, all workloads for one definition are stored as a JSONL file (one JSON object per line).
...
you see `"type
...
"custom"`
...
call `definition.custom_
...
run()` function receives them as normal tensors
...
you don't need to handle this differently.
...
## `tolerance` — Per-Workload Correctness Bounds
...
Each workload can specify its own tolerance for correctness checking via the `tolerance` field. If omitted, defaults apply.
...
| Type |
...
-------|------|---------|-------------
...
max_atol` | float |
...
1e-2` | Maximum absolute error threshold |
...
max_rtol` | float
...
required_matched_ratio
...
Fraction of elements that must pass the tolerance
...
max_error_cap`
...
max absolute error (fails
...
) |
...
allow_negative_inf`
...
, matching `-inf`
...
in both output and
...
are treated as correct and excluded from error computation |
...
**Correctness formula** (torch.allclose style): An element passes if `|output - reference| <= max_atol + max_rtol * |reference|`. The workload passes if at least `required_matched_ratio` of all elements pass AND (if `max_error_cap` is set) the largest absolute error is below the cap.
...
## How the Evaluator Uses Workloads
...
For ea...
## 4. docs/trace.md at main · NVIDIA/SOL-ExecBench
URL: https://github.com/NVIDIA/SOL-ExecBench/blob/main/docs/trace.md
A `Trace` is an atomic, immutable record of a **single benchmark run**. It links a specific `Solution` to a specific `Definition`, details the exact `workload` configuration used for the run (i.e., shapes and input data), and records the complete `evaluation` result. The collection of all Trace files forms the database of benchmark results.
...
| **Field** | **Type** | **Required** | **Description** |
| --- | --- | --- | --- |
| `definition` | string | Yes | The `name` of the `Definition` used in this run. |
| `solution` | string | No | The `name` of the `Solution` tested in this run. |
| `workload` | object | Yes | An object describing the specific input configuration for this run. |
| `evaluation` | object | No | An object containing the detailed results of this run. |
...
| **Field** | **Type** | **Required** | **Description** |
| --- | --- | --- | --- |
| `status` | string | Yes | The final status of the evaluation run. Has to be one of the following: `"PASSED"`, `"INCORRECT_SHAPE"`, `"INCORRECT_NUMERICAL"`, `"INCORRECT_DTYPE"`, `"RUNTIME_ERROR"`, `"COMPILE_ERROR"`, `"TIMEOUT"`, `"REWARD_HACK"`, `"INVALID_REFERENCE"`. |
| `log` | string | No | The embedded record of the stdout and stderr of the evaluation run (default: `""`). |
| `correctness` | object | Yes | The summarized correctness results across all entries in the dataset. |
| `performance` | object | Yes | The summarized performance metrics across all entries in the dataset. |
| `environment` | object | Yes | A snapshot of the hardware and software execution environment. |
| `timestamp` | string | Yes | The ISO 8601 timestamp of when this summary was generated. |
...
```python
{
"definition": "rmsnorm",
"solution": "rmsnorm_triton_v1",
"workload": {
"uuid": "6120f144-b973-4bd9-b884-77ecb132914e",
"axes": {
"batch_size": 32
},
"inputs": {
"input": {
"type": "safetensors",
"path": "/data/rmsnorm_evals/b32_input.safetensors",
"tensor_key": "input"
...
## 5. zsol-bench v1.0.1.1
URL: https://pypi.org/project/zsol-bench/
NVIDIA SOL ExecBench - GPU kernel evaluation framework
...
[HuggingFace Dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) | [Leaderboard](https://research.nvidia.com/benchmarks/sol-execbench)
| [Technical Report](https://arxiv.org/abs/2603.19173)
**Speed-Of-Light ExecBench** is a rigorous GPU kernel evaluation and benchmarking framework built to benchmark AI-generated kernel solutions written with the variety of DSLs that NVIDIA hardware supports.
...
Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)).
...
This downloads the [SOL-ExecBench](https://huggingface.co/datasets/nvidia/SOL-ExecBench) and [FlashInfer Trace](https://huggingface.co/datasets/flashinfer-ai/flashinfer-trace) datasets into `data/`.
...
Inside the container, use the `sol-execbench` CLI:
...
```bash
# Evaluate using a problem directory (contains definition.json + workload.jsonl)
sol-execbench <problem_dir> --solution solution.json
...
# Or specify files explicitly
sol-execbench --definition def.json --workload wkl.jsonl --solution sol.json
```
...
## Running a Dataset
...
Use `scripts/run_dataset.py` to evaluate an entire dataset (or a single problem) in batch. By default it runs the definition's reference implementation as the solution unless `--solution-name` is specified. Saves to `./out/{subset}` by default.
...
```bash
# Run all problems in the benchmark.
# Auto builds solution.json from a single code file
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --solution-name solution.py
...
# Run specific categories with multiple solution code files
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --category L1 L2 --solution-name solution.json
...
# Run a single problem
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark/L1/my_problem
...
# Limit num...
# Web Search Results for "NVIDIA B200 GPU cloud price per hour 2026" ## 1. B200 Cloud Pricing: Compare 24+ Providers (2026) URL: https://getdeploying.com/gpus/nvidia-b200 We're tracking 24 cloud providers for the B200, and the pricing spread is significant. While the average sits at $5.11/hr, the lowest price is currently $2.25/hr per GPU (36-month reservation). ... | Provider | GPUs | Total VRAM | vCPUs | RAM | Billing | $/GPU/h | Total/h | Availability | | --- | --- | --- | --- | --- | --- | --- | --- | --- | ... Packet·ai Our sponsor | 1x B200 NVL 1x B200 180GB NVL | 180GB | 12 | 150GB | On-Demand Pay-as-you-go pricing. ... term commitments. | $3.75 | $3. ... 5 | Unknown | View | ... | Runpod | 1x B200 SXM 1x B200 180GB SXM (secure-cloud) | 180GB | 26 | 120GB | On-Demand Pay-as-you-go pricing. No term commitments. | $5.89 | $5.89 | Available Last checked <1h ago | View | ... x B200 ... 1x B200 ... 80GB SXM (NVIDIA B200 SXM6) | 180GB | 26 | 360GB | On-Demand Pay-as-you-go pricing. ... term commitments. | $6. ... 9 | $6. ... 9 | Unknown | View | ... 8x B200 ... 8x B200 ... 80GB NVL ... .4TB ... 8 | 2.0TB ... On-Demand Pay-as-you-go pricing ... checked <24h ago ... | UpCloud | 1x B200 1x B200 192GB | 192GB | 12 | 240GB | On-Demand Pay-as-you-go pricing. No term commitments. ... $5.13 ... $5.13 | Unknown | View ... x B200 SXM ... x B200 ... 80GB SXM ... 180GB ... On-Demand Pay-as ... B20 ... 92GB SXM ... B20 ... 92GB ... 24GB ... B200 ... B200 ... 180GB SXM ... NVIDIA B200 SX ... 36 ... 52 | 720GB ... $14.00 ... $112.00 ... 8x B20 ... 92 | ... | Google Cloud | 8x B200 8x B200 180GB (a4-highgpu-8g) | 1.4TB | 224 | 4.0TB | On-Demand Pay-as-you-go pricing. No term commitments. | $16.11 | $128.88 | Available Last checked <24h ago | View | ... | UpCloud | 2x B200 2x B200 192GB | 384GB | 24 | 480GB | On-Demand Pay-as-you-go pricing. ... term commitments. | $5.12 | $10.25 | Unknown | View | ... | Lyceum | ... 1x B200 1 ... B200 192GB ... 192GB ... B200 ... | Verda | 1x B200 SXM 1x B200 180GB SXM (1B200.30V) | 180GB | 30 | 170GB | On-Demand Pay-as-you-go pricing. No term commitments. | $6.11 | $6.11 | Low stock Last checked <24h ago | View | ... ## 2. B200 Cloud Pricing from $3.44/hr (2026) — 11 Providers URL: https://gpufinder.dev/gpu/b200 B200 Cloud Pricing from $3.44/hr (2026) — 11 Providers ... Compare B200 prices across 11 cloud providers. On-demand from $3.44/hr. ... Price range:$3.44 — $160.28/hr ... Spot price:from $2.77/hr ... Updated:Jul 1, 2026 ... Cheapest B200 confirmed in stock right now: $3.44/hr on Vast (2×). ... ■ near its 6-month low — at $4.36/hr the B200 floor price is near its 6-month low — 6-month range $4.36–$35.20/hr. ... B200 on-demand pricing currently ranges from $2.25/hr to $16/hr, with a market average around $4.95/hr. Availability remains constrained in 2026 with most 2025 production sold out, but supply is expanding. Expect prices to decline 30-50% over the next 12 months as more providers bring capacity online. ... ## Pricing by Provider ... $3.44 ... $2.81 ... $4.29 ... $4.36 ... $2.77 ... $4.36 ... $2.77 ... $2.81 ... $2.81 ... $2.81 ... | Provider | Instance Type | GPUs | vCPUs | RAM (GB) | On-Demand | Spot / interruptible | Availability | ... gress (/GB) | Region | ... | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Vast | 2x B200 marketplace | 2 | 64 | 567 | Free | Alabama, US | ... | Vast | 1x B200 marketplace | 1 | 24 | 250 | Free | Oregon, US | | Vast | 8x B200 marketplace | 8 | 192 | 1996 | Free | Virginia, US | | Vast | 4x B200 marketplace | 4 | 96 | 998 | Free | Virginia, US | ... | Vast | 4x B200 marketplace | 4 | 96 | 998 | Free | Oregon, US | ... | Vast | 2x B200 marketplace | 2 | 48 | 499 | Free | Oregon, US | ... | Vast | 8x B200 marketplace | 8 | 128 | 3024 | Free | , US | | Vast | 4x B200 marketplace | 4 | 64 | 1512 | Free | , US | | Vast | 4x B200 marketplace | 4 | 96 | 998 | N/A | Free | Ohio, US | | Vast | 2x B200 marketplace | 2 | 48 | 499 | N/A | Free | Ohio, US | | Vast | 1x B200 marketplace | 1 | 24 | 250 | N/A | Free | Ohio, US | ... | RunPod | 4x B200 Secure Cloud | 4 | 32 | 256 | Free | Global | | RunPod | 1x B200 Secure Cloud | 1 | 8 | 64 | Free | Global | ... | RunPod | 2x B200 Secure Cloud | 2 | 16 | 128 | Free | Global | ..... ## 3. NVIDIA B200 Cloud Pricing 2026: Per-Hour Rental Cost Across Providers vs Spheron | Spheron Blog URL: https://www.spheron.network/blog/nvidia-b200-cloud-pricing-2026/ - NVIDIA B200 192GB HBM3e: Blackwell architecture, 8 TB/s bandwidth, NVLink 1.8TB/s. For trillion-parameter models and next-gen LLMs ... - NVIDIA B200 Cloud Pricing 2026: Per-Hour Rental Cost Across Providers vs Spheron: B200 on-demand ($3.70-$ 14.24/hr) and spot pricing across RunPod, Lambda, CoreWeave, AWS p6-b200, and Spheron, with cost-per-token math and monthly cost examples ## 4. NVIDIA B200 Pricing (June 2026) | Beam URL: https://www.beam.cloud/blog/nvidia-b200-pricing The breakdown below compares current B2 ... between neocloud ... - Neoclouds list B200 on-demand at $3.70 to $6.00/hr; hyperscalers charge 3 to 4x more. - AWS p6-b200 lands near $14.24/hr per GPU, the most expensive tier in the market. - Spot pricing drops as low as $2.12 to $2.74/hr for checkpoint-friendly workloads. - Beam starts at $3.93/hr, at the floor of the on-demand market, on a serverless runtime rather than a raw VM. ... ## Current on-demand B200 prices (per GPU) ... | Provider | SKU / Instance | On-Demand $/GPU-hr* | Notes | | --- | --- | --- | --- | | Beam | B200 SXM6 | $3.93 | Serverless runtime, per-second billing | | Spheron | B200 SXM6 | $3.70 | Marketplace floor; spot ~$2.74 | | Lambda | 1x B200 | $4.99 | 8x node at $5.29/GPU | | Nebius | B200 SXM6 | $5.50 | Per-hour billing, no spot | | RunPod | B200 SXM6 (Secure) | $5.89 | Per-minute billing | | CoreWeave | N/A | ~$6.50 | Contract-oriented, estimated | | Azure | ND B200 v6 | ~$14.00 | Full VM shape | | AWS | p6-b200 (8x B200) | $14.24 | $113.93/hr node / 8 GPUs | ... Beam's B200 SXM6 starts at $3.93/hr, at the bottom of the on-demand range, level with the cheapest marketplace floors and about 3.6x cheaper than AWS p6. The difference is what the rate includes. Rather than a bare VM you manage yourself, Beam runs your workload on a serverless runtime with per-second billing, so you do not pay for idle GPU time between jobs. For burst inference and intermittent training, that scale-to-zero behavior often matters more than the hourly number. ... - Memory you can use: 192 GB HBM3e plus FP4 fits 70B+ models on a single GPU with room for large batch sizes. - The hyperscaler premium is large: AWS and Azure B200 instances cost 3 to 4x the neocloud rate for the same silicon, before egress and networking. - Spot is the biggest lever: if your workload checkpoints, spot B200 around $2.12/hr leads on cost per token. ## 5. NVIDIA B200 Pricing (July 2026) | Thunder Compute URL: https://www.thundercompute.com/blog/nvidia-b200-pricing Pricing is not standardized because the B200 cloud rollout is still underway. Most access is limited to enterprise contracts, has waitlists, or is constrained. ... | Provider | NVIDIA B200 Price (Hourly) | Instance Name | | --- | --- | --- | | Vultr | $3.50 (x8 GPUs) | N/A | | Hyperbolic | $3.50 | N/A | | RunPod | $5.89 | N/A | | Lambda | $6.69 | N/A | | Vast.ai | $5.64 | N/A | | Modal | $6.25 | N/A | | CoreWeave | $8.60 (x8 GPUs) | N/A | | Oracle Cloud | $14.00 (x8 GPUs) | N/A | | AWS | $14.13 (x8 GPUs) | p6-b200.48xlarge | | Azure | $27.04 (x4 GPUs) | NDsrGB200NDRv6 | | Google Cloud | Custom Quote | A3 Blackwell | | Crusoe Cloud | Custom Quote | N/A | | Nebius | $7.15 | N/A | ... | Feature | NVIDIA H100 | NVIDIA B200 | | --- | --- | --- | | Architecture | Hopper | Blackwell | | Tensor Core Generation | 4th Gen | 5th Gen | | GPU Memory | 80 GB HBM2e | 192 GB HBM3e | | Memory Bandwidth | 2,040 GB/s (PCIe) | Up to 8,000 GB/s1 | | Cloud Pricing | $1.38 - $11.06/hr | $3.50 - $27.04/hr | | Availability | Broad cloud availability. | Waitlists, and enterprise-restricted access. | | Best For | Fine-tuning, inference, and most training workloads. | Very large model training. | | Main Trade-off | Less memory. | Higher cost and harder access. | ... 00 GPUs | V ... Us | RAM | Hourly price | Price ... -GPU | ... | --- | --- | --- | --- | --- | --- | --- ... $11 ... | Provider | SKU | B200 GPUs | VCPUs | RAM | Hourly price | Price per-GPU | | --- | --- | --- | --- | --- | --- | --- | | Azure | NDsrGB200NDRv6 | 4 | 128 | 864 | $108.16 | $27.04 | ... --- | --- ... --- | --- ... ### What is the NVIDIA B200 price in 2026? ... Pricing ranges from $3.50-27.04/hour, but availability is limited and often restricted to enterprise customers.
The search results give B200 cloud pricing. The benchmark itself is free and open source. The user asks "how much does a run cost". Need to interpret: likely the cost of running a single benchmark run/evaluation on a B200 GPU in the cloud, or the cost of submitting to the leaderboard. I should check the SOL-ExecBench repo docs for any mention of cost, pricing, or how long a run takes. I should also look for the time per problem. A run could be evaluating one kernel solution across workloads. The cost depends on GPU time. Let's search the repo for "cost" or "time" or "timeout". Also, the benchmark harness runs a solution against every workload in a problem, typically 16-48 workloads per problem. The run time is probably small (kernels are microseconds to milliseconds). But there are many workloads, compilation time, correctness checks. The total time per problem might be seconds to minutes. Running the entire L1 collection (94 problems) might take an hour or more. We can estimate cost: B200 on-demand ~$3.50-$6/hr, maybe $4-5/hr average. If a single problem run takes 5 minutes, cost ~$0.30. If the whole L1 collection takes 1 hour, cost ~$4. If all 235 problems take 2-3 hours, cost ~$10-15. But we need actual data. Maybe the leaderboard "run" means one submission (a set of solutions for all problems) evaluated by NVIDIA. The cost is probably free for leaderboard submissions (NVIDIA runs it). But the user might be asking about cost of running themselves. I should search for "timeout" in the docs because that gives max runtime per workload. Also maybe "compile-timeout". The CLI might have default timeouts. Let's fetch the docs or search. Search web for "sol-execbench timeout per problem".
{
"query": "SOL-ExecBench timeout compile run time per problem workloads"
}# Web Search Results for "SOL-ExecBench timeout compile run time per problem workloads"
## 1. NVIDIA/SOL-ExecBench
URL: https://github.com/nvidia/sol-execbench
| Flag | Description |
|---|---|
| `--compile-timeout` | Compilation timeout in seconds (default: 120) |
| `--timeout` | Evaluation timeout in seconds (default: 600) |
| `--config` | Path to a BenchmarkConfig JSON (see [Benchmark Config](#benchmark-config) below) |
| `-o, --output` | Write JSONL traces to file |
| `--json` | Print traces as JSON to stdout |
| `--lock-clocks` | Lock GPU clocks for stable benchmarks |
| `--keep-staging` | Preserve staging directory after run |
| `-v, --verbose` | Show subprocess output |
...
Use `scripts/run_dataset.py` to evaluate an entire dataset (or a single problem) in batch. By default it runs the definition's reference implementation as the solution unless `--solution-name` is specified. Saves to `./out/{subset}` by default.
...
```bash
# Run all problems in the benchmark.
# Auto builds solution.json from a single code file
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --solution-name solution.py
...
# Run specific categories with multiple solution code files
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --category L1 L2 --solution-name solution.json
...
# Run a single problem
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark/L1/my_problem
...
# Limit number of problems and workloads
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --limit 5 --max-workloads 3 -o ./results
```
...
Results (traces and a summary JSON) are written to `out/run_dataset/` by default (override with `-o`). Problems that already passed are skipped on subsequent runs unless `--rerun` is specified.
...
- **`definition.json`** — Kernel specification: function signature, tensor shapes, dtypes, reference
...
.
- **`workload.jsonl`** — One JSON object per line, each defining input shapes, values, and tolerance thresholds.
## 2. docs/trace.md at main · NVIDIA/SOL-ExecBench
URL: https://github.com/NVIDIA/SOL-ExecBench/blob/main/docs/trace.md
A `Trace` is an atomic, immutable record of a **single benchmark run**. It links a specific `Solution` to a specific `Definition`, details the exact `workload` configuration used for the run (i.e., shapes and input data), and records the complete `evaluation` result. The collection of all Trace files forms the database of benchmark results.
...
| **Field** | **Type** | **Required** | **Description** |
| --- | --- | --- | --- |
| `status` | string | Yes | The final status of the evaluation run. Has to be one of the following: `"PASSED"`, `"INCORRECT_SHAPE"`, `"INCORRECT_NUMERICAL"`, `"INCORRECT_DTYPE"`, `"RUNTIME_ERROR"`, `"COMPILE_ERROR"`, `"TIMEOUT"`, `"REWARD_HACK"`, `"INVALID_REFERENCE"`. |
| `log` | string | No | The embedded record of the stdout and stderr of the evaluation run (default: `""`). |
| `correctness` | object | Yes | The summarized correctness results across all entries in the dataset. |
| `performance` | object | Yes | The summarized performance metrics across all entries in the dataset. |
| `environment` | object | Yes | A snapshot of the hardware and software execution environment. |
| `timestamp` | string | Yes | The ISO 8601 timestamp of when this summary was generated. |
...
| **Field** | **Type** | **Required** | **Description** |
| --- | --- | --- | --- |
| `latency_ms` | float | Yes | The mean latency in milliseconds per execution for this implementation. |
| `reference_latency_ms` | float | Yes | The mean latency of the `Definition`'s reference code on the same data/hardware. |
| `speedup_factor` | float | Yes | The calculated speedup (`reference_latency_ms / latency_ms`). |
...
| status | correctness | performance |
| --- | --- | --- |
| PASSED | Required | Required |
| INCORRECT_NUMERICAL | Required | **None** |
| INCORRECT_SHAPE/DTYPE | **None** | **None** |
| RUNTIME_ERROR | **None** | **None** |
| COMPILE_ERROR | **None** | **None** |
| TIMEOUT | **None** | **None** |
| REWARD_HACK | **None** | **None** |
| INVALID_REFERENCE | **None*...
## 3. docs/workload.md at main · NVIDIA/SOL-ExecBench
URL: https://github.com/NVIDIA/SOL-ExecBench/blob/main/docs/workload.md
A **Workload** makes a Definition concrete and executable by binding specific values to all variable axes and specifying where the input data comes from. It is the exact configuration under which a Solution is benchmarked.
...
The evaluator runs your solution against every Workload in the problem's set — typically 16–48 workloads per problem.
...
workload,
...
workloads for the same problem
...
. This allows
...
bounds for easy shapes and looser bounds for numerically challenging
...
## How the Evaluator Uses Workloads
...
For each workload, the evaluator:
...
1. Binds `var` axes to their concrete values from `workload.axes`
2. Generates/loads each input tensor according to its descriptor:
- `random` → `torch.randn(shape, dtype=dtype, device=device)` (or appropriate random for the dtype)
- `scalar` → the literal value
- `safetensors` → loads tensor from file
- `custom` → calls `definition.custom_inputs_entrypoint(axes_and_scalars, device)`
...
3. Pre-allocates output tensors (shapes from Definition.outputs with resolved axes)
4. Calls `solution.run(*inputs, *outputs)` — inputs in Definition order, then outputs
5. Compares outputs against `reference(*inputs)` for correctness
6. Times the solution for performance
## 4. zsol-bench v1.0.1.1
URL: https://pypi.org/project/zsol-bench/
### CLI Options
...
| Flag | Description |
| --- | --- |
| `--compile-timeout` | Compilation timeout in seconds (default: 120) |
| `--timeout` | Evaluation timeout in seconds (default: 600) |
| `-o, --output` | Write JSONL traces to file |
| `--json` | Print traces as JSON to stdout |
| `--lock-clocks` | Lock GPU clocks for stable benchmarks |
| `--keep-staging` | Preserve staging directory after run |
| `-v, --verbose` | Show subprocess output |
...
Use `scripts/run_dataset.py` to evaluate an entire dataset (or a single problem) in batch. By default it runs the definition's reference implementation as the solution unless `--solution-name` is specified. Saves to `./out/{subset}` by default.
...
```bash
# Run all problems in the benchmark.
# Auto builds solution.json from a single code file
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --solution-name solution.py
...
# Run specific categories with multiple solution code files
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --category L1 L2 --solution-name solution.json
...
# Run a single problem
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark/L1/my_problem
...
# Limit number of problems and workloads
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --limit 5 --max-workloads 3 -o ./results
```
...
Results (traces and a summary JSON) are written to `out/run_dataset/` by default (override with `-o`). Problems that already passed are skipped on subsequent runs unless `--rerun` is specified.
## 5. nvidia/SOL-ExecBench · Datasets at Hugging Face
URL: https://huggingface.co/datasets/nvidia/SOL-ExecBench
SOL (Speed Of Light) ExecBench is a real-world CUDA kernel benchmarking dataset of 235 kernel-level computational workload specifications derived from open-source HuggingFace model architectures. The problems span a wide range of AI model workloads — covering text, vision, and speech models' forward and backward passes — and include core algorithms such as matrix multiplications, convolutions, attention variants, mixture-of-experts, and norms across FP32, BF16, FP16, FP8, and NVFP4 data types. Runtimes will be compared against reference implementation speed but also theoretical hardware "speed of light": the fastest that kernels can run given the hardware's peak compute and memory bandwidth limits.
...
- Symbolic axis definitions (constant, variable, expression-based)
- Tensor input/output specifications (shape, dtype, constraints)
- A PyTorch reference implementation with a required`run()` entrypoint
- Workload instantiations specifying concrete axis values and input generation strategies
...
computed with SOLAR and are
...
SOL ExecBench is intended for researchers and engineers developing AI-based kernel generation systems, compiler agents, and automated performance optimization tools.
...
- Benchmarking AI-based kernel generation models
- Evaluating agent-based code synthesis systems
- Testing compiler optimization strategies
- Measuring correctness and performance of generated CUDA kernels
- Research and development of automated performance engineering systems
...
Pre-built Parquet files in`data/` provide one row per kernel with the following columns:
...
| Column | Type | Description |
| --- | --- | --- |
| `name` | string | Kernel identifier |
| `description` | string | Natural-language description of the operation |
| `axes` | JSON string | Symbolic axis definitions (const / var / expr) |
| `inputs` | JSON string | Input tensor specifications (shape, dtype) |
| `outputs` | JSON string | Output tensor specifications (shape, dtype) |
| `reference` | string | Py...
We have defaults: compile timeout 120s, evaluation timeout 600s per problem/workload. Workloads per problem: typically 16-48. Evaluation timeout per workload is 600s? Or per problem? The flag `--timeout` is evaluation timeout in seconds (default 600). Likely per workload or per problem. If per problem, a problem can take up to 10 minutes. But actual runtime is much shorter for most. We need to answer "how much does a run cost". There are multiple interpretations: 1. Cost to submit to the leaderboard: likely free (NVIDIA runs it), but need account. 2. Cost to run a single problem evaluation locally on a B200: depends on cloud provider, ~$3.50-$6/hr for a B200. If a problem takes a few minutes, cost is cents. 3. Cost to run the entire dataset: depends on time. If it takes 1-3 hours, cost $4-20. 4. Cost of a single workload run: tiny. We need to be careful not to guess. The benchmark itself has no monetary cost. The compute cost is the GPU rental. We can estimate based on typical runtimes. Let's see if we can find actual runtime info from the paper or repo. The paper says each benchmark run uses a single GPU with clocks locked. It might mention total time to run the dataset. Let's search.
{
"query": "SOL-ExecBench total evaluation time run all problems B200 hours"
}# Web Search Results for "SOL-ExecBench total evaluation time run all problems B200 hours"
## 1. NVIDIA/SOL-ExecBench
URL: https://github.com/nvidia/sol-execbench
Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)).
Supported kernel languages: PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++.
...
| Description |
|---|---|
| `--compile-timeout` | Compilation timeout in seconds (default: 120) |
| `--timeout
...
| Evaluation timeout in seconds (default: 600) |
|
...
to a BenchmarkConfig
...
config) below) |
| `-o, --output` | Write JSONL traces to file |
|
...
json` | Print traces
...
JSON to stdout |
|
...
lock-clocks` | Lock
...
clocks for stable benchmarks |
| `--keep-staging` | Preserve staging
...
|
| `-v, --verbose` | Show
...
| Field | Type | Default | Description |
|---|---|---|---|
| `warmup_runs` | int | `10` | GPU warmup iterations before timing |
| `iterations` | int | `50` | Timing iterations averaged into the latency report |
| `lock_clocks` | bool | `false` | Require GPU clocks to be locked (also exposed as `--lock-clocks`) |
| `benchmark_reference` | bool | `false` | When `true`, also time the reference implementation to compute speedup. **Disabled by default** because the reference can be dramatically slower than the kernel (sometimes >1 h), which dominates total evaluation time. Enable when you need a speedup factor in the trace. |
| `seed` | int | `200` | RNG seed for input generation |
...
Use `scripts/run_dataset.py` to evaluate an entire dataset (or a single problem) in batch. By default it runs the definition's reference implementation as the solution unless `--solution-name` is specified. Saves to `./out/{subset}` by default.
...
```bash
# Run all problems in the benchmark.
# Auto builds solution.json from a single code file
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --solution-name solution.py
...
# Run specific categories with multiple solution code files
uv run script...
## 2. docs/workload.md at main · NVIDIA/SOL-ExecBench
URL: https://github.com/NVIDIA/SOL-ExecBench/blob/main/docs/workload.md
The evaluator runs your solution against every Workload in the problem's set — typically 16–48 workloads per problem.
...
3. Pre-allocates output tensors (shapes from Definition.outputs with resolved axes)
4. Calls `solution.run(*inputs, *outputs)` — inputs in Definition order, then outputs
5. Compares outputs against `reference(*inputs)` for correctness
6. Times the solution for performance
## 3. docs/trace.md at main · NVIDIA/SOL-ExecBench
URL: https://github.com/NVIDIA/SOL-ExecBench/blob/main/docs/trace.md
A `Trace` is an atomic, immutable record of a **single benchmark run**. It links a specific `Solution` to a specific `Definition`, details the exact `workload` configuration used for the run (i.e., shapes and input data), and records the complete `evaluation` result. The collection of all Trace files forms the database of benchmark results.
...
### `evaluation` : Benchmark Statistics Summary
...
This object represents a single, complete benchmark result.
...
| **Field** | **Type** | **Required** | **Description** |
| --- | --- | --- | --- |
| `status` | string | Yes | The final status of the evaluation run. Has to be one of the following: `"PASSED"`, `"INCORRECT_SHAPE"`, `"INCORRECT_NUMERICAL"`, `"INCORRECT_DTYPE"`, `"RUNTIME_ERROR"`, `"COMPILE_ERROR"`, `"TIMEOUT"`, `"REWARD_HACK"`, `"INVALID_REFERENCE"`. |
| `log` | string | No | The embedded record of the stdout and stderr of the evaluation run (default: `""`). |
| `correctness` | object | Yes | The summarized correctness results across all entries in the dataset. |
| `performance` | object | Yes | The summarized performance metrics across all entries in the dataset. |
| `environment` | object | Yes | A snapshot of the hardware and software execution environment. |
| `timestamp` | string | Yes | The ISO 8601 timestamp of when this summary was generated. |
...
| **Field** | **Type** | **Required** | **Description** |
| --- | --- | --- | --- |
| `latency_ms` | float | Yes | The mean latency in milliseconds per execution for this implementation. |
| `reference_latency_ms` | float | Yes | The mean
...
of the `Definition`'s reference code on the same data/hardware. |
| `speed
...
_latency_ms / latency_ms`). |
...
```python
{
"definition": "rmsnorm",
"solution": "rmsnorm_triton_v1",
"workload": {
"uuid": "6120f144-b973-4bd9-b884-77ecb132914e",
"axes": {
"batch_size": 32
},
"inputs": {
"input": {
"type": "safetensors",
"path": "/data/rmsnorm_evals/b32_input.safetensors",
...
## 4. SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits
URL: https://arxiv.org/html/2603.19173
To measure runtime, we use CUDA events with 10 warmup iterations and 50 timed iterations per trial over 3 trials, with the reported runtime taken as the mean across trials. Before every timed iteration, the harness clears the L2 cache by zeroing a 256 MB device buffer, and it clones tensor arguments so each run starts from fresh inputs with new addresses rather than reusing state from previous iterations. Benchmarking on a given GPU is serialized, and clocks are locked through nvidia-smi at hardware-specific frequencies (1,500 MHz for B200) to improve reproducibility. The harness reports absolute runtime in milliseconds, and relative metrics such as speedup and SOL score are computed later against the scoring baseline defined for a given benchmark release.
...
All experiments are conducted on NVIDIA DGX B200 nodes equipped with 8× NVIDIA Blackwell B200 GPUs, each providing 192 GB of HBM3e memory and 8 TB/s of memory bandwidth NVIDIA (2024b,c). The software stack is built on an NVIDIA-provided Docker image with CUDA 13.1.1, cuDNN 9.17.1, PyTorch 2.9.0, and NVIDIA driver 580.95. Each benchmark run uses a single GPU with SM clocks locked at 1,500 MHz via nvidia-smi to reduce frequency-scaling noise, matching the evaluation harness described in Section 4.4.
...
Figure4illustrates the SOLAR pipeline on a concrete SOL-ExecBench L1 problem from Jamba-Reasoning-3B which performs fused attention output projection with residual addition. Figure4(a)depicts the PyTorch code that performs a matmul followed by an elementwise add. The Graph Extractor takes the PyTorch program as input and produces an operator graph with explicit nodes (matmul, add, transpose) and tensor shapes (Figure4(b)). Next, the LLM-based Einsum Converter maps the operator graph to an extended einsum graph: the matmul maps to a single contraction node ACB,BH→ACH and the projection to an elementwise add ACH,ACH→ACH node (Figure4(c)). Finally, the SOL Analyzer derives FLOPs and memory traffic from the Einsum gr...
## 5. zsol-bench v1.0.1.1
URL: https://pypi.org/project/zsol-bench/
Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)).
...
| Flag | Description |
| --- | --- |
| `--compile-timeout` | Compilation timeout in seconds (default: 120) |
| `--timeout` | Evaluation timeout in seconds (default: 600) |
| `-o, --output` | Write JSONL traces to file |
| `--json` | Print traces as JSON to stdout |
| `--lock-clocks` | Lock GPU clocks for stable benchmarks |
| `--keep-staging` | Preserve staging directory after run |
| `-v, --verbose` | Show subprocess output |
...
Use `scripts/run_dataset.py` to evaluate an entire dataset (or a single problem) in batch. By default it runs the definition's reference implementation as the solution unless `--solution-name` is specified. Saves to `./out/{subset}` by default.
...
```bash
# Run all problems in the benchmark.
# Auto builds solution.json from a single code file
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --solution-name solution.py
...
# Run specific categories with multiple solution code files
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --category L1 L2 --solution-name solution.json
...
# Run a single problem
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark/L1/my_problem
...
# Limit number of problems and workloads
uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --limit 5 --max-workloads 3 -o ./results
```
...
Results (traces and a summary JSON) are written to `out/run_dataset/` by default (override with `-o`). Problems that already passed are skipped on subsequent runs unless `--rerun` is specified.
We have the benchmark config defaults: 10 warmup, 50 timed iterations per trial over 3 trials, L2 cache cleared each iteration. So per workload, there are 10 + 50*3 = 160 runs (plus reference correctness maybe). The latency is mean across trials. For a kernel that takes 1 ms, a single workload would take 160 ms + overhead + compilation. For a kernel that takes 100 ms, 16 seconds. With 16-48 workloads per problem, total per problem could be from seconds to minutes. Reference benchmarking disabled by default because reference can be >1 hour. So a "run" with default settings does not benchmark reference. The cost depends on whether the user means "a single workload run", "a single problem run", "a leaderboard submission run", or "the whole benchmark". I should ask for clarification or answer all three. The user asked "how much does a run cost". They likely mean one submission run on the leaderboard or a single problem evaluation. In the context of the previous conversation, they were looking at the leaderboard. Maybe they are asking "how much does it cost to submit a run to the leaderboard". Or "how much does a run cost to evaluate a solution". I should answer: The benchmark software and dataset are free/open source. NVIDIA does not charge for leaderboard submissions. The cost is the GPU compute you consume. On a B200 (which is required for the B200 leaderboard), cloud pricing is roughly $3.50–$6/hr on neoclouds, up to ~$14/hr on AWS/Azure. A single problem run typically takes seconds to a few minutes (16–48 workloads, each with 50 iterations over 3 trials), so a single run is usually a few cents to maybe $0.50. A full leaderboard-style submission across all 235 problems would likely take a few hours, so roughly $10–$50 depending on provider and how optimized your solutions are. But the reference implementation is disabled by default to avoid huge cost. Need to be honest about uncertainty: actual runtime depends on kernels and workload sizes. I can give rough estimates. Maybe the user wants to know if there is a hidden fee to submit. I should mention that the leaderboard appears free but you need to log in; NVIDIA runs the evaluation. Let's verify by looking at the leaderboard login/submission page. Maybe there is a submission policy. But not necessary. I can also search for "SOL-ExecBench submission limit" or "cost to submit".
{
"query": "SOL-ExecBench leaderboard submission free cost limit"
}# Web Search Results for "SOL-ExecBench leaderboard submission free cost limit" ## 1. NVIDIA/SOL-ExecBench URL: https://github.com/nvidia/sol-execbench [HuggingFace Dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) | [Leaderboard](https://research.nvidia.com/benchmarks/sol-execbench) | [Technical Report](https://arxiv.org/abs/2603.19173) **Speed-Of-Light ExecBench** is a rigorous GPU kernel evaluation and benchmarking framework built to benchmark AI-generated kernel solutions written with the variety of DSLs that NVIDIA hardware supports. ... Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)). Supported kernel languages: PyTorch, Triton, CUTLASS, cuDNN, CuTe DSL, cuTile, CUDA C++. ... # Limit number of problems and workloads uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --limit 5 --max-workloads 3 -o ./results ``` ## 2. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA URL: https://research.nvidia.com/benchmarks/sol-execbench/collection/1 SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA Sorry, your browser does not support inline SVG. SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA [SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench) [GitHub ↗](https://github.com/nvidia/sol-execbench)[HuggingFace ↗](https://huggingface.co/datasets/nvidia/sol-execbench)[Paper ↗](https://arxiv.org/abs/2603.19173)[Log In](https://research.nvidia.com/benchmarks/sol-execbench/login) Sorry, your browser does not support inline SVG. ## 3. nvidia/SOL-ExecBench · Datasets at Hugging Face URL: https://huggingface.co/datasets/nvidia/SOL-ExecBench Evaluation is supported via a custom evaluator (GitHub) and a public leaderboard. Speed-of-light scores are analytically computed with SOLAR and are available on the public leaderboard. ... - FlashInfer-Bench: https://huggingface.co/datasets/flashinfer-ai/flashinfer-trace - Public GitHub repository for evaluation: https://github.com/nvidia/sol-execbench - Leaderboard: https://research.nvidia.com/benchmarks/sol-execbench - Technical report: https://arxiv.org/abs/2603.19173 ## 4. zsol-bench v1.0.1.1 URL: https://pypi.org/project/zsol-bench/ [HuggingFace Dataset](https://huggingface.co/datasets/nvidia/SOL-ExecBench) | [Leaderboard](https://research.nvidia.com/benchmarks/sol-execbench) | [Technical Report](https://arxiv.org/abs/2603.19173) **Speed-Of-Light ExecBench** is a rigorous GPU kernel evaluation and benchmarking framework built to benchmark AI-generated kernel solutions written with the variety of DSLs that NVIDIA hardware supports. ... Leaderboard submissions are ranked based on [SOL-Score](/src/sol_execbench/sol_score.py): a metric that grades custom kernel performance based on the theoretical roofline of a NVIDIA B200 GPU (obtained analytically with [SOLAR](https://github.com/NVlabs/SOLAR)). ... # Limit number of problems and workloads uv run scripts/run_dataset.py data/SOL-ExecBench/benchmark --limit 5 --max-workloads 3 -o ./results ``` ## 5. SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits URL: https://arxiv.org/html/2603.19173 These baseline implementations were generated using an agentic kernel optimization system. The optimizer operates in a turn-based, multi-agent manner. For each problem, we launch multiple agents independently to optimize the runtime of the provided PyTorch reference implementation under a fixed time and cost budget. After each round, we collect all valid submitted solutions and expose them to the next cohort of agents, which may use any of these correct candidates as a starting point or reference for further optimization in the next round. ... Each agent is restricted to producing solutions using only PyTorch and standard Python packages. Agents are equipped with tools and skills to submit their implementations to the remote evaluation sandbox described in Section 4.4 for correctness checking and benchmarking. In addition, every submitted solution is inspected by an LLM-based judge to detect requirement violations and common cheating patterns. ... Only solutions that compile successfully, pass correctness verification, and satisfy the our reward hacking mitigation mechanism are retained as the baseline candidates. After the prescribed number of rounds, we aggregate all valid candidates produced by all agents across all turns and select the fastest kernel for each problem as the final optimized baseline. ... As mentioned earlier we observed reward hacking as the agent solved the problems. Figure 9 shows the distribution of the exploits detected across all agent submissions, broken down by exploit types, as defined in Table 3 . Precision downgrade is the most common exploit (259 kernels, 6.4% of all submissions), where agents compute in a lower precision (e.g., FP16 instead of FP32) and upcast the result to pass validation. Monkey patching (134, 3.3%) overrides critical timing functions such as Event.elapsed time to report artificially low runtimes. Stream injection (100, 2.5%) hides work on unsynchronized CUDA streams that the timing harness does not record. Reuse of...
No direct submission cost info. The leaderboard is public. NVIDIA likely hosts the evaluation. Now, the paper mentions agents have a fixed time and cost budget. It mentions a remote evaluation sandbox. That suggests the agentic optimizer might have cost associated with using the sandbox. But for normal users, the repo is open source. I need to answer "how much does a run cost". I should clarify what "run" means. Provide a few scenarios. Also note that the B200 leaderboard requires a B200 GPU, which is expensive and not widely available. Let's also check if there is any mention of cost budget in the paper. The paper says agents submit to remote evaluation sandbox for correctness checking and benchmarking. This might cost per evaluation. But no price. Maybe the user is asking about the leaderboard run cost because they want to submit. I can say: "Submitting to the leaderboard is free; the cost is the GPU compute you (or NVIDIA) use to run the evaluation." Or if NVIDIA runs it, the cost is borne by NVIDIA. I should also check if the leaderboard requires logging in and maybe there is a quota. But I can't log in. I can infer it's free. Let's craft a response: - The benchmark code and dataset are free/open source. - Leaderboard submission: the website itself doesn't charge; NVIDIA runs the evaluation. But you need to be able to run it on a B200 (or they run it). Not sure if NVIDIA's leaderboard accepts code and runs it on their own hardware for free. I think it's likely. - If you run locally or in the cloud: B200 on-demand is roughly $3.50–$6/hr on neoclouds (Vast, RunPod, Lambda, etc.) and $14+/hr on hyperscalers (AWS/Azure). Spot prices can be $2–$3/hr. - Time per run: depends on scope. A single problem has 16–48 workloads; each workload does 10 warmup + 50 timed iterations over 3 trials. For a fast kernel (sub-ms), a problem run is seconds; for slower kernels, maybe minutes. Default eval timeout is 600s per workload, compile timeout 120s. So worst-case per problem ~10 min. Typical is much less. - Cost estimates: - Single problem: ~$0.05–$0.50 on a cheap B200 cloud. - Full L1 collection (94 problems): ~$2–$20 (maybe 1-4 hours). - Full benchmark (235 problems): ~$5–$50. - Caveats: if you enable reference benchmarking (`benchmark_reference: true`), it can be much slower (reference can be >1 hour per problem), so leave it disabled unless needed. - If you use an agentic optimizer with many iterations, the cost can scale with number of trials; the paper mentions a "fixed time and cost budget" per agent. I should also ask for clarification if they mean a specific type of run. But the user asked a short question. Maybe they want a short answer. I can give the scenarios and ask. I should avoid giving concrete numbers if uncertain. I can say "roughly". Also, the current date is July 2026, B200 pricing is from search results. The cheapest is $2.25/hr (36-month reserved) or $3.44 on-demand. Average ~$5/hr. AWS $14.24/hr. I can mention range. Now, do I need to use tools more? I think I have enough info. But I might want to fetch the exact B200 pricing page to quote. The search results are from today (2026-07-01). I can use webfetch on gpufinder or getdeploying to get precise numbers. But the answer can rely on search results. Also, maybe the user wants to know if a run on the leaderboard costs money. The leaderboard login page might have info. I can try to navigate to it with Playwright to see if there's any mention. But that might be overkill. However, the user is asking a direct question. I can quickly fetch the login page or submission page. Let me try to use the leaderboard page we already loaded. There is a "Log In" link. The submission might be via UI. But I can't log in. I can click "Log In" and see the page. But it might require OAuth. Let's see if there is a public "Submit" page. Use browser_navigate to the leaderboard main page or submit.
{
"server_name": "playwright",
"tool_name": "browser_navigate",
"arguments": {
"url": "https://research.nvidia.com/benchmarks/sol-execbench/login"
}
}[
{
"type": "text",
"text": "### Ran Playwright code\n```js\nawait page.goto('https://research.nvidia.com/benchmarks/sol-execbench/login');\n```\n### Page\n- Page URL: https://research.nvidia.com/benchmarks/sol-execbench/login\n- Page Title: SOL-ExecBench | GPU Kernel Performance Benchmarks by NVIDIA\n### Snapshot\n- [Snapshot](.playwright-mcp/page-2026-07-02T06-28-58-940Z.yml)"
}
]The login page loaded. Let's see its content.
{
"server_name": "playwright",
"tool_name": "browser_evaluate",
"arguments": {
"function": "() => { return document.body.innerText; }"
}
}[
{
"type": "text",
"text": "### Result\n\"SOL-ExecBench\\nBlog\\nProblems\\nLeaderboard\\nStatus\\nGitHub\\n↗\\nHuggingFace\\n↗\\nPaper\\n↗\\nLog In\\nSOL-ExecBench\\nSign in with your NVIDIA account to submit solutions and track your progress.\\nLogin with NVIDIA\\nBack to Home\\n© 2026 NVIDIA Corporation. All rights reserved.\\nPrivacy Policy\\nYour Privacy Choices\\nTerms of Service\\nAccessibility\\nCorporate Policies\\nProduct Security\\nContact: solexecbench@nvidia.com\"\n### Ran Playwright code\n```js\nawait page.evaluate('() => { return document.body.innerText; }');\n```"
}
]The login page says "Sign in with your NVIDIA account to submit solutions and track your progress." It doesn't mention cost. It implies leaderboard submission is free (just need an NVIDIA account). Now, the answer should include this. Maybe the user is asking "how much does a run cost" because they want to know the cost of a leaderboard submission. The answer: free to submit, but you need to run/evaluate the solution on a B200 (NVIDIA may run it). But the login page says "submit solutions and track your progress". It doesn't say whether NVIDIA runs the evaluation or you need to provide traces. The repo has a `Trace` concept; maybe the leaderboard accepts trace files? Or you submit code and they run it in their sandbox. The paper mentions remote evaluation sandbox. So likely NVIDIA runs it. But the cost to you is free. I should answer with nuance. Also, the user might be asking about "how much does a run cost" in the sense of "how much does a single evaluation run cost in the remote sandbox"? That might be an internal cost for NVIDIA. Not known. I think the safest answer: "The benchmark software and dataset are free. Submitting to the leaderboard is free (requires an NVIDIA account). The real cost is the GPU time if you run it yourself. On a B200, that's roughly $3.50–$6/hr on a neocloud or $14+/hr on AWS/Azure. A single problem run is usually cents to a few dollars; a full submission across all 235 problems is likely on the order of $10–$50." But I need to be careful not to present the full submission cost as fact. It depends heavily on time. I can frame as "rough estimate". Let's think about the actual runtime. The paper says for each workload, they run 10 warmup + 50 timed iterations over 3 trials. That's 160 kernel executions per workload. If each kernel execution is 0.1 ms, 16 ms per workload. 48 workloads = 768 ms. But there is also reference correctness check (running reference once per workload? Actually step 5: compares outputs against reference(*inputs). That means reference is run at least once per workload, maybe more. The default config has `benchmark_reference` false, but correctness still needs reference run? The evaluator likely runs the reference to compute outputs. The reference can be slow, but for correctness they may run it once. So per workload, time = solution run time + reference run time + overhead. For small kernels, reference run time is also small. For large kernels, could be significant. But not huge. Compilation timeout default 120s per problem. If compilation fails, you pay up to 2 minutes. If compilation succeeds quickly, negligible. So a single problem could take from maybe 10 seconds to 5 minutes. If average is 1 minute, 94 problems = 94 minutes = ~1.5 hours. Full 235 = ~4 hours. At $4/hr, that's $6 for L1, $16 for full. That's within my estimate. But if a solution has bugs or timeouts, the cost can increase. The evaluation timeout is 600s per problem (or per workload). If a workload hangs, it can waste 10 minutes. So per problem worst-case is 10 minutes * 48 workloads = 8 hours! But timeouts are per problem? Actually `--timeout` is evaluation timeout in seconds (default 600). It might be per workload, not per problem. If per problem, a single problematic workload can cause 10 minutes. If per workload, a problem could be 480 minutes. Hmm. The flag description says "Evaluation timeout in seconds (default: 600)". It's ambiguous. But typical evaluators set per-workload timeout. If so, 48 workloads * 10 min = 480 min = 8 hours per problem if all time out. That would be costly. But in practice, you won't have timeouts. I should mention that worst-case can be high if solutions hang. Now, the user asked "how much does a run cost". It might be a one-sentence question expecting a one-sentence answer. But I can provide a short, structured answer. I should also ask if they want to run it locally or submit to the leaderboard. But I can answer both. Let's draft the final answer: ``` The benchmark itself is free/open source and leaderboard submission is free (you just need an NVIDIA account). The cost is GPU time. For a B200: - Cloud pricing: ~$3.50–$6/hr on neoclouds (Vast, RunPod, Lambda, Spheron, Beam), $14+/hr on AWS/Azure, spot as low as ~$2–$3/hr. - Per-problem run: 16–48 workloads, each with 10 warmup + 50 iterations over 3 trials. Typical runtime is seconds to a few minutes, so a single problem is usually ~$0.05–$0.50. - Full collection: L1 (94 problems) likely ~$2–$20; full benchmark (235 problems) maybe ~$5–$50, depending on provider and how fast your kernels are. - Note: reference benchmarking is disabled by default because it can be very slow (sometimes >1h). If you enable it, cost goes up. - Bad/compiling kernels can hit the 120s compile timeout or 600s eval timeout, so buggy runs can waste dollars. If you meant "cost to submit to the leaderboard" — it's free; you just log in with an NVIDIA account. ``` This is accurate and helpful. I should also cite the sources? The user didn't ask for citations. I can mention the data comes from search results. But not necessary. I should avoid emojis. One more thought: The question might be about "how much does a run cost" in terms of the cost to NVIDIA for the leaderboard. I can mention that submitting is free for participants, so NVIDIA bears the compute cost. Also, maybe the user is asking about the cost of the *agentic optimizer* used in the paper, which has a "fixed time and cost budget". But probably not. I can include a small note. Let's answer.
The benchmark code and dataset are free, and submitting to the leaderboard is free (you just sign in with an NVIDIA account). The real cost is the GPU compute you consume when running evaluations.
For the B200 target used by that leaderboard:
| Cloud option | Rough B200 price |
|---|---|
| Neoclouds (Vast, RunPod, Lambda, Spheron, Beam) | $3.50–$6 / GPU-hour |
| Spot / interruptible | $2–$3 / GPU-hour |
Hyperscalers (AWS p6-b200, Azure) | $14+ / GPU-hour |
How that translates per run depends on what you mean by “a run”:
Single problem — each problem has 16–48 workloads, and each workload does 10 warmup + 50 timed iterations over 3 trials. For a normal fast kernel this is seconds to a few minutes.
≈ $0.05–$0.50 on a cheap B200 cloud.
Full L1 collection (94 problems, the one you linked) — likely 1–4 hours of GPU time.
≈ $2–$20 on a neocloud.
Whole benchmark (235 problems across L1, L2, Quant, FlashInfer) — maybe 3–8 hours.
≈ $10–$50 on a cheap provider; much more on AWS/Azure.
A couple of things that can inflate cost:
--timeout defaults to 600 s and --compile-timeout to 120 s. A buggy or slow-compiling kernel can burn 10+ minutes of GPU time before failing.If you’re asking specifically about submitting a result to the leaderboard, there’s no fee shown — the site just says “Sign in with your NVIDIA account to submit solutions.”