NVSHMEM Knowledge Base + Claude Code Skill

A Claude Code skill for multi-GPU / GPU-initiated-communication work, built from how two production libraries actually use NVSHMEM rather than from a rewrite of the vendor manual.

NVIDIA's documentation tells you what nvshmem_put_nbi does. It does not tell you how to structure a library around it, which of the hundreds of routines matter in practice, or what a team learns after shipping. That is what is here — reverse-engineered from DeepEP and UltraEP, both MIT-licensed and both vendored in full, so every claim is anchored to code you can read.

What's Here

1. nvshmem-purified-docs/ — a clean-room NVSHMEM knowledge base (5.5k lines)

   Seven topic files organised by what you have to get right, not by API family: bootstrap and lifecycle, symmetric memory, data movement, ordering and synchronisation, collectives, environment tuning, and what lives below the API. Plus a symbol index covering all 55 NVSHMEM symbols the two projects use, annotated with which project uses each.

   That last annotation is the part no vendor index can give you: 26 of the 55 are used by both projects. Two independent teams converging on a routine is strong evidence it is core; something only one uses is situational. The vendor manual documents hundreds of routines with equal weight.

   Derived entirely from MIT-licensed source. Verified to share zero 10-word sequences with NVIDIA's documentation.

2. Two production case studies, deliberately contrasting (6.7k lines + 20.1k lines of vendored source)

   DeepEP and UltraEP solve neighbouring problems in MoE training and made opposite NVSHMEM choices. Reading them side by side answers a question the reference cannot: which shape should my library have? ~950 file:line citations, all verified against the vendored trees.

3. The skill itself — SKILL.md plus topical guides on the API surface, environment variables, and correctness traps. Everything is searched with grep rather than loaded into context.

Two Production Case Studies

That last row is the deciding question, and ultraep-docs/ultraep-vs-deepep.md develops it into an ordered set of questions to ask about your own workload. Read that one first if you are choosing an approach rather than reading an existing codebase.

Two things worth knowing before reading the DeepEP guides:

• Its hot path depends on NVSHMEM-internal interfaces, not public API. Read it for the patterns, not as a template. This is a considered trade for a specific regime, and the comparison guide says what that regime is.
• A nvshmemi_ prefix does not make a symbol NVSHMEM's. DeepEP defines nvshmemi_ibgda_put_nbi_warp, nvshmemi_ibgda_quiet, nvshmemi_get_p2p_ptr and others itself under that prefix — and UltraEP's ultra_ep::nvshmem::* wrappers are likewise its own. Both guides tag every symbol, verified by cross-checking the source trees rather than inferring from the name.

Why grep, not a manual

Every question below is answered by a literal string in this repository:

cd nvshmem_skill/references

# Which NVSHMEM routines do production libraries actually call?
cat nvshmem-purified-docs/INDEX.md            # 55 symbols, 26 used by both projects

# How does a real library bootstrap NVSHMEM under torch.distributed?
grep -n -A20 'unique-ID' nvshmem-purified-docs/01-bootstrap-and-lifecycle.md

# What does fence guarantee that quiet doesn't?
grep -n -B2 -A10 'fence' nvshmem-purified-docs/04-ordering-and-synchronisation.md

# Which env vars does shipping code set, and why?
grep -n -B2 "os.environ\['NVSHMEM" deepep-docs/src/deep_ep/buffer.py
grep -n 'NVSHMEM' ultraep-docs/src/ultra_ep/runtime.py

# UltraEP's entire NVSHMEM dependency is one header — read it in a minute
cat ultraep-docs/src/csrc/utils/nvshmem.cuh

Structure

nvshmem-metal-markdown/
├── README.md
├── scrape_nvshmem_docs.py                 # optional: fetch NVIDIA's docs locally (see below)
├── html_to_markdown.py                    # optional: convert them to markdown
└── nvshmem_skill/                         # the portable skill
    ├── SKILL.md                           # program structure, build/launch, debugging, traps
    └── references/
        ├── nvshmem-purified-docs/         # THE clean-room knowledge base
        │   ├── INDEX.md                   #   55-symbol index, both/one-project annotated
        │   ├── 01-bootstrap-and-lifecycle.md
        │   ├── 02-symmetric-memory.md
        │   ├── 03-data-movement.md
        │   ├── 04-ordering-and-synchronisation.md
        │   ├── 05-collectives.md
        │   ├── 06-environment-and-tuning.md
        │   └── 07-below-the-api.md
        │
        ├── nvshmem-api.md                 # API-surface search guide
        ├── env-and-config.md              # environment variables
        ├── patterns-and-pitfalls.md       # correctness traps
        │
        ├── deepep-docs/                   # case study 1: below the API
        │   ├── INDEX.md
        │   ├── deepep-architecture.md
        │   ├── deepep-kernel-patterns.md
        │   ├── deepep-ibgda-direct.md
        │   └── src/                       # DeepEP v1.2.1, MIT (8.9k lines)
        │
        └── ultraep-docs/                  # case study 2: above the API
            ├── INDEX.md
            ├── ultraep-vs-deepep.md       # THE decision guide
            ├── ultraep-architecture.md
            ├── ultraep-balancing-and-p2p.md
            └── src/                       # UltraEP v1.0.0, MIT (11.2k lines)

Using the Skill

cp -r nvshmem_skill ~/.claude/skills/nvshmem

It activates automatically for NVSHMEM work. Questions it answers from local content:

• "Why does my program hang in nvshmem_barrier_all?"
• "What's the difference between nvshmem_fence and nvshmem_quiet?"
• "How do I bootstrap NVSHMEM from an existing torch.distributed world?"
• "Should I use nvshmem_ptr peer stores or put_nbi here?"
• "How do I allocate symmetric memory when CUDA graphs forbid dynamic allocation?"
• "Which environment variables should I set before init, and why?"
• "What does an RMA put actually do at the hardware level?"

Optional: adding NVIDIA's own documentation locally

NVIDIA's NVSHMEM documentation is proprietary — "All rights reserved", and its SLA forbids redistribution and derivative works — so it is not included here. The skill is designed to work without it.

If you want it locally anyway, two scripts fetch and convert it into the same grep-able shape:

./scrape_nvshmem_docs.py     # 49 pages, random 2-5s between requests
./html_to_markdown.py        # -> nvshmem_skill/references/nvshmem-docs/

Dependencies: requests, beautifulsoup4, lxml. Both outputs are gitignored.

The conversion is a structural walk of the Sphinx DOM rather than a generic HTML-to-text pass, because the parts that matter are the parts generic converters break:

• Function signatures are reassembled from the code.descname / em / span.sig-paren spans Sphinx splits them across, and emitted as fenced code so each prototype stays greppable on one line.
• Parameter names are recovered. NVIDIA's generator emits parameter documentation as an unlabelled run of paragraphs — the names exist only in the prototype. Where the counts line up (24 matches, 0 mismatches), the name is spliced back in.
• Underscores are never escaped, so nvshmem_put stays searchable.
• Tables become GitHub pipe tables; the env-var and constant pages are almost entirely tables.

Verified against the HTML source: 49/49 pages, tables 29→29, code blocks and signatures 686→686, text coverage within 0.85–1.6× on 46/49 pages (the three outliers are link indexes).

Provenance and licensing

Two exclusions worth stating plainly:

• NVIDIA's documentation is not redistributed here, for the licence reasons above. The purified knowledge base was written from MIT-licensed code, not from that documentation, and is checked mechanically: a 10-word n-gram comparison against the vendor text (code fences and symbol tables stripped, so only prose is compared) returns zero overlapping sequences.
• deepep-docs/src/csrc/kernels/ibgda_device.cuh is not included. Alone in the DeepEP tree, that file is a fork of NVSHMEM's own non_abi/device/pt-to-pt/ibgda_device.cuh and carries NVIDIA's copyright under the NVSHMEM SLA rather than DeepEP's MIT licence — its own header says so. Clone DeepEP upstream at v1.2.1 to follow the citations that reference it.

For anything beyond the scope of what these two libraries exercise, NVIDIA's official documentation is the authoritative reference.

Verification

Everything published here is checked mechanically rather than asserted:

Known limitations:

• The purified knowledge base is narrow by construction: it covers what these two libraries use. Large parts of NVSHMEM — most of the RMA/AMO/signal type-generic families, NVSHMEM4Py, the CuTe and Numba device DSLs — are out of scope. Use NVIDIA's docs for those.
• Both case studies are MoE communication libraries on NVLink+InfiniBand hardware. Patterns are labelled with a copy / do-not-copy verdict, but the sample is two projects, not a survey.