commitwork Coverage Survey

Language coverage matrix

Rows are ordered by GitHub Octoverse 2025 rank.

Reading the last column

native lint collects the per-language tools that exist for exactly one row: shellcheck for Shell, clippy for Rust, hlint for Haskell, sobelow for Elixir. All four now run. Semgrep shows hollow for Elixir for a specific reason set out below, and blank for Haskell because no Semgrep Haskell rules exist at all.

The Elixir lane reports a clean scan of a file it never parsed

The narrow fix on our side is one line — admit note-level notifications whose descriptor id is Missing plugin. But the upstream behaviour is worse than the SARIF suggested, and worth reporting.

The text output asserts a scan that did not happen

Two runs of the same Semgrep build. The first genuinely analysed a Rust file; the second could not parse Elixir at all. Every line but the finding count is identical.

There is a warning line, but the summary block contradicts it on three counts and then affirms success. --error does not change the exit code. For contrast, a ruleset that 404s exits 7 and emits notifications at level: error — so Semgrep already grades one "the analysis did not happen" case correctly, and this one differently.

Where it originates

Missing-plugin errors are handled in a branch of semgrep/output.py that deliberately bypasses _handle_semgrep_error, carrying an unresolved maintainer note:

# THINK: These perhaps should not be errors but reported as "skipped"

The summary counters sit further down the same file and count rules and targets that were selected, not ones that were analysed — so rules skipped for a missing plugin still register under "Rules run".

A hypothesis I tested and discarded

The counter is derived as executed_rule_count or len(filtered_rules), which in Python falls through on a zero count — an appealing one-line explanation. A mixed scan of one Python rule and two Elixir rules should then have reported "Rules run: 1". It reported 3. The fallback is not the mechanism, and the real cause is that the count never excludes skipped rules in the first place.

Extending Rust and Elixir ourselves

Yes for Rust, and the precedent is already in the tree. manifests/semgrep-taint/ is a locally authored rule pack wired into the sast check, written because the three registry packs produced byte-identical output with and without the Pro engine — the fleet was paying for a capability its own rule selection could not exercise.

One constraint on where Rust rules go

They cannot join manifests/semgrep-taint/. That pack's entry condition, enforced by monitor/test/semgrep-taint.test.mjs, is that every rule must find something the OSS engine cannot — the test exists so the Semgrep account dependency stays load-bearing rather than decorative. Rust rules that work fine on OSS would violate it. They need a sibling pack with its own stated entry condition.

Wired this session

Both requested lanes were already written and already in the scheduled group. Neither needed a manifest edit — they were unwired only in the sense that the binary was absent.

Verified by running, not by resolving

shell-lint returns 8 findings across 9 shell scripts — 1 warning, 6 info, 1 style, of which six are SC2086, the unquoted-expansion class the manifest names as the injection surface. Under the manifest's own severity policy that is 0 high and 1 medium. actions-actionlint returns ran:true with 0 findings, which is a clean result carrying its own witness rather than an empty file.

Bearer needed a different route, not a fix

The brew path fails: no bottle exists for macOS 27, and the source build is refused because Xcode 26.6 is too old. That reads like a dead dependency and is not one — the project is actively maintained, with v2.1.1 released on 24 August 2026 and a push the same day. Only the packaging was blocked.

It ships a darwin_arm64 tarball with a checksums.txt alongside it, so the lane was unblocked by installing the pinned release and verifying its SHA-256 before extracting — not by piping an install script into a shell, which the repository's own unpinned-fetch gate would refuse.

Still blocked

joern — and the earlier reading here, that it was never attempted and needs a JVM and Scala toolchain, is now superseded by measurement. Probed 2026-08-30: joern-scan 4.0.610 is installed via Homebrew, builds a CPG, runs ScanPass and exits 0. It also returns zero results on a file containing strcpy(buf, argv[1]) and system(sprintf(...)), logging “Attempting to determine flows from empty list of sources” thirteen times. The full query bundle is present (sql-injection, shell-exec, xss-servlet and the rest) but resolves no sources or sinks for the input.

That is a worse state than unavailable, and the distinction matters here: a tool that cannot run reports tool-unavailable and the repository treats it as unscanned, whereas this one exits 0 with an empty result and would publish as a clean scan. The lane stays unwired deliberately until the query bundle can be shown to fire on a known-vulnerable input — an exit code is not a finding, and a green cell earned this way would be the exact defect this survey exists to expose.

One operational note worth acting on: govulncheck installed to ~/go/bin and bearer to ~/.local/bin, neither of which is on the default sweep path. Both resolve in an interactive shell and may not resolve under launchd.

Full inventory

Classification is the repository's own, read from LANE_KINDS in monitor/extractors.mjs rather than judged here. Of 43 published lanes, 38 can raise a security finding and 5 carry why: 'not-a-vulnerability' — they are executable scanners that will run, produce output, and never report a security issue, because that is not what they measure.

Registry entries with no execution path

Docker Bench and Falco are reachable only through bin/audit.mjs, gated on a Docker host and a production runtime. ScoutSuite is named only as an alternative inside Prowler's registry entry. Grype is version-tracked by the preflight and now installed, but drives no lane. Cosign is declared and never invoked — the actual signing in cra/attest.mjs uses Node's built-in ed25519, so any claim that this fleet signs with Sigstore is wrong today.

One arguable classification

The accessibility lane is marked as able to report, because the repository files it under policy with no not-a-vulnerability exemption and it reaches the severity sum. A WCAG failure is a conformance defect rather than a security issue, so this is the one row where the repository's own marker and the plain reading of "security issue" diverge.

On comparing languages

The row order above carries a caveat that should travel with it. No public dataset measures lines of code authored on GitHub — GitHub computes per-repository Linguist byte counts but has never published them in aggregate.

Worse for the delta column: Octoverse changed its metric between editions. The 2024 report ranked by total activity across commits, issues, pull requests and reviews; the 2025 report ranked by distinct monthly contributors. Rank movement across that boundary compares two different measurements, so the arrows are directional and not evidence.

The better denominator is already in the tree

Global popularity answers where the world's code is, not whether this fleet's coverage is adequate. An unscanned Elixir service you actually operate outranks PHP's global position every time.

monitor/codeql-coverage.mjs already derives per-language file counts from stored SARIF, using each lane's appliesIfSourceExt as a language-aware denominator — built because a naive ratio reports every polyglot repository as degraded. Its recorded fleet measurement is a better language axis than anything public, and it says something this matrix cannot:

python  200 runs   median 1.00   2 below 0.90
ruby     44 runs   median 1.00   0 below 0.90
csharp    3 runs   median 1.00   0 below 0.90
cpp       6 runs                 2 below 0.90 — worst 0.64
swift     2 runs                 1 below 0.90 — 0.67

The C++ and Swift lanes are wired, installed, and not reading the files they claim to cover. A support matrix can only show that a lane exists; this shows whether it looked.

Loose thread