Home
Cognium
cognium-dev 3.19.4 · benchmark date April 22, 2026

cognium-dev SAST Benchmarks

Static analysis only. No LLM. Published snapshot.

cognium-dev is an MIT-licensed semantic static analysis engine. These benchmarks measure the static analysis engine only—no LLM verification layer is involved. The dated summary and machine-readable result artifacts are available in the cogniumhq/sast-benchmarks repository for independent review.

Results

Benchmarks by Language

Java (6 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
OWASP Benchmark 1,415 708 707 0 0 100% 0% 100%
Juliet Test Suite 243 122 121 0 0 100% 0% 100%
SecuriBench Micro 123 60 60 1 2 96.8% 1.6% 97.7%
CWE-Bench-Java 120 61 — — 59 50.8% — 50.8%
WebGoat 29 26 — — 3 89.7% — 89.3%
DVJA 7 7 — — 0 100% — 100%

Node.js / TypeScript (3 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
NodeGoat 14 14 — — 0 100% — 100%
Juice Shop 14 14 — — 0 100% — 100%
NodeJS Synthetic 25 23 — — 2 92.0% — 92.9%

Python (2 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
PyGoat 26 23 — — 3 88.5% — 90.0%
DVPWA 6 6 — — 0 100% — 100%

Rust (2 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
Rust Synthetic 50 46 — — 4 92.0% — 92.3%
CWE-Bench-Rust 30 28 — — 2 93.3% — 94.4%

Other Languages (3 benchmarks)

Benchmark Tests TP TN FP FN TPR FPR Score
Bash Synthetic 31 31 — — 0 100% — 100%
HTML/JS Synthetic 30 30 — — 0 100% — 100%
Firing Range 40 35 — 2 3 92.1% — 92.1%

Legend: TP = True Positives · TN = True Negatives · FP = False Positives · FN = False Negatives · TPR = True Positive Rate · FPR = False Positive Rate

Summary

Results by Language

Language Perfect (100%) Near-perfect (90%+) Total Benchmarks
Java 3 4 6
Node.js / TypeScript 2 3 3
Python 1 2 2
Rust 0 2 2
Bash 1 1 1
HTML/JS 1 1 1
Total 8 13 16
Deep Dive

CWE-Bench-Java by Category

CWE Category Detected Missed Rate
CWE-022 Path Traversal 37 / 55 18 67.3%
CWE-078 Command Injection 6 / 13 7 46.2%
CWE-079 XSS 13 / 31 18 41.9%
CWE-094 Code Injection 5 / 21 16 23.8%
Methodology

How we measured

  • cognium-dev is a semantic static analyzer with inter-procedural taint tracking
  • All results are from static analysis only—no LLM involvement in detection or verification
  • Each benchmark's source dataset is linked to its origin: OWASP Benchmark, NIST Juliet Test Suite, CWE-Bench-Java, and others
  • CWE-Bench-Java uses per-project binary detection: each project contains one CVE, scored as detected or not
  • The published summary, CSV, JSON, and CWE breakdown are in cogniumhq/sast-benchmarks/results/2026-04-22/
Limitations

Known gaps

  • SSTI (Server-Side Template Injection) is not currently in cognium-dev's CWE coverage—this causes the PyGoat false negative
  • Firing Range has 2 false positives in the escape/ category (escaped output flagged) and 3 false negatives in cors/ (CORS misconfigurations not detected)
  • CWE-Bench-Java uses per-project detection, not per-CVE counts—a single missed sink in a complex project counts as a full miss
  • These benchmarks test static analysis only—the full cognium-dev + LLM verification pipeline (SAST+LLM) produces different results, published separately
  • The historical one-command benchmark harness is not present in the current source tree, so this dated result set is auditable but not currently reproducible with a single published command
Audit

Inspect the published snapshot

The April 22, 2026 result set includes a human-readable summary plus JSON and CSV artifacts. A fresh full rerun remains blocked until the historical benchmark harness is restored and published.

git clone https://github.com/cogniumhq/sast-benchmarks
cd sast-benchmarks/results/2026-04-22
ls summary.md results.csv results.json cwe-bench-java-breakdown.csv

Review the published result artifacts and the current rerun status.