On 2 August 2026, the Commission can impose Article 101 fines on general-purpose AI model providers for documentation and related Chapter V gaps. Article 5 prohibited practices have applied since 2 February 2025 (Art. 113(a), Chapters I and II). Fines reach up to €15M or 3% of worldwide annual turnover, whichever is higher, for GPAI providers (Article 101) and up to €35M or 7%, whichever is higher, for Article 5 prohibited-practice violations (Article 99(3)).
We read the public documentation of every major GPAI provider with material EU exposure and graded it against six obligation categories drawn from the Act. Across 12 providers, the average score is 49.6% on a 0-to-3 scale. Google (DeepMind) leads at 83.3%; xAI (Grok) trails at 11.1%. No provider scores the maximum on every obligation. The leaderboard, provider evidence files, and scoring code are open source: anyone can re-run the benchmark and dispute a score with a public URL we missed.
What we measured
We score six public-documentation categories. Article cites below match Regulation (EU) 2024/1689 (not the legacy key names inside the JSON schema):
| Article | Category | Question |
|---|---|---|
| 50(2) | Content disclosure | Are synthetic outputs marked in a machine-readable format? (Chapter IV transparency; includes GPAI systems.) |
| 51 + 52 | Systemic-risk classification | Is there public evidence of threshold assessment (Art. 51) and Commission notification where required (Art. 52)? |
| 53(1)(a) | Technical documentation | Does public material meet Annex XI completeness for model technical documentation? |
| 53(1)(c) | Copyright transparency | Is there a published copyright policy and a workable rights-reservation / opt-out path? |
| 53(1)(b) | Downstream information | Is integrator documentation sufficient under Annex XII? (Training-content summary under Art. 53(1)(d) is adjacent; this bucket scores deployer-facing integration docs.) |
| 55 | Systemic risk | Evaluations, adversarial testing, incident reporting, cybersecurity: for models at or above the ≥10²⁵ FLOPs presumption in Art. 51(2). |
Each obligation is scored 0 (no evidence), 1 (partial), 2 (adequate), or 3 (exceeds). Only public documentation counts: model cards, terms of service, API docs, transparency reports.
Article 55 is marked N/A for providers we treat as below the systemic-risk threshold (Mistral, Hugging Face SmolLM, Stability AI, Cohere, AI21 Labs, Inflection AI, Amazon Titan/Nova). Those providers are scored out of 15 instead of 18. The five scored on Article 55 in this release are Google, Meta, Anthropic, OpenAI, and xAI.
| Score | Label | Standard |
|---|---|---|
| 0 | No evidence | Provider does not address the obligation publicly. |
| 1 | Partial | Mentioned without verifiable detail. |
| 2 | Adequate | Specific, sourced documentation exists. |
| 3 | Exceeds | Machine-readable, regularly updated, audited. |
What this is not
- Not a runtime test. We are not running model outputs through ComplyEdge's compliance engine. That is a different, complementary measurement: see our runtime benchmark.
- Not a legal verdict. Until 2 August 2026, GPAI documentation obligations are not yet enforceable with fines. We are scoring readiness, not adjudicating non-compliance.
- Not exhaustive. Twelve providers in scope: every vendor with material EU GPAI exposure we could verify from public sources.
All twelve scores carry verification_status: needs_review in the working dataset. We publish the rankings with that flag visible. Dispute a score by opening a PR with a public URL we missed.
The leaderboard
Data as of July 11, 2026. Generated from scripts/benchmark/results/benchmark_latest.json (generated_at 2026-07-11T00:28:58Z).
| Rank | Provider | Class | 50(2) | 51/52 | 53(a) | 53(c) | 53(b) | 55 | Aggregate | % |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Google (DeepMind) | closed_api | 2 | 2 | 3 | 2 | 3 | 3 | 15 / 18 | 83.3% |
| 2 | Meta (Llama) | open_weights | 1 | 2 | 3 | 2 | 3 | 3 | 14 / 18 | 77.8% |
| 3 | Anthropic | closed_api | 0 | 2 | 3 | 1 | 3 | 3 | 12 / 18 | 66.7% |
| 4 | Mistral AI | open_weights | 1 | 2 | 3 | 1 | 3 | N/A | 10 / 15 | 66.7% |
| 5 | Hugging Face (SmolLM) | open_weights | 1 | 0 | 3 | 3 | 2 | N/A | 9 / 15 | 60.0% |
| 6 | OpenAI | closed_api | 1 | 1 | 2 | 1 | 2 | 2 | 9 / 18 | 50.0% |
| 7 | Stability AI | open_weights | 1 | 0 | 2 | 1 | 3 | N/A | 7 / 15 | 46.7% |
| 8 | Cohere | closed_api | 1 | 0 | 2 | 0 | 3 | N/A | 6 / 15 | 40.0% |
| 9 | AI21 Labs | open_weights | 1 | 0 | 2 | 0 | 2 | N/A | 5 / 15 | 33.3% |
| 10 | Inflection AI | closed_api | 1 | 0 | 1 | 0 | 3 | N/A | 5 / 15 | 33.3% |
| 11 | Amazon (Titan/Nova) | closed_api | 1 | 0 | 1 | 0 | 2 | N/A | 4 / 15 | 26.7% |
| 12 | xAI (Grok) | closed_api | 0 | 0 | 1 | 0 | 1 | 0 | 2 / 18 | 11.1% |
What the data shows
- Article 51/52 classification and notification is the industry floor. Seven of twelve providers have zero public evidence of systemic-risk threshold assessment or Commission notification. Even top scorers stop at partial (max score 2 / 3 in this release).
- Machine-readable text disclosure is functionally absent. Article 50(2) averages 0.92 / 3. Image generation has converged on C2PA credentials. Text outputs are a different story: Anthropic and xAI score zero on content disclosure.
- The distribution is bimodal. Five providers score zero on copyright transparency. xAI is the only provider scoring zero on four obligations simultaneously.
- Open-weights does not predict lower scores. Meta scores 3/3 on Article 55. Most open-weights vendors in this set are below the FLOPs threshold, so Article 55 is N/A.
- Technical documentation is the high-water mark. Average 2.17 / 3: five providers score the maximum on Art. 53(1)(a).
Why we publish this
ComplyEdge's product is a runtime compliance engine: OPA/Rego on the hot path, with the rule identifier and legal citation written when a rule fires. The benchmark is not the product. It is a measurement of the documentation gap the product operates in.
- Deployers. Obligations cascade from the model provider to the deployer. The deployer cannot inherit what was never published: 50.4 percentage points below full compliance industry-wide.
- Acquirers. Average GPAI provider scores 49.6% on public documentation; no provider exceeds 84%.
- Regulators. Public data shows where industry stands on documentation readiness. Article 51/52 classification and notification is the gap that remains after 2 August 2026 Article 101 fining powers began.
Reproduce it
git clone https://github.com/ComplyEdge/complyedge
cd complyedge
python scripts/benchmark/benchmark_runner.py
python scripts/benchmark/leaderboard_renderer.py
cat scripts/benchmark/results/leaderboard.mdRepository: github.com/ComplyEdge/complyedge
Provider evidence: providers/
Benchmark code: scripts/benchmark/
Leaderboard JSON: benchmark_latest.json