Do Foundation Model Providers Comply with the Draft EU AI Act

We assess whether score foundation model providers are likely to compl with the EU Parliament's AI Act requirements in their draft from June 2023. We interpret the Act text (as described below) and use public information to make these assessments. None of this analysis is legally binding, and many aspects of this analysis are likely to diverge from the status in 2025-2026 when the Act is likely to go into effect (e.g. the EU AI Act's finalized requirements, the foundation model providers, the flagship foundation models of each provider, the appropriate interpretation and standards for a given requirement).

Methodology.

1. We identify 22 requirements in the EU Parliament's draft AI Act that are directed towards foundation model providers in requirements.md.
2. We select 12 of these 22 requirements to evaluate, justifying why we excluded the remaining 10, in requirements.md.
3. We design 5-point rubrics for each of 12 requirements in rubrics.md.
4. We select 10 major foundation providers (e.g. OpenAI, Google) to assess, designating a flagship foundation model that represents the provider for the purposes of our evaluation.
5. We rate all 10 providers for all 12 requirements, producing the final scores in results.png. Every underlying score comes with an explicit justification for why and provenance for the information we used to make our decisions in /grades.