The problem

When a team puts a third-party model into production, it inherits everything that model does not say about itself: what data it was trained on, which biases are known, under what conditions it stops being reliable, under which licence it can be used. That information lives in the model card, a document nobody is required to complete and which, in practice, is filled in very unevenly. The risk is not theoretical: it is the difference between being able to answer an audit and not.

How it works

01

Fetch

Downloads the model card and licence metadata from the Hugging Face Hub.

02

Extraction

Looks for six mandatory fields — limitations, bias, training data, licence, context length and benchmarks — first with regular expressions over the headed sections, and only falls back to a model call when the regex fails.

03

Scoring

Computes a score from 0 to 100 based on the fields found, weightable per field because not every consumer values each gap the same way.

04

Report

Generates a documentation risk report in Markdown, field by field.

05

CI gate

Runs in batch across several models and fails the build when the score drops below the configured threshold.

The decision that matters most

The regex → model order is a deliberate cost decision: the common case should not pay for an inference. Most well-structured model cards resolve without calling any model, and the call is reserved for ambiguous cases. The second decision is turning it into a CI gate rather than a report someone reads occasionally. A quality metric that blocks nothing ends up ignored.

What it does not do

It measures whether a model is documented, not whether it is good. A well-written model card for a mediocre model scores high, and that is the intended behaviour: this is a documentation risk tool, not a performance benchmark.

What I learned

That a good share of real technical risk sits not in the model, but in what was never documented about it. And that a check only changes a team's behaviour when it has automatic consequences.