Document classification AI benchmark

What we compare (and what we do not)

We compare two families of approach to document classification: a generalist LLM via API versus a specialised model deployed in a controlled environment (local / private). Dimensions are typing accuracy, p50/p95 latency and approximate € cost per document.

We do not publish vendor rankings or third-party metrics. Any numeric result on this page will come exclusively from Magnum’s anonymised test set, when it is ready for publication.

Experiment methodology

The protocol is designed to be reproducible in a client pilot and, later, in a public report without identifiable data.

• Corpus: anonymised real documents (invoices, delivery notes, credit notes, statements, other) with human reference labels
• Metrics: accuracy / F1 per class, latency, estimated cost per document and reject / human-review rate
• Conditions: same prompt or typing policy, same timeouts and the same definition of “success”
• Train/eval split when the specialised model is tuned; the generalist is evaluated zero-shot or with the same instructions

Results pending / shadow experiment

There are still no public figures on this page. We are preparing the anonymised dataset and measurement protocol. We will publish results only when Magnum’s test set is ready and reviewed; until then we do not show invented percentages or performance tables.

If your organisation wants to contribute anonymised documents or stress-test the framework on your typology, get in touch: the shadow experiment accepts partners under NDA.

How to use the framework in a buy decision

Before choosing “the most powerful model”, define classes, cost of error and whether data may leave the perimeter. A generalist may suffice at low volume; a local model often wins when latency, privacy or unit cost under high volume matter.

Magnum applies the same criterion in the business case and pilot validation: if there is no measured ROI, we do not scale.