Protege Develops Specialized Benchmarks for AI in Medical Coding and Clinical Documentation

A LinkedIn post from Protege describes new medical benchmarks the company is using to evaluate AI performance in clinical documentation and coding. According to the post, Protege builds “uncontaminated, evaluation-ready” EMR datasets linked to bills that have successfully passed payer submission, including raw clinical notes, submitted billing codes, and ancillary non-billed codes.

The post indicates that all datasets were held out of model pretraining at the patient level to reduce data contamination and benchmark inflation, a common concern for public coding datasets. It also suggests that the benchmarks focus on payer-approved codes rather than merely submitted claims, aiming to align model evaluation with actual reimbursement and compliance outcomes.

According to the post, Protege worked with Vals AI to evaluate models on assigning primary and secondary ICD codes and maximizing compliant code sets, with expert coder review for validation. Reported results show models reaching about 88% accuracy on clinical documentation tasks but only 56% on medical coding, implying a performance gap in the structured reasoning required for billing.

The post characterizes medical coding as both an evidence extraction and optimization problem, factoring in disease severity, comorbidities, procedural context, and institution-specific SOPs. For investors, this framing suggests a potentially defensible niche for Protege as a data and evaluation provider in healthcare AI, where rigorous, payer-aligned benchmarks may be critical for managing financial, compliance, and administrative risk.

If these benchmarks gain traction among healthcare AI developers and providers, Protege could benefit from recurring demand for specialized datasets tied directly to revenue-cycle and coding workflows. The results highlighted in the post also point to ongoing limitations of current AI models in complex billing tasks, which may sustain demand for improved data infrastructure and evaluation frameworks rather than rapid full automation of medical coders in the near term.