When the Model Is the Vendor: What Happens to Compliance When You Don’t Own the Weights

By Alex

On 23 December 2025, the FDA cleared UpDoc V1.0 under 510(k) K253281 — prescription software that manages insulin for adults with type 2 diabetes, built on a large language model. It is a landmark clearance. And according to Innolitics’ reading of the public record, that record does not identify the foundation model, the prompt stack, the temperature, the top-p, the model version, the runtime orchestration pattern, the validation harness, or the configuration management system.

A device was authorised. Which model it runs is, from the outside, unknown.

That is not an oversight by a regulator caught off guard. It is the first visible symptom of a problem every regulated-software regime is about to meet: these frameworks assume you can describe what you shipped, pin its version, and reproduce its behaviour. A manufacturer building on somebody else’s foundation model can struggle with all three, because the component’s release schedule belongs to a vendor with no obligation to anyone’s clearance.

The documentation was written for a model you trained yourself

Look at what the FDA asks for. In a real presubmission exchange, the agency required “an engineering description of the underlying model architecture and how it was trained” — input dimensions and patient demographics, network architecture down to layers and activation functions, the development approach including transfer learning and regularisation and loss functions, sampling methods, distribution across covariates, acquisition conditions, internal performance data, post-processing. The stated reason: “This information is required to help us understand the underlying functionality and complexity of your device.”

Every item on that list is answerable if you trained the model. Almost none of it is answerable if you called an API.

The same gap runs through validation. The FDA expects evidence that “your test dataset includes data that hasn’t been used to train the foundation model to avoid data contamination.” That is a reasonable requirement and, for a model whose training corpus is undisclosed, an unfalsifiable one. You cannot prove a negative about a set you have never seen. The best a manufacturer can do is construct data that did not exist publicly before the model’s cutoff — which is real work, and still an argument rather than a proof.

This is a supply-chain problem, and the industry already has a name for the artifact meant to solve it. An AI bill of materials records models with version, lineage, weights identifier, licence and provenance; datasets with source, licence, preprocessing, splits and known biases; frameworks and dependencies. CISA and its G7 partners pushed supply-chain transparency guidance for AI systems through 2026, and AIBOMs are moving from optional security artifact toward procurement requirement.

But an AIBOM, as one practitioner guide puts it, documents “the structure, provenance, and relationships of an AI system — not the raw weights or the proprietary algorithm itself.” The transparency artifact stops precisely where the opacity starts. It is a faithful record that a component exists and cannot be characterised. Software supply-chain security matured on the assumption that a dependency is inspectable and pinnable — you can read log4j, hash it, diff two versions. Weights break each of those. We have traced how configuration became executable over twenty years; this is the same movement, one layer further out, with a component that resists inspection by construction.

Question 24

On 18 August 2026, CDRH’s Digital Health Center of Excellence published Considerations for the Regulation of Generative AI-Enabled Medical Devices — a discussion paper with 26 numbered questions, open for comment until 19 October under docket FDA-2026-N-7874. It explicitly does not represent draft or final guidance. It is the agency thinking out loud, which makes it more revealing than guidance usually is.

Question 24 addresses changes “initiated by the foundation model developer rather than the device manufacturer.”

That single clause is the whole problem stated in regulatory language. Predetermined Change Control Plans, finalised in August 2025, were built so a manufacturer could pre-authorise its own planned modifications and deploy inside a validated envelope without a new clearance. The design assumes the manufacturer initiates the change. When the model provider silently swaps what answers a stable endpoint, the change arrives from outside the envelope entirely, and no PCCP was written for it.

The agency’s tentative answer tells you how unsettled this is: it asks whether nightly performance reruns would be enough to detect unexpected model changes by cloud providers. Read that as an engineering proposal and it is monitoring standing in for a configuration guarantee — running your validation suite every night because you cannot otherwise know whether the component under it is still the one you cleared.

The paper also names data contamination, benchmark saturation and lack of representativeness in public benchmarking assets as failure modes that need prospective checking, and it insists on evaluating “the final user-facing device, as configured and intended to be deployed for real-world use” — not the model in isolation, the assembled system with its prompts and guardrails.

Two architectures that actually clear

Faced with all this, the manufacturers who have gotten through have converged on two strategies. Both work by shrinking the ungovernable component until it stops being load-bearing.

Freeze it. The FDA has said it “strongly recommend[s]” a clear account of how a model is deployed and called a “frozen application” desirable. One device description in a presubmission spells out what that looks like in practice: “Llama 2 OTS is downloaded and all inference is run locally on OTS hardware without any requirement to communicate to the internet. The weights are frozen (non-adaptive).” Open weights, local inference, dependencies locked in a container. This is the only configuration where a manufacturer can honestly claim the artifact it validated is the artifact in the field — and note that it rules out the hosted frontier models entirely. Determinism gets engineered on top: temperature set to zero, seed pinned, test cases run repeatedly to confirm outputs are reproducible. The FDA asks for “best efforts” toward determinism, which is a quiet admission that the property is not natively available.

Fence it. UpDoc’s clearance shows the second route. Its authorised change protocol requires that modifications “preserve deterministic insulin dosing logic” and maintain “exact data handling, and auditability.” Conversation outputs pass through schema and safety checks before reaching provider-configured clinical logic. The language model handles the conversation; a deterministic calculator makes the dosing decision. The predicate device, Hygieia’s d-Nav (K181916), was a conventional insulin dose calculator — and the clearance essentially argues that the new device is that same calculator with a better front door.

Nobody has yet cleared a device where a hosted third-party model makes the clinical decision. That absence looks technological and is actually structural: the evidence such a submission would need cannot currently be produced. Teams working through this on real projects will recognise the shape from ordinary device certification work — the hard part was never the algorithm, it was proving what the algorithm was on the day you tested it. The category boundary itself keeps moving too, as we covered when software became a medical device.

Europe hands the manufacturer the whole bag

The EU arrived at the same problem from the opposite direction and assigned the liability more bluntly. Under MDCG 2025-6 guidance, the MDR or IVDR manufacturer is the AI Act provider; the clinic using the device is merely a deployer. General-purpose AI obligations have applied since 2 August 2025 with enforcement from 2 August 2026, high-risk obligations apply in full from August 2026, and integration with MDR/IVDR conformity assessment lands 2 August 2028.

So the company that fine-tuned somebody else’s model carries the full weight of high-risk conformity for a component it did not build. Its one formal lever is that GPAI providers owe downstream providers technical documentation and must publish a summary of training content. Whether that summary is specific enough to satisfy a notified body examining a Class II device is, at present, an open question — and it is being answered contract by contract, which is why AI-BOM clauses and provenance warranties are showing up in procurement rather than in standards. Anyone mapping their obligations across regimes will find this sits alongside the compliance audit work, not apart from it.

What the regulator is really weighing

The most consequential sentence in the August paper is not about models at all. It is this: “CDRH is considering whether it is appropriate to accept greater premarket uncertainty regarding a GenAI-enabled device’s benefit-risk profile through greater reliance on postmarket monitoring.”

Strip out the register and it says: we may not be able to know enough before approval any more, so we are weighing whether to find out afterwards instead. That is a genuine trade, not a retreat — postmarket surveillance catches things trials never will. But it moves the burden of proof from a fixed point in time to a continuous obligation, and it moves cost from the submission to the operating budget. A manufacturer that clears a device on those terms owns a monitoring commitment for the life of the product.

The FDA has authorised more than 1,350 AI-enabled devices, roughly double the 2022 count, and has said it plans to start tagging the ones that incorporate foundation models — LLMs through multimodal architectures — so clinicians and patients can tell when such a component is present. Sit with the implication. The regulator is announcing a project to find out which of its own authorised devices contain a foundation model, because right now the paperwork does not say.

The tagging is announced, not implemented. The discussion paper is questions, not rules. Comments close on 19 October. Whatever comes out of that docket will set the terms for every company that wants to build a regulated product on a model it does not own — which, before long, will be most of them.