On 24 March 2026, two versions of LiteLLM that stole credentials were live on PyPI. For about 40 minutes. LiteLLM is not a model and not a prompt. It is the library many applications use to talk to their models. That is the AI supply chain: everything that sits between someone else's source and your application.
Classic supply chain attacks hit code you did not write. With AI there is more: components nobody can read, components that change without a release, and components that act on their own. This post is the overview: six ways in, one documented incident for each, and what to do first. The deep dive on model files is in Open-Source Models and Their Risks.
What belongs to the AI supply chain
The supply chain is everything you did not build yourself that still decides what your application does. For an AI system that is six building blocks:
The route from the sources to your application. At each of the six stations someone can swap something out without it showing at the end.
- Data. Training data, fine-tuning data and everything a system looks up at runtime (RAG// RAGRetrieval-Augmented Generation: before answering, the system searches its own knowledge base and hands the hits to the model.ExampleA support bot that looks things up in the manual before every answer.).
- Models. Weights and architecture, usually from a hub such as Hugging Face, plus adapters and converted variants.
- Packages. Frameworks and their dependencies, from PyTorch down to the smallest helper library.
- Infrastructure. Hubs, package indexes, build pipelines and the credentials for them.
- Providers and APIs. Hosted models, gateways and the service providers behind them.
- Agent tools. MCP servers// MCP serverAn extension through which an AI agent connects tools and data. MCP stands for Model Context Protocol.ExampleA server that lets the agent send email or access GitHub., skills, plugins and the prompt templates that steer behaviour.
This classification is not made up for this post. OWASP lists the supply chain as a risk of its own for LLM applications (LLM03). MITRE ATLAS has the technique AI Supply Chain Compromise with separate sub-techniques for hardware, software, data, models, container registries and agent tools.
What is new about it
Much of this is familiar from the software world. Four things are different.
1 // Components nobody can read
Source code can be reviewed, and a diff shows every change. Weights are millions to billions of numbers, and nobody can tell from them what was trained in. OWASP calls models binary black boxes for which static inspection offers little assurance, and notes that there are currently no strong provenance assurances for published models.
The same goes for the data behind them. In 2025 Anthropic, together with the UK AI Security Institute and the Alan Turing Institute, showed that 250 poisoned documents are enough for a backdoor, in models from 600 million to 13 billion parameters. The backdoor they tested was deliberately simple. If you feed a system your own knowledge through RAG, you inherit the problem at runtime: in the PoisonedRAG study, five injected texts per target question in a knowledge base of millions of texts were enough for a 90% success rate.
2 // Components that change without a release
A package only changes when someone publishes a new version. A model behind an API can change while nothing about your call changes. Researchers from Stanford and Berkeley compared two states of the same service in 2023: GPT-4 correctly identified 84% of the prime numbers it was asked about in March, and 51% in June. Their conclusion: the behaviour of the "same" service can change substantially in a short time, which is why it needs continuous monitoring.
It does not even have to be the model. In September 2025 Anthropic disclosed that three infrastructure bugs had intermittently degraded Claude's response quality over several weeks. In the worst hour, 16% of the requests to one of the models were affected (postmortem). Anthropic stresses that it never reduces quality on purpose. For you as a customer the effect is the same: the result changes, your code does not.
Fixed model versions help against this. OpenAI calls them snapshots, and at Anthropic every model ID stands for a pinned version, except for the aliases of older models.
3 // Language as a carrier
In classic software a comment is harmless. In an AI system text is a control: prompt templates, rule files and tool descriptions determine what the model does, and hardly anyone reviews them like code. In 2025 Invariant Labs described tool poisoning: instructions in the description of an MCP tool that the user does not see but the model reads. Then there is the "rug pull": a server changes the description after it has been approved. Pillar Security showed the same pattern for the rule files of coding assistants, hidden with invisible Unicode characters.
Both were research findings. Pillar documented the first attempt in the wild in August 2026: a single account opened 23 pull requests against other people's projects within 74 minutes, most of them meant to wire in a rigged MCP server (Deadbugz). The server behaved normally for three calls and then steered the agent towards SSH keys and cloud credentials. At the time of the analysis none of the pull requests had been merged, and four were still open.
4 // Components that act on their own
A package manager installs what the file says. An agent decides for itself what to install and which tool to call. That has two consequences. First, dependencies appear that no human chose: in a study of 16 LLMs, 19.7% of the recommended packages did not exist. In early 2026 Aikido found one such invented package name in 237 GitHub repositories, spread through copied agent skills, and registered it defensively itself.
Second, the agent becomes the attacker's tool. In the Nx incident of August 2025 the attackers, according to Wiz, invoked the AI command-line tools installed on the machine to read out the file system. A year later an npm worm added malicious hooks to .claude/settings.json and .vscode/tasks.json so that its code runs automatically on the next start, and passed off its commits as those of a coding assistant (Aikido).
Six ways in, six pieces of evidence
For each of the six building blocks there is at least one documented case.
| Building block | What happened | What it shows |
|---|---|---|
| 1 Data | July 2026: a crafted dataset led to code execution in the dataset processing at Hugging Face, with access to internal datasets and credentials. | Data is not just content. It can be the attack itself. |
| 2 Models | February 2024: JFrog finds around 100 models on Hugging Face that execute code when loaded. May 2026: a fake repository reaches the number one trending spot (HiddenLayer). | Popularity is not a seal of approval. More in the deep dive. |
| 3 Packages | December 2022: the package torchtriton on PyPI carries the name of a dependency of PyTorch-nightly and steals SSH keys (PyTorch). The pattern is called dependency confusion// Dependency confusionA public package carries the name of an internal one and gets installed in its place.ExampleInternally there is acme-utils 1.2. An attacker publishes acme-utils 99.0 on PyPI, and pip picks the higher version.. |
A single package name is enough. |
| 4 Infrastructure | March 2026: attackers overwrite 76 of 77 version tags of the GitHub Action of the security scanner Trivy (Aqua Security). LiteLLM suspects that its own incident a few days later originated in that Trivy dependency of its CI, which is not confirmed. | The tools that are meant to protect you are part of the chain too. |
| 5 Providers and APIs | January 2025: Wiz finds an openly accessible DeepSeek database with over a million log lines, including chat history and secret keys. November 2025: at Mixpanel, an analytics provider used by OpenAI, names, email addresses and organisation IDs of API users are exported. Chats and keys were not affected. | Behind every provider there is a supply chain of its own that you cannot see. |
| 6 Agent tools | September 2025: an npm package poses as Postmark's MCP server and from version 1.0.16 BCCs emails to an external server (Postmark). February 2026: 13.4% of 3,984 audited agent skills have a critical security issue (Snyk). | A tool for the agent is a dependency with power of attorney. |
What to do first
Nobody secures six building blocks in a day. Five steps give you the most, in this order.
- 1. Build an inventory. Write down what you load: models with their revision, datasets, packages, providers, MCP servers, prompt templates. For the software part there is a format for this, the SBOM// SBOMSoftware Bill of Materials: a machine-readable list of every component in a piece of software, with versions.ExampleA JSON file with entries such as
torch 2.6.0ornumpy 1.26.4.. For AI, the G7 Cybersecurity Working Group proposed minimum elements in May 2026, published by Germany's BSI: SBOM for AI, Minimum Elements. - 2. Pin everything. A commit hash for models, hashes for packages, a fixed model version for APIs, a fixed version for MCP servers, prompts in your own repository. What cannot change by itself cannot surprise you.
- 3. Check before anything runs. Scan model files and load them without network access first. How to do that in practice is in the deep dive.
- 4. Limit permissions. An agent, a build pipeline and an inference server get only the access they need, and no open path to the outside. Then a compromised component is an incident and not a disaster. More in the defense playbook.
- 5. Watch behaviour. A fixed set of test cases runs against every model, before every change and regularly afterwards. For API models this is the only check you have.
| Building block | First measure | Tools |
|---|---|---|
| Data | Record where it comes from, pin datasets to a revision, restrict the sources for runtime lookups. | Dataset revisions, review |
| Models | Allow Safetensors only, scan before loading, load by commit hash. | ModelAudit, Fickling, ModelScan |
| Packages | One single package index, install with hashes only, review every new dependency, including the one your assistant suggested. | pip --require-hashes, uv, pip-audit |
| Infrastructure | Tokens with minimal permissions, pin actions and images to a commit instead of a tag, rotate credentials completely. | zizmor, Syft, Grype |
| Providers and APIs | Use a fixed model version, log the model ID, run test cases as a regression test, separate keys per service. | promptfoo, garak |
| Agent tools | Treat MCP servers and skills like dependencies: read the source, pin the version, limit permissions. Keep prompts in your own repository. | Git, lockfiles, review |
Two traps with packages
pip-audit finds known vulnerabilities, not malicious packages. Its documentation says so explicitly. And --extra-index-url is unsafe for private packages according to the pip documentation, because pip picks the best version across all indexes. By default uv stops at the first index that has the package.
What regulation says
Three sets of rules touch the AI supply chain. None of them covers it completely.
- Cyber Resilience Act. Manufacturers of products with digital elements have to draw up a software bill of materials in a commonly used, machine-readable format, covering at least the top-level dependencies (Regulation (EU) 2024/2847, Annex I Part II). They do not have to publish it. It applies from 11 December 2027, and the reporting obligations have applied since 11 September 2026. Whether an individual model falls under it depends on the product.
- AI Act. Providers of general-purpose AI models have to make information and documentation available to those who integrate their model (Article 53 of Regulation (EU) 2024/1689, applicable since 2 August 2025). For freely licensed models with public weights this only applies if they carry systemic risk. So open models, of all things, may come without that documentation. For high-risk systems, Article 25 requires written agreements with suppliers. Anyone who makes tools or components publicly available under a free and open-source licence is exempt, providers of GPAI models are not. After the amendment of July 2026 this applies from 2 December 2027 at the earliest, and only from 2 August 2028 for Annex I systems.
- Recommendations. In its guide SP 800-218A, NIST advises verifying the integrity and provenance of every acquired model before using it. Since December 2025 OWASP has had a separate Top 10 for agentic applications, with the agents' supply chain as item ASI04. Neither is binding.
The short version
- The AI supply chain has six building blocks. Most teams secure one or two of them.
- What is new is not the attack through a supplier. What is new is that components cannot be read, change without a release, are steered by language and act on their own.
- Almost all incidents of the last two years came through legitimate channels: an official package, a real build, a tool with a good name.
- Inventory, pin, check, limit permissions, watch behaviour. In that order.
Sources:
Want to know which of the six ways in are open in your organisation? Let's talk. In 30 minutes, with no obligation, we walk through your supply chain one building block at a time.