Open-Source Models and Their Risks: Checking Model Files

An AI model is a neural network: an architecture that defines how to compute, and millions to billions of learned numbers, the weights. The file it ships in can hold more: instructions that run when the file is loaded. If you load someone else's .pt file unchecked, in the worst case you start a program that somebody else wrote.

Hugging Face hosts more than two million public models, and most teams pull them with the same trust they give an npm package. This post is the deep dive on model files: which formats can execute what, what scanners miss, and how to vet a third-party model before it reaches production. For the overview of the whole chain, from data to agent tools, see The New Risk in the AI Supply Chain.

When a model file becomes a program

Many model formats are built on Python's pickle. Pickle does not just store values. It stores a sequence of instructions for rebuilding an object. Through the __reduce__ method an object may name any function with any arguments, and the loader calls it. The Python documentation is blunt about it: the module is not secure, and you should only unpickle data you trust.

import os, pickle

class Model:
    def __reduce__(self):
        # any command
        return (os.system, ("id",))

# looks like a model
pickle.dump(Model(), open("model.pkl", "wb"))
# runs the command
pickle.load(open("model.pkl", "rb"))

So the format decides what can happen at load time:

Two things tend to get lost here. First: weights_only=True has been the default since PyTorch 2.6 and restricts the unpickler. It is not a guarantee: CVE-2025-32434 allowed code execution with the flag set, up to version 2.5.1. Second: Safetensors protects you from code at load time, not from what the weights have learned.

Code in the architecture: Keras Lambda layers

Pickle is not the only way to get code into a model. Keras has Lambda layers: building blocks that hook an arbitrary Python expression into the model as a computation step. According to the Keras documentation they are saved as Python bytecode, which it calls "potentially unsafe" itself.

CERT/CC describes the consequence plainly: loading a third-party Keras model can mean running third-party code. Keras 3 therefore refuses by default to load Lambda layers whose function is stored as bytecode (safe_mode). Two gaps remain. In Keras 2 and in Keras 3 before version 3.12, that check does not apply to classic .h5 files, which is also how CERT/CC describes it. Only since 3.12 from October 2025 does Keras pass safe_mode through to the H5 loader. And the protection itself was bypassed in 2025: CVE-2025-1550 allowed code execution with safe_mode switched on.

Changing the format does not help here: Safetensors stores weights only, and the architecture still lives in the Keras file or in code. What helps is looking at the list of layers before the model is loaded. ModelScan looks specifically for Lambda layers in .h5 files (install it with the extras modelscan[tensorflow,h5py]). Any Lambda or custom layer in a model you did not build yourself is a reason to ask questions.

What has already happened with model files

In February 2024 JFrog found around 100 models on Hugging Face that executed code when loaded, one of them opening a reverse shell// Reverse shellThe compromised machine opens a connection to the attacker and hands over a command line.. JFrog itself notes that most of these payloads// PayloadThe part of an attack that does the actual damage. Here: the injected code. are probably tests by researchers or bug bounty hunters. The mechanism works either way.

Three developments since 2025 make things harder.

1 // Scanners can be fooled

Hugging Face scans uploaded files, but says itself that this is not 100% foolproof. In February 2025 ReversingLabs described two models (nullifAI) that were compressed with 7z instead of ZIP and whose pickle stream breaks right after the payload. They were not flagged as unsafe, and the payload still ran. In the same year Sonatype reported four and JFrog three more ways around the picklescan scanner. A green tick on the platform is a hint, not a finding.

2 // A name is not proof of origin

PoisonGPT demonstrated the principle in 2023: a deliberately falsified model, uploaded under EleuterAI instead of EleutherAI. In 2025 Unit 42 described Model Namespace Reuse: when an author account is deleted or transferred, someone else can register the old name again, and every pipeline that loads models by name alone pulls the stranger's model. According to Unit 42 this also affected Google's Vertex AI and Microsoft's Azure AI Foundry. In May 2026 a repository imitating an OpenAI release reached the number one trending spot on Hugging Face before it was disabled. According to HiddenLayer the download numbers were almost certainly inflated artificially.

3 // Adapters and converters

A model rarely comes alone. LoRA adapters// LoRA adapterA small add-on file that adapts a base model to a task without retraining it.ExampleAn adapter that tunes a general language model for support replies. are shared just like models, and researchers showed that a backdoor can be trained into an adapter once and then merged with other adapters without any further training. That is why the OWASP Top 10 for LLM Applications lists vulnerable LoRA adapters as a supply chain risk of their own. In 2024 HiddenLayer showed that Hugging Face's Safetensors conversion service could be hijacked to open pull requests in the name of the official bot. Anything that has passed through someone else's pipeline is a new artefact.

Risk, measure, tool

The short version as a table. Every tool named here is open source.

Packages, providers and agent tools come with risks of their own. Those are covered in the overview: The New Risk in the AI Supply Chain.

An intake pipeline in five gates

Individual tools do little if a model reaches production without passing them. The remedy is a fixed intake process: five gates that every artefact has to pass, however good the source's reputation is.

• 1. Quarantine. Download into an isolated staging environment, never straight into production. The first load happens without network access.
• 2. Provenance. Organisation, account age, history, model card. Load by commit hash, not by name.
• 3. Integrity. SHA-256 against the published value and, where available, the signature. OpenSSF Model Signing signs a manifest of all files of a model. NVIDIA says it has been signing all of its own models in the NGC catalog this way since March 2025.
• 4. Scan. Static scan of the files, the layer list for Keras, the chat template for GGUF, an audit of the dependencies.
• 5. Approval. Only now does the model move into the internal registry.

Try it yourself: the scan in three minutes

You need a terminal (macOS or Linux, WSL on Windows) and Python 3.10 to 3.12. The three scanners come from pip, ideally into a fresh virtual environment:

python3 -m venv .venv
source .venv/bin/activate
pip install fickling modelscan modelaudit

Then build a harmless test file. It uses the same trick as above, but the command only prints one word:

python3 - <<'EOF'
import os, pickle

class Model:
    def __reduce__(self):
        return (os.system, ("echo pwned",))

pickle.dump(Model(), open("test.pkl", "wb"))
EOF

Now check the file without loading it:

# standard library: shows the instructions in the pickle
python3 -m pickletools test.pkl

# translates the pickle back into Python and rates it
fickling --check-safety -p test.pkl

# scan and assign a severity
modelscan -p test.pkl
modelaudit test.pkl

All three scanners fire. Fickling calls the file "overtly malicious", and ModelScan reports:

Unsafe operator found:
  - Severity: CRITICAL
  - Description: Use of unsafe operator 'system' from module 'posix'

posix is the module behind os.system on macOS and Linux. What matters is what you did not do: you never opened the file with pickle.load().

On a real project

The tools stay the same. Load with a pinned version, here using the transformers library. org/model stands for the repository name on Hugging Face, and you find the commit hash in its history:

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "org/model",
    # full commit hash, not "main"
    revision="877b84a8f93f2d619faa2a6e514a32beef88ab0a",
    use_safetensors=True,     # no pickle
    trust_remote_code=False,  # the default, leave it
)

The commit hash is more than cosmetics here. If the repository has no safetensors file and the revision is not pinned, transformers tries an automatic conversion through the pull-request branch of Hugging Face's conversion bot, the very service from “Silent Sabotage”. With a pinned commit hash that does not happen. And without use_safetensors=True the library quietly falls back to the pickle file.

The checks from gates 3 and 4 run in the project folder that holds your requirements.txt:

# Gate 3: compare the hash with the published value
sha256sum model.safetensors

# Gate 4: fetch the repo from the Hub, scan it statically
modelaudit hf://org/model

# Gate 4: check dependencies for known vulnerabilities
pip install pip-audit
pip-audit -r requirements.txt

# Gate 4: generate the bill of materials and check it
syft . -o cyclonedx-json=sbom.json
grype sbom:./sbom.json

Three notes. Syft and Grype are standalone programs, not Python packages. On a Mac you install them with brew install syft grype. On older macOS versions sha256sum is called shasum -a 256. And once every line of your requirements.txt carries a hash, pip-audit --require-hashes -r requirements.txt enforces strict mode. Without hashes that call aborts.

A signature can only be verified if the publisher has released one. Then you need the signature file, the publisher's expected identity and their login provider:

pip install model-signing
model_signing verify ./model --signature model.sig \
  --identity "releases@example.org" \
  --identity_provider "https://accounts.example.org"

Adapters and converted models take the same path. What changes for API models, MCP servers and prompt templates is covered in the overview: The New Risk in the AI Supply Chain.

What no tool solves

Three things remain, however good the pipeline is.

• Weights cannot be read. No static scanner finds a trained-in backdoor. Only your own tests and watching behaviour in operation help against that.
• Correct answers prove nothing. A payload runs at load time. What the model answers afterwards can look entirely normal.
• Trust is not a control. The nastiest cases came through legitimate channels: a repository at the top of the trending list, a model under a deleted author's name, the platform's own conversion service.

Where I would start if nothing is in place yet:

• Make Safetensors the only permitted format and require a reason for every exception.
• Pin every model by commit hash and lock every dependency by hash.
• Put a scanner into CI before a model is allowed into the registry.
• Block outbound traffic from the inference server by default. A model server has no reason to call unknown hosts.
• Send prompts, adapters and MCP servers down the same review track as code.

Sources:

Want to know what is really inside your model and agent supply chain? Let's talk. In 30 minutes, with no obligation, we walk through the five gates for your stack.