Create Model Ops and Supply Chain Security Cheat Sheet by Tryingtobeabetterprogrammer · Pull Request #2058 · OWASP/CheatSheetSeries

Tryingtobeabetterprogrammer · 2026-03-09T08:04:21Z

Initial draft covering unsafe deserialization risks, model scanning (modelscan), and migration to safetensors. Fixes #1781.

You're A Rockstar

Thank you for submitting a Pull Request (PR) to the Cheat Sheet Series.

🚩 If your PR is related to grammar/typo mistakes, please double-check the file for other mistakes in order to fix all the issues in the current cheat sheet.

Please make sure that for your contribution:

In case of a new Cheat Sheet, you have used the Cheat Sheet template.
All the markdown files do not raise any validation policy violation, see the policy.
All the markdown files follow these format rules.
All your assets are stored in the assets folder.
All the images used are in the PNG format.
Any references to websites have been formatted as [TEXT](URL)
You verified/tested the effectiveness of your contribution (e.g., the defensive code proposed is really an effective remediation? Please verify it works!).
The CI build of your PR pass, see the build status here.

If your PR is related to an issue, please finish your PR text with the following line:

This PR fixes issue #1781.

AI Tool Usage Disclosure (required for all PRs)

Please select one of the following options:

I have NOT used any AI tool to generate the contents of this PR.
I have used AI tools to generate the contents of this PR. I have verified
the contents and I affirm the results. The LLM used is Gemini 3 Flash
and the prompt used is Help me draft a new OWASP cheat sheet for Model Ops and Supply Chain Security, focusing on Pickle risks, Safetensors, and model scanning.

Thank you again for your contribution 😃

Initial draft covering unsafe deserialization risks, model scanning (modelscan), and migration to safetensors. Fixes OWASP#1781.

mackowski

Looks like a good start IMO

jmanico

I would rename this "Machine Learning Model Supply Chain Security Cheat Sheet"
The real issue isn't just pip install model, it's that model loading often triggers dynamic code execution through custom layers, preprocessing hooks, or reduce methods in Pickle. HuggingFace from_pretrained() calls can execute arbitrary code from config.json custom class references. That's worth calling out explicitly.
On prompt injection in tool models, I'd actually recommend keeping this out of scope or mentioning it only briefly with a cross-reference to a prompt injection cheat sheet.
add clean-label poisoning, where the attacker manipulates training examples that are correctly labeled but positioned to shift decision boundaries.
One significant gap: model format conversion attacks. When organizations convert between formats (PyTorch → ONNX → TensorRT, for example), the conversion pipeline itself can introduce vulnerabilities. ONNX custom operators can embed arbitrary code. TensorFlow SavedModel's tf.py_function wraps arbitrary Python. This deserves its own subsection.
Another: SBOM for models. The cheat sheet should reference the concept of an ML Bill of Materials, tracking not just the model artifact but the training data lineage, framework versions, and dependency graph. This aligns with NIST SP 800-218 (SSDF) and the broader software supply chain security direction OWASP is moving toward.
Also missing: weight-level integrity verification. Hash-pinning the whole file is necessary but not sufficient. Techniques exist to modify individual weight tensors while preserving file-level checksums if the hash isn't computed correctly. The cheat sheet should specify hashing the serialized artifact at rest and verifying before every load, not just at download time.

Tryingtobeabetterprogrammer · 2026-03-16T13:33:13Z

Hi @jmanico, thank you so much for this incredibly detailed and high-level feedback!
This is a masterclass in AI security. I'm especially excited to add the sections on Model Format Conversion Attacks and the ML-BOM (aligning with NIST SP 800-218). I’ll also update the title as requested and move Prompt Injection to 'Out of Scope' with a cross-reference to keep the focus tight.

…Learning_Model_Supply_Chain_Security_Cheat_Sheet.mdMachine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md

… to Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md

Added sections on Model Format Conversion Attacks, Clean-Label Model Poisoning, Weight-Level Integrity Verification, and Model Bill of Materials to enhance guidance on Model Supply Chain Security.

jmanico · 2026-03-17T04:47:58Z

Nice work on the updates! The rename, format conversion attacks section, ML-BOM, and clean-label poisoning additions all look good.

A few things to address before this is ready to merge:

The HuggingFace from_pretrained() risk still needs to be called out. Most developers assume it is safe, but config.json can reference custom classes that execute arbitrary code on load. This is probably the most common real-world attack vector and worth a dedicated example alongside the existing torch.load one.
The Safetensors section should note that while it prevents code execution during deserialization, it does not solve the entire trust problem. A malicious actor can still distribute a Safetensors file alongside a config.json that references custom code. Worth a brief caveat.
It would be great to add a short code example showing how to actually run modelscan or fickling on a model file. Cheat sheets work best when developers can copy and paste something actionable.
A references section at the end would bring this in line with other cheat sheets in the series. Links to NIST SP 800-218, CycloneDX, SPDX, the HuggingFace security docs, modelscan, and the OWASP Top 10 for LLM Applications would all be useful here.
The CI build checkbox is unchecked. Please make sure the build passes before the next review round.

You are making great progress on this. Looking forward to the next revision!

Added sections on Model Format Conversion Attacks, Clean-Label Model Poisoning, Weight-Level Integrity Verification, Model Bill of Materials (ML-BOM), HuggingFace `from_pretrained()` RCE Risk, Security Note on Safetensors, and Security Scanning Tools.

Tryingtobeabetterprogrammer · 2026-03-18T04:26:23Z

Hi @jmanico, I've addressed all the feedback from the previous review:

HuggingFace Risk: Added a section on from_pretrained() RCE risks with a Python code example.
Safetensors Caveat: Included a note about the config.json trust issue even when using safe weight formats.
Actionable Tools: Added modelscan and fickling CLI examples.
References: Added a References section with links to NIST, HuggingFace, CycloneDX, and OWASP documentation.
CI/CD: Fixed Markdown linting/spacing to ensure the build passes.

The PR also includes the This PR fixes issue #1781 line. Ready for your final review!

jmanico · 2026-03-18T04:43:41Z

Can I ask for a few more things?

Broken link on line 60: [OWASP Top 10 for LLM Applications] is missing its URL
Remove the emoji from the ⚠️ Security Note header
Add torch.load(weights_only=True) as a mitigation. This is the most important PyTorch security change (default since 2.6) and the cheat sheet currently skips from "unsafe pickle" to "safetensors" with nothing in between
Add MITRE ATLAS and OWASP AISVS references. These are the two most relevant frameworks for this topic
Update NIST reference from SP 800-218 to SP 800-218A (the generative AI supplement)
Add a clear defensive statement: "Never set trust_remote_code=True in production"
Mention Hugging Face's built-in malware scanning as a registry-level control
Check fickling maintenance status; consider noting it's less actively maintained than ModelScan

Updated the cheat sheet to include links and emphasize security notes regarding model loading and trust issues.

Tryingtobeabetterprogrammer · 2026-03-18T05:27:39Z

Hi @jmanico! I've completed the final round of polishing based on your suggestions:

Links Fixed: Added the URL for the OWASP Top 10 for LLM Applications on Line 55.
PyTorch Updates: Added the weights_only=True mitigation and noted the new default behavior in version 2.6.
Hugging Face Security: Included the defensive statement regarding trust_remote_code in production and mentioned registry-level malware scanning.
References: Updated NIST to SP 800-218A and added MITRE ATLAS and OWASP AISVS.
Tooling: Added the maintenance status note for Fickling.
Cleanup: Removed the emoji from the header as requested.

All checks are passing. Ready for final review and merge!

Copilot

Pull request overview

Adds a new cheat sheet to the OWASP Cheat Sheet Series focused on ML model artifact supply-chain risks (notably unsafe deserialization), and practical mitigations like safetensors migration, pre-ingestion scanning, and integrity/provenance controls.

Changes:

Introduces a new “Machine Learning Model Supply Chain Security” cheat sheet covering pickle/RCE risks, conversion attack surface, and ML-BOM concepts.
Adds example snippets for unsafe vs. safer model loading and registry-level controls.
Documents suggested tooling for static model scanning (ModelScan / Fickling).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-26T09:36:18Z

+
+### Model Bill of Materials (ML-BOM)
+
+Aligning with **NIST SP 800-218 (SSDF)**, an ML-BOM provides a verifiable record of the model's supply chain.


The document cites NIST SP 800-218 (SSDF) in the ML-BOM section, but the References list links to NIST SP 800-218A. Please make the reference entry match the cited publication (or adjust the earlier citation) so readers can find the correct source.

Suggested change

Aligning with **NIST SP 800-218 (SSDF)**, an ML-BOM provides a verifiable record of the model's supply chain.

Aligning with **NIST SP 800-218A (SSDF)**, an ML-BOM provides a verifiable record of the model's supply chain.

Copilot · 2026-03-26T09:36:19Z

+# Machine Learning Model Supply Chain Security Cheat Sheet
+
+## Introduction
+
+Machine Learning (ML) models are frequently treated as static data, but in many common formats (like Python's Pickle), they are actually executable code. This "Model-as-Code" reality introduces significant supply chain risks, where malicious actors can embed "Pickle Bombs" or backdoors into pre-trained models.


PR metadata says this PR fixes #1781 (an update to the existing Secure AI/ML Model Ops cheat sheet), but this change only adds a new supply-chain-focused cheat sheet and does not update cheatsheets/Secure_AI_Model_Ops_Cheat_Sheet.md. If the intent is to close #1781, please either update the existing Model Ops cheat sheet (or clarify in the PR description why adding this new file resolves the issue).

Copilot · 2026-03-26T09:36:19Z

+
+## Scope and Specific Controls
+
+### Out of Scope: Prompt Injection


Per the contributing guide, headings should have a blank line after them. Add a blank line after this ### Out of Scope: Prompt Injection heading.

Suggested change

### Out of Scope: Prompt Injection

### Out of Scope: Prompt Injection

Copilot · 2026-03-26T09:36:19Z

+
+In clean-label attacks, malicious samples are correctly labeled to evade human audit while strategically shifting the model's decision boundaries. This creates "triggers" where the model performs normally on most data but fails or misclassifies specific inputs chosen by the attacker.
+
+### Weight-Level Integrity Verification


Per the contributing guide, headings should have a blank line after them. Add a blank line after ### Weight-Level Integrity Verification before the paragraph/list that follows.

Suggested change

### Weight-Level Integrity Verification

### Weight-Level Integrity Verification

Copilot · 2026-03-26T09:36:20Z

+* [NIST SP 800-218A: AI-Specific Secure Software Development](https://doi.org/10.6028/NIST.SP.800-218A)
+* [MITRE ATLAS Framework](https://atlas.mitre.org/)
+* [OWASP AI Security Verification Standard (AISVS)](https://owasp.org/www-project-ai-security-verification-standard/)
+* [Hugging Face Security Documentation](https://huggingface.co/docs/hub/security)
+* [CycloneDX ML-BOM Specification](https://cyclonedx.org/capabilities/mlbom/)
+* [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/)
+* [ModelScan GitHub Repository](https://github.com/protectai/modelscan)


The References list uses * bullets, but markdownlint config enforces MD004 style "dash". Switch these bullets to - to avoid lint failures.

Suggested change

* [NIST SP 800-218A: AI-Specific Secure Software Development](https://doi.org/10.6028/NIST.SP.800-218A)

* [MITRE ATLAS Framework](https://atlas.mitre.org/)

* [OWASP AI Security Verification Standard (AISVS)](https://owasp.org/www-project-ai-security-verification-standard/)

* [Hugging Face Security Documentation](https://huggingface.co/docs/hub/security)

* [CycloneDX ML-BOM Specification](https://cyclonedx.org/capabilities/mlbom/)

* [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/)

* [ModelScan GitHub Repository](https://github.com/protectai/modelscan)

- [NIST SP 800-218A: AI-Specific Secure Software Development](https://doi.org/10.6028/NIST.SP.800-218A)

- [MITRE ATLAS Framework](https://atlas.mitre.org/)

- [OWASP AI Security Verification Standard (AISVS)](https://owasp.org/www-project-ai-security-verification-standard/)

- [Hugging Face Security Documentation](https://huggingface.co/docs/hub/security)

- [CycloneDX ML-BOM Specification](https://cyclonedx.org/capabilities/mlbom/)

- [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/)

- [ModelScan GitHub Repository](https://github.com/protectai/modelscan)

Copilot · 2026-03-26T09:36:20Z

+
+While `safetensors` prevents code execution during weight loading, it does **not** solve the trust problem. A malicious repository can still bundle a safe `.safetensors` weight file with a malicious `config.json` that triggers code execution via the `trust_remote_code` flag. Always audit the repository files beyond just the weights.
+
+## Security Scanning Tools


There should be a blank line after headings per the contributing guide. Add a blank line between ## Security Scanning Tools and the following ### 1. ... heading.

Suggested change

## Security Scanning Tools

## Security Scanning Tools

Copilot · 2026-03-26T09:36:20Z

+- **Digital Signatures:** Ensure the ML-BOM itself is cryptographically signed and linked to the model hash to prevent tampering.
+- **Guidance:** Integrate ML-BOM generation into the CI/CD pipeline using standardized formats like CycloneDX or SPDX.
+
+### HuggingFace `from_pretrained()` RCE Risk


The heading uses "HuggingFace" but elsewhere in this cheat sheet you use "Hugging Face" (and that's the standard spelling). Use a consistent name to avoid confusion and improve searchability.

Suggested change

### HuggingFace `from_pretrained()` RCE Risk

### Hugging Face `from_pretrained()` RCE Risk

mackowski · 2026-03-26T10:11:59Z

@Tryingtobeabetterprogrammer please address markdown linter failures and Copilot review

Added sections on model supply chain security, including prompt injection, model format conversion attacks, clean-label model poisoning, weight-level integrity verification, ML-BOM, and HuggingFace RCE risk mitigation.

mackowski · 2026-04-27T08:24:10Z

Linter is still failing

Create Model Ops and Supply Chain Security Cheat Sheet

148a78e

Initial draft covering unsafe deserialization risks, model scanning (modelscan), and migration to safetensors. Fixes OWASP#1781.

andrzejsydor approved these changes Mar 15, 2026

View reviewed changes

mackowski previously approved these changes Mar 15, 2026

View reviewed changes

mackowski requested review from jmanico and szh March 15, 2026 17:57

jmanico reviewed Mar 15, 2026

View reviewed changes

Rename Model_Ops_and_Supply_Chain_Security_Cheat_Sheet.md to Machine_…

a811cf0

…Learning_Model_Supply_Chain_Security_Cheat_Sheet.mdMachine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md

Tryingtobeabetterprogrammer dismissed mackowski’s stale review via a811cf0 March 16, 2026 13:36

Tryingtobeabetterprogrammer added 4 commits March 16, 2026 19:14

Rename cheat sheet to clarify focus on ML models

c2b8333

Merge branch 'OWASP:master' into feat-model-ops-cheat-sheet

74c8c81

Replacing Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md…

c044c97

… to Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md

Expand Model Supply Chain Security Cheat Sheet

98489e1

Added sections on Model Format Conversion Attacks, Clean-Label Model Poisoning, Weight-Level Integrity Verification, and Model Bill of Materials to enhance guidance on Model Supply Chain Security.

Enhance Machine Learning Model Supply Chain Security Cheat Sheet

062d506

Updated the cheat sheet to include links and emphasize security notes regarding model loading and trust issues.

mackowski requested a review from Copilot March 26, 2026 09:32

Copilot started reviewing on behalf of mackowski March 26, 2026 09:33 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

Nawte mentioned this pull request Apr 19, 2026

Update: Secure AI/ML Model Ops Cheat Sheet #1781

Open

Enhance Machine Learning Model Supply Chain Security Cheat Sheet

c5fde15

Added sections on model supply chain security, including prompt injection, model format conversion attacks, clean-label model poisoning, weight-level integrity verification, ML-BOM, and HuggingFace RCE risk mitigation.


		### Model Bill of Materials (ML-BOM)

		Aligning with NIST SP 800-218 (SSDF), an ML-BOM provides a verifiable record of the model's supply chain.

	Aligning with NIST SP 800-218 (SSDF), an ML-BOM provides a verifiable record of the model's supply chain.
	Aligning with NIST SP 800-218A (SSDF), an ML-BOM provides a verifiable record of the model's supply chain.


		## Scope and Specific Controls

		### Out of Scope: Prompt Injection


		In clean-label attacks, malicious samples are correctly labeled to evade human audit while strategically shifting the model's decision boundaries. This creates "triggers" where the model performs normally on most data but fails or misclassifies specific inputs chosen by the attacker.

		### Weight-Level Integrity Verification


		While `safetensors` prevents code execution during weight loading, it does not solve the trust problem. A malicious repository can still bundle a safe `.safetensors` weight file with a malicious `config.json` that triggers code execution via the `trust_remote_code` flag. Always audit the repository files beyond just the weights.

		## Security Scanning Tools

	### HuggingFace `from_pretrained()` RCE Risk
	### Hugging Face `from_pretrained()` RCE Risk

Conversation

Tryingtobeabetterprogrammer commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

You're A Rockstar

AI Tool Usage Disclosure (required for all PRs)

Uh oh!

mackowski left a comment

Choose a reason for hiding this comment

Uh oh!

jmanico left a comment

Choose a reason for hiding this comment

Uh oh!

Tryingtobeabetterprogrammer commented Mar 16, 2026

Uh oh!

jmanico commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tryingtobeabetterprogrammer commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmanico commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tryingtobeabetterprogrammer commented Mar 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

mackowski commented Mar 26, 2026

Uh oh!

mackowski commented Apr 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Tryingtobeabetterprogrammer commented Mar 9, 2026 •

edited

Loading

jmanico commented Mar 17, 2026 •

edited

Loading

Tryingtobeabetterprogrammer commented Mar 18, 2026 •

edited

Loading

jmanico commented Mar 18, 2026 •

edited

Loading