From 148a78ebf6b773bfa598e0c669765e56b431faa9 Mon Sep 17 00:00:00 2001 From: Tryingtobeabetterprogrammer Date: Mon, 9 Mar 2026 13:15:52 +0530 Subject: [PATCH 1/8] Create Model Ops and Supply Chain Security Cheat Sheet Initial draft covering unsafe deserialization risks, model scanning (modelscan), and migration to safetensors. Fixes #1781. --- ...s_and_Supply_Chain_Security_Cheat_Sheet.md | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 templates/Model_Ops_and_Supply_Chain_Security_Cheat_Sheet.md diff --git a/templates/Model_Ops_and_Supply_Chain_Security_Cheat_Sheet.md b/templates/Model_Ops_and_Supply_Chain_Security_Cheat_Sheet.md new file mode 100644 index 0000000000..714438ca20 --- /dev/null +++ b/templates/Model_Ops_and_Supply_Chain_Security_Cheat_Sheet.md @@ -0,0 +1,49 @@ +# Model Ops and Supply Chain Security Cheat Sheet + +## Introduction + +Machine Learning (ML) models are frequently treated as static data, but in many common formats (like Python's Pickle), they are actually executable code. This "Model-as-Code" reality introduces significant supply chain risks, where malicious actors can embed "Pickle Bombs" or backdoors into pre-trained models. + +## Primary Risks + +### Unsafe Deserialization + +Loading a model using standard Python libraries (like `torch.load` or `pickle.load`) can execute arbitrary code hidden within the model file. A hacker can trigger a reverse shell or data exfiltration the moment a developer "loads" a downloaded model. + +### Model Poisoning and Backdoors + +Attackers can subtly alter model weights so that the model performs normally on most data but triggers a specific, malicious behavior when it sees a "trigger" input. + +## Mitigation Strategies + +### 1. Mandate Safe Serialization (Safetensors) + +Whenever possible, transition from `.pkl` or `.pth` (Pickle-based) formats to the **Safetensors** format. + +- **Why:** Safetensors is a "data-only" format. It contains no executable instructions, making it physically impossible to hide a script inside. + +### 2. Pre-Ingestion Scanning + +Treat every third-party model as "Untrusted Code." + +- **Tooling:** Use specialized scanners like `modelscan` or `fickling` to inspect the internal instruction stack (opcodes) of a model for malicious triggers. +- **Environment:** Always perform scanning and initial testing in a network-isolated sandbox. + +### 3. Provenance and Integrity + +- **Hash Pinning:** Store and verify the SHA-256 hash of every model used in production. +- **Signed Registries:** Only pull models from registries that support cryptographic signing and identity verification. + +## Code Examples + +### Unsafe vs. Safe Loading + +```python +# UNSAFE: Risk of arbitrary code execution +import torch +model = torch.load('malicious_model.pkl') + +# SAFE: Only loads numeric tensors +from safetensors.torch import load_file +weights = load_file('safe_model.safetensors') +``` From a811cf0984d15ee8fa324211a01b010f0c1a91af Mon Sep 17 00:00:00 2001 From: Tryingtobeabetterprogrammer Date: Mon, 16 Mar 2026 19:06:30 +0530 Subject: [PATCH 2/8] Rename Model_Ops_and_Supply_Chain_Security_Cheat_Sheet.md to Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.mdMachine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md --- ...> Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename templates/{Model_Ops_and_Supply_Chain_Security_Cheat_Sheet.md => Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md} (100%) diff --git a/templates/Model_Ops_and_Supply_Chain_Security_Cheat_Sheet.md b/templates/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md similarity index 100% rename from templates/Model_Ops_and_Supply_Chain_Security_Cheat_Sheet.md rename to templates/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md From c2b833378d4de9887bb03ad45ac587f404c6b24f Mon Sep 17 00:00:00 2001 From: Tryingtobeabetterprogrammer Date: Mon, 16 Mar 2026 19:14:10 +0530 Subject: [PATCH 3/8] Rename cheat sheet to clarify focus on ML models --- .../Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/templates/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md b/templates/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md index 714438ca20..8c45314508 100644 --- a/templates/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md +++ b/templates/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md @@ -1,4 +1,4 @@ -# Model Ops and Supply Chain Security Cheat Sheet +# Machine Learning Model Supply Chain Security Cheat Sheet ## Introduction From c044c97ab5fa6c70323d630feb891695ca78ab60 Mon Sep 17 00:00:00 2001 From: Tryingtobeabetterprogrammer Date: Mon, 16 Mar 2026 19:43:10 +0530 Subject: [PATCH 4/8] Replacing Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md to Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md --- .../Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {templates => cheatsheets}/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md (100%) diff --git a/templates/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md b/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md similarity index 100% rename from templates/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md rename to cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md From 98489e1f27a803fc9b44beb6f292e0e4ff230aa7 Mon Sep 17 00:00:00 2001 From: Tryingtobeabetterprogrammer Date: Mon, 16 Mar 2026 21:57:05 +0530 Subject: [PATCH 5/8] Expand Model Supply Chain Security Cheat Sheet Added sections on Model Format Conversion Attacks, Clean-Label Model Poisoning, Weight-Level Integrity Verification, and Model Bill of Materials to enhance guidance on Model Supply Chain Security. --- ...Model_Supply_Chain_Security_Cheat_Sheet.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md b/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md index 8c45314508..dc2d30ad45 100644 --- a/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md +++ b/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md @@ -47,3 +47,28 @@ model = torch.load('malicious_model.pkl') from safetensors.torch import load_file weights = load_file('safe_model.safetensors') ``` + + +## Scope and Specific Controls + +### Out of Scope: Prompt Injection +This cheat sheet focuses on **Model Supply Chain Security** (the integrity of the model artifact itself). **Prompt Injection**, jailbreaking, and direct LLM output manipulation are out of scope. For guidance on those topics, refer to the [OWASP Top 10 for LLM Applications]. + +### Model Format Conversion Attacks +The process of converting models between frameworks (e.g., PyTorch → ONNX → TensorRT) creates an attack surface. +- **Custom Operator Injection:** Formats like ONNX support custom operators that can be weaponized to execute arbitrary code during model initialization. +- **Guidance:** Use sandboxed environments for conversion and perform security scans on the model both **before** and **after** the format shift. + +### Clean-Label Model Poisoning +In clean-label attacks, malicious samples are correctly labeled to evade human audit while strategically shifting the model's decision boundaries. This creates "triggers" where the model performs normally on most data but fails or misclassifies specific inputs chosen by the attacker. + +### Weight-Level Integrity Verification +Standard file-level hashing at the time of download is a "point-in-time" check and is insufficient for long-term security. +- **Load-Time Verification:** Hashes must be verified every time the model is loaded from disk into memory to protect against "at-rest" tampering. +- **Tensor-Level Hashing:** For high-security models, implement integrity checks on individual serialized weight tensors. + +### Model Bill of Materials (ML-BOM) +Aligning with **NIST SP 800-218 (SSDF)**, an ML-BOM provides a verifiable record of the model's supply chain. +- **Lineage Tracking:** Document the base model, fine-tuning datasets, and framework versions. +- **Digital Signatures:** Ensure the ML-BOM itself is cryptographically signed and linked to the model hash to prevent tampering. +- **Guidance:** Integrate ML-BOM generation into the CI/CD pipeline using standardized formats like CycloneDX or SPDX. From c54a255eb92e6f00571a11e81a9c6116c162b6a7 Mon Sep 17 00:00:00 2001 From: Tryingtobeabetterprogrammer Date: Tue, 17 Mar 2026 15:11:22 +0530 Subject: [PATCH 6/8] Expand Machine Learning Model Supply Chain Security Cheat Sheet Added sections on Model Format Conversion Attacks, Clean-Label Model Poisoning, Weight-Level Integrity Verification, Model Bill of Materials (ML-BOM), HuggingFace `from_pretrained()` RCE Risk, Security Note on Safetensors, and Security Scanning Tools. --- ...Model_Supply_Chain_Security_Cheat_Sheet.md | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md b/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md index dc2d30ad45..d4441bff5b 100644 --- a/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md +++ b/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md @@ -55,11 +55,13 @@ weights = load_file('safe_model.safetensors') This cheat sheet focuses on **Model Supply Chain Security** (the integrity of the model artifact itself). **Prompt Injection**, jailbreaking, and direct LLM output manipulation are out of scope. For guidance on those topics, refer to the [OWASP Top 10 for LLM Applications]. ### Model Format Conversion Attacks + The process of converting models between frameworks (e.g., PyTorch → ONNX → TensorRT) creates an attack surface. - **Custom Operator Injection:** Formats like ONNX support custom operators that can be weaponized to execute arbitrary code during model initialization. - **Guidance:** Use sandboxed environments for conversion and perform security scans on the model both **before** and **after** the format shift. ### Clean-Label Model Poisoning + In clean-label attacks, malicious samples are correctly labeled to evade human audit while strategically shifting the model's decision boundaries. This creates "triggers" where the model performs normally on most data but fails or misclassifies specific inputs chosen by the attacker. ### Weight-Level Integrity Verification @@ -68,7 +70,49 @@ Standard file-level hashing at the time of download is a "point-in-time" check a - **Tensor-Level Hashing:** For high-security models, implement integrity checks on individual serialized weight tensors. ### Model Bill of Materials (ML-BOM) + Aligning with **NIST SP 800-218 (SSDF)**, an ML-BOM provides a verifiable record of the model's supply chain. - **Lineage Tracking:** Document the base model, fine-tuning datasets, and framework versions. - **Digital Signatures:** Ensure the ML-BOM itself is cryptographically signed and linked to the model hash to prevent tampering. - **Guidance:** Integrate ML-BOM generation into the CI/CD pipeline using standardized formats like CycloneDX or SPDX. + +### HuggingFace `from_pretrained()` RCE Risk + +Many developers assume the `transformers` library is inherently safe. However, the `from_pretrained()` method reads a `config.json` file that can reference custom model classes. +- **The Attack:** If `trust_remote_code=True` is set, the library will download and execute arbitrary Python code (e.g., a `modeling_*.py` file) from the repository immediately upon loading. +- **Example:** +```python +# DANGEROUS: Executes arbitrary code from the remote repo +from transformers import AutoModel +model = AutoModel.from_pretrained("malicious-user/repo-name", trust_remote_code=True) +``` + +### ⚠️ Security Note on Safetensors + +While `safetensors` prevents code execution during weight loading, it does **not** solve the trust problem. A malicious repository can still bundle a safe `.safetensors` weight file with a malicious `config.json` that triggers code execution via the `trust_remote_code` flag. Always audit the repository files beyond just the weights. + +## Security Scanning Tools +### 1. ModelScan (Protect AI) + +Scans models for unsafe "opcodes" without executing them. + +```bash +pip install modelscan +modelscan -p ./path_to_model/model.pkl +``` + +### 2. Fickling (Trail of Bits) + +A static analysis tool that decompiles Python pickles to identify malicious intent safely. + +```bash +pip install fickling +fickling my_model.pth +``` + +## References +* [NIST SP 800-218: Secure Software Development Framework (SSDF)](https://doi.org/10.6028/NIST.SP.800-218) +* [Hugging Face Security Documentation](https://huggingface.co/docs/hub/security) +* [CycloneDX ML-BOM Specification](https://cyclonedx.org/capabilities/mlbom/) +* [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) +* [ModelScan GitHub Repository](https://github.com/protectai/modelscan) From 062d5061c5754ef2b06e4a0e22a8eb3188927360 Mon Sep 17 00:00:00 2001 From: Tryingtobeabetterprogrammer Date: Wed, 18 Mar 2026 10:51:57 +0530 Subject: [PATCH 7/8] Enhance Machine Learning Model Supply Chain Security Cheat Sheet Updated the cheat sheet to include links and emphasize security notes regarding model loading and trust issues. --- ...Model_Supply_Chain_Security_Cheat_Sheet.md | 25 ++++++++++++++++--- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md b/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md index d4441bff5b..241a427016 100644 --- a/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md +++ b/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md @@ -48,11 +48,10 @@ from safetensors.torch import load_file weights = load_file('safe_model.safetensors') ``` - ## Scope and Specific Controls ### Out of Scope: Prompt Injection -This cheat sheet focuses on **Model Supply Chain Security** (the integrity of the model artifact itself). **Prompt Injection**, jailbreaking, and direct LLM output manipulation are out of scope. For guidance on those topics, refer to the [OWASP Top 10 for LLM Applications]. +This cheat sheet focuses on **Model Supply Chain Security** (the integrity of the model artifact itself). **Prompt Injection**, jailbreaking, and direct LLM output manipulation are out of scope. For guidance on those topics, refer to the [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/). ### Model Format Conversion Attacks @@ -81,13 +80,26 @@ Aligning with **NIST SP 800-218 (SSDF)**, an ML-BOM provides a verifiable record Many developers assume the `transformers` library is inherently safe. However, the `from_pretrained()` method reads a `config.json` file that can reference custom model classes. - **The Attack:** If `trust_remote_code=True` is set, the library will download and execute arbitrary Python code (e.g., a `modeling_*.py` file) from the repository immediately upon loading. - **Example:** + ```python # DANGEROUS: Executes arbitrary code from the remote repo from transformers import AutoModel model = AutoModel.from_pretrained("malicious-user/repo-name", trust_remote_code=True) ``` -### ⚠️ Security Note on Safetensors +**Defensive Statement:** Never set `trust_remote_code=True` in production environments. + +**Registry Controls:** Use Hugging Face's built-in malware scanning and "Pickle Scan" badges to verify model safety at the registry level before downloading. + +### Mitigation: Use `weights_only=True` +Starting with PyTorch 2.6, `torch.load()` defaults to `weights_only=True`. This restricts unpickling to a safe subset of Python objects, preventing arbitrary code execution while still using the `.pth` format. + +```python +# Safe loading in modern PyTorch +weights = torch.load("model.pth", weights_only=True) +``` + +### Security Note on Safetensors While `safetensors` prevents code execution during weight loading, it does **not** solve the trust problem. A malicious repository can still bundle a safe `.safetensors` weight file with a malicious `config.json` that triggers code execution via the `trust_remote_code` flag. Always audit the repository files beyond just the weights. @@ -104,6 +116,8 @@ modelscan -p ./path_to_model/model.pkl ### 2. Fickling (Trail of Bits) A static analysis tool that decompiles Python pickles to identify malicious intent safely. +Fickling is a specialized tool designed to analyze and "de-pickle" Python object streams to identify malicious opcodes. It can also be used to create "safe" versions of existing pickle files. +**Note:** Fickling is currently less actively maintained than ModelScan; prioritize ModelScan for up-to-date opcode coverage. ```bash pip install fickling @@ -111,7 +125,10 @@ fickling my_model.pth ``` ## References -* [NIST SP 800-218: Secure Software Development Framework (SSDF)](https://doi.org/10.6028/NIST.SP.800-218) + +* [NIST SP 800-218A: AI-Specific Secure Software Development](https://doi.org/10.6028/NIST.SP.800-218A) +* [MITRE ATLAS Framework](https://atlas.mitre.org/) +* [OWASP AI Security Verification Standard (AISVS)](https://owasp.org/www-project-ai-security-verification-standard/) * [Hugging Face Security Documentation](https://huggingface.co/docs/hub/security) * [CycloneDX ML-BOM Specification](https://cyclonedx.org/capabilities/mlbom/) * [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) From c5fde15871763322d9e4f90e920077c28db9a740 Mon Sep 17 00:00:00 2001 From: Tryingtobeabetterprogrammer Date: Mon, 20 Apr 2026 14:31:19 +0530 Subject: [PATCH 8/8] Enhance Machine Learning Model Supply Chain Security Cheat Sheet Added sections on model supply chain security, including prompt injection, model format conversion attacks, clean-label model poisoning, weight-level integrity verification, ML-BOM, and HuggingFace RCE risk mitigation. --- ...Model_Supply_Chain_Security_Cheat_Sheet.md | 23 +++++++++++++------ 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md b/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md index 241a427016..9758eacba5 100644 --- a/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md +++ b/cheatsheets/Machine_Learning_Model_Supply_Chain_Security_Cheat_Sheet.md @@ -51,12 +51,15 @@ weights = load_file('safe_model.safetensors') ## Scope and Specific Controls ### Out of Scope: Prompt Injection + This cheat sheet focuses on **Model Supply Chain Security** (the integrity of the model artifact itself). **Prompt Injection**, jailbreaking, and direct LLM output manipulation are out of scope. For guidance on those topics, refer to the [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/). ### Model Format Conversion Attacks The process of converting models between frameworks (e.g., PyTorch → ONNX → TensorRT) creates an attack surface. + - **Custom Operator Injection:** Formats like ONNX support custom operators that can be weaponized to execute arbitrary code during model initialization. + - **Guidance:** Use sandboxed environments for conversion and perform security scans on the model both **before** and **after** the format shift. ### Clean-Label Model Poisoning @@ -64,6 +67,7 @@ The process of converting models between frameworks (e.g., PyTorch → ONNX → In clean-label attacks, malicious samples are correctly labeled to evade human audit while strategically shifting the model's decision boundaries. This creates "triggers" where the model performs normally on most data but fails or misclassifies specific inputs chosen by the attacker. ### Weight-Level Integrity Verification + Standard file-level hashing at the time of download is a "point-in-time" check and is insufficient for long-term security. - **Load-Time Verification:** Hashes must be verified every time the model is loaded from disk into memory to protect against "at-rest" tampering. - **Tensor-Level Hashing:** For high-security models, implement integrity checks on individual serialized weight tensors. @@ -71,8 +75,11 @@ Standard file-level hashing at the time of download is a "point-in-time" check a ### Model Bill of Materials (ML-BOM) Aligning with **NIST SP 800-218 (SSDF)**, an ML-BOM provides a verifiable record of the model's supply chain. + - **Lineage Tracking:** Document the base model, fine-tuning datasets, and framework versions. + - **Digital Signatures:** Ensure the ML-BOM itself is cryptographically signed and linked to the model hash to prevent tampering. + - **Guidance:** Integrate ML-BOM generation into the CI/CD pipeline using standardized formats like CycloneDX or SPDX. ### HuggingFace `from_pretrained()` RCE Risk @@ -92,6 +99,7 @@ model = AutoModel.from_pretrained("malicious-user/repo-name", trust_remote_code= **Registry Controls:** Use Hugging Face's built-in malware scanning and "Pickle Scan" badges to verify model safety at the registry level before downloading. ### Mitigation: Use `weights_only=True` + Starting with PyTorch 2.6, `torch.load()` defaults to `weights_only=True`. This restricts unpickling to a safe subset of Python objects, preventing arbitrary code execution while still using the `.pth` format. ```python @@ -104,6 +112,7 @@ weights = torch.load("model.pth", weights_only=True) While `safetensors` prevents code execution during weight loading, it does **not** solve the trust problem. A malicious repository can still bundle a safe `.safetensors` weight file with a malicious `config.json` that triggers code execution via the `trust_remote_code` flag. Always audit the repository files beyond just the weights. ## Security Scanning Tools + ### 1. ModelScan (Protect AI) Scans models for unsafe "opcodes" without executing them. @@ -126,10 +135,10 @@ fickling my_model.pth ## References -* [NIST SP 800-218A: AI-Specific Secure Software Development](https://doi.org/10.6028/NIST.SP.800-218A) -* [MITRE ATLAS Framework](https://atlas.mitre.org/) -* [OWASP AI Security Verification Standard (AISVS)](https://owasp.org/www-project-ai-security-verification-standard/) -* [Hugging Face Security Documentation](https://huggingface.co/docs/hub/security) -* [CycloneDX ML-BOM Specification](https://cyclonedx.org/capabilities/mlbom/) -* [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) -* [ModelScan GitHub Repository](https://github.com/protectai/modelscan) +- [NIST SP 800-218A: AI-Specific Secure Software Development](https://doi.org/10.6028/NIST.SP.800-218A) +- [MITRE ATLAS Framework](https://atlas.mitre.org/) +- [OWASP AI Security Verification Standard (AISVS)](https://owasp.org/www-project-ai-security-verification-standard/) +- [Hugging Face Security Documentation](https://huggingface.co/docs/hub/security) +- [CycloneDX ML-BOM Specification](https://cyclonedx.org/capabilities/mlbom/) +- [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) +- [ModelScan GitHub Repository](https://github.com/protectai/modelscan)