« Back to Glossary Index

The .safetensors file format is a secure and efficient method for storing tensors, commonly used in machine learning applications. Developed by Hugging Face, it offers a safer alternative to traditional serialization methods like Python’s pickle, which can execute arbitrary code and pose security risks.

Key Features:

• Safety: Unlike pickle, .safetensors files do not allow for code execution, mitigating the risk of executing malicious code embedded in model weights.
• Efficiency: The format is designed for fast, zero-copy operations, enabling quick loading and saving of tensor data.
• Cross-Framework Compatibility: .safetensors supports multiple machine learning frameworks, including PyTorch, TensorFlow, and others, facilitating seamless integration across different platforms.

Applications/Use Cases:

• Model Storage: Safely storing and sharing machine learning models without the risk of executing malicious code.
• Data Serialization: Efficiently saving and loading large tensor datasets across various machine learning frameworks.
• Model Distribution: Distributing pre-trained models in a secure and framework-agnostic manner.

Installation:

To install the safetensors library, use pip:

pip install safetensors

Usage Example:

Here’s how to save and load tensors using the safetensors library in Python:

import torch
from safetensors import safe_open
from safetensors.torch import save_file

# Define tensors
tensors = {
    "weight1": torch.zeros((1024, 1024)),
    "weight2": torch.zeros((1024, 1024))
}

# Save tensors to a .safetensors file
save_file(tensors, "model.safetensors")

# Load tensors from the .safetensors file
loaded_tensors = {}
with safe_open("model.safetensors", framework="pt", device="cpu") as f:
    for key in f.keys():
        loaded_tensors[key] = f.get_tensor(key)

« Back to Glossary Index