Sparse Autoencoder
https://github.com/openai/sparse_autoencoder- Category
- AI Tools
- Rank
- No. 1573Tools index
Previous survey · No. 1582 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- openai
- GitHub
- 603 stars
- Date
About
OpenAI's reference implementation of sparse autoencoders for interpretability research on large language models.
What it does
It converts GPT2-small activation vectors into sparse latent features, then reconstructs the original vectors. The package includes pretrained weights, several activation-location families, training machinery, and a browser viewer for inspecting feature behavior across token sequences.
Why it's ranked here
This is a useful research artifact because it joins runnable model code, public pretrained weights, distributed training techniques, and a feature viewer. Its value is strongest as a concrete implementation to study or compare against. The deliberately bare-bones training workflow limits its use as a complete experimentation platform.
What's good
The model supports standard rectified activations, fixed-count top-feature selection, tied or separate encoder and decoder weights, and optional input normalization. Published weights cover multiple GPT2-small layers and activation locations. The training code demonstrates operator sharding, data parallelism, sparse kernels, and dead-feature tracking. The viewer exposes feature examples and activation statistics.
Tradeoffs
Training omits checkpointing, data loading, and validation. The authors explicitly say the code is not designed for easy extension and may require profiling and modification for efficient use. Training imports Triton although the declared package dependencies list only Blobfile, PyTorch, and TransformerLens. Published model metadata also leaves training-token counts unfinished for newer weight families.
How to use it well
Use it when you already work with PyTorch and GPT2 activations, and need pretrained sparse features or a concrete baseline for implementing similar training elsewhere. Start by extracting cached activations, loading a matching public weight family, encoding and reconstructing them, then inspect features in the viewer. Do not treat it as a managed training pipeline, dataset system, or validation framework.
Technical notes+
sparse_autoencoder/model.py defines Autoencoder, TopK, tied decoder weights, normalization, serialization, and dead-latent buffers. sparse_autoencoder/loss.py combines normalized reconstruction error with normalized L1 pressure. sparse_autoencoder/paths.py maps supported GPT2-small layers and activation locations to public Azure Blob weights. sparse_autoencoder/train.py implements NCCL communication, operator sharding, data parallelism, global top-k selection, and a sparse FastAutoencoder; its header documents missing production features. sparse_autoencoder/kernels.py supplies custom Triton sparse matrix operations and autograd support. sae-viewer/src/App.tsx and sae-viewer/src/feed.tsx implement a React hash-routed feature browser, while sae-viewer/src/utils.ts caches fetched values in local storage.
Observed
- Primary implementation
- Python package built around PyTorch, with custom Triton training kernels.
- Viewer
- React and TypeScript browser interface using hash-based routes.
- Installation
- Install directly from the Git repository with pip.
- Packaging
- Setuptools build backend with the sparse_autoencoder package included.
- Declared dependencies
- Blobfile 2.0.2, PyTorch 2.1.0, and TransformerLens 1.9.1.
- Public artifacts
- Pretrained GPT2-small autoencoder weights are addressed through Azure Blob paths.
Read from README.md, pyproject.toml, sparse_autoencoder/loss.py, sparse_autoencoder/model.py, sparse_autoencoder/paths.py, sparse_autoencoder/train.py, sae-viewer/tailwind.config.js, sparse_autoencoder/kernels.py, sparse_autoencoder/__init__.py, sparse_autoencoder/explanations.py, sae-viewer/src/App.tsx, sae-viewer/src/feed.tsx, sae-viewer/src/types.ts, sae-viewer/src/utils.ts, sae-viewer/src/index.tsx.
What it can do
Train sparse autoencoders on language model activations
Neural network activation data from large language models → Trained sparse autoencoder models
Extract interpretable features from language model representations
Language model hidden states or embeddings → Sparse feature representations
Reconstruct neural network activations from sparse codes
Sparse feature codes → Reconstructed activation vectors
Analyze feature sparsity patterns in language models
Text inputs and corresponding model activations → Sparsity metrics and activation statistics
Visualize learned sparse features
Trained autoencoder weights and activations → Feature visualization plots and analysis
Evaluate reconstruction quality of sparse representations
Original and reconstructed activation vectors → Reconstruction loss metrics and quality scores
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.