Pretrained SAE Models

GPT-2 Small (Layer 3)

• Architecture: TopK SAE
• Weight Tying: Yes
• Hidden Size: 24,576
• Non-zero Fraction: 3%
• Training Dataset: 1.1M Activations
• Context Window: 1,024
• Eval Loss: 0.06

Download Model Download Config

GPT-2 Small (Layer 3)

• Architecture: TopK SAE
• Weight Tying: No
• Hidden Size: 24,576
• Non-zero Fraction: 3%
• Training Dataset: 1.1M Activations
• Context Window: 1,024
• Eval Loss: 0.06

Download Model Download Config

GPT-2 Small (Layer 3)

• Architecture: TopK SAE
• Weight Tying: No
• Hidden Size: 24,576
• Non-zero Fraction: 3%
• Training Dataset: 2M Activations
• Context Window: 1,024
• Eval Loss: 0.15

Download Model Download Config

GPT-2 Small (Layer 12)

• Architecture: TopK SAE
• Weight Tying: No
• Hidden Size: 24,576
• Non-zero Fraction: 3%
• Training Dataset: 1.1M Activations
• Context Window: 1,024
• Eval Loss: 0.14

Download Model Download Config

GPT-2 Medium (Last Layer)

• Architecture: TopK SAE
• Weight Tying: No
• Hidden Size: 32,768
• Non-zero Fraction: 2%
• Training Dataset: 1.1M Activations
• Context Window: 1,024
• Eval Loss: 0.06

Download Model Download Config