Instella MoE 16B A3B Pretrain
Instella-MoE-16B-A3B-Pretrain is the base pre-training checkpoint of AMD's Instella-MoE model, a fully open Mixture-of-Experts language model with 16 billion total parameters and 2.8 billion active parameters per token. Trained from scratch on AMD Instinct MI300X and MI325X GPUs, it uses Gated Multi-head Latent Attention and FarSkip-Collective communication for efficient large-scale training, serving as the starting checkpoint for later mid-training and fine-tuning stages.
- Type
- Text
- Weights
- Open
- Downloads
- 740
- Likes
- 6
- Pipeline
- text generation
- License
- other
Model hub
amd/Instella-MoE-16B-A3B-Pretrain
- Repo
- amd/Instella-MoE-16B-A3B-Pretrain
- Library
- transformers
- Access
- Public
Technical tags and languages
Papers
1
More from AMD
8
-
273k DL
DeepSeek-R1-MXFP4
License mit · 272,995 downloads · by amd
-
799 DL
Instella MoE 16B A3B Base
Text
Instella-MoE-16B-A3B-Base is the long-context checkpoint AMD designates as the final base model in its Instel…
-
652 DL
Instella MoE 16B A3B DPO
Text
Instella-MoE-16B-A3B-DPO is the direct preference optimization checkpoint of AMD's Instella-MoE model, a full…
-
638 DL
Instella MoE 16B A3B Midtrain
Text
Instella-MoE-16B-A3B-Midtrain is the mid-training checkpoint of AMD's Instella-MoE model, a fully open Mixtur…
-
1.9k DL
Instella MoE 16B A3B SFT
Text
Instella-MoE-16B-A3B-SFT is the supervised fine-tuning checkpoint of AMD's Instella-MoE model, a fully open M…
-
3.2k DL
Instella MoE 16B A3B Think
Text
Instella-MoE-16B-A3B-Think is the final reinforcement-learning tuned checkpoint of AMD's Instella-MoE model,…
-
9k DL
MiniMax-M3-MXFP4
Multimodal
image text to text · License other · 89,605 downloads · by amd
-
608 DL
Nitro-E
Image
Nitro-E is an efficiency-focused reasoning and coding model optimized for AMD hardware. It offers long contex…
About AMD
1
Semiconductor company designing CPUs, GPUs, adaptive processors, and AI accelerators for PCs, gaming, data centers, embedded systems, and high-performance computing.