Paper
M6: A Chinese Multimodal Pretrainer
In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 bill · Authors: Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang… · Published 2021-03-01 · arXiv:2103.00823
- Published
- 2021-03-01
- Authors
- 25
- arXiv
- 2103.00823
- Category
- cs.CL
Abstract
arXiv:2103.00823
Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, Jie Zhang, Jianwei Zhang…
In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 billion and 100 billion parameters, and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding performance in comparison with strong baselines. Furthermore, we specifically design a downstream task of text-guided image generation, and show that the finetuned M6 can create high-quality images with high resolution and abundant details.
Related papers
16
-
Hupan Laboratory
-
National University of Singapore
-
Zhejiang University
-
Fudan University
-
Beijing University of Posts and Telecommunications
-
Chinese Academy of Sciences
-
Beijing Jiaotong University
-
Chinese Academy of Sciences
-
University of Chinese Academy of Sciences
-
Tsinghua University