Promote Sign in Sign up

Repository

CSC6203-LLM

CSC6203: Selected Topics in CS III (Large Language Models)

FreedomIntelligence
Repository

README

CSC6203: Selected Topics in CS III (Large Language Models)

Basic info of the course

Large language models (LLMs) have utterly transformed the field of natural language processing (NLP) in the last 3-4 years. They form the basis of state-of-art systems and become ubiquitous in solving a wide range of natural language understanding and generation tasks. With the unprecedented potential and capabilities, these models also give rise to new ethical and scalability challenges. This course aims to cover cutting-edge research topics centering around pre-trained language models. We will discuss their technical foundations (BERT, GPT, T5 models, mixture-of-expert models, retrieval-based models), emerging capabilities (knowledge, reasoning, few-shot learning, in-context learning), fine-tuning and adaptation, system design, as well as security and ethics. We will cover each topic and discuss important papers in depth. Students will be expected to routinely read and present research papers and complete a research project at the end.

This is an advanced graduate course and all the students are expected to have taken machine learning and NLP courses before and are familiar with deep learning models such as Transformers.

Learning goals:

This course is intended to prepare you for performing cutting-edge research in natural language processing, especially topics related to pre-trained language models. We will discuss the state-of-the-art, their capabilities and limitations.
Practice your research skills, including reading research papers, conducting literature survey, oral presentations, as well as providing constructive feedback.
Gain hands-on experience through the final project, from brainstorming ideas to implementation and empirical evaluation and writing the final paper.

Course Outline

Date Topics Recommended Reading Pre-Lecture Questions Lecture Note Events Deadlines Feedback Providers
Sep. 8th Introduction to Large Language Models (LLMs) Open AI's blog On the Opportunities and Risks of Foundation Models Sparks of Artificial General Intelligence: Early experiments with GPT-4 What is ChatGPT and how to use? [slide]() [note]() Assignment Xidong Wang
Sep. 8th Language models and beyond [A Neural Probabilistic Language Model](A Neural Probabilistic Language Model) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Training language models to follow instructions with human feedback What is language model and why is it important? [slide]() [note]() Assignment Xidong Wang
TBD Architecture engineering and scaling law: Transformer and beyond nanoGPT GitHub Attention Is All You Need HuggingFace's course on Transformers Scaling Laws for Neural Language Models The Transformer Family Version 2.0 On Position Embeddings in BERT Why does Transformer become the backbone of PLMs? [slide]() [note]() TBD TBD
TBD Training LLMs from scratch LLMZoo, LLMFactory Training language models to follow instructions with human feedback LLaMA: Open and Efficient Foundation Language Models Llama 2: Open Foundation and Fine-Tuned Chat Models How to train LLMs from scratch? [slide]() [note]() TBD TBD
TBD Efficiency in LLM llama2.c GitHub Efficient Transformers: A Survey FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity [Towards a Unified View of Parameter-Efficient Transfer Learning](Towards a Unified View of Parameter-Efficient Transfer Learning) how to make LLMs train/inference faster? [slide]() [note]() TBD TBD
TBD Prompt engineering Best practices for prompt engineering with OpenAI API prompt engineering What is in-context learning and how to better prompt LLMs? [slide]() [note]() TBD TBD
TBD Mid review of final project N/A N/A [slide]() [note]() TBD TBD
TBD Knowledge and Reasoning MMLU/ C-eval, knowledge injection, RAG/ mathmatical reasoning, Natural Language Reasoning, A Survey Can LLMs reason? [slide]() [note]() TBD TBD
TBD Multimodal LLMs CLIP, Learning Transferable Visual Models From Natural Language Supervision MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models Stable Diffusion Can LLMs see? [slide]() [note]() TBD TBD
TBD LLMs in vertical domains Large Language Models Encode Clinical Knowledge Capabilities of GPT-4 on Medical Challenge Problems Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models HuatuoGPT, Medical-NLP, ChatLaw Can LLMs be a mature doctors/lawyer? [slide]() [note]() TBD TBD
TBD Tools and Large language models ToolBench, AgentBench Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks LLM Powered Autonomous Agents How are LLMs used in planning? [slide]() [note]() TBD TBD
TBD Privacy, bias, fairness, Toxicity and Holistic Evaluation On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Survey of Hallucination in Natural Language Generation Extracting Training Data from Large Language Models In which aspects should we evaluate LLMs? [slide]() [note]() TBD TBD
TBD Alignment, Limitations, and broader Impact Superalignment GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks Theory of Mind Might Have Spontaneously Emerged in Large Language Models What are LLMs' limitations? [slide]() [note]() TBD TBD
TBD Guest lecture N/A TBD [slide]() [note]() TBD TBD
TBD In-class presentation (extended class) N/A how to solve real-world problems using LLMs [slide]() [note]() TBD TBD

Teaching Assistant

  • Xidong Wang (Leading TA)
  • Fei Yu
  • Zhengyang Tang
  • Junying Chen
  • Juhao Liang
  • Zhuoheng Ma

Course Project

All projects will be archived and open-sourced. Please specify reasons if you decide not to do so.

Default Projects

Customized Projects

[Call for Sponsorship]()

Awards

  • Best Research Award
  • Best Presentation Award
  • Best Poster Award
  • Promising Research Award
  • TA favorites

Anyone who gets one of the above awards would likely get expected recomendations in any occasion.

Problems unsolved

Bonus

  • Anyone who releases a public LLM-related paper with permission from his/her supervisors could get 5 bonus marks (total marks should not exceed 100).

References

COS 597G (Fall 2022): Understanding Large Language Models by Danqi
CS25: Transformers United V2
CS324 - Large Language Models