Adding New Knowledge to LLMs

• Curate high-quality datasets and generate synthetic data using NVIDIA NeMo Curator.
• Rigorously evaluate LLM performance with benchmarks (MMLU), LLM-as-a-judge, and the NeMo Evaluator.
• Inject new domain-specific knowledge into LLMs using Continued Pretraining (CPT).
• Teach LLMs new skills and align them to specific tasks with Supervised Fine-Tuning (SFT).
• Align model behavior to human preferences for style, tone, and safety using Direct Preference Optimization (DPO).
• Compress and optimize LLMs for efficient deployment using Quantization, Pruning, and Knowledge Distillation with TensorRT-LLM and NeMo.
• Apply end-to-end model customization workflows to solve real-world problems.
• Topics Covered
  

Module 1: Data Curation and Synthetic Data Generation

• Learn to prepare large-scale, high-quality datasets using NVIDIA NeMo Curator.
• Perform essential data curation tasks: text cleaning, filtering, and PII removal.
• Generate high-quality synthetic Question-Answer pairs to create robust datasets for Supervised Fine-Tuning (SFT).
• Understand the importance of data quality in the LLM development lifecycle.

Module 2: Evaluating Large Language Models

• Explore multiple LLM evaluation techniques, from simple "eyeballing" to systematic, quantitative methods.
• Evaluate models against industry-standard benchmarks like MMLU.
• Implement LLM-as-a-judge for nuanced, automated evaluation.
• Use the NeMo Evaluator microservice to compare zero-shot vs. few-shot (in-context learning) performance.
• Track and visualize evaluation experiments using MLflow.

Module 3: Customizing LLMs

• Dive into three key customization techniques: CPT, SFT, and DPO.
• Use Continued Pretraining (CPT) to teach a model new knowledge about a specific domain.
• Apply Supervised Fine-Tuning (SFT) to teach a model new skills, such as solving math problems in a different language.
• Utilize Direct Preference Optimization (DPO) to align a model's conversational style to human preferences (e.g., formal vs. informal, specific dialects).
• Gain hands-on experience with the NeMo framework for all customization tasks.

Module 4: Optimizing LLMs for Deployment

• Learn to compress and accelerate LLMs for efficient inference.
• Apply Post-Training Quantization (PTQ) to reduce model size and memory usage using TensorRT-LLM, focusing on the FP8 format.
• Use Depth Pruning to reduce model size by removing entire layers.
• Employ Knowledge Distillation to recover performance lost during pruning by training a smaller "student" model to mimic a larger "teacher" model.
• Evaluate the performance vs. accuracy trade-offs of each optimization technique.

Module 5: Interactive Assessment

• Apply your knowledge in a hands-on coding assessment.
• Use Direct Preference Optimization (DPO) to align a Llama 3.1 8B model to a unique conversational style (Shakespearean English).
• Demonstrate your ability to prepare a preference dataset, run an alignment job with NeMo-RL, and evaluate the final model.
• Earn a certificate of competency by successfully completing the assessment.

• Familiarity with Python programming and Jupyter notebooks.
• Basic understanding of Large Language Models and their applications.
• Conceptual knowledge of deep learning and neural networks.
• Tools, libraries, frameworks used: Python, NVIDIA NeMo, NVIDIA TensorRT-LLM, Docker, MLflow