ch05

Jan 24, 2025

2e79d7b · Jan 24, 2025

This branch is 7 commits ahead of, 97 commits behind rasbt/LLMs-from-scratch:main.

Name	Name	Last commit message	Last commit date
parent directory ..
01_main-chapter-code	01_main-chapter-code	sdf	Jan 24, 2025
02_alternative_weight_loading	02_alternative_weight_loading	fixed num_workers (rasbt#229 )	Jun 19, 2024
03_bonus_pretraining_on_gutenberg	03_bonus_pretraining_on_gutenberg	Update README.md	Aug 10, 2024
04_learning_rate_schedulers	04_learning_rate_schedulers	Add and link bonus material (rasbt#84 )	Mar 23, 2024
05_bonus_hparam_tuning	05_bonus_hparam_tuning	total training iters may equal to warmup_iters (rasbt#301 )	Aug 6, 2024
06_user_interface	06_user_interface	Add user interface to ch06 and ch07 (rasbt#366 )	Sep 22, 2024
07_gpt_to_llama	07_gpt_to_llama	[minor] typo & comments (rasbt#441 )	Nov 18, 2024
08_memory_efficient_weight_loading	08_memory_efficient_weight_loading	update mmap section	Oct 14, 2024
README.md	README.md	Memory efficient weight loading (rasbt#401 )	Oct 14, 2024

README.md

02_alternative_weight_loading contains code to load the GPT model weights from alternative places in case the model weights become unavailable from OpenAI
03_bonus_pretraining_on_gutenberg contains code to pretrain the LLM longer on the whole corpus of books from Project Gutenberg
04_learning_rate_schedulers contains code implementing a more sophisticated training function including learning rate schedulers and gradient clipping
05_bonus_hparam_tuning contains an optional hyperparameter tuning script
06_user_interface implements an interactive user interface to interact with the pretrained LLM
07_gpt_to_llama contains a step-by-step guide for converting a GPT architecture implementation to Llama 3.2 and loads pretrained weights from Meta AI
08_memory_efficient_weight_loading contains a bonus notebook showing how to load model weights via PyTorch's load_state_dict method more efficiently