🤩Thank you @SylphAI-Inc | Source

LLM-engineer-handbook

🔥 Large Language Models(LLM) have taken the NLP community AI community the Whole World by storm. The LLM space is complicated! This repo provides a curated list to help you navigate; it includes a collection of Large Language Model frameworks and tutorials, covering model training, serving, fine-tuning, and building LLM applications.

Table of Content

• Applications
• Pretraining
• Fine-tuning
• Serving
• Datasets
• Benchmarks
• Learn LLM Applications
• Understand LLM
• Social Accounts

Applications

Build & Auto-optimize

• AdalFlow - The library to build & auto-optimize LLM applications, from Chatbot, RAG, to Agent. It is AI-first with PyTorch-like design patterns.

• dspy - DSPy: The framework for programming—not prompting—foundation models.

Build

• LlamaIndex — A Python library for augmenting LLM apps with data.
• LangChain — A popular Python/JavaScript library for chaining sequences of language model prompts.
• Haystack - Python framework that allows you to build applications powered by LLMs

Prompt Optimization

• AutoPrompt - A framework for prompt tuning using Intent-based Prompt Calibration
• PromptFify - A library for prompt engineering that simplifies NLP tasks (e.g., NER, classification) using LLMs like GPT.

Others

• LiteLLM - Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format.

Pretraining

• PyTorch - PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing.
• TensorFlow - TensorFlow is an open source machine learning library developed by Google.
• JAX - Google’s library for high-performance computing and automatic differentiation.
• tinygrad - A minimalistic deep learning library with a focus on simplicity and educational use, created by George Hotz.
• micrograd - A simple, lightweight autograd engine for educational purposes, created by Andrej Karpathy.

Fine-tuning

• Transformers - Hugging Face Transformers is a popular library for Natural Language Processing (NLP) tasks, including fine-tuning large language models.
• Unsloth - Finetune Llama 3.2, Mistral, Phi-3.5 & Gemma 2-5x faster with 80% less memory!
• LitGPT - 20+ high-performance LLMs with recipes to pretrain, finetune, and deploy at scale.

Serving

• TorchServe - An open-source model serving library developed by AWS and Facebook specifically for PyTorch models, enabling scalable deployment, model versioning, and A/B testing.

• TensorFlow Serving - A flexible, high-performance serving system for machine learning models, designed for production environments, and optimized for TensorFlow models but also supports other formats.

• Ray Serve - Part of the Ray ecosystem, Ray Serve is a scalable model-serving library that supports deployment of machine learning models across multiple frameworks, with built-in support for Python-based APIs and model pipelines.

• NVIDIA Triton Inference Server - A high-performance inference server supporting multiple ML/DL frameworks (TensorFlow, PyTorch, ONNX, etc.), optimized for GPU deployments, and ideal for both cloud and on-premises serving.

• vllm - An optimized, high-throughput serving engine for large language models, designed to efficiently handle massive-scale inference with reduced latency.

• sglang - SGLang is a fast serving framework for large language models and vision language models.

• LitServe - LitServe is a lightning-fast serving engine for any AI model of any size. Flexible. Easy. Enterprise-scale.

Prompt Management

• Opik - Opik is an open-source platform for evaluating, testing and monitoring LLM applications

Datasets

Use Cases

• Datasets - A vast collection of ready-to-use datasets for machine learning tasks, including NLP, computer vision, and audio, with tools for easy access, filtering, and preprocessing.
• Argilla - A UI tool for curating and reviewing datasets for LLM evaluation or training.
• distilabel - A library for generating synthetic datasets with LLM APIs or models.

Fine-tuning

• LLMDataHub - A quick guide (especially) for trending instruction finetuning datasets
• LLM Datasets - High-quality datasets, tools, and concepts for LLM fine-tuning.

Pretraining

• IBM LLMs Granite 3.0 - Full list of datasets used to train IBM LLMs Granite 3.0

Benchmarks

• lighteval - A library for evaluating local LLMs on major benchmarks and custom tasks.

Agent

• TravelPlanner - paper A Benchmark for Real-World Planning with Language Agents

Understand LLM

Prompt Engineering

• Brown, Tom B. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).

Reasoning & Planning

• Kambhampati, Subbarao, et al. "LLMs can't plan, but can help planning in LLM-modulo frameworks." arXiv preprint arXiv:2402.01817 (2024).

Learn LLM

Training

• Chip's Blog - Chip Huyen's blog on training LLMs, including the latest research, tutorials, and best practices.
• Lil'Log - Lilian Weng(OpenAI)'s blog on machine learning, deep learning, and AI, with a focus on LLMs and NLP.
• Decoding ML - Content on building production GenAI, RecSys and MLOps applications.

Books

• Build a Large Language Model from Scratch by Sebastian Raschka
• Hands-On Large Language Models: Build, Tune, and Apply LLMs by Jay Alammar , Maarten Grootendorst
• LLM Engineer's Handbook: Master the art of engineering large language models from concept to production by Paul Iusztin , Maxime Labonne

Applications

General

• AdalFlow documentation - Includes tutorials from building RAG, Agent, to LLM evaluation and fine-tuning.

• CS224N - Stanford course covering NLP fundamentals, LLMs, and PyTorch-based model building, led by Chris Manning and Shikhar Murty.

• LLM-driven Data Engineering - A playlist of 6 lectures by Zach Wilson on how LLMs will impact data pipeline development

• LLM Course by Maxime Labonne - An end-to-end course for AI and ML engineers on open source LLMs.

Agent

1. Lectures

• LLM Agents MOOC - A playlist of 11 lectures by the Berkeley RDI Center on Decentralization & AI, featuring guest speakers like Yuandong Tian, Graham Neubig, Omar Khattab, and others, covering core topics on Large Language Model agents.

2. Projects

• OpenHands - Open source agents for developers by AllHands.
• CAMEL - First LLM multi-agent framework and an open-source community dedicated to finding the scaling law of agents. by CAMEL-AI.
• swarm - Educational framework exploring ergonomic, lightweight multi-agent orchestration. Managed by OpenAI Solution team.
• AutoGen - A programming framework for agentic AI 🤖 by Microsoft.

Auto-optimization

• TextGrad - Automatic ''Differentiation'' via Text -- using large language models to backpropagate textual gradients.

Social Accounts

• Chip Huyen
• Damien Benveniste, PhD
• Jim Fan
• Li Yin
• Paul Iusztin LLM engineering & LLMOps
• Armand Ruiz AI product at IBM
• Pascal Biese Daily LLM papers
• Maxime Labonne LLM fine-tuning
• Sebastian Raschka Blogs :Ahead of AI

Contributing

This is an active repository and your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are not an instant fit for this repo, you could vote for them by adding 👍 to them.

---

If you have any question about this opinionated list, do not hesitate to contact Li Yin