Hands-on labIntermediate
LLM Course
Hugging Face
A practical path through Transformers, tokenizers, datasets, fine-tuning, and modern LLM workflows.
- Time
- 12 chapters
transformersdatasetsfine-tuning
Read the ToolDix guide
Browse 20 learning paths
AI learning path
LLMs, RAG & Evaluation
Understand transformer-based language systems, build retrieval with citations, and evaluate quality before treating a demo as a product.
For AI application developers and technical product teams
24 original lessons on ToolDix, roughly 494 minutes end to end
Start the courseCourse outline
- 1The LLM Application Stack: From Model to ProductThe four layers between a raw model and a shipped product, why most bugs are not in the model, and how orchestration and data layers determine real-world quality.Beginner18 min
- 2Context Windows and Token Economics for BuildersHow context windows work, where token cost concentrates in RAG, prompt caching strategies, and failure modes when context runs out.Beginner16 min
- 3Embeddings: How Meaning Becomes VectorsWhat embeddings are, why cosine similarity works, how semantically similar text clusters in vector space even without shared vocabulary, and why embeddings are the foundation of RAG.Intermediate18 min
- 4Structured Output and Function CallingJSON schema-constrained output, function calling round trips, why structured output matters for RAG citations, and validation patterns when models emit malformed structures.Intermediate20 min
- 5Choosing a Model: Cost, Latency, and Quality TradeoffsModel tiers from fast-and-cheap to frontier, routing strategies, cost-latency-quality as three axes you trade off per task, not a single "best" model.Intermediate20 min
- 6Chunking Strategies: Fixed, Semantic, and StructuralFixed-size chunking with overlap, semantic chunking via embedding similarity, structural/document-aware chunking, tradeoffs of each, and how bad chunking silently caps retrieval quality.Intermediate22 min
- 7Choosing and Benchmarking an Embedding ModelCompare embedding models across dimensions, vector size, retrieval quality, cost, and latency. Build your own labeled query set to validate a model choice on your domain before committing to production.Intermediate20 min
- 8Vector Index Internals: HNSW, IVF, and QuantizationWhy exact nearest-neighbor search doesn't scale, how HNSW and IVF trade accuracy for speed, and how quantization compresses vectors. Learn to tune recall-speed tradeoffs and pick the right index for your use case.Advanced24 min
- 9Hybrid Search: Combining BM25 and Vector RetrievalBM25 excels at exact-match queries; vectors excel at semantic queries. Learn why pure vector search fails on product codes and error IDs, and how to merge both retrieval methods using reciprocal rank fusion.Intermediate20 min
- 10Re-ranking: Cross-Encoders and Late InteractionBi-encoders are fast but less accurate. Cross-encoders are slower but can re-rank retrieved candidates with much higher precision. Learn when to use each and how late-interaction models offer a middle ground.Advanced20 min
- 11Query Transformation: Rewriting, Expansion, and HyDERaw user questions are often conversational, vague, or off-topic. Learn to rewrite queries for clarity, expand them with synonyms, and use HyDE (Hypothetical Document Embeddings) to generate better search terms before retrieval.Intermediate18 min
- 12Build a Citation-Aware RAG PipelineDesign and implement a complete RAG system with provenance tracking: from document ingestion with metadata tags, through chunking with source IDs, to retrieval and generation with citation enforcement and verification.Intermediate26 min
- 13Multi-Hop and Agentic RetrievalHow to decompose complex questions into sub-questions and chain retrieval passes together, with cost and error-compounding tradeoffs.Advanced22 min
- 14RAG Beyond Text: Tables, Code, and Structured DataWhy naive text-chunking fails for tables and code; chunking strategies for each; and when to route to text-to-SQL or metadata filters instead.Advanced22 min
- 15GraphRAG and Knowledge-Graph-Augmented RetrievalWhy graphs surface connected facts that chunks alone miss: entity-relationship retrieval, community summarization, and tradeoffs with plain RAG.Advanced22 min
- 16Why Evaluate: Demo Quality vs Product QualityWhy RAG systems that look great on 5 hand-picked questions fail in production. The five maturity levels of evaluation and why 'it feels right' is not a strategy.Beginner16 min
- 17Building a Golden Evaluation SetMining real queries from production logs, curating for coverage, versioning like code, and re-running on every pipeline change.Intermediate20 min
- 18Evaluate Retrieval and Answers SeparatelyWhy end-to-end scoring hides failures, how to score retrieval and generation independently, and worked examples of controlled comparisons that isolate which layer changed.Advanced22 min
- 19Retrieval Metrics: Recall, Precision, MRR, and nDCGPrecisely define Recall@k, Precision@k, Mean Reciprocal Rank, and normalized Discounted Cumulative Gain with worked numeric examples. Learn when each metric matters and how to avoid common pitfalls in choosing k.Intermediate20 min
- 20Generation Quality and Faithfulness MetricsScore answer relevance and faithfulness (is every claim grounded in evidence). Learn claim-level scoring, hallucination detection via evidence matching, and automated frameworks like Ragas.Advanced22 min
- 21LLM-as-Judge: Design, Bias, and CalibrationDesign an LLM judge with an explicit rubric and structured output. Recognize position bias, verbosity bias, and self-preference bias. Learn why every judge must be calibrated against human labels before deployment.Advanced22 min
- 22Human Evaluation: Rubrics, Sampling, and CalibrationDesign rubrics that humans can apply consistently. Choose sampling strategies. Use inter-rater agreement to detect ambiguous rubrics vs. genuinely hard cases. Feed human labels into LLM-judge calibration.Intermediate20 min
- 23A/B Testing and Regression Testing for LLM FeaturesOnline A/B testing for RAG features: traffic splitting, quality + business metrics, statistical significance. Offline regression testing via CI: run golden sets on every code change, gate deploys on metric thresholds.Advanced22 min
- 24Observability, Cost, and Latency Monitoring in ProductionMonitor four production signals worth alerting on. Understand why golden-set scores measured once at deploy time miss slow drift. Implement tracing for retrieval and prompts. Close the course with a synthesis of how evaluation and monitoring fit together.Advanced22 min
Optional, after the lessons
Further reading from primary sources
Published by other organizations and kept here for reference. The lessons above are the ToolDix course; these are where to go once you want the vendor's own documentation.
8 resources
Hands-on labIntermediate
OpenAI Cookbook
OpenAI
Focused examples for building, evaluating, and operating AI-assisted applications.
- Time
- Pick one recipe
APIsevaluationproduction
Read the ToolDix guide
CourseIntermediate
Full Stack LLM Bootcamp
Full Stack Deep Learning
A practitioner-focused course covering prompt engineering, augmented language models, LLMOps, deployment, user experience, and product development.
- Time
- Lectures, labs, and projects
LLM applicationsRAGLLMOps
Read the ToolDix guide
Hands-on labAdvanced
RAG Evaluation
Hugging Face
A practical notebook for creating a synthetic question set and evaluating retrieval-augmented answers with automated judges and explicit metrics.
- Time
- One notebook
RAG evaluationsynthetic dataLLM judges
Read the ToolDix guide
Hands-on labIntermediate
RAG from Scratch with LlamaIndex
LlamaIndex
An official starter path through document loading, indexing, retrieval, querying, and inspection of a compact RAG application.
- Time
- Starter tutorial
LlamaIndexretrievalcitations
Read the ToolDix guide
Hands-on labIntermediate
Build a RAG Agent with LangChain
LangChain
A guided implementation of retrieval and agentic RAG using loaders, embeddings, vector stores, tools, and a model-driven workflow.
- Time
- Guided tutorial
LangChainagentic RAGretrieval tools
Read the ToolDix guide
Classic readingIntermediate
Evaluation Best Practices
OpenAI Developer Documentation
Guidance for defining objectives, collecting representative cases, selecting graders, and continuously evaluating model behavior.
- Time
- Guide and examples
evalsgraderstest datasets
Read the ToolDix guide
CourseAdvanced
Stanford CS336: Language Modeling from Scratch
Stanford NLP
An end-to-end technical course on data pipelines, tokenization, transformer implementation, training, scaling, evaluation, and systems efficiency for language models.
- Time
- Lectures · assignments · systems work
transformerstraining systemsscaling
Read the ToolDix guide