This document contains evolving ideas for course content.
Foundation of modern NLP - representing words as dense vectors in continuous space.
-
Word2Vec
- Skip-gram and CBOW architectures
- Training techniques
- Applications and limitations
-
GloVe (Global Vectors)
- Matrix factorization approach
- Comparison with Word2Vec
- Use cases
Suggestions:
- Include hands-on exercises for training embeddings on custom datasets
- Visualize embeddings using t-SNE or UMAP
- Discuss semantic relationships and analogies
Modern architecture that revolutionized NLP through attention mechanisms.
- BERT (Bidirectional Encoder Representations from Transformers)
- Self-attention mechanism
- Pre-training and fine-tuning paradigm
- Masked language modeling
- Applications: classification, NER, QA
Suggestions:
- Cover the evolution: from RNNs to Transformers
- Hands-on with Hugging Face Transformers library
- Explore variants (RoBERTa, DistilBERT, etc.)
- Discuss computational costs and efficiency
This topic needs further specification. Possible interpretations:
- Universal Vectors?
- Uncertainty Visualization?
- A specific tool or framework?
Action needed: Please clarify what "uv" refers to in the context of this NLP course.
Advanced prompting technique for improving reasoning in large language models.
- Zero-shot and few-shot CoT
- Step-by-step reasoning
- Applications in complex problem-solving
- Limitations and failure modes
Suggestions:
- Demonstrate with practical examples (math, logic, commonsense reasoning)
- Compare with other prompting strategies
- Explore variants (Self-Consistency, Tree of Thoughts)
Other important areas that could complement this course:
- Tokenization techniques (BPE, WordPiece, SentencePiece)
- Evaluation metrics for NLP tasks
- Fine-tuning strategies and transfer learning
- Ethical considerations in NLP
- Recent developments (LLMs, RAG, agents)
This is a living document. Topics will be expanded and refined as the course takes shape.