Based on the architecture on the graph, my implementation might be wrong because they draw the layer normalization between the residual blocks. You have Pre-Ln and Post-Ln -> https://arxiv.org/abs/2002.04745
Andrej Karpathy's videos are pretty amazing. Andrej is building GPT2 from scratch. You might want to check your implementation against his. 😃 https://www.youtube.com/watch?v=kCc8FmEb1nY
Based on the architecture on the graph, my implementation might be wrong because they draw the layer normalization between the residual blocks. You have Pre-Ln and Post-Ln -> https://arxiv.org/abs/2002.04745
Andrej Karpathy's videos are pretty amazing. Andrej is building GPT2 from scratch. You might want to check your implementation against his. 😃 https://www.youtube.com/watch?v=kCc8FmEb1nY