Assignment HW6: Transformers

In this assignment, you will complete the implementation of a transformer model. All the resources you need are provided in the starter repo for this assignment, which is available in the course GitHub org. The README file in the starter repo contains a description of the task.

To complete the assignment, add and/or alter code at the locations of the string “TODO” in the source files train_transformer.py, transformer_model.py, and generate_text.py.

To turn in this assignment, push changes to your private assignment repo in the course GitHub org. Grading will be based largely on completeness and correctness.

Optional extension exercises

  1. Generate a much larger knowledge base and train a fact-learning transformer on it.
  2. Play around with the architecture of the transformer, altering for example the number of layers, number of heads, and the model dimension. What is the effect on training time and accuracy?
  3. Extend the knowledge base and training procedure so that the transformer can answer yes/no questions such as “Is the cup cylindrical?”.

AI Use

The AI policy for this assignment is very similar to the previous ones:

You are permitted unlimited AI use if desired. However, it is recommended that you do not use AI for this assignment, except when you have no other way to make progress. You will learn and understand the concepts of transformer neural networks by completing these questions manually as much as possible. Therefore it is recommended to turn off completions in your development environment.