This was my first time formally doing research and writing a paper. The process was very strenuous, yet rewarding. I had to go through about 60 papers to find the information I needed, which took a lot of time. Oftentimes I would read several papers which didn't really have the information I was looking for, making me feel like I'm wasting my time; but then I would find a goldmine of a paper that had so much good information and helped me both understand the topic better and answered some crucial questions I had.

Additionally I learned a whole lot about exactly why and how language model training is so energy intensive. I read about the GPUs that are used to train the models, how to estimate the number of FLOPs (floating point operations) a model would take to train and how the dataset size and model architecture affect the energy consumption. I found out that training a state of the art model can cost hundreds of millions of dollars in electricity alone. When accounting for other factors such as researcher salaries the costs can easily go into the billions.

Overall, this project was a great learning experience. I garnered a lot of knowledge about language models and the energy consumption of training them. I also now have a better understanding of how to write a research paper and how to find the information I need.