Facebook OPT finetuning
Added 2022-05-23 09:00:23 +0000 UTCYes, you read the title right, I am currently in the process of making "yet another" model. This time I am working on the facebook OPT models. They do have a difference compared to my normal models, they are released under a CC-BY-NC-SA licence. This is not because I don't "like" to release my models under an MIT licence, but because facebook decided to release their OPT models under an non-commercial license.
I already built the OPT-2.7B model with a new dataset called "Pike", which contains around 2500 ebooks, and is 20% bigger than the Janeway model. I am waiting for some input on how the 2.7B performs, and some fixes from the huggingface community to make their software more stable, but it might mean I might develop a 13B and even an 30B model if I can manage to get it working. If you are one of my supporters or patreons, just send me a message and I will give you access to the models as soon as I finished building them.
Note that building these models are quite costly, and the 30B only runs on 2x A6000 machines due to their size, so it might be that it might take a while to build them. I do want to thank my biggest sponsors KoboldAI, Vast.ai, RunPod.io and Wes for their support, could not do this without them.
