Something funky with Dreambooth on the last build
Added 2022-12-15 02:54:38 +0000 UTCIt seen that there is a bug with some hardware + Accelerator Library + Float32 with Float16 calculations. It only happen once some numbers get to really low values.
I'm almost sure it something to do with the way Accelerator handle some variables.
I did some tests and it seen fixed, but will test a little more tomorrow to make sure, since it happen at random at some point in training, it can take some time between each test.
Thanks for all the reports on this and feedback, it made it faster to find the problem. Will also take a look the other callers to see if this may also happen on the regular prompt2img.
Comments
Can you copy or cut and paste one into your models\models_sd folder then restart the UI and select it from the models list? There's also an option in the configuration tab to change the models folder (so you could try temporarily changing it to that) but I haven't used that yet.
cool1
2022-12-15 18:11:53 +0000 UTCI'm working with GUI 0.62, stable-diffusion-v1-5. I did a Dreambooth fit with 1000 steps. I generated a preview every 100 steps and the picture seems overfit at 1000 steps, with the best image being about 700 steps in. I had Dreambooth save a model every 100 steps. Looking in the folders "...models\dreambooth\jrr", I see the saved models (jrr_100.pt, jrr_200.pt, etc.). Can anyone tell me how to select one of these model to use for my renders?
Jamie
2022-12-15 16:56:01 +0000 UTCAre you using a older .pt file? For now on the PT files will alwasys work, but older ones will not. You need to generate a new .PT file.
GRisk
2022-12-15 13:57:05 +0000 UTC
