Lora Training - Learning Rate and Scheduler

Hey guys, extending the Lora experimentation from last time, I want to dive into the differences between the different schedulers and try out a lower learning rate while exploring at two different network rank and alphas.

If all that sounds like nonsense to you, basically I am going to train my AI slower, for a longer period, experimenting with two different configs.

Lets look at a bunch of XY images! (Large Image Warning)

This is our original image with no Lora influence:

And once again here is an image from our training data:

ahegao, 1girl, solo, long_hair, black_hair, horror_\(theme\), close-up

We are trying to train the `ahegao` face, with hopes of applying the face to an image, and keeping the image as close to the original as possible while changing the face expression to this one. Hopefully we can come close to something.

• Learning Rate: 1e-5
• 64 Rank and Alpha
• Scheduler: Constant

• Learning Rate: 1e-5
• 64 Rank and Alpha
• Scheduler: Cosine

• Learning Rate: 1e-5
• 64 Rank and Alpha
• Scheduler: Linear

• Learning Rate: 1e-5
• 128 Rank and Alpha
• Scheduler: Constant

• Learning Rate: 1e-5
• 128 Rank and Alpha
• Scheduler: Cosine

• Learning Rate: 1e-5
• 128 Rank and Alpha
• Scheduler: Linear

That's a lot of pictures! So right off the bat, comparing the first 3 at 64 rank and alpha vs the last 3 at 128 rank and alpha, the pose is not tilted to an angle at the extremes of the training at 240 epochs, that's a good thing because we are looking to match the original picture on the right side as close as we can, and the alignment matching is one extra piece.

But unfortunately, for all the training, as we go further into training, the more nude the character becomes. If our target was a lewd Lora this might be ok, but we are looking for simply a change in the face.

In both 64 and 128 Constant training runs, the very tip of 240 epochs at 1.0 strength seem to be the closest to the training images' poses and closeup distance, so that indicates to me that Constant might overfit to the training images more readily.

Comparing Cosine and Linear, to my eye, they look nearly identical in both 64 and 128, but there is a slight ever so slight tilt toward Cosine holding the armor better through strength 0.9-0.7 in 64 and 0.5 in 128.

Lets take a closer look in between the epochs we were looking at earlier and see if there are any particular ones that stand out.

64 x Constant
Looking at the previous grid, it looks like there might be something between epochs 70 and 80, so lets dig in.

73 and 80 look pretty on target for least amount of changes while having the target face expression.

64 x Cosine
Going to to try epoch 90-100:

And 91 and 100 seem to be most on target.

64 x Linear
Nothing was ever found in the investigation that met the minimum set by the earlier finds.

128 x Constant
I don't think there will be any armored ahegao faces so I am going to look for a good lewd nude.

229 at 0.8 strength looks pretty lewd

128 x Cosine

151 at 0.9 look on target.

128 x Linear

129 and 128 at 1.0 strength

Coming up I will take a look at all of the best epochs and compare them against each other against a random set of prompts and seeds to see which ones perform consistently.