How to create an original character LoRA [NoobAI XL Training] NoobAI XL Character training

So far, I have explained character training for SD1.5, SDXL, Pony, and IllustriousXL. This time, I would like to use “NoobAI XL” for character training. Since NoobAI XL is based on Illustrious-XL, it is essentially a model similar to Illustrious-XL. The model used for training is the conventional ε-Pred (epsilon-prediction) model “noobai-XL v1.1 eps-pred”.

If you don’t know how to install Kohya ss GUI for LoRA training or how to create a dataset, you should first read the following articles.

How to create an original character LoRA [Preparation] Kohya SS GUI Installation and basic operations (New GUI supported)

🔗Read later

How to create an original character LoRA [Dataset] Making a training image and caption

🔗Read later

Also, if you are new to LoRA training, I recommend starting with the SD1.5 model, which finishes training quickly.

How to create an original character LoRA [SD1.5 Training] SD1.5 Character Training

🔗Read later

About NoobAI XL

Laxhar Dream Lab · Hugging Face

https://huggingface.co/Laxhar

🔗External Link

NoobAI XL is a generative model based on (SDXL) developed by Laxhar Dream Lab. It’s simple, but I’ve summarized the outline below.

• Developer: Laxhar Dream Lab
• Architecture: Based on Illustrious‑XL v0 (SDXL)
• Resolution: SDXL standard area (example: 768×1344～1344×768, 1024×1024)
• Prompt: Danbooru tags
• Extension compatibility: Compatible with major tools such as LoRA, ControlNet, Img2Img
• Prediction format: Supports both ε‑Pred (epsilon-prediction) (noise prediction) and V‑Pred (v-prediction) (velocity/color prediction)
• License: Fair AI Public License 1.0-SD

Training with the Default Values of Kohya ss GUI

Now, let’s try training using the default parameters, as is customary.

The base training model we will use this time is “noobai‑XL v1.1 eps-pred”.

Laxhar/noobai-XL-1.1 · Hugging Face

https://huggingface.co/Laxhar/noobai-XL-1.1

🔗External Link

Dataset

The dataset is based on the data created in “How to Create Original Character LoRA【Dataset Edition】Production of training Images and Captions”. If you want to train with the same dataset, it is available on Patreon, but only paid supporters can download it.

DataSet sample for training LoRA | Digital Creative AI | PATREON

https://www.patreon.com/posts/125348409

🔗External Link

　

Default Parameters

Once the dataset is ready, use the following parameters for training. We have slightly modified the values for training with the NoobAI XL model. The parts that need to be input or changed are marked in red.

• Pretrained model name or path：NoobAI-XL-v1.1.safetensors
• Trained Model output name：DCAI_Girl_Def_NoobAI ※Model output name
• Instance prompt：dcai-girl ※Although the value is ignored in the captioning method used this time, it will cause an error if not entered.
• Class prompt：1girl ※Entered for the same reason as above.
• Repeats：5 [Default: 40] ※Because the original image for training is 100, and we want to make the total images 500.
• Presets：none
• LoRA type：Standard
• Train batch size：1
• Epoch：1
• Max train epoch：0
• Max train steps：1600
• Save every N epochs：1
• Seed：123 [Default: 0 = Random] ※I put in an arbitrary number to control the parameters.
• LR Scheduler：cosine
• Optimizer：AdamW8bit
• Learning rate：0.0001 (1e-4)
• Text Encoder learning rate：0.00005 (5e-5) [Default: 0.0001 (1e-4)] ※Changed to the recommended default value in the official documentation.
• Unet learning rate：0.0001 (1e-4)
• LR warmup (% of total steps)：10
• Max resolution：1024, 1024 [Default: 512, 512] ※Resolution of SDXL
• Network Rank (Dimension)：8
• Network Alpha：1

Test Generation Using the Trained LoRA

　

　

The image above was generated using the LoRA trained with default settings. The image below was generated without using LoRA, but with the same settings.

Looking at this image, it seems to have been well trained, but many images with different colors and shapes of costumes were generated. The generation settings of A1111 WebUI are as follows.

masterpiece, best quality, amazing quality, dcai-girl, 1girl, upper body, solo, looking at viewer, short hair, orange hair, brown eyes, animal ears, blue dress,  (white thighhighs, thigh strap, brown footwear:0.7), meadow, sky, mountain, day, newest, extremely detailed background, extremely detailed foreground <lora:DCAI_Girl_Def_NoobAI:1>

nsfw, lowres, worst quality, low quality, bad anatomy, bad hands, 4koma, comic, greyscale, censored, jpeg artifacts, overly saturated, overly vivid

checkpoint: NoobAI-XL-v1.1.safetensors
Steps: 25
Sampler: DPM++ SDE
Schedule type: Karras
CFG scale: 3
Seed: 2985336440
Size: 1344x768
ADetailer: ON
Hires. fix: Upscaler: 4x-UltraSharp / Step: 10 / Denoising strength: 0.25 / Upscale by: 2
VAE: sdxl.vae.safetensors

Incorporating Convolution into Standard LoRA

Convolution is an operation that extracts “local features (patterns, edges, shapes)” in an image. Simply put, it traces each part of the image with a “small filter (kernel)” to extract features, and this process is layered to gradually convert it into higher-order information (faces, eyes, clothes, composition, etc.).

In Standard LoRA, corrections are made to the Linear layer (centered on Attention). Since the Conv2D layer is not subject to correction, it is said that “fine visual features such as wrinkles in clothes, movement of hair, and decorations are difficult to learn.” Additional settings are required to target the Conv2D layer.

How to Incorporate Convolution into Standard LoRA with Kohya ss GUI

The way to incorporate it is not so difficult, you can enable it just by entering the following code in the “Additional parameters” of the Advanced section of the parameters.

--network_args use_conv=True conv_dim=32 conv_alpha=16

　

Basically, you can use the same values for “conv_dim” and “conv_alpha” as for Network Rank (Dimension) / Network Alpha. Also, if you do not enter conv_alpha, the default will be conv_alpha=1.0.

About fp16 and bf16

This is an explanation of fp16 (Half Precision Floating Point) and bf16 (Brain Floating Point 16). fp stands for Floating Point, which refers to floating point numbers. General computers are calculated with fp32 (single precision floating point). Halving this to achieve speed and memory efficiency results in fp16 (half precision floating point). The format “bf16” developed by Google Brain, Google’s artificial intelligence research group, for TPUs achieves the speed and VRAM efficiency of fp16 while behaving similarly to fp32. Although the calculation accuracy is slightly inferior to fp32, AI (especially deep learning) values “being able to calculate a lot” over “perfect decimal accuracy”, so using bf16 allows for fast and stable training.

However, bf16 requires chips from Ampere onwards for Nvidia GPUs. You will need GPUs like A100/H100 or RTX 4090/4080. RTX 30xx is Ampere, but officially, bf16 is “limited support” and may not behave correctly, and furthermore, the learning time may be longer.

Below is a simple summary of the differences between fp16 and bf16.

How to set bf16

To set bf16, switch the “Mixed precision” in the Resource Selection section of the Accelerate launch to bf16.

　

Once switched, you can choose the save method in the “Save precision” in the Model section. If you have a GPU of RTX40xx or higher for personal use with LoRA, save it as bf16, and if you need to publish on the internet or need versatility, save it as fp16.

　

Standard LoRA/Conv and bf16-based LoRA Training

Now, let’s use the Standard LoRA/Conv and bf16 I just explained to train. The parts that need to be input or changed are written in red. We are also using the “Prodigy+Cosine Annealing” introduced in the previous Illustrious article. Prodigy’s settings were slow to learn with NoobAI XL, so I set d_coef=2.0 to make learning more intensive.

Training Parameters

• Pretrained model name or path：NoobAI-XL-v1.1.safetensors
• Trained Model output name：DCAI_Girl_NoobAI ※Model output name
• Instance prompt：dcai-girl ※In the caption method used this time, the value is ignored, but if not entered, an error will occur
• Class prompt：1girl ※Input for the same reason as above.
• Repeats：5 [Default: 40] ※Because the original image is 100 and we want to make the total image 500.
• Presets：none
• LoRA type：Standard
• Train batch size：1
• Epoch：5 [Default: 1] ※To adjust the total steps with Epoch
• Max train epoch：0
• Max train steps：0 [Default: 1600] ※To adjust the total steps with Epoch
• Save every N epochs：0 [Default: 1] ※Because there was no need to see the progress
• Seed：123 [Default: 0 = Random] ※Enter an arbitrary number to control the parameters
• LR Scheduler：cosine ※Ignored because we use CosineAnnealingLR
• LR Scheduler type：CosineAnnealingLR
• LR scheduler extra arguments：T_max=2500 eta_min=1e-6
• Optimizer：Prodigy [Default: AdamW8bit]
• Optimizer extra arguments：weight_decay=0.05 d_coef=2.0 betas=0.9,0.99 use_bias_correction=True
• Learning rate：1.0 [Default: 0.0001 (1e-4)] ※Prodigy recommended value
• Text Encoder learning rate：1.0 [Default: 0.0001 (1e-4)] ※Prodigy recommended value
• Unet learning rate：1.0 [Default: 0.0001 (1e-4)] ※Prodigy recommended value
• LR warmup (% of total steps)：0 [Default: 10] ※Because it causes an error when used with CosineAnnealingLR
• Max resolution：1024, 1024 [Default: 512, 512] ※SDXL resolution
• Network Rank (Dimension)：32 [Default: 8]
• Network Alpha：16 [Default: 1]
• Additional parameters：--network_args use_conv=True conv_dim=32 conv_alpha=16
• clip_skip：0 [Default: 1]
• CrossAttention：sdpa [Default: xformers] ※Although the training speed drops, the quality is good

The TensorBord graph trained with the above parameters turned out as follows. In this training, the average loss value decreased steadily and ultimately reached 0.0672. The middle “d*lr” is a value that has been automatically adjusted.

　

Training Results

　

The generation parameters of A1111 WebUI are the same as those trained with the default settings of LoRA.

I think we were able to train with good quality and detail.

Applying to the Same Lineage Checkpoint Model

You can apply the LoRA you trained this time to the NoobAI XL (ε‑pred) lineage checkpoint. Below is a sample of applying LoRA. The generation parameters use the following parameters.

masterpiece, best quality, amazing quality, dcai-girl, 1girl, upper body, solo, looking at viewer, short hair, orange hair, brown eyes, animal ears, blue dress,  (white thighhighs, thigh strap, brown footwear:0.7), open arms, happy, meadow, sky, ocean, beach, day, newest, extremely detailed background, extremely detailed foreground <lora:DCAI_Girl_NoobAI:1>

nsfw, lowres, worst quality, low quality, bad anatomy, bad hands, 4koma, comic, greyscale, censored, jpeg artifacts, overly saturated, overly vivid

Steps: 25
Sampler: DPM++ SDE
Schedule type: Karras
CFG scale: 3
Seed: 1878814088
Size: 1344x768
ADetailer: ON
Hires. fix: Upscaler: 4x-UltraSharp / Step: 10 / Denoising strength: 0.25 / Upscale by: 2
VAE: sdxl.vae.safetensors

Although some images have strange hand expressions, I was able to generate images with detailed and beautiful.

The final LoRA results are published on Civitai, so if you’re interested, try downloading them.

DCAI Girl – NoobAI XL (ε‑Pred) | Civit AI

https://civitai.com/models/1807357

🔗External Link

Conclusion

This time, T have explained in detail about character LoRA training using the NoobAI XL (ε‑pred) model. Perhaps because NoobAI XL has not been around for long, there are not many derivative models. However, this model, based on Illustrious-XL, combines ease of use and versatility, such as high-resolution output of SDXL and support for Danbooru-style prompts.

In addition, by incorporating Convolution into Standard LoRA, I have enhanced the expressiveness of details and introduced advanced training methods such as bf16 and Prodigy＋CosineAnnealingLR. From the actual generation results, I was able to confirm stable quality and detail reproducibility.