How to create an original character LoRA [Illustrious-XL Training] Illustrious-XL Character training
![How to create an original character LoRA [Illustrious-XL Training] Illustrious-XL Character training featured Image](/_next/image?url=https%3A%2F%2Fdca.data-hub-center.com%2Fcontent%2Fuploads%2F2025%2F07%2Feye_catch_original-character-lora-illustrious-character-training-en.jpg&w=3840&q=80)
This article follows the previous one on Pony and will use “Illustrious-XL” to train about the character LoRA. Illustrious-XL is a generation model based on Stable Diffusion XL (SDXL), specialized for high-quality illustrations and animations. It combines hybrid input of Danbooru tags and natural language, high-resolution generation capability, and extended compatibility. From v2 onwards, its stability and ease of use have been further enhanced. If you don’t know how to install the Kohya ss GUI for LoRA training or how to create a dataset, you should first read the articles below.
![Featured image of How to create an original character LoRA [Preparation] Kohya SS GUI Installation and basic operations](https://dca.data-hub-center.com/content/uploads/2025/03/eye_catch_kohya-ss-gui-installation-basic-operations-en.jpg)
![Featured image of How to create an original character LoRA [Dataset] Making a training image and caption](https://dca.data-hub-center.com/content/uploads/2025/03/eye_catch_training-image-and-caption-en.jpg)
Also, if you are new to LoRA training, I recommend starting with the SD1.5 model, which completes training quickly.
![Featured image of How to create an original character LoRA [SD1.5 Training] SD1.5 Character Training](https://dca.data-hub-center.com/content/uploads/2025/04/eye_catch_original-character-lora-sd15-character-training-en.jpg)
About Illustrious-XL
Illustrious-XL is a generative model based on (SDXL) developed by OnomaAI. It started from checkpoints such as Kohaku XL-beta5 and has been retrained with their unique illustration-specific dataset. Here is a simple summary of the model.
- Developer: OnomaAI
- Architecture: SDXL-based (Large-scale UNet + two-stage cross-attention structure)
- Resolution: Native 1536×1536px generation possible. Supports non-standard aspect ratios between 512 and 1536px
- Hybrid Prompt: Can generate detailed and intuitive works with Danbooru tags + natural language input
- Extended Compatibility: Compatible with major tools such as LoRA, ControlNet, Img2Img, etc.
- Stability: From v2.0 onwards, it is a “stable” checkpoint with less variation in generation and high fidelity
- License: v0.1: Fair Public AI + TOS / v1.0~: MIT + CreativeML Open-Rail++
Features by Version
v0.1 (Released in November 2024):- The first public version. Centered on Danbooru tag training, it has a passing basic performance but also some instability
- Supports native 1536×1536px, exceeding the standard resolution of the conventional SDXL
- Evolved to support hybrid prompts of Danbooru tags + English natural language
- Enhanced compatibility with extension modules such as LoRA and ControlNet
- Released as a minimal “raw” pre-trained model, also suitable for fine-tuning purposes externally
- An enhanced version of v1.0, with more refined composition and color expressiveness due to dataset readjustment
- Adopted cosine annealing training schedule, improving the stability of the generated results
- Strengthened stability and consistency of color tone and composition in 1536px output, ensuring more reliable expression as per the prompt
- Continues to support SDXL compatible extensions such as LoRA/ControlNet
Training with the Default Values of Kohya ss GUI
In DCAI character training, let’s try training using the default parameters that have become a tradition.
The training base model we will use this time is “Illustrious-XL-v2.0”.
Dataset
The dataset is based on the data created in “How to create an original character LoRA [Dataset] Making a training image and caption”. If you want to train with the same dataset, it is available on Patreon, but only paid supporters can download it.
Illustrious-XL can train up to 1536×1536px, but let’s try the usual 1024×1024px first.
Default Parameters
Once the dataset is ready, use the following parameters for training. I have slightly modified the values for training the Illustrious-XL model. The parts that need to be input or changed are marked in red.
- Pretrained model name or path：Illustrious-XL-v2.0.safetensors
- Trained Model output name：DCAI_Girl_Illutorious_Def ※Model output name
- Instance prompt：dcai-girl ※Although the value is ignored in the captioning method used this time, it will cause an error if not entered.
- Class prompt：1girl ※Entered for the same reason as above.
- Repeats：5 [Default: 40] ※Because the original image for training is 100 images, and I want to make the total images 500.
- Presets：none
- LoRA type：Standard
- Train batch size：1
- Epoch：1
- Max train epoch：0
- Max train steps：1600
- Save every N epochs：1
- Seed：123 [Default: 0 = Random] ※I put in an arbitrary number to control the parameters.
- LR Scheduler：cosine
- Optimizer：AdamW8bit
- Learning rate：0.0001 (1e-4)
- Text Encoder learning rate：0.00005 (5e-5) [Default: 0.0001 (1e-4)] ※Changed to the recommended default value in the official documentation.
- Unet learning rate：0.0001 (1e-4)
- LR warmup (% of total steps)：10
- Max resolution：1024, 1024 [Default: 512, 512] ※Resolution of SDXL
- Network Rank (Dimension)：8
- Network Alpha：1
- clip_skip：2 [Default: 1] ※Recommended value for Illustrious-XL
Test Generation Using the Trained LoRA
The image above was generated using the LoRA trained with default settings. The image below was generated without using LoRA, but with the same settings.
As you can see, in the default settings, only the costume is slightly affected, and the face and other features are hardly affected by LoRA. The generation settings of A1111 WebUI are as follows.
Promptmasterpiece, best quality, amazing quality, dcai-girl, 1girl, solo, looking at viewer, solo, short hair, orange hair, brown eyes, animal ears, blue dress, meadow, sky, day, newest, extremely detailed background, extremely detailed foreground worst quality, low quality, bad quality, bad details, oldest, very displeasing, displeasing, explicit, nsfw, bad anatomy, artistic error, lowres, bad hands, multiple views, by bad artist, jpeg artifacts, anatomical nonsense, bad perspective, bad proportions, bad reflection, ugly, poorly drawn face, deformed eyes, deformed handscheckpoint: Illustrious-XL-v2.0.safetensors
Steps: 28
Sampler: Euler a
Schedule type: Automatic
CFG scale: 5.5
Seed: 181736684
Size: 1344x768
ADetailer: ON
Hires. fix: Upscaler: 4x-UltraSharp / Step: 10 / Denoising strength: 0.25 / Upscale by: 2
VAE: sdxl.vae.safetensorsAbout Cosine Annealing
Cosine Annealing is a type of PyTorch scheduler that changes the learning rate according to the cosine function as training progresses. It is used to effectively promote convergence while avoiding drastic changes in the learning rate.
How to Use Cosine Annealing in Kohya_ss GUI
To use Cosine Annealing in Kohya_ss GUI, you need to have Kohya_ss GUI version v25.1.0 or higher. If you don’t know how to update, please refer to the article below.
To set it up, select CosineAnnealingLR from the “LR Scheduler type” in the Parameters area of Kohya_ss GUI, and enter the following command in “LR scheduler extra arguments”.
T_max=1600 eta_min=1e-6The commands that can be entered in extra arguments are as follows.
- T_max (int)：(Required argument) The number of steps until the learning rate reaches its minimum. Usually, it matches max_train_steps.
- eta_min (float)：Enter the minimum learning rate. (Default: 0.0)
- last_epoch (int)：The epoch at the time of resumption (usually ignored) (Default: -1)
Please note that Cosine Annealing and warmup cannot be used together, so if you do not set “LR warmup (% of total steps)” to 0, you will get an error.
“LR Scheduler” is overwritten, so use constant, or Cosine Annealing is not written in LoRA’s Metadata, so use cosine similar to it with warmup set to 0 for training.
LoRA Training with Cosine Annealing and Prodigy
Now, let’s try character training of Illustrious-XL using the Cosine Annealing scheduler introduced earlier and the Prodigy optimizer introduced in Pony’s Character Training.
Training Parameters
- Pretrained model name or path：Illustrious-XL-v2.0.safetensors
- Trained Model output name：DCAI_Girl_Illustrious-XL_V20 ※Model output name
- Instance prompt：dcai-girl ※Although the value is ignored in the caption method used this time, it is necessary to enter it to avoid errors.
- Class prompt：1girl ※Entered for the same reason as above.
- Repeats：5 [Default：40] ※Because the original image is 100 and we want to make the total image 500.
- Presets：none
- LoRA type：Standard
- Train batch size：1
- Epoch：5 [Default：1] ※To adjust the total steps with Epoch
- Max train epoch：0
- Max train steps：0 [Default：1600] ※To adjust the total steps with Epoch
- Save every N epochs：0 [Default：1] ※Because there was no need to see the progress
- Seed：123 [Default：0 = Random] ※Enter an arbitrary number to control the parameters
- LR Scheduler：constant [Default：cosine] ※Ignored because we use CosineAnnealingLR
- LR Scheduler type：CosineAnnealingLR
- LR scheduler extra arguments：T_max=1600 eta_min=1e-6
- Optimizer：Prodigy [Default：AdamW8bit]
- Learning rate：1.0 [Default：0.0001 (1e-4)] ※Recommended value for Prodigy
- Text Encoder learning rate：1.0 [Default：0.0001 (1e-4)] ※Recommended value for Prodigy
- Unet learning rate：1.0 [Default：0.0001 (1e-4)] ※Recommended value for Prodigy
- LR warmup (% of total steps)：0 [Default：10] ※Because it causes an error when used with CosineAnnealingLR
- Max resolution：1024, 1024 [Default：512, 512] ※Resolution of SDXL
- Network Rank (Dimension)：32
- Network Alpha：16
- clip_skip：2 [Default：1] ※Recommended value for Illustrious-XL
- CrossAttention：sdpa [Default：xformers] ※Although the training speed decreases, the quality is good
The TensorBord graph trained with the above parameters turned out as follows. It’s not much different from the last Pony character training graph, but it has dropped to about 0.065 in the initial stage. The LarningRate is properly in the shape of a Cosine. The upper “lr/d*lr/group1” is a value that has been automatically adjusted.
Training Results
The generation parameters of A1111 WebUI are the same as those trained with the default settings of LoRA.
As a result, I think it has been well reproduced. If you are concerned about the simple background, try specifying the background prompt in detail.
Applying to the Same Lineage Checkpoint Model
Just like Pony, you can apply the LoRA you trained this time to the Illustrious-XL lineage checkpoint. Below is a sample of applying LoRA. The generation parameters use the following parameters.
Promptmasterpiece, best quality, amazing quality, dcai-girl, 1girl, solo, looking at viewer, solo, short hair, orange hair, brown eyes, animal ears, blue dress, dynamic, open mouth, crystal staff, healing magic, green aura, meadow, grass, flower, sky, mountain, newest, extremely detailed background, extremely detailed foreground <lora:DCAI_Girl_Illustrious-XL_V20:1>worst quality, low quality, bad quality, bad details, oldest, very displeasing, displeasing, explicit, nsfw, bad anatomy, artistic error, lowres, bad hands, multiple views, by bad artist, jpeg artifacts, anatomical nonsense, bad perspective, bad proportions, bad reflection, ugly, poorly drawn face, deformed eyes, deformed hands, extra digit, fewer digits, textSteps: 28
Sampler: DPM++ SDE
Schedule type: Karras
CFG scale: 5.5
Seed: 311095029
Size: 1344x768
ADetailer: ON
Hires. fix: Upscaler: 4x-UltraSharp / Step: 10 / Denoising strength: 0.25 / Upscale by: 2
VAE: sdxl.vae.safetensors
As you can see from the last “Illustrious-XL – v0.1”, it is confirmed that Illustrious-XL-v2.0 has some degree of backward compatibility.
The LoRA of the final results is available on Civitai for those interested to download.
Conclusion
This time, I introduced character training using Illustrious-XL. Compared to Pony, Illustrious-XL has a better prompt response, and I thought it was easier to create a free composition like the sample of the checkpoint model of the same lineage. Also, the combination of Prodigy + CosineAnnealingLR allows you to easily and simply achieve high-quality training without worrying about difficult settings, making it easy to use even for beginners.
![Image of Corel PaintShop Pro 2023 Ultimate | Powerful Photo Editing & Graphic Design Software + Creative Suite | Amazon Exclusive ParticleShop + 5 Brush Starter Pack [PC Download]](https://m.media-amazon.com/images/I/81hyRE69BDL._AC_SX679_.jpg)
