DCAI
Loading Light/Dark Toggl

How to create an original character LoRA [Illustrious-XL Training] Illustrious-XL Character training

⏱️14min read
📅 Jul 10, 2025
Category:📂 Advanced
How to create an original character LoRA [Illustrious-XL Training] Illustrious-XL Character training featured Image
Supported by

This article follows the previous one on Pony and will use “Illustrious-XL” to train about the character LoRA. Illustrious-XL is a generation model based on Stable Diffusion XL (SDXL), specialized for high-quality illustrations and animations. It combines hybrid input of Danbooru tags and natural language, high-resolution generation capability, and extended compatibility. From v2 onwards, its stability and ease of use have been further enhanced. If you don’t know how to install the Kohya ss GUI for LoRA training or how to create a dataset, you should first read the articles below.

Also, if you are new to LoRA training, I recommend starting with the SD1.5 model, which completes training quickly.

PR
Image of One by Wacom Small Graphics Drawing Tablet 8.3 x 5.7 Inches, Portable Versatile for Students and Creators, Ergonomic 2048 Pressure Sensitive Pen Included, Compatible with Chromebook Mac and Windows
One by Wacom Small Graphics Drawing Tablet 8.3 x 5.7 Inches, Portable Versatile for Students and Creators, Ergonomic 2048 Pressure Sensitive Pen Included, Compatible with Chromebook Mac and Windows
🔗Amazon-Usa Link
Image of XPPen Updated Deco 01 V3 Drawing Tablet-16384 Levels of Pressure Battery-Free Stylus, 10x6 Inch OSU Graphic Tablet, 8 Hotkeys for Digital Art, Teaching, Gaming Drawing Pad for Chrome, PC, Mac, Android
XPPen Updated Deco 01 V3 Drawing Tablet-16384 Levels of Pressure Battery-Free Stylus, 10x6 Inch OSU Graphic Tablet, 8 Hotkeys for Digital Art, Teaching, Gaming Drawing Pad for Chrome, PC, Mac, Android
🔗Amazon-Usa Link

About Illustrious-XL

Illustrious-XL is a generative model based on (SDXL) developed by OnomaAI. It started from checkpoints such as Kohaku XL-beta5 and has been retrained with their unique illustration-specific dataset. Here is a simple summary of the model.

  • Developer: OnomaAI
  • Architecture: SDXL-based (Large-scale UNet + two-stage cross-attention structure)
  • Resolution: Native 1536×1536px generation possible. Supports non-standard aspect ratios between 512 and 1536px
  • Hybrid Prompt: Can generate detailed and intuitive works with Danbooru tags + natural language input
  • Extended Compatibility: Compatible with major tools such as LoRA, ControlNet, Img2Img, etc.
  • Stability: From v2.0 onwards, it is a “stable” checkpoint with less variation in generation and high fidelity
  • License: v0.1: Fair Public AI + TOS / v1.0~: MIT + CreativeML Open-Rail++

Features by Version

v0.1 (Released in November 2024):
  • The first public version. Centered on Danbooru tag training, it has a passing basic performance but also some instability
v1.0 (Trained in July 2024, Released by CivitAI):
  • Supports native 1536×1536px, exceeding the standard resolution of the conventional SDXL
  • Evolved to support hybrid prompts of Danbooru tags + English natural language
  • Enhanced compatibility with extension modules such as LoRA and ControlNet
  • Released as a minimal “raw” pre-trained model, also suitable for fine-tuning purposes externally
v1.1 (Exclusive to Tensor.Art):
  • An enhanced version of v1.0, with more refined composition and color expressiveness due to dataset readjustment
v2.0-STABLE (Released in April 2025):
  • Adopted cosine annealing training schedule, improving the stability of the generated results
  • Strengthened stability and consistency of color tone and composition in 1536px output, ensuring more reliable expression as per the prompt
  • Continues to support SDXL compatible extensions such as LoRA/ControlNet

Training with the Default Values of Kohya ss GUI

In DCAI character training, let’s try training using the default parameters that have become a tradition.

The training base model we will use this time is “Illustrious-XL-v2.0”.

Dataset

The dataset is based on the data created in “How to create an original character LoRA [Dataset] Making a training image and caption”. If you want to train with the same dataset, it is available on Patreon, but only paid supporters can download it.

Illustrious-XL can train up to 1536×1536px, but let’s try the usual 1024×1024px first.

Dataset image sample list 

Default Parameters

Once the dataset is ready, use the following parameters for training. I have slightly modified the values for training the Illustrious-XL model. The parts that need to be input or changed are marked in red.

  • Pretrained model name or path:Illustrious-XL-v2.0.safetensors
  • Trained Model output name:DCAI_Girl_Illutorious_Def ※Model output name
  • Instance prompt:dcai-girl ※Although the value is ignored in the captioning method used this time, it will cause an error if not entered.
  • Class prompt:1girl ※Entered for the same reason as above.
  • Repeats:5 [Default: 40] ※Because the original image for training is 100 images, and I want to make the total images 500.
  • Presets:none
  • LoRA type:Standard
  • Train batch size:1
  • Epoch:1
  • Max train epoch:0
  • Max train steps:1600
  • Save every N epochs:1
  • Seed:123 [Default: 0 = Random] ※I put in an arbitrary number to control the parameters.
  • LR Scheduler:cosine
  • Optimizer:AdamW8bit
  • Learning rate:0.0001 (1e-4)
  • Text Encoder learning rate:0.00005 (5e-5) [Default: 0.0001 (1e-4)] ※Changed to the recommended default value in the official documentation.
  • Unet learning rate:0.0001 (1e-4)
  • LR warmup (% of total steps):10
  • Max resolution:1024, 1024 [Default: 512, 512] ※Resolution of SDXL
  • Network Rank (Dimension):8
  • Network Alpha:1
  • clip_skip:2 [Default: 1] ※Recommended value for Illustrious-XL

Test Generation Using the Trained LoRA

The image above was generated using the LoRA trained with default settings. The image below was generated without using LoRA, but with the same settings.

LoRA sample with default values 
LoRA sample with default values (without LoRA) 

As you can see, in the default settings, only the costume is slightly affected, and the face and other features are hardly affected by LoRA. The generation settings of A1111 WebUI are as follows.

Prompt
masterpiece, best quality, amazing quality, dcai-girl, 1girl, solo, looking at viewer, solo, short hair, orange hair, brown eyes, animal ears, blue dress, meadow, sky, day, newest, extremely detailed background, extremely detailed foreground 
Negative-Prompt
worst quality, low quality, bad quality, bad details, oldest, very displeasing, displeasing, explicit, nsfw, bad anatomy, artistic error, lowres, bad hands, multiple views, by bad artist, jpeg artifacts, anatomical nonsense, bad perspective, bad proportions, bad reflection, ugly, poorly drawn face, deformed eyes, deformed hands
checkpoint: Illustrious-XL-v2.0.safetensors
Steps: 28
Sampler: Euler a
Schedule type: Automatic
CFG scale: 5.5
Seed: 181736684
Size: 1344x768
ADetailer: ON
Hires. fix: Upscaler: 4x-UltraSharp / Step: 10 / Denoising strength: 0.25 / Upscale by: 2
VAE: sdxl.vae.safetensors

About Cosine Annealing

Cosine Annealing is a type of PyTorch scheduler that changes the learning rate according to the cosine function as training progresses. It is used to effectively promote convergence while avoiding drastic changes in the learning rate.

How to Use Cosine Annealing in Kohya_ss GUI

To use Cosine Annealing in Kohya_ss GUI, you need to have Kohya_ss GUI version v25.1.0 or higher. If you don’t know how to update, please refer to the article below.

To set it up, select CosineAnnealingLR from the “LR Scheduler type” in the Parameters area of Kohya_ss GUI, and enter the following command in “LR scheduler extra arguments”.

T_max=1600 eta_min=1e-6

The commands that can be entered in extra arguments are as follows.

  • T_max (int):(Required argument) The number of steps until the learning rate reaches its minimum. Usually, it matches max_train_steps.
  • eta_min (float):Enter the minimum learning rate. (Default: 0.0)
  • last_epoch (int):The epoch at the time of resumption (usually ignored) (Default: -1)

Please note that Cosine Annealing and warmup cannot be used together, so if you do not set “LR warmup (% of total steps)” to 0, you will get an error.

“LR Scheduler” is overwritten, so use constant, or Cosine Annealing is not written in LoRA’s Metadata, so use cosine similar to it with warmup set to 0 for training.

Kohya_ss GUI settings 

LoRA Training with Cosine Annealing and Prodigy

Now, let’s try character training of Illustrious-XL using the Cosine Annealing scheduler introduced earlier and the Prodigy optimizer introduced in Pony’s Character Training.

Training Parameters

  • Pretrained model name or path:Illustrious-XL-v2.0.safetensors
  • Trained Model output name:DCAI_Girl_Illustrious-XL_V20 ※Model output name
  • Instance prompt:dcai-girl ※Although the value is ignored in the caption method used this time, it is necessary to enter it to avoid errors.
  • Class prompt:1girl ※Entered for the same reason as above.
  • Repeats:5 [Default:40] ※Because the original image is 100 and we want to make the total image 500.
  • Presets:none
  • LoRA type:Standard
  • Train batch size:1
  • Epoch:5 [Default:1] ※To adjust the total steps with Epoch
  • Max train epoch:0
  • Max train steps:0 [Default:1600] ※To adjust the total steps with Epoch
  • Save every N epochs:0 [Default:1] ※Because there was no need to see the progress
  • Seed:123 [Default:0 = Random] ※Enter an arbitrary number to control the parameters
  • LR Scheduler:constant [Default:cosine] ※Ignored because we use CosineAnnealingLR
  • LR Scheduler type:CosineAnnealingLR
  • LR scheduler extra arguments:T_max=1600 eta_min=1e-6
  • Optimizer:Prodigy [Default:AdamW8bit]
  • Learning rate:1.0 [Default:0.0001 (1e-4)] ※Recommended value for Prodigy
  • Text Encoder learning rate:1.0 [Default:0.0001 (1e-4)] ※Recommended value for Prodigy
  • Unet learning rate:1.0 [Default:0.0001 (1e-4)] ※Recommended value for Prodigy
  • LR warmup (% of total steps):0 [Default:10] ※Because it causes an error when used with CosineAnnealingLR
  • Max resolution:1024, 1024 [Default:512, 512] ※Resolution of SDXL
  • Network Rank (Dimension):32
  • Network Alpha:16
  • clip_skip:2 [Default:1] ※Recommended value for Illustrious-XL
  • CrossAttention:sdpa [Default:xformers] ※Although the training speed decreases, the quality is good

The TensorBord graph trained with the above parameters turned out as follows. It’s not much different from the last Pony character training graph, but it has dropped to about 0.065 in the initial stage. The LarningRate is properly in the shape of a Cosine. The upper “lr/d*lr/group1” is a value that has been automatically adjusted.

TensorBord graph of LoRA training using Cosine Annealing and Prodigy 

Training Results

Illustrious-XL-v2.0-LoRA学習の最終結果 

The generation parameters of A1111 WebUI are the same as those trained with the default settings of LoRA.

As a result, I think it has been well reproduced. If you are concerned about the simple background, try specifying the background prompt in detail.

Applying to the Same Lineage Checkpoint Model

Just like Pony, you can apply the LoRA you trained this time to the Illustrious-XL lineage checkpoint. Below is a sample of applying LoRA. The generation parameters use the following parameters.

Prompt
masterpiece, best quality, amazing quality, dcai-girl, 1girl, solo, looking at viewer, solo, short hair, orange hair, brown eyes, animal ears, blue dress, dynamic, open mouth, crystal staff, healing magic, green aura, meadow, grass, flower, sky, mountain, newest, extremely detailed background, extremely detailed foreground <lora:DCAI_Girl_Illustrious-XL_V20:1>
Negative-Prompt
worst quality, low quality, bad quality, bad details, oldest, very displeasing, displeasing, explicit, nsfw, bad anatomy, artistic error, lowres, bad hands, multiple views, by bad artist, jpeg artifacts, anatomical nonsense, bad perspective, bad proportions, bad reflection, ugly, poorly drawn face, deformed eyes, deformed hands, extra digit, fewer digits, text
Steps: 28
Sampler: DPM++ SDE
Schedule type: Karras
CFG scale: 5.5
Seed: 311095029
Size: 1344x768
ADetailer: ON
Hires. fix: Upscaler: 4x-UltraSharp / Step: 10 / Denoising strength: 0.25 / Upscale by: 2
VAE: sdxl.vae.safetensors
Illustrious-XL lineage checkpoint sample - Featureless Mix Illustrious
Featureless Mix Illustrious - v3.0
Illustrious-XL lineage checkpoint sample - Nova Anime XL
Nova Anime XL - IL v9.0
Illustrious-XL lineage checkpoint sample - Plant Milk 🌿 - Model Suite
Plant Milk 🌿 - Model Suite - Walnut
Illustrious-XL lineage checkpoint sample - Sweet-mix
Sweet-mix - Illustrious/XL v1.4
Illustrious-XL lineage checkpoint sample - ZUKI anime ILL
ZUKI anime ILL - v5.0
Illustrious-XL lineage checkpoint sample - Rillusm · Realistic IL🔮
Rillusm · Realistic IL🔮 - v2.1
Illustrious-XL lineage checkpoint sample - iLustMix
iLustMix - v7.0 Cinematic
Illustrious-XL lineage checkpoint sample - Illustrious-XL
Illustrious-XL - v0.1

As you can see from the last “Illustrious-XL – v0.1”, it is confirmed that Illustrious-XL-v2.0 has some degree of backward compatibility.

The LoRA of the final results is available on Civitai for those interested to download.

Conclusion

This time, I introduced character training using Illustrious-XL. Compared to Pony, Illustrious-XL has a better prompt response, and I thought it was easier to create a free composition like the sample of the checkpoint model of the same lineage. Also, the combination of Prodigy + CosineAnnealingLR allows you to easily and simply achieve high-quality training without worrying about difficult settings, making it easy to use even for beginners.

PR
Image of Corel PaintShop Pro 2023 Ultimate | Powerful Photo Editing & Graphic Design Software + Creative Suite | Amazon Exclusive ParticleShop + 5 Brush Starter Pack [PC Download]
Corel PaintShop Pro 2023 Ultimate | Powerful Photo Editing & Graphic Design Software + Creative Suite | Amazon Exclusive ParticleShop + 5 Brush Starter Pack [PC Download]
🔗Amazon-Usa Link
Image of Adobe Creative Cloud Photography Plan 1TB (Photoshop + Lightroom) | 12-month Subscription with auto-renewal
Adobe Creative Cloud Photography Plan 1TB (Photoshop + Lightroom) | 12-month Subscription with auto-renewal
🔗Amazon-Usa Link
Supported by