What Is LTX-2.3 IC-LoRA? How to Control ComfyUI Video Generation with Union Control
This time I'll introduce how to use "LTX Video IC-LoRA". With IC-LoRA, you can control the videos you generate to a certain degree. In this article I'll focus on "Union Control" out of the many available IC-LoRAs. I won't cover every IC-LoRA, but the basic usage is the same for the others. Let's start by mastering Union Control.
What You'll Learn in This Article
- How IC-LoRA (In-Context LoRA) works, and how it differs from a regular LoRA
- The types of IC-LoRA released for LTX-2.3, and what each one is for
- When to use IC-LoRA versus LoRA, and the recommended weights
- The models required for the official ComfyUI template "LTX-2.3 IC-LoRA Union Control" and where to place them
- How to use the official "LTX-2.3 IC-LoRA Union Control" template and the results it produces
- How to create a control video with the subgraph blueprint "Video Depth Estimation (MoGe)"
- The structure of the DCAI custom workflow aimed at high-quality video (💎members only)
- The resolution rules specific to Union Control, and a list of usable resolutions (💎members only)
- Frame interpolation with the latest models using ComfyUI core nodes (💎members only)
What Is IC-LoRA?
IC-LoRA (In-Context LoRA) is an image generation framework for diffusion transformers (DiT) such as the FLUX / Qwen-Image / Z-Image series, Wan, and the LTX series. Without changing the architecture of an existing DiT model at all, it enables a wide variety of image generation tasks simply through how the training data is presented.
IC-LoRA has been shown to achieve quality on par with full-parameter fine-tuning or large-scale LoRA fine-tuning, using roughly 1/50 the amount of data and roughly 1/10 the GPU hours. In other words, its greatest strength is that you can "add image generation ability specialized for a specific task afterwards, with a small amount of data and at low cost".
The same concept is used not only for image generation but also in the "LTX Video IC-LoRA" introduced here. IC-LoRA in LTX is a special adapter that conditions video generation based on reference inputs such as an existing video clip or a control signal like a depth map. Each adapter is trained for a specific task, such as structural control, VFX, video restoration, creative transformation, or image quality enhancement. Whereas a regular LoRA changes the style globally, IC-LoRA performs targeted, reference-driven manipulation while preserving the identity of the original scene.
IC-LoRA vs LoRA in LTX-2.3: When to Use Each
In LTX-2.3, the recommended split between IC-LoRA and LoRA is as follows.
- IC-LoRA: structural control, video resolution, VFX, and creative transformation
- LoRA: visual style
✅If you want to use IC-LoRA and LoRA at the same time, the recommended weights are IC-LoRA 0.5-1.0 + LoRA 0.5-0.8.
Full List of LTX-2.3 IC-LoRAs
As of writing, the officially released IC-LoRAs are as follows. Every link goes to a model card on Hugging Face.
- 🔗Union Control: Handles three types of control signals in a unified way: Canny (edges), Depth, and Pose. It delivers fine-grained Image to Video control based on the spatial layout of the control video (reference video).
- 🔗Motion Control: Lets you precisely control the movement of objects and regions along colored trajectories (splines) drawn on the control video.
- 🔗Ingredients: Generates video while maintaining visual consistency, based on a reference sheet image that collects characters, props, and backgrounds.
- 🔗Pixel Spatial Upscaler: An adapter for creative upscaling that enlarges low-resolution video 2x or 4x while synthesizing new detail.
- 🔗In-Outpainting: Supports removing and replacing masked regions (inpainting) as well as expanding the canvas horizontally and vertically (outpainting).
- 🔗Water Simulation: Naturally adds water effects such as rivers, waves, rain, and waterfalls to a control video that has no water.
- 🔗Colorization: Restores natural color to grayscale footage while preserving subject identity and composition.
- 🔗Decompression: Removes compression artifacts typical of low-bitrate video, such as block noise, color bleeding, and ringing.
- 🔗Deblurring: Restores focus and sharpness to out-of-focus footage while preserving the structure of the subject and background.
- 🔗Day to Night: Converts daytime footage into a night scene while retaining composition, camera work, and subject motion.
- 🔗Instant Shave: Removes beards, mustaches, and stubble from people in the footage while leaving expressions, motion, and lighting untouched.
- 🔗Cross-Eyed: Changes the eyes in close-up footage to a cross-eyed (esotropic) state while preserving expression and camera framing.
- 🔗HDR (Beta): Enables high dynamic range video generation, such as converting 8-bit SDR footage into 16-bit HDR footage.
- 🔗Dub-It (Beta): Provides lip-sync (dubbing) functionality that naturally synchronizes mouth movement to audio.
Exploring the LTX-2.3 IC-LoRA Union Control Template
If this is your first time touching LTX, the article below covers the basics, so please read that first.
As always, I recommend updating ComfyUI to the latest version. This time I verified everything using 🔗v0.29.0.
"LTX-2.3 IC-LoRA Union Control" generates video using an input image (the initial frame) and a control video (depth map) created from a reference video. The main nodes are consolidated in the subgraph "First-Last-Frame to Video (LTX-2.3)".
First, let's check the example workflow for "LTX-2.3 IC-LoRA Union Control" from the templates.
Open the template list and select Video from GENERATION TYPE in the left-hand menu.
Once the video-related templates appear, choose "LTX-2.3 IC-LoRA Union Control".
When you open the template, any missing models are shown. Download them as instructed and you can run it as-is. ✅If you're not sure what to do, the next section explains this in detail, so please refer to it.
The official documentation is below.
Downloading the Models
These are basically the same as the regular LTX-2.3 models, but this template uses a distilled checkpoint model. Get the files below and place them in the specified folders under ComfyUI/models. When you open the workflow in ComfyUI, a download button appears on the nodes, so you can also get them from there.
Placement example:
ComfyUI/
├── 📁 models/
│ ├── 📁 checkpoints/
│ │ └── ltx-2.3-22b-distilled-fp8.safetensors
│ ├── 📁 loras/
│ │ ├── ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors
│ │ └── gemma-3-12b-it-abliterated_lora_rank64_bf16.safetensors
│ ├── 📁 text_encoders/
│ │ └── gemma_3_12B_it_fp4_mixed.safetensors
│ └── 📂 geometry_estimation/
│ └── moge_2_vitl_normal_fp16.safetensors
This workflow uses a distilled checkpoint model. A combination of the regular Dev Fp8 model + a distilled LoRA also works without issue.
IC-LoRA LoRA (Gemma abliterated) Text Encoder Geometry Estimation
Geometry Estimation refers to the capability—and the model category—that estimates 3D shape information (depth maps, normal maps, point maps, and so on) from a single 2D image.
Input Materials for "LTX-2.3 IC-LoRA Union Control"
The two input materials used with LTX-2.3 IC-LoRA Union Control are linked in the official documentation. Download them in advance.
About the "LTX-2.3 IC-LoRA Union Control" Nodes
This workflow is made up of the subgraph blueprint "Video Depth Estimation (MoGe)", which generates a control video from a reference video, and the main subgraph "First-Last-Frame to Video (LTX-2.3)". ✅I covered the basic LTX-2.3 nodes in my previous article, so here I'll focus on the nodes specific to IC-LoRA.
Inside the Video Depth Estimation (MoGe) subgraph
Get Video Components
A node that breaks the input video down into four elements—image frames, audio, frame rate (fps), and bit depth—and extracts them. Here it is used to extract only the image frames from the video and pass them to the downstream depth estimation by MoGe.
Resize Images by Longer Edge (DEPRECATED)
A node that resizes the longer edge (longer_edge) of an image to a specified number of pixels and automatically calculates the shorter edge while preserving the aspect ratio. As the name says, it is now deprecated, and officially the recommendation is to replace it with the "Resize Image/Mask" node (with resize_type set to "scale longer dimension").
Run MoGe Inference
A node that performs 3D shape estimation on an input image with the MoGe model. From a single image it estimates depth, normals, camera field of view (FOV), masks, and more all at once, and outputs them as geometry data (moge_geometry) to be passed to the downstream Render MoGe Geometry.
Render MoGe Geometry
A node that writes out the geometry data estimated by Run MoGe Inference as actual viewable images. You can select which types to output from color depth map, grayscale depth map, normal map (OpenGL/DirectX format), and mask. Here it is used to output the color depth map, the grayscale depth map, and the mask.
Inside the First-Last-Frame to Video (LTX-2.3) subgraph
Get IC-LoRA Parameters
A node that extracts IC-LoRA-specific parameters such as reference_downscale_factor from the safetensors metadata of the IC-LoRA model loaded by the LoRA Loader. The extracted parameters are connected to LTXVAddGuide and used to automatically apply the guide image processing that differs per IC-LoRA (such as downscaling the reference image).
LTXVAddGuide
A node that conditions the latent being generated, using a reference image or video as a guide. In addition to positive/negative conditioning, VAE, and latent, you provide inputs such as frame_idx, which specifies the frame position where conditioning starts, and strength, which specifies the strength of the conditioning. Connecting the output of Get IC-LoRA Parameters to iclora_parameters automatically absorbs the differences in guide processing required by each IC-LoRA.
About Subgraph Blueprints
A "subgraph blueprint" in ComfyUI is a subgraph saved to a file in a reusable form. A saved blueprint can be recalled at any time when you start a new workflow. The "Video Depth Estimation (MoGe)" used this time is also a blueprint, so you can call it up with the following steps.
- Double-click anywhere on the canvas and the quick search palette appears.
- There is a category filter below the search box, so select "Blueprints".
- The left column lists subgraph blueprint categories, so select "Conditioning & Preprocessors".
- A list of blueprints appears on the right, and selecting "Video Depth Estimation (MoGe)" loads the blueprint.
About the Video Depth Estimation (MoGe) Subgraph
Looking at the internal structure, it converts the input video into a concatenated image, scales the longer edge to 2048, and then outputs a color depth map, a grayscale depth map, and a mask.
✅It isn't configured in this node, but depending on the settings you can also output a normal map.
Here are samples of each output.
How to Use "LTX-2.3 IC-LoRA Union Control"
Now let's actually use the template. Using the workflow is simple.
- Load the initial frame image: Load the material
the_forgotten_gate.pnginto "Load Image". - Load the control video: Load the material
stone_ruins.mp4into "Load Video". - Load each model: Check that the models in the subgraph "First-Last-Frame to Video (LTX-2.3)" are loaded correctly.
- Run the generation: Once the input image/video and each model are loaded correctly, run it with the "Run" button.
After a while, the generated result appears in "Save Video".
Results from the Official Workflow
The generated result came out as follows. The audio is muted, so unmute it if you want to listen.✅This sample has been downscaled for the web.
As a reference for generation speed, in my environment (RTX3090) it generated in 224 seconds. (I did not use SageAttention.) ✅It isn't a fair comparison since the content generated and the workflow structure differ, but the LTX-2.3 in my previous article (the "LTX-2.3: Image to Video" template) took 182 seconds.
Customizing the Official Workflow
From here I'll introduce an improved version aimed at higher-quality video, built on top of the official ComfyUI "LTX-2.3 IC-LoRA Union Control" workflow. For ease of work, I split the workflow into three parts: control video, initial frame, and video generation. (Two of them are workflows I introduced previously in "What Is Wan 2.2 Fun Control? How to Control Video Generation as Intended".) The customizations are as follows.
- Separating the reference video WF: Rather than doing everything in a single workflow, splitting the process lets you focus on each task.
- Generating the initial frame: Generate the initial frame based on the Depth/Canny/Pose you created.
- Separating the generation stages: Change generation into two stages to control high-resolution output.
- Frame interpolation: Use frame interpolation nodes to raise the frame rate of the output video.
- Consistency support with LoRA: Use a LoRA to improve character consistency.
- Live preview during video generation: Check the progress with a live preview while generating.
The workflows and input materials are published on Patreon. Only paid supporters can view and download them.
Here is a video sample generated with this custom workflow. The three at the top are the reference video, the control video, and the initial frame, respectively. The video settings are approximately Full HD at 60fps.
The control video was generated using 🔗a video by Amorn_mimi from Pixabay.
Ghosting can be seen in the result, but at this point I haven't found a way to solve it.
Summary
In this article I covered everything from an overview of LTX-2.3's IC-LoRA to the steps for generating video with the official ComfyUI template "LTX-2.3 IC-LoRA Union Control".
- IC-LoRA is an adapter that controls generation based on reference input while preserving the identity of the original scene.
- In LTX-2.3, the recommended split is for IC-LoRA to handle structural control and VFX, and for LoRA to handle visual style.
- Union Control is an IC-LoRA that handles three types of control signals—Canny, Depth, and Pose—in a unified way.
- With the official template, you can try generating just by loading an initial frame image and a reference video.
- The control video can be created with the subgraph blueprint "Video Depth Estimation (MoGe)".
In the paid section, I also explain the DCAI custom workflow split up by task, the resolution rules specific to Union Control, and frame interpolation with core nodes. If you're interested, please take a look at the main content as well.
Thank you for reading to the end.
If you found this even a little helpful, please support by giving it a “Like”!