Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models
概念滑块：用于扩散模型精确控制的 LoRA 适配器

Artistic users of text-to-image diffusion models [4, 37, 36, 9, 19] often need finer control over the visual attributes and concepts expressed in a generated image than currently possible. Using only text prompts, it can be challenging to precisely modulate continuous attributes such as a person’s age or the intensity of the weather, and this limitation hinders creators’ ability to adjust images to match their vision [43]. In this paper, we address these needs by introducing interpretable Concept Sliders that allow nuanced editing of concepts within diffusion models. Our method empowers creators with high-fidelity control over the generative process as well as image editing. Our code and trained sliders will be open sourced.
文本到图像扩散模型[4,37,36,9,19]的艺术用户往往需要对生成图像中表达的视觉属性和概念进行比目前更精细的控制。仅使用文本提示，要精确调节连续属性（如人的年龄或天气的强度）可能具有挑战性，而这种限制阻碍了创作者调整图像以符合其视觉的能力[43]。在本文中，我们引入了可解释的概念滑块，允许对扩散模型中的概念进行细致入微的编辑，从而满足了这些需求。我们的方法能让创作者对生成过程和图像编辑进行高保真控制。我们的代码和训练有素的滑块将开源。

Concept Sliders solve several problems that are not well-addressed by previous methods. Direct prompt modification can control many image attributes, but changing the prompt often drastically alters overall image structure due to the sensitivity of outputs to the prompt-seed combination [38, 44, 22]. Post-hoc techniques such PromptToPrompt [13] and Pix2Video [3] enable editing visual concepts in an image by inverting the diffusion process and modifying cross-attentions. However, those methods require separate inference passes for each new concept and can support only a limited set of simultaneous edits. They require engineering a prompt suitable for an individual image rather than learning a simple generalizable control, and if not carefully prompted, they can introduce entanglement between concepts, such as altering race when modifying age (see Appendix). In contrast, Concept Sliders provide lightweight plug-and-play adaptors applied to pre-trained models that enable precise, continuous control over desired concepts in a single inference pass, with efficient composition (Figure 6) and minimal entanglement (Figure 11).
概念滑块解决了以往方法未能很好解决的几个问题。直接修改提示语可以控制许多图像属性，但由于输出对提示语-种子组合的敏感性，改变提示语往往会大幅改变整体图像结构[38,44,22]。事后技术，如 PromptToPrompt[13]和 Pix2Video[3 ]，可以通过反转扩散过程和修改交叉注意力来编辑图像中的视觉概念。但是，这些方法需要对每个新概念进行单独的推理，而且只能支持有限的同时编辑。这些方法需要设计一个适合单个图像的提示，而不是学习一个简单的通用控制，而且如果不仔细提示，可能会引入概念之间的纠缠，例如在修改年龄时改变种族（见附录）。相比之下，概念滑动器提供了轻量级的即插即用适配器，适用于预先训练好的模型，能够在单次推理中对所需概念进行精确、连续的控制，并具有高效的组合（图6）和最小的纠缠（图11）。

Each Concept Slider is a low-rank modification of the diffusion model. We find that the low-rank constraint is a vital aspect of precision control over concepts: while finetuning without low-rank regularization reduces precision and generative image quality, low-rank training identifies the minimal concept subspace and results in controlled, high-quality, disentangled editing (Figure 11). Post-hoc image editing methods that act on single images rather than model parameters cannot benefit from this low-rank framework.
每个概念滑块都是对扩散模型的低阶修正。我们发现，低秩约束是对概念进行精确控制的一个重要方面：虽然不进行低秩正则化的微调会降低精确度和生成图像的质量，但低秩训练可以识别最小概念子空间，从而实现可控的、高质量的分离编辑（图11）。对单幅图像而非模型参数采取行动的事后图像编辑方法无法从这种低秩框架中受益。

Concept Sliders also allow editing of visual concepts that cannot be captured by textual descriptions; this distinguishes it from prior concept editing methods that rely on text [7, 8]. While image-based model customization methods [25, 38, 6] can add new tokens for new image-based concepts, those are difficult to use for image editing. In contrast, Concept Sliders allow an artist to provide a handful of paired images to define a desired concept, and then a Concept Slider will then generalize the visual concept and apply it to other images, even in cases where it would be infeasible to describe the transformation in words.
概念滑块还允许编辑文字描述无法捕捉的视觉概念；这使它有别于之前依赖文字的概念编辑方法[7,8]。虽然基于图像的模型定制方法[25,38,6]可以为基于图像的新概念添加新标记，但这些方法很难用于图像编辑。相比之下，概念滑块允许艺术家提供一些配对图像来定义所需的概念，然后概念滑块就会概括视觉概念并将其应用到其他图像中，即使在无法用文字描述转换的情况下也是如此。

Other generative image models, such as GANs, have previously exhibited latent spaces that provide highly disentangled control over generated outputs. In particular, it has been observed that StyleGAN [20] stylespace neurons offer detailed control over many meaningful aspects of images that would be difficult to describe in words [45]. To further demonstrate the capabilities of our approach, we show that it is possible to create Concept Sliders that transfer latent directions from StyleGAN’s style space trained on FFHQ face images [20] into diffusion models. Notably, despite originating from a face dataset, our method successfully adapts these latents to enable nuanced style control over diverse image generation. This showcases how diffusion models can capture the complex visual concepts represented in GAN latents, even those that may not correspond to any textual description.
其他生成式图像模型，如 GANs，以前也展示过潜在空间，可对生成的输出结果提供高度分散的控制。特别是，人们已经观察到，StyleGAN[20]风格空间神经元可以对图像的许多有意义的方面进行详细控制，而这些方面是很难用语言来描述的[45]。为了进一步证明我们方法的能力，我们展示了创建概念滑块的可能性，这种概念滑块可以将在 FFHQ 人脸图像[20]上训练的 StyleGAN 风格空间的潜在方向转移到扩散模型中。值得注意的是，尽管我们的方法源自人脸数据集，但却成功地调整了这些潜在方向，从而实现了对不同图像生成的细微风格控制。这展示了扩散模型如何捕捉 GAN 潜在变量所代表的复杂视觉概念，甚至是那些可能与任何文本描述都不对应的概念。

We demonstrate that the expressiveness of Concept Sliders is powerful enough to address two particularly practical applications—enhancing realism and fixing hand distortions. While generative models have made significant progress in realistic image synthesis, the latest generation of diffusion models such as Stable Diffusion XL [36] are still prone to synthesizing distorted hands with anatomically implausible extra or missing fingers [31], as well as warped faces, floating objects, and distorted perspectives. Through a perceptual user study, we validate that a Concept Slider for “realistic image” as well as another for “fixed hands” both create a statistically significant improvement in perceived realism without altering image content.
我们证明了概念滑块的表现力足以解决两个特别实际的应用问题--增强逼真度和修复手部变形。虽然生成模型在逼真图像合成方面取得了重大进展，但最新一代的扩散模型（如稳定扩散 XL 模型[36]）仍然容易合成出扭曲的手掌、解剖学上难以置信的多余或缺失的手指[31]，以及扭曲的人脸、漂浮的物体和扭曲的视角。通过对用户的感知研究，我们验证了 "逼真图像 "概念滑块和 "固定手部 "概念滑块都能在不改变图像内容的情况下显著提高感知逼真度。

Concept Sliders are modular and composable. We find that over 50 unique sliders can be composed without degrading output quality. This versatility gives artists a new universe of nuanced image control that allows them to blend countless textual, visual, and GAN-defined Concept Sliders. Because our method bypasses standard prompt token limits, it empowers more complex editing than achievable through text alone.
概念滑块是模块化和可组合的。我们发现，在不降低输出质量的情况下，可以组成 50 多个独特的滑块。这种多功能性为艺术家提供了一个细微图像控制的新领域，使他们能够将无数文字、视觉和 GAN 定义的概念滑块融合在一起。由于我们的方法绕过了标准提示符的限制，因此可以实现比单纯文本更复杂的编辑。

Concept Sliders are a method for fine-tuning LoRA adaptors on a diffusion model to enable concept-targeted image control as shown in Figure 2. Our method learns low-rank parameter directions that increase or decrease the expression of specific attributes when conditioned on a target concept. Given a target concept $c_{t}$ and model $\theta$, our goal is to obtain $\theta^{*}$ that modifies the likelihood of attributes $c_{+}$ and $c_{-}$ in image $X$ when conditioned on $c_{t}$ - increase likelihood of attribute $c_{+}$ and decrease likelihood of attribute $c_{-}$.
如图2 所示，"概念滑块 "是一种在扩散模型上微调 LoRA 适配器的方法，可实现以概念为目标的图像控制。我们的方法可以学习低等级参数方向，在目标概念的条件下增加或减少特定属性的表达。给定一个目标概念 $c_{t}c\_\{t\}$ 和模型 $\theta \backslash theta$ ，我们的目标是获得 ${\theta }^{\ast }\backslash theta^\{*\}$ ，当以 $c_{t}c\_\{t\}$ 为条件时，它可以修改图像 $XX$ 中属性 $c_{+}c\_\{+\}$ 和 $c_{-}c\_\{-\}$ 的可能性--增加属性 $c_{+}c\_\{+\}$ 的可能性，减少属性 $c_{-}c\_\{-\}$ 的可能性。

Where $P_{\theta}(X|c_{t})$ represents the distribution generated by the original model when conditioned on $c_{t}$. Expanding $P(c_{+}|X)=\frac{P(X|c_{+})P(c_{+})}{P(X)}$, the gradient of the log probability $\nabla\log P_{\theta^{*}}(X|c_{t})$ would be proportional to:
其中 $P_{\theta }(X|c_t)P\_\{\backslash theta\}(X|c\_\{t\})$ 代表原始模型在 $c_{t}c\_\{t\}$ 条件下产生的分布。将 $P(c_{+}|X)=\frac{P(X|c_{+})P(c_{+})}{P(X)}P(c\_\{+\}|X)=\backslash frac\{P(X|c\_\{+\})P(c\_\{+\})\}\{P(X)\}$ 展开，对数概率 $\nabla \log P_{{\theta }^{\ast }}(X|c_t)\backslash nabla\backslash log\; P\_\{\backslash theta^\{*\}\}(X|c\_\{t\})$ 的梯度将与之成正比：

Based on Tweedie’s formula [5] and the reparametrization trick of [15], we can introduce a time-varying noising process and express each score (gradient of log probability) as a denoising prediction $\epsilon(X,c_{t},t)$. Thus Eq. 6 becomes:
根据特威迪公式[5]和[15]的重参数化技巧，我们可以引入一个时变噪声过程，并将每个分数（对数概率梯度）表示为去噪预测值 $ϵ(X,c_t,t)\backslash epsilon(X,c\_\{t\},t)$ 。因此，公式6变为

The proposed score function in Eq. 7 shifts the distribution of the target concept $c_{t}$ to exhibit more attributes of $c_{+}$ and fewer attributes of $c_{-}$. In practice, we notice that a single prompt pair can sometimes identify a direction that is entangled with other undesired attributes. We therefore incorporate a set of preservation concepts $p\in\mathcal{P}$ (for example, race names while editing age) to constrain the optimization. Instead of simply increasing $P_{\theta}(c_{+}|X)$, we aim to increase, for every $p$, $P_{\theta}((c_{+},p)|X)$, and reduce $P_{\theta}((c_{-},p)|X)$. This leads to the disentanglement objective:
公式7中提议的评分函数会改变目标概念 $c_{t}c\_\{t\}$ 的分布，使其表现出更多的 $c_{+}c\_\{+\}$ 属性和更少的 $c_{-}c\_\{-\}$ 属性。在实践中，我们注意到，单个提示对有时会识别出一个与其他不受欢迎的属性纠缠在一起的方向。因此，我们加入了一组保护概念 $p\in 𝒫p\backslash in\backslash mathcal\{P\}$ （例如，在编辑年龄的同时编辑种族名称）来限制优化。我们的目标不是简单地增加 $P_{\theta }(c_{+}|X)P\_\{\backslash theta\}(c\_\{+\}|X)$ ，而是为每个 $pp$ 增加 $P_{\theta }((c_{+},p)|X)P\_\{\backslash theta\}((c\_\{+\},p)|X)$ ，并减少 $P_{\theta }((c_{-},p)|X)P\_\{\backslash theta\}((c\_\{-\},p)|X)$ 。这就引出了拆分目标：

The disentanglement objective in Equation 8 finetunes the Concept Slider modules while keeping pre-trained weights fixed. Crucially, the LoRA formulation in Equation 4 introduces a scaling factor $\alpha$ that can be modified at inference time. This scaling parameter $\alpha$ allows adjusting the strength of the edit, as shown in Figure 1. Increasing $\alpha$ makes the edit stronger without retraining the model. Previous model editing method [7], suggests a stronger edit by retraining with increased guidance $\eta$ in Eq. 8. However, simply scaling $\alpha$ at inference time produces the same effect of strengthening the edit, without costly retraining.
等式8中的解缠目标是对概念滑块模块进行微调，同时保持预先训练的权重不变。最重要的是，等式4中的 LoRA 公式引入了一个可在推理时修改的缩放因子 $\alpha \backslash alpha$ 。如图1 所示，该缩放参数 $\alpha \backslash alpha$ 可以调整编辑强度。增加 $\alpha \backslash alpha$ 可以在不重新训练模型的情况下增强编辑强度。之前的模型编辑方法[7]建议，通过增加公式8 中的指导 $\eta \backslash eta$ 进行再训练，以获得更强的编辑效果。然而，只需在推理时缩放 $\alpha \backslash alpha$ ，就能产生同样的加强编辑效果，而无需昂贵的重新训练。

We evaluate our approach primarily on Stable Diffusion XL [36], a high-resolution 1024-pixel model, and we conduct additional experiments on SD v1.4 [37]. All models are trained for 500 epochs. We demonstrate generalization by testing sliders on diverse prompts - for example, we evaluate our "person" slider on prompts like "doctor", "man", "woman", and "barista". For inference, we follow the SDEdit technique of Meng et al. [30]: to maintain structure and semantics, we use the original pre-trained model for the first $t$ steps, setting the LoRA adaptor multipliers to 0 and retaining the pre-trained model priors. We then turn on the LoRA adaptor for the remaining steps.

One of the most interesting aspects of a large-scale generative model such as Stable Diffusion XL is that, although their image output can often suffer from distortions such as warped or blurry objects, the parameters of the model contains a latent capability to generate higher-quality output with fewer distortions than produced by default. Concept Sliders can unlock these abilities by identifying low-rank parameter directions that repair common distortions.

We analyze the two key components of our method to verify that they are both necessary: (1) the disentanglement formulation and (2) low-rank adaptation. Table 3 shows quantitative measures on 2500 images, and Figure 11 shows qualitative differences. In both quantitative and quantitative measures, we find that the disentanglement objective from Eq.8 success in isolating the edit from unwanted attributes (Fig.11.c); for example without this objective we see undesired changes in gender when asking for age as seen in Table 3, Interference metric which measures the percentage of samples with changed race/gender when making the edit. The low-rank constraint is also helpful: it has the effect of precisely capturing the edit direction with better generalization (Fig.11.d); for example, note how the background and the clothing are better preserved in Fig.11.b. Since LORA is parameter-efficient, it also has the advantage that it enables lightweight modularity. We also note that the SDEdit-inspired inference technique allows us to use a wider range of alpha values, increasing the editing capacity, without losing image structure. We find that SDEdit’s inference technique expands the usable range of alpha before coherence declines relative to the original image. We provide more details in the Appendix.
我们分析了我们方法的两个关键组成部分，以验证它们都是必要的：(1) 解缠公式和 (2) 低阶适应。表3显示了 2500 幅图像的定量测量结果，图11显示了定性差异。在定量和定量测量中，我们发现公式8中的解缠目标成功地将编辑与不需要的属性隔离开来（图11.c）；例如，如果没有这个目标，我们会在询问年龄时看到不需要的性别变化，如表3 所示，干扰度量用于测量编辑时种族/性别发生变化的样本百分比。低秩约束也很有帮助：它可以精确捕捉编辑方向，并具有更好的泛化效果（图11.d）；例如，请注意图11.b 中背景和服装是如何得到更好的保留的。我们还注意到，受 SDEdit 启发的推理技术允许我们在不丢失图像结构的情况下使用更大范围的 alpha 值，从而提高了编辑能力。我们发现，相对于原始图像，SDEdit 的推理技术在一致性下降之前扩大了 alpha 值的可用范围。我们将在附录中提供更多细节。

While the disentanglement formulation reduces unwanted interference between edits, we still observe some residual effects as shown in Table 3 for our sliders. This highlights the need for more careful selection of the latent directions to preserve, preferably an automated method, in order to further reduce edit interference. Further study is required to determine the optimal set of directions that minimizes interference while retaining edit fidelity. We also observe that while the inference SDEdit technique helps preserve image structure, it can reduce edit intensity compared to the inference-time method, as shown in Table 1. The SDEdit approach appears to trade off edit strength for improved structural coherence. Further work is needed to determine if the edit strength can be improved while maintaining high fidelity to the original image.
虽然解缠公式减少了编辑之间不必要的干扰，但我们仍然观察到一些残余效应，如表3中我们的滑块所示。这说明需要更仔细地选择要保留的潜在方向，最好是采用自动方法，以进一步减少编辑干扰。我们需要进一步研究，以确定既能最大限度减少干扰，又能保持编辑保真度的最佳方向集。我们还观察到，虽然推理 SDEdit 技术有助于保留图像结构，但与推理时方法相比，它可以降低编辑强度，如表1 所示。SDEdit 方法似乎是以编辑强度换取结构一致性的改善。要确定是否能在保持与原始图像高度保真度的同时提高编辑强度，还需要进一步的工作。

Concept Sliders are a simple and scalable new paradigm for interpretable control of diffusion models. By learning precise semantic directions in latent space, sliders enable intuitive and generalized control over image concepts. The approach provides a new level of flexiblilty beyond text-driven, image-specific diffusion model editing methods, because Concept Sliders allow continuous, single-pass adjustments without extra inference. Their modular design further enables overlaying many sliders simultaneously, unlocking complex multi-concept image manipulation.
概念滑块是对扩散模型进行可解释控制的一种简单、可扩展的新模式。通过学习潜在空间中的精确语义方向，滑块可以对图像概念进行直观和通用的控制。由于概念滑块可以进行连续的单次调整，无需额外的推理，因此这种方法提供了一种全新的灵活性，超越了文本驱动的、针对特定图像的扩散模型编辑方法。其模块化设计还能同时叠加多个滑块，从而实现复杂的多概念图像处理。

We have demonstrated the versatility of Concept Sliders by measuring their performance on Stable Diffusion XL and Stable Diffusion 1.4. We have found that sliders can be created from textual descriptions alone to control abstract concepts with minimal interference with unrelated concepts, outperforming previous methods. We have demonstrated and measured the efficacy of sliders for nuanced visual concepts that are difficult to describe by text, derived from small artist-created image datasets. We have shown that Concept Sliders can be used to transfer StyleGAN latents into diffusion models. Finally, we have conducted a human study that verifies the high quality of Concept Sliders that enhance and correct hand distortions. Our code and data will be made publicly available.
我们在 Stable Diffusion XL 和 Stable Diffusion 1.4 上测量了概念滑块的性能，从而证明了概念滑块的多功能性。我们发现，仅凭文字描述就能创建滑块，从而控制抽象概念，并将无关概念的干扰降到最低，这一点优于以往的方法。对于难以用文字描述的细微视觉概念，我们通过小型艺术家创作的图像数据集，展示并测量了滑块的功效。我们还证明了概念滑块可用于将 StyleGAN 潜在变量转移到扩散模型中。最后，我们还进行了一项人体研究，验证了概念滑块在增强和纠正手部变形方面的高品质。我们的代码和数据将公开发布。

We thank Jaret Burkett (aka Ostris) for the continued discussion on the image slider method and for sharing their eye size dataset. RG and DB are supported by Open Philanthropy.
我们感谢 Jaret Burkett（又名 Ostris）对图像滑块方法的持续讨论，并分享了他们的眼睛大小数据集。RG 和 DB 得到了 Open Philanthropy 的支持。

Our methods are available as open-source code. Source code, trained sliders, and data sets for reproducing our results can be found at sliders.baulab.info and at https://github.com/rohitgandikota/sliders.
我们的方法以开放源代码的形式提供。可在sliders.baulab.info和https://github.com/rohitgandikota/sliders 上找到源代码、训练有素的滑块以及用于重现我们的结果的数据集。