AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
AnyV2V：适用于任何视频到视频编辑任务的免调优框架

In this work, we focus on leveraging latent diffusion-based (Rombach et al., 2022) I2V generation models for video editing. Given an input first frame $I_{1}$, a text prompt $\mathbf{s}$ and a noisy video latent $\mathbf{z}_{t}$ at time step $t$, I2V generation models recover a less noisy latent $\mathbf{z}_{t-1}$ using a denoising model $\epsilon_{\theta}(\mathbf{z}_{t},I_{1},\mathbf{s},t)$ conditioned on both $I_{1}$ and $\mathbf{s}$. The denoising model $\epsilon_{\theta}$ contains a set of spatial and temporal self-attention layers, where the self-attention operation can be formulated as:
在这项工作中，我们专注于利用基于潜在扩散（Rombach et al.， 2022）的 I2V 生成模型进行视频编辑。给定输入第一帧 $I_{1}$ 、文本提示 $\mathbf{s}$ 和时间步 $\mathbf{z}_{t}$ $t$ 的噪声视频潜伏，I2V 生成模型 $\mathbf{z}_{t-1}$ 使用以 $I_{1}$ 和 $\mathbf{s}$ 为条件的去噪模型 $\epsilon_{\theta}(\mathbf{z}_{t},I_{1},\mathbf{s},t)$ 恢复噪声较小的潜伏。去噪模型 $\epsilon_{\theta}$ 包含一组空间和时间的自注意力层，其中自注意力作可以表述为：

Report issue for preceding element

	$\displaystyle Q=W^{Q}z,K=W^{K}z,V=W^{V}z,$		(1)
	$\displaystyle\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}(\frac{QK^{\top}}{\sqrt{d}})V,$		(2)

where $z$ is the input hidden state to the self-attention layer and $W^{Q}$, $W^{K}$ and $W^{V}$ are learnable projection matrices that map $z$ onto query, key and value vectors, respectively. For spatial self-attention, $z$ represents a sequence of spatial tokens from each frame. For temporal self-attention, $z$ is composed of tokens located at the same spatial position across all frames.
其中 $z$ 是自我注意层的输入隐藏状态 和 $W^{Q}$ ， $W^{K}$ 是 $W^{V}$ 可学习的投影矩阵，分别 $z$ 映射到 query、key 和 value 向量。对于空间自我注意， $z$ 表示来自每个帧的空间标记序列。对于时间自我注意， $z$ 它由位于所有帧中相同空间位置的标记组成。

Report issue for preceding element

The denoising process for I2V generation models from $\mathbf{z}_{t}$ to $\mathbf{z}_{t-1}$ can be achieved using the DDIM (Song et al., 2020) sampling algorithm. The reverse process of DDIM sampling, known as DDIM inversion (Mokady et al., 2023; Dhariwal & Nichol, 2021), allows obtaining $\mathbf{z}_{t+1}$ from $\mathbf{z}_{t}$ such that $\mathbf{z}_{t+1}=\sqrt{\frac{\alpha_{t+1}}{\alpha_{t}}}\mathbf{z}_{t}+(\sqrt{\frac{1}{\alpha_{t+1}}-1}-\sqrt{\frac{1}{\alpha_{t}}-1})\cdot\epsilon_{\theta}(\mathbf{z}_{t},x_{0},\mathbf{s},t)$, where $\alpha_{t}$ is derived from the variance schedule of the diffusion process.
I2V 生成模型的 $\mathbf{z}_{t}$ 去噪过程 $\mathbf{z}_{t-1}$ 可以使用 DDIM （Song et al.， 2020） 采样算法来实现。DDIM 采样的相反过程，称为 DDIM 反转（Mokady等 人，2023 年;Dhariwal & Nichol， 2021）允许从 $\mathbf{z}_{t}$ 中得到 $\mathbf{z}_{t+1}$ ， $\mathbf{z}_{t+1}=\sqrt{\frac{\alpha_{t+1}}{\alpha_{t}}}\mathbf{z}_{t}+(\sqrt{\frac{1}{\alpha_{t+1}}-1}-\sqrt{\frac{1}{\alpha_{t}}-1})\cdot\epsilon_{\theta}(\mathbf{z}_{t},x_{0},\mathbf{s},t)$ 其中 $\alpha_{t}$ 是从扩散过程的方差时间表中得出的。

Report issue for preceding element

Tumanyan et al. (2023a) proposed PnP diffusion features for image editing, based on the observation that intermediate convolution features $f$ and self-attention scores $A=\mathrm{Softmax}(\frac{QK^{\top}}{\sqrt{d}})$ in a text-to-image (T2I) denoising U-Net capture the semantic regions (e.g. legs or torso of a human body) during the image generation process.
Tumanyan 等 人 （2023a） 提出了用于图像编辑的 PnP 扩散特征，其依据是观察到文本到图像 （T2I） 去噪 U-Net 中的中间卷积特征 $f$ 和自注意力分数 $A=\mathrm{Softmax}(\frac{QK^{\top}}{\sqrt{d}})$ 在图像生成过程中捕获了语义区域（例如人体的腿部或躯干）。

Report issue for preceding element

Given an input source image $I^{S}$ and a target prompt $P$, PnP first performs DDIM inversion to obtain the image’s corresponding noise $\{\mathbf{z}^{S}_{t}\}_{t=1}^{T}$ at each time step $t$. It then collects the convolution features $\{f^{l}_{t}\}$ and attention scores $\{A^{l}_{t}\}$ from some predefined layers $l$ at each time step $t$ of the backward diffusion process $\mathbf{z}_{t-1}^{S}=\epsilon_{\theta}(\mathbf{z}_{t}^{S},\varnothing,t)$, where $\varnothing$ denotes the null text prompt during denoising.
给定一个输入源图像 $I^{S}$ 和一个目标提示 $P$ 符 ，PnP 首先执行 DDIM 反转，以获得图像在每个时间步 $t$ 的相应噪声 $\{\mathbf{z}^{S}_{t}\}_{t=1}^{T}$ 。然后，它在向后扩散过程 $\mathbf{z}_{t-1}^{S}=\epsilon_{\theta}(\mathbf{z}_{t}^{S},\varnothing,t)$ 的每个时间步 $t$ 从一些预定义的层 $l$ 中收集卷积特征 $\{f^{l}_{t}\}$ 和注意力分数 $\{A^{l}_{t}\}$ ，其中 $\varnothing$ 表示去噪期间的空文本提示。

Report issue for preceding element

To generate the edited image $I^{*}$, PnP starts from the initial noise of the source image (i.e. $\mathbf{z}_{T}^{*}=\mathbf{z}_{T}^{S}$) and performs feature injection during denoising: $\mathbf{z}_{t-1}^{*}=\epsilon_{\theta}(\mathbf{z}^{*}_{t},P,t,\{f^{l}_{t},A^{l}_{t}\})$, where $\epsilon_{\theta}(\cdot,\cdot,\cdot,\{f^{l}_{t},A^{l}_{t}\})$ represents the operation of replacing the intermediate feature and attention scores $\{f^{l*}_{t},A^{l*}_{t}\}$ with $\{f^{l}_{t},A^{l}_{t}\}$. This feature injection mechanism ensures $I^{*}$ to preserve the layout and structure from $I^{S}$ while reflecting the description in $P$. To control the feature injection strength, PnP also employs two thresholds $\tau_{f}$ and $\tau_{A}$ such that the feature and attention scores are only injected in the first $\tau_{f}$ and $\tau_{A}$ denoising steps. Our method extends this feature injection mechanism to I2V generation models, where we inject features in convolution, spatial, and temporal attention layers. We show the detailed design of AnyV2V in Section 4.
为了生成编辑后的图像 $I^{*}$ ，PnP 从源图像的初始噪声（即 $\mathbf{z}_{T}^{*}=\mathbf{z}_{T}^{S}$ ）开始，并在降噪过程中执行特征注入： $\mathbf{z}_{t-1}^{*}=\epsilon_{\theta}(\mathbf{z}^{*}_{t},P,t,\{f^{l}_{t},A^{l}_{t}\})$ ，其中 $\epsilon_{\theta}(\cdot,\cdot,\cdot,\{f^{l}_{t},A^{l}_{t}\})$ 表示将中间特征和注意力分数 $\{f^{l*}_{t},A^{l*}_{t}\}$ 替换为 $\{f^{l}_{t},A^{l}_{t}\}$ 的作。此功能注入机制可确保 $I^{*}$ 在反映 中的描述 $I^{S}$ 时保留布局和结构 $P$ 。为了控制特征注入强度，PnP 还采用了两个阈值 $\tau_{f}$ ， $\tau_{A}$ 这样特征和注意力分数仅在第一步 $\tau_{f}$ 和 $\tau_{A}$ 降噪步骤中注入。我们的方法将这种特征注入机制扩展到 I2V 生成模型，我们在卷积、空间和时间注意力层中注入特征。我们在第 4 节中展示了 AnyV2V 的详细设计。

Report issue for preceding element

In visual manipulation, controllability is a key element in performing precise editing. AnyV2V enables more controllable video editing by utilizing image editing models to modify the video’s first frame. This strategic approach enables highly accurate modifications in the video and is compatible with a broad spectrum of image editing models, including other deep learning models that can perform image style transfer (Gatys et al., 2015; Ghiasi et al., 2017; Lötzsch et al., 2022; Wang et al., 2024a), mask-based image editing (Nichol et al., 2022; Avrahami et al., 2022), image inpainting (Suvorov et al., 2021; Ku et al., 2022), identity-preserving image editing (Wang et al., 2024b), and subject-driven image editing (Chen et al., 2023c; Li et al., 2023; Gu et al., 2023a).
在视觉作中，可控性是执行精确编辑的关键要素。AnyV2V 通过利用图像编辑模型修改视频的第一帧，实现更可控的视频编辑。这种战略方法可以对视频进行高度准确的修改，并与广泛的图像编辑模型兼容，包括其他可以执行图像风格迁移的深度学习模型（Gatys等 人，2015 年;Ghiasi et al.， 2017;Lötzsch 等 人，2022 年;Wang et al.， 2024a）中，基于蒙版的图像编辑（Nichol et al.， 2022;Avrahami等 人，2022 年），图像修复（Suvorov等 人，2021 年;Ku et al.， 2022）、身份保留图像编辑（Wang et al.， 2024b）和主体驱动的图像编辑（Chen et al.， 2023c;Li et al.， 2023;Gu et al.， 2023a）。

Report issue for preceding element

To ensure the generated videos from the I2V generation model follow the general structure as presented in the source video, we employ DDIM inversion to obtain the latent noise of the source video at each time step $t$. Specifically, we perform the inversion without text prompt condition but with the first frame condition. Formally, given a source video $V^{S}=\{I_{1},I_{2},I_{3},...,I_{n}\}$, we obtain the inverted latent noise for time step $t$ as:
为了确保从 I2V 生成模型生成的视频遵循源视频中呈现的一般结构，我们采用 DDIM 反演来获取源视频在每个时间步 $t$ 的潜噪声。具体来说，我们在没有文本提示条件的情况下执行反转，但使用第一个帧条件。形式上，给定一个源视频 $V^{S}=\{I_{1},I_{2},I_{3},...,I_{n}\}$ ，我们得到时间步 $t$ 长的倒置潜在噪声：

Report issue for preceding element

$$\mathbf{z}^{S}_{t}=\mathrm{DDIM\_Inv}(\epsilon_{\theta}(\mathbf{z}_{t+1},I_{1},\varnothing,t)),$$

(3)

where $\text{DDIM\_Inv}(\cdot)$ denotes the DDIM inversion operation as described in Appendix 3. In ideal cases, the latent noise $\mathbf{z}^{S}_{T}$ at the final time step $T$ (initial noise of the source video) should be used as the initial noise for sampling the edited videos. In practice, we find that due to the limited capability of certain I2V models, the edited videos denoised from the last time step are sometimes distorted. Following Li et al. (2023), we observe that starting the sampling from a previous time step $T^{\prime}<T$ can be used as a simple workaround to fix this issue.
其中 $\text{DDIM\_Inv}(\cdot)$ ，表示附录 3 中描述的 DDIM 反演作。在理想情况下，应使用最后时间步 $T$ 的潜伏噪声 $\mathbf{z}^{S}_{T}$ （源视频的初始噪声）作为对编辑后的视频进行采样的初始噪声。在实践中，我们发现，由于某些 I2V 模型的能力有限，从最后一个时间步去噪的编辑视频有时会失真。根据 Li et al. （2023） 的说法，我们观察到从前一个时间步 $T^{\prime}<T$ 开始采样可以作为解决此问题的简单解决方法。

Report issue for preceding element

Our empirical observation (Section 5.4) suggests that I2V generation models already have some editing capabilities by only using the edited first frame and DDIM inverted noise as the model input. However, we find that this simple approach is often unable to correctly preserve the background in the edited first frame and the motion in the source video, as the conditional signal from the source video encoded in the inverted noise is limited.
我们的实证观察（第 5.4 节）表明，仅使用编辑后的第一帧和 DDIM 反转噪声作为模型输入，I2V 生成模型已经具有一些编辑功能。然而，我们发现这种简单的方法通常无法正确保留编辑后的第一帧中的背景和源视频中的运动，因为以倒置噪声编码的源视频的条件信号是有限的。

Report issue for preceding element

To enforce consistency with the source video, we perform feature injection in both convolution layers and spatial attention layers in the denoising U-Net. During the video sampling process, we simultaneously denoise the source video using the previously collected DDIM inverted latents $\mathbf{z}^{S}_{t}$ at each time step $t$ such that $\mathbf{z}^{S}_{t-1}=\epsilon_{\theta}(\mathbf{z}^{S}_{t},I_{1},\varnothing,t)$. We preserve two types of features during source video denoising: convolution features $f^{l_{1}}$ before skip connection from the $l_{1}^{\text{th}}$ residual block in the U-Net decoder, and the spatial self-attention scores $\{A_{s}^{l_{2}}\}$ from $l_{2}=\{l_{low},l_{low+1},...,l_{high}\}$ layers. We collect the queries $\{Q_{s}^{l_{2}}\}$ and keys $\{K_{s}^{l_{2}}\}$ instead of directly collecting $A_{s}^{l_{2}}$ as the attention score matrices are parameterized by the query and key vectors. We then replace the corresponding features during denoising the edited video in both the normal denoising branch and the negative prompt branch for classifier-free guidance (Ho & Salimans, 2022). We use two thresholds $\tau_{conv}$ and $\tau_{sa}$ to control the convolution and spatial attention injection to only happen in the first $\tau_{conv}$ and $\tau_{sa}$ steps during video sampling.
为了加强与源视频的一致性，我们在去噪 U-Net 的卷积层和空间注意力层中都执行特征注入。在视频采样过程中，我们在每个时间步 $t$ 长使用先前收集的 DDIM 反转潜伏 $\mathbf{z}^{S}_{t}$ 物同时对源视频进行去噪，以便 $\mathbf{z}^{S}_{t-1}=\epsilon_{\theta}(\mathbf{z}^{S}_{t},I_{1},\varnothing,t)$ .在源视频去噪过程中，我们保留了两种类型的特征：来自 U-Net 解码器中 $l_{1}^{\text{th}}$ 残差块的跳过连接之前的卷积特征 $f^{l_{1}}$ ，以及来自层的空间 $l_{2}=\{l_{low},l_{low+1},...,l_{high}\}$ 自我注意分数 $\{A_{s}^{l_{2}}\}$ 。我们收集查询 $\{Q_{s}^{l_{2}}\}$ 和键 $\{K_{s}^{l_{2}}\}$ ，而不是直接收集 $A_{s}^{l_{2}}$ ，因为注意力分数矩阵是由查询和键向量参数化的。然后，我们在正常降噪分支和否定提示分支中替换编辑后的视频时的相应特征，以实现无分类器的指导（Ho & Salimans））。我们使用两个阈值， $\tau_{conv}$ 并 $\tau_{sa}$ 控制卷积和空间注意力注入仅发生在视频采样期间的 first $\tau_{conv}$ 和 $\tau_{sa}$ steps。

Report issue for preceding element

The spatial feature injection mechanism described in Section 4.3 significantly enhances the background and overall structure consistency of the edited video. While it also helps maintain the source video motion to some degree, we observe that the edited videos will still have a high chance of containing incorrect motion compared to the source video. On the other hand, we notice that I2V generation models, or video diffusion models in general, are often initialized from pre-trained T2I models and continue to be trained on video data. During the training process, parameters in the spatial layers are often frozen or set to a lower learning rate such that the pre-trained weights from the T2I model are less affected, and the parameters in the temporal layers are more extensively updated during training. Therefore, it is likely that a large portion of the motion information is encoded in the temporal layers of the I2V generation models. Concurrent work (Bai et al., 2024) also observes that features in the temporal layers show similar characteristics with optical flow (Horn & Schunck, 1981), a pattern that is often used to describe the motion of the video.
4.3 节中描述的空间特征注入机制显着增强了编辑视频的背景和整体结构一致性。虽然它还在一定程度上有助于保持源视频的运动，但我们观察到，与源视频相比，编辑后的视频仍然很有可能包含不正确的运动。另一方面，我们注意到 I2V 生成模型或一般的视频扩散模型通常是从预先训练的 T2I 模型初始化的，并继续在视频数据上进行训练。在训练过程中，空间层中的参数通常会被冻结或设置为较低的学习率，以便 T2I 模型中的预训练权重受到的影响较小，并且时态层中的参数在训练期间得到更广泛的更新。因此，很大一部分运动信息很可能是在 I2V 生成模型的时间层中编码的。同时进行的工作（Bai 等人，2024）还观察到，时间层中的特征显示出与光流相似的特征（Horn & Schunck，1981），这种模式通常用于描述视频的运动。

Report issue for preceding element

To better reconstruct the source video motion in the edited video, we propose to also inject the temporal attention features in the video generation process. Similar to spatial attention injection, we collect the source video temporal self-attention queries $Q^{l_{3}}_{t}$ and keys $K^{l_{3}}_{t}$ from some U-Net decoder layers represented by $l_{3}$ and inject them into the edited video denoising branches. We also only apply temporal attention injection in the first $\tau_{ta}$ steps during sampling.
为了更好地重建编辑后的视频中的源视频运动，我们建议在视频生成过程中也注入时间注意力特征。与空间注意力注入类似，我们从以 U-Net 为代表的一些 U-Net 解码层中收集源视频时态自注意力查询 $Q^{l_{3}}_{t}$ 和键 $K^{l_{3}}_{t}$ ， $l_{3}$ 并将它们注入到编辑后的视频去噪分支中。我们也只在采样的第一步 $\tau_{ta}$ 应用时间注意力注入。

Report issue for preceding element

Overall, combining the spatial and temporal feature injection mechanisms, we replace the editing branch features $\{f^{*l_{1}},Q_{s}^{*l_{2}},K_{s}^{*l_{2}},Q_{t}^{*l_{3}},K_{t}^{*l_{3}}\}$ with the features from the source denoising branch:
总的来说，结合空间和时间特征注入机制，我们将编辑分支特征 $\{f^{*l_{1}},Q_{s}^{*l_{2}},K_{s}^{*l_{2}},Q_{t}^{*l_{3}},K_{t}^{*l_{3}}\}$ 替换为来自源降噪分支的特征：

Report issue for preceding element

$$\mathbf{z}^{*}_{t-1}=\epsilon_{\theta}(\mathbf{z}^{*}_{t},I^{*},\mathbf{s}^{*},t\ ;\{f^{l_{1}},Q_{s}^{l_{2}},K_{s}^{l_{2}},Q_{t}^{l_{3}},K_{t}^{l_{3}}\}),$$

(4)

where $\epsilon_{\theta}(\cdot\ ;\{f^{l_{1}},Q_{s}^{l_{2}},K_{s}^{l_{2}},Q_{t}^{l_{3}},K_{t}^{l_{3}}\})$ denotes the feature replacement operation across different layers $l_{1},l_{2},l_{3}$. Our proposed spatial and temporal feature injection scheme enables tuning-free adaptation of I2V generation models for video editing. Our experimental results demonstrate that each component in our design is crucial to the accurate editing of source videos. We showcase more qualitative results for the effectiveness of our model components in Section 5.
其中 $\epsilon_{\theta}(\cdot\ ;\{f^{l_{1}},Q_{s}^{l_{2}},K_{s}^{l_{2}},Q_{t}^{l_{3}},K_{t}^{l_{3}}\})$ 表示跨不同层 $l_{1},l_{2},l_{3}$ 的特征替换作。我们提出的空间和时间特征注入方案支持对 I2V 生成模型进行无调谐适应，用于视频编辑。我们的实验结果表明，我们设计中的每个组件对于源视频的准确编辑都至关重要。我们在第 5 节中展示了模型组件有效性的更多定性结果。

Report issue for preceding element

We employ AnyV2V on three off-the-shelf I2V generation models: I2VGen-XL¹¹1We use the version provided in https://huggingface.co/ali-vilab/i2vgen-xl. (Zhang et al., 2023c), ConsistI2V (Ren et al., 2024) and SEINE (Chen et al., 2023d). For all I2V models, we use $\tau_{conv}=0.2T$, $\tau_{sa}=0.2T$ and $\tau_{ta}=0.5T$, where $T$ is the total number of sampling steps. We use the DDIM (Song et al., 2020) sampler and set $T$ to the default values of the selected I2V models. Following PnP (Tumanyan et al., 2023b), we set $l_{1}=4$ for convolution feature injection and $l_{2}=l_{3}=\{4,5,6,...,11\}$ for spatial and temporal attention injections. During sampling, we apply text classifier-free guidance (CFG) (Ho & Salimans, 2022) for all models with the same negative prompt “Distorted, discontinuous, Ugly, blurry, low resolution, motionless, static, disfigured, disconnected limbs, Ugly faces, incomplete arms” across all edits. To obtain the initial edited frames in our implementation, we use a set of image editing model candidates including prompt-based image editing model InstructPix2Pix (Brooks et al., 2023), style transfer model Neural Style Transfer (NST) (Gatys et al., 2015), subject-driven image editing model AnyDoor (Chen et al., 2023c), and identity-driven image editing model InstantID (Wang et al., 2024b). We experiment with only the successfully edited frames, which is crucial for our method. We conducted all the experiments on a single Nvidia A6000 GPU. To edit a 16-frame video, it requires around 15G GPU memory and around 100 seconds for the whole inference process. We refer readers to Appendix A for more discussions on our implementation details and hyperparameter settings.
我们在三种现成的 I2V 生成模型上使用 AnyV2V：I2VGen-XL¹¹1 我们使用 https://huggingface.co/ali-vilab/i2vgen-xl.（Zhang et al.， 2023c）、ConsistI2V（任 et al.， 2024）和 SEINE（Chen et al.， 2023d）中提供的版本。对于所有 I2V 模型，我们使用 $\tau_{conv}=0.2T$ ， $\tau_{sa}=0.2T$ 和 $\tau_{ta}=0.5T$ ，其中 $T$ 是采样步骤的总数。我们使用 DDIM （Song et al.， 2020） 采样器并设置为 $T$ 所选 I2V 模型的默认值。在 PnP （Tumanyan et al.， 2023b） 之后，我们设置 $l_{1}=4$ 了卷积特征注入以及 $l_{2}=l_{3}=\{4,5,6,...,11\}$ 空间和时间注意力注入。在采样过程中，我们应用无文本分类器指导（CFG）（Ho & Salimans，2022）适用于所有模型，并在所有编辑中使用相同的负面提示“扭曲、不连续、丑陋、模糊、低分辨率、静止、毁容、断开的肢体、丑陋的面孔、不完整的手臂”。为了在我们的实现中获得初始编辑的帧，我们使用了一组图像编辑模型候选者，包括基于提示的图像编辑模型 InstructPix2Pix（Brooks等 人，2023 年）、风格迁移模型神经风格迁移 （NST）（Gatys等人 ，2015 年）、主题驱动的图像编辑模型 AnyDoor（Chen et al.， 2023c）和身份驱动的图像编辑模型 InstantID（Wang et al.， 2024b）。我们只对成功编辑的帧进行实验，这对我们的方法至关重要。我们在单个 Nvidia A6000 GPU 上进行了所有实验。 要编辑 16 帧的视频，它需要大约 15G 的 GPU 内存和大约 100 秒的整个推理过程。我们建议读者参考附录 A，以了解有关我们的实现细节和超参数设置的更多讨论。

Report issue for preceding element

• 1.

  Prompt-based Editing: allows users to manipulate video content using only natural language. This can include descriptive prompts or instructions. With the prompt, Users can perform a wide range of edits, such as incorporating accessories, spawning or swapping objects, adding effects, or altering the background.

  Report issue for preceding element
  
  1. 基于提示的编辑：允许用户仅使用自然语言作视频内容。这可以包括描述性提示或说明。通过提示，用户可以执行广泛的编辑，例如合并附件、生成或交换对象、添加效果或更改背景。
• 2.

  Reference-based Style Transfer: In the realm of style transfer tasks, the artistic styles of Monet and Van Gogh are frequently explored, but in real-life examples, users might want to use a distinct style based on one particular artwork. In reference-based style transfer, we focus on using a style image as a reference to perform video editing. The edited video should capture the distinct style of the referenced artwork.

  Report issue for preceding element
  
  2. 基于参考的风格迁移：在风格迁移任务领域，经常探索莫奈和梵高的艺术风格，但在现实生活中，用户可能希望根据一件特定的艺术品使用不同的风格。在基于引用的样式传输中，我们重点介绍使用样式图像作为参考来执行视频编辑。编辑后的视频应捕捉到引用图稿的独特风格。
• 3.

  Subject-driven Editing: In subject-driven video editing, we aim at replacing an object in the video with a target subject based on a given subject image while maintaining the video motion and persevering the background.

  Report issue for preceding element
  
  3. 主体驱动的编辑：在主体驱动的视频编辑中，我们的目标是根据给定的主体图像将视频中的对象替换为目标主体，同时保持视频运动和背景的持久性。
• 4.

  Identity Manipulation: Identity manipulation allows the user to manipulate video content by replacing a person with another person’s identity in the video based on an input image of the target person.

  Report issue for preceding element
  
  4. 身份纵：身份纵允许用户根据目标人的输入图像，通过在视频中将一个人替换为另一个人的身份来纵视频内容。

As shown in Figure 3 and 4, AnyV2V can perform the following video editing tasks: (1) prompt-based editing, (2) reference-based style transfer, (3) subject-driven editing, and (4) identity manipulation. we compare AnyV2V against three baseline models Tune-A-Video (Wu et al., 2023b), TokenFlow (Geyer et al., 2023) and FLATTEN (Cong et al., 2023) for (1) prompt-based editing. Since there exists no publicly available baseline method for task (2) (3) (4), we evaluate the performance of three I2V generation models under AnyV2V. We included a more comprehensive evaluation in Appendix B.
如图 3 和图 4 所示，AnyV2V 可以执行以下视频编辑任务：（1） 基于提示的编辑，（2） 基于参考的样式迁移，（3） 主题驱动的编辑，以及 （4） 身份作。我们将 AnyV2V 与三个基线模型 Tune-A-Video （Wu et al.， 2023b）、TokenFlow （Geyer et al.， 2023） 和 FLATTEN （Cong et al.， 2023） 进行了比较，以进行 （1） 基于提示的编辑。由于任务 （2） （3） （4） 没有公开可用的基线方法，我们评估了三种 I2V 生成模型在 AnyV2V 下的性能。我们在附录 B 中纳入了更全面的评估。

Report issue for preceding element

Effectiveness of Temporal Feature Injection According to the results, after disabling temporal feature injection in AnyV2V (I2VGen-XL), while we observe a slight increase in the CLIP-Image score value, the edited videos often demonstrate less adherence to the motion presented in the source video. For example, in the second frame of the “couple sitting” case (3${}^{\text{rd}}$ row, 2${}^{\text{nd}}$ column in the right panel in Figure 6), the motion of the woman raising her leg in the source video is not reflected in the edited video without applying temporal injection. On the other hand, even when the style of the video is completely changed, AnyV2V (I2VGen-XL) with temporal injection is still able to capture this nuance motion in the edited video.
时间特征注入的有效性 根据结果，在 AnyV2V （I2VGen-XL） 中禁用时间特征注入后，虽然我们观察到 CLIP-Image 分数值略有增加，但编辑后的视频通常表现出对源视频中呈现的运动的依从性较低。例如，在 “couple sitting” 案例的第二帧（图 6 右侧面板的 3 ${}^{\text{rd}}$ 行，2 ${}^{\text{nd}}$ 列）中，源视频中女性抬腿的动作在未应用时间注入的情况下不会反映在编辑后的视频中。另一方面，即使视频的风格完全改变，具有时间注入的 AnyV2V （I2VGen-XL） 仍然能够在编辑后的视频中捕捉到这种细微的动作。
Effectiveness of Spatial Feature Injection As shown in Table 3, we observe a drop in the CLIP-Image score after removing the spatial feature injection mechanisms from our model, indicating that the edited videos are not smoothly progressed across consecutive frames and contain more appearance and motion inconsistencies. Further illustrated in the third row of Figure 6, removing spatial feature injection will often result in incorrect subject appearance and pose (as shown in the “ballet dancing” case) and degenerated background appearance (evident in the “couple sitting” case). These observations demonstrate that directly generating edited videos from the DDIM inverted noise is often not enough to fully preserve the source video structures, and the spatial feature injection mechanisms are crucial for achieving better editing results.
空间特征注入的有效性 如表 3 所示，我们从模型中删除空间特征注入机制后观察到 CLIP-Image 分数下降，这表明编辑后的视频在连续帧中进展不顺畅，并且包含更多的外观和运动不一致。在图 6 的第三行中进一步说明，去除空间特征注入通常会导致不正确的主体外观和姿势（如 “ballet dance” 案例所示）和退化的背景外观（在 “couple sitting” 案例中很明显）。这些观察结果表明，直接从 DDIM 倒置噪声生成编辑后的视频通常不足以完全保留源视频结构，空间特征注入机制对于获得更好的编辑结果至关重要。

Report issue for preceding element

DDIM Inverted Noise as Structural Guidance Finally, we observe a further decrease in CLIP-Image scores and a significantly degraded visual appearance in both examples in Figure 6 after replacing the initial DDIM inverted noise with random noise during sampling. This indicates that the I2V generation models become less capable of animating the input image when the editing prompt is completely out-of-domain and highlights the importance of the DDIM inverted noise as the structural guidance of the edited videos.
DDIM 反向噪声作为结构导向 最后，在采样过程中用随机噪声替换初始 DDIM 反转噪声后，我们观察到图 6 中两个示例的 CLIP-Image 分数进一步下降，视觉外观明显下降。这表明当编辑提示完全超出域时，I2V 生成模型对输入图像进行动画处理的能力变得较弱，并突出了 DDIM 倒置噪声作为编辑视频的结构指导的重要性。

Report issue for preceding element