ArXiv | Poster | Video | Project

• The proposed style-disentangled Transformer (SDT) generates online handwritings with conditional content and style.
  提出的风格解耦 Transformer（SDT）能够根据条件内容与风格生成在线手写体。
• Existing RNN-based methods mainly focus on capturing a person’s overall writing style, neglecting subtle style inconsistencies between characters written by the same person. In light of this, SDT disentangles the writer-wise and character-wise style representations from individual handwriting samples for enhancing imitation performance.
  现有基于 RNN 的方法主要关注捕捉个人的整体书写风格，忽略了同一个人书写不同字符时存在的细微风格不一致性。针对这一问题，SDT 从单个手写样本中解耦了作者级和字符级风格表示，以增强模仿性能。
• We extend SDT and introduce an offline-to-offline framework for improving the generation quality of offline Chinese handwritings.
  我们扩展了 SDT，并引入了一个离线到在线的框架，用于提高离线中文手写体的生成质量。

Overview of our SDT

Three samples of online characters with writing orders
带有书写顺序的在线字符样本三例

• [2024/11/26] 🎉🎉🎉 Release of the implementations of Content Score and Style Score.
  [2024/11/26] 🎉🎉🎉 Content Score 和 Style Score 的实现已发布。
• [2024/07/01] 🎉🎉🎉 A new state-of-the-art method for handwritten text generation, named One-DM, is accepted by ECCV 2024.
  [2024/07/01] 🎉🎉🎉 一种名为 One-DM 的新手写文本生成尖端方法已被 ECCV 2024 录用。
• [2024/01/07] Add a tutorial and code for synthesizing handwriting with user-customized styles, more information can be found here.
  [2024/01/07] 添加了用于合成用户自定义风格手写体的教程和代码，更多信息可以在这里找到。
• [2023/12/15] 🎉🎉🎉 This work is reported by a top bilibili video blogger with 2.7 million followers and received nearly one million views.
  [2023/12/15] 🎉🎉🎉 该工作被一位拥有 270 万粉丝的顶级 bilibili 视频博主报道，并获得了近 100 万次观看。
• [2023/10/10] The author is invited to give a talk (in Chinese) by CSIG (China Society of Image and Graphics).
  [2023/10/10] 作者受邀由中国图象图形学学会（CSIG）进行了一场中文演讲。
• [2023/06/14] This work is reported by Synced (机器之心).
  [2023/06/14] 本工作由 Synced（机器之心）报道。
• [2023/04/12] Initial release of the datasets, pre-trained models, training and testing codes.
  [2023/04/12] 数据集、预训练模型、训练和测试代码的初始发布。
• [2023/02/28] 🎉🎉🎉 Our SDT is accepted by CVPR 2023.
  [2023/02/28] 🎉🎉🎉 我们的 SDT 被 CVPR 2023 录用。

• Online Chinese handwriting generation
  在线中文手写生成

• Applications to various scripts
  应用于多种脚本

• Extension on offline Chinese handwriting generation
  扩展离线中文手写生成

conda create -n sdt python=3.8 -y
conda activate sdt
# install all dependencies
conda env create -f environment.yml

SDT/
│
├── train.py - main script to start training
├── test.py - generate characters via trained model
├── evaluate.py - evaluation of generated samples
│
├── configs/*.yml - holds configuration for training
├── parse_config.py - class to handle config file
│
├── data_loader/ - anything about data loading goes here
│   └── loader.py
│
├── model_zoo/ - pre-trained content encoder model
│
├── data/ - default directory for storing experimental datasets
│
├── model/ - networks, models and losses
│   ├── encoder.py
│   ├── gmm.py
│   ├── loss.py
│   ├── model.py
│   └── transformer.py
│
├── saved/
│   ├── models/ - trained models are saved here
│   ├── tborad/ - tensorboard visualization
│   └── samples/ - visualization samples in the training process
│
├── trainer/ - trainers
│   └── trainer.py
│  
└── utils/ - small utility functions
    ├── util.py
    └── logger.py - set log dir for tensorboard and logging output

We provide Chinese, Japanese and English datasets in Google Drive | Baidu Netdisk PW:xu9u. Please download these datasets, uzip them and move the extracted files to /data.
我们提供了中文、日文和英文数据集，存储在 Google Drive | Baidu Netdisk PW:xu9u。请下载这些数据集，解压并将解压后的文件移动到/data 目录。

Note: Please download these weights, and move them to /model_zoo.
注意：请下载这些权重，并将它们移动到 /model_zoo。

Training  训练

• To train the SDT on the Chinese dataset, run this command:
  要在中文数据集上训练 SDT，请运行此命令：

python train.py --cfg configs/CHINESE_CASIA.yml --log Chinese_log

• To train the SDT on the Japanese dataset, run this command:
  训练 SDT 模型使用日语数据集，运行以下命令：

python train.py --cfg configs/Japanese_TUATHANDS.yml --log Japanese_log

• To train the SDT on the English dataset, run this command:
  训练 SDT 模型使用英语数据集，运行以下命令：

python train.py --cfg configs/English_CASIA.yml --log English_log

Qualitative Test  定性测试

• To generate online Chinese handwritings with our SDT, run this command:
  要使用我们的 SDT 模型生成在线中文手写体，运行以下命令：

python test.py --pretrained_model checkpoint_path --store_type online --sample_size 500 --dir Generated/Chinese

• To generate offline Chinese handwriting images with our SDT, run this command:
  使用我们的 SDT 生成离线中文手写图像，运行以下命令：

python test.py --pretrained_model checkpoint_path --store_type offline --sample_size 500 --dir Generated_img/Chinese

• To generate online Japanese handwritings with our SDT, run this command:
  使用我们的 SDT 生成在线日文手写，运行以下命令：

python test.py --pretrained_model checkpoint_path --store_type online --sample_size 500 --dir Generated/Japanese

• To generate offline Japanese handwriting images with our SDT, run this command:
  使用我们的 SDT 生成离线日文手写图像，运行以下命令：

python test.py --pretrained_model checkpoint_path --store_type offline --sample_size 500 --dir Generated_img/Japanese

• To generate online English handwritings with our SDT, run this command:
  使用我们的 SDT 生成在线英文手写，运行以下命令：

python test.py --pretrained_model checkpoint_path --store_type online --sample_size 500 --dir Generated/English

• To generate offline English handwriting images with our SDT, run this command:
  要使用我们的 SDT 生成离线英文手写图像，请运行以下命令：

python test.py --pretrained_model checkpoint_path --store_type offline --sample_size 500 --dir Generated_img/English

Quantitative Evaluation  定量评估

• To evaluate the generated handwritings, you need to set data_path to the path of the generated handwritings (e.g., Generated/Chinese), and run this command:
  要评估生成的手写体，您需要将 data_path 设置为生成手写体的路径（例如，Generated/Chinese），然后运行以下命令：

python evaluate.py --data_path Generated/Chinese --metric DTW

• To calculate the Content Score of generated handwritings, you need to set data_path to the path of the generated handwritings (e.g., Generated/Chinese), and run this command:
  要计算生成手写体的内容得分，您需要将 data_path 设置为生成手写体的路径（例如，Generated/Chinese），然后运行以下命令：

python evaluate.py --data_path Generated/Chinese --metric Content_score --pretrained_model model_zoo/chinese_content_iter30k_acc95.pth

• To calculate the Style Score of generated handwritings, you need to set data_path to the path of the generated handwriting images (e.g., Generated_img/Chinese), and run this command:

python evaluate.py --data_path Generated_img/Chinese --metric Style_score --pretrained_model models_zoo/chinese_style_iter60k_acc999.pth

We are delighted to discover that P0etry-rain has proposed a pipeline that involves initially converting the generated results by our SDT to TTF format, followed by the development of software to enable flexible adjustments in spacing between paragraphs, lines, and characters. Below, we present TTF files, software interface and the printed results. More details can be seen in #78.
我们很高兴地发现 P0etry-rain 提出了一个流程，该流程首先将我们 SDT 生成的结果转换为 TTF 格式，然后开发软件以实现段落、行和字符间距的灵活调整。下面我们展示了 TTF 文件、软件界面和打印结果。更多细节请参见 #78。

• TTF File  TTF 文件

• Software Interface  软件界面

• Printed Results  打印结果

If you find our work inspiring or use our codebase in your research, please cite our work:
如果我们的工作对你有所启发或你在研究中使用了我们的代码库，请引用我们的工作：

@inproceedings{dai2023disentangling,
  title={Disentangling Writer and Character Styles for Handwriting Generation},
  author={Dai, Gang and Zhang, Yifan and Wang, Qingfeng and Du, Qing and Yu, Zhuliang and Liu, Zhuoman and Huang, Shuangping},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
  pages={5977--5986},
  year={2023}
}