Train with PyTorch Trainer

🤗 Transformers provides a Trainer class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The Trainer API supports a wide range of training options and features such as logging, gradient accumulation, and mixed precision.

Start by loading your model and specify the number of expected labels. From the Yelp Review dataset card, you know there are five labels.

By default, the weights are loaded in full precision (torch.float32) regardless of the actual data type the weights are stored in such as torch.float16. Set torch_dtype="auto" to load the weights in the data type defined in a model’s config.json file to automatically load the most memory-optimal data type.

>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5, torch_dtype="auto")

Training hyperparameters

Next, create a TrainingArguments class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training hyperparameters, but feel free to experiment with these to find your optimal settings.

Specify where to save the checkpoints from your training:

>>> from transformers import TrainingArguments

>>> training_args = TrainingArguments(output_dir="test_trainer")

Evaluate

Trainer does not automatically evaluate model performance during training. You’ll need to pass Trainer a function to compute and report metrics. The 🤗 Evaluate library provides a simple accuracy function you can load with the evaluate.load (see this quicktour for more information) function:

>>> import numpy as np
>>> import evaluate

>>> metric = evaluate.load("accuracy")

Call compute on metric to calculate the accuracy of your predictions. Before passing your predictions to compute, you need to convert the logits to predictions (remember all 🤗 Transformers models return logits):

>>> def compute_metrics(eval_pred):
...     logits, labels = eval_pred
...     predictions = np.argmax(logits, axis=-1)
...     return metric.compute(predictions=predictions, references=labels)

If you’d like to monitor your evaluation metrics during fine-tuning, specify the eval_strategy parameter in your training arguments to report the evaluation metric at the end of each epoch:

>>> from transformers import TrainingArguments, Trainer

>>> training_args = TrainingArguments(output_dir="test_trainer", eval_strategy="epoch")

Trainer

Create a Trainer object with your model, training arguments, training and test datasets, and evaluation function:

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... )

Then fine-tune your model by calling train():

>>> trainer.train()

Train a TensorFlow model with Keras
训练 TensorFlow 模型使用 Keras

You can also train 🤗 Transformers models in TensorFlow with the Keras API!
您还可以使用 Keras API 在 TensorFlow 中训练🤗 Transformers 模型！

Loading data for Keras  加载数据以供 Keras 使用

When you want to train a 🤗 Transformers model with the Keras API, you need to convert your dataset to a format that Keras understands. If your dataset is small, you can just convert the whole thing to NumPy arrays and pass it to Keras. Let’s try that first before we do anything more complicated.
当你想使用 Keras API 训练🤗 Transformers 模型时，你需要将你的数据集转换为 Keras 理解的格式。如果你的数据集很小，你可以直接将其全部转换为 NumPy 数组并传递给 Keras。在我们进行更复杂的事情之前，让我们先试试这个方法。

First, load a dataset. We’ll use the CoLA dataset from the GLUE benchmark, since it’s a simple binary text classification task, and just take the training split for now.
首先，加载一个数据集。我们将使用 GLUE 基准测试中的 CoLA 数据集，因为它是一个简单的二分类文本分类任务，现在只取训练集。

from datasets import load_dataset

dataset = load_dataset("glue", "cola")
dataset = dataset["train"]  # Just take the training split for now

Next, load a tokenizer and tokenize the data as NumPy arrays. Note that the labels are already a list of 0 and 1s, so we can just convert that directly to a NumPy array without tokenization!
接下来，加载一个分词器并将数据分词成 NumPy 数组。注意，标签已经是 0 和 1 的列表，所以我们可以直接将其转换为 NumPy 数组，无需分词！

from transformers import AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")
tokenized_data = tokenizer(dataset["sentence"], return_tensors="np", padding=True)
# Tokenizer returns a BatchEncoding, but we convert that to a dict for Keras
tokenized_data = dict(tokenized_data)

labels = np.array(dataset["label"])  # Label is already an array of 0 and 1

Finally, load, compile, and fit the model. Note that Transformers models all have a default task-relevant loss function, so you don’t need to specify one unless you want to:
最后，加载并微调模型。请注意，所有 Transformers 模型都有一个默认与任务相关的损失函数，除非您想指定一个，否则不需要指定。

from transformers import TFAutoModelForSequenceClassification
from tensorflow.keras.optimizers import Adam

# Load and compile our model
model = TFAutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased")
# Lower learning rates are often better for fine-tuning transformers
model.compile(optimizer=Adam(3e-5))  # No loss argument!

model.fit(tokenized_data, labels)

This approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why? Because the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn’t handle “jagged” arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole dataset. That’s going to make your array even bigger, and all those padding tokens will slow down training too!
这种方法对于较小的数据集效果很好，但对于较大的数据集，你可能会发现它开始变成一个问题。为什么？因为标记后的数组和标签必须全部加载到内存中，而且 NumPy 不处理“锯齿形”数组，所以每个标记后的样本都必须填充到整个数据集中最长样本的长度。这将使你的数组更大，而且所有这些填充标记也会减慢训练速度！

Loading data as a tf.data.Dataset
加载数据为 tf.data.Dataset

If you want to avoid slowing down training, you can load your data as a tf.data.Dataset instead. Although you can write your own tf.data pipeline if you want, we have two convenience methods for doing this:
如果您想避免减慢训练速度，您可以将数据作为 tf.data.Dataset 加载。尽管您可以选择编写自己的 tf.data 管道，但我们为此提供了两种便捷方法：

• prepare_tf_dataset(): This is the method we recommend in most cases. Because it is a method on your model, it can inspect the model to automatically figure out which columns are usable as model inputs, and discard the others to make a simpler, more performant dataset.
  准备 tf_dataset(): 这是我们推荐在大多数情况下使用的方法。因为它是在您的模型上的一个方法，它可以检查模型以自动确定哪些列可以作为模型输入，并丢弃其他列以创建一个更简单、性能更好的数据集。
• to_tf_dataset: This method is more low-level, and is useful when you want to exactly control how your dataset is created, by specifying exactly which columns and label_cols to include.
  to_tf_dataset：此方法更底层，当您需要精确控制数据集的创建方式，指定确切要包含的 columns 和 label_cols 时，非常有用。

Before you can use prepare_tf_dataset(), you will need to add the tokenizer outputs to your dataset as columns, as shown in the following code sample:
在您可以使用 prepare_tf_dataset() 之前，您需要将分词器输出添加到您的数据集作为列，如下代码示例所示：

def tokenize_dataset(data):
    # Keys of the returned dictionary will be added to the dataset as columns
    return tokenizer(data["text"])


dataset = dataset.map(tokenize_dataset)

Remember that Hugging Face datasets are stored on disk by default, so this will not inflate your memory usage! Once the columns have been added, you can stream batches from the dataset and add padding to each batch, which greatly reduces the number of padding tokens compared to padding the entire dataset.
记住，Hugging Face 数据集默认存储在磁盘上，所以这不会增加你的内存使用！一旦添加了列，您可以从数据集中流式传输批次并添加填充到每个批次，这比填充整个数据集大大减少了填充标记的数量。

>>> tf_dataset = model.prepare_tf_dataset(dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer)

Note that in the code sample above, you need to pass the tokenizer to prepare_tf_dataset so it can correctly pad batches as they’re loaded. If all the samples in your dataset are the same length and no padding is necessary, you can skip this argument. If you need to do something more complex than just padding samples (e.g. corrupting tokens for masked language modelling), you can use the collate_fn argument instead to pass a function that will be called to transform the list of samples into a batch and apply any preprocessing you want. See our examples or notebooks to see this approach in action.
注意，在上面的代码示例中，您需要将分词器传递给 prepare_tf_dataset ，以便它可以在加载时正确地填充批次。如果您的数据集中的所有样本长度相同且不需要填充，则可以跳过此参数。如果您需要执行比仅填充样本更复杂的操作（例如，对掩码语言模型中的标记进行损坏），则可以使用 collate_fn 参数传递一个函数，该函数将被调用来将样本列表转换为批次并应用您想要的任何预处理。请参阅我们的示例或笔记本，以查看此方法的实际应用。

Once you’ve created a tf.data.Dataset, you can compile and fit the model as before:
一旦创建了 tf.data.Dataset ，您就可以像以前一样编译和调整模型：

model.compile(optimizer=Adam(3e-5))  # No loss argument!

model.fit(tf_dataset)

Trainer takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a 🤗 Transformers model in native PyTorch.

At this point, you may need to restart your notebook or execute the following code to free some memory:

from accelerate.utils.memory import clear_device_cache
del model
del trainer
clear_device_cache()

Next, manually postprocess tokenized_dataset to prepare it for training.

1. Remove the text column because the model does not accept raw text as an input:

   Copied

   >>> tokenized_datasets = tokenized_datasets.remove_columns(["text"])

2. Rename the label column to labels because the model expects the argument to be named labels:

   Copied

   >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels")

3. Set the format of the dataset to return PyTorch tensors instead of lists:

   Copied

   >>> tokenized_datasets.set_format("torch")

Then create a smaller subset of the dataset as previously shown to speed up the fine-tuning:

>>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))

DataLoader

Create a DataLoader for your training and test datasets so you can iterate over batches of data:

>>> from torch.utils.data import DataLoader

>>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)
>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)

Load your model with the number of expected labels:

>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5)

Optimizer and learning rate scheduler

Create an optimizer and learning rate scheduler to fine-tune the model. Let’s use the AdamW optimizer from PyTorch:

>>> from torch.optim import AdamW

>>> optimizer = AdamW(model.parameters(), lr=5e-5)

Create the default learning rate scheduler from Trainer:

>>> from transformers import get_scheduler

>>> num_epochs = 3
>>> num_training_steps = num_epochs * len(train_dataloader)
>>> lr_scheduler = get_scheduler(
...     name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
... )

Lastly, specify device to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.

>>> import torch
>>> from accelerate.test_utils.testing import get_backend

>>> device, _, _ = get_backend() # automatically detects the underlying device type (CUDA, CPU, XPU, MPS, etc.)
>>> model.to(device)

Great, now you are ready to train! 🥳

Training loop

To keep track of your training progress, use the tqdm library to add a progress bar over the number of training steps:

>>> from tqdm.auto import tqdm

>>> progress_bar = tqdm(range(num_training_steps))

>>> model.train()
>>> for epoch in range(num_epochs):
...     for batch in train_dataloader:
...         batch = {k: v.to(device) for k, v in batch.items()}
...         outputs = model(**batch)
...         loss = outputs.loss
...         loss.backward()

...         optimizer.step()
...         lr_scheduler.step()
...         optimizer.zero_grad()
...         progress_bar.update(1)

Evaluate

Just like how you added an evaluation function to Trainer, you need to do the same when you write your own training loop. But instead of calculating and reporting the metric at the end of each epoch, this time you’ll accumulate all the batches with add_batch and calculate the metric at the very end.

>>> import evaluate

>>> metric = evaluate.load("accuracy")
>>> model.eval()
>>> for batch in eval_dataloader:
...     batch = {k: v.to(device) for k, v in batch.items()}
...     with torch.no_grad():
...         outputs = model(**batch)

...     logits = outputs.logits
...     predictions = torch.argmax(logits, dim=-1)
...     metric.add_batch(predictions=predictions, references=batch["labels"])

>>> metric.compute()