掩码语言建模

掩码语言建模（Masked language modeling）预测序列中被掩码的 token，模型可以双向关注 token。这意味着模型可以完全访问左侧和右侧的 token。掩码语言建模非常适合需要对整个序列进行良好上下文理解的任务。BERT 就是一个掩码语言模型的例子。

本指南将向您展示如何：

在 ELI5 数据集的 r/askscience 子集上微调 DistilRoBERTa。
使用微调后的模型进行推理。

要查看与此任务兼容的所有架构和检查点，我们建议查阅任务页面

在开始之前，请确保已安装所有必要的库：

pip install transformers datasets evaluate

我们建议您登录您的 Hugging Face 账户，以便上传并与社区共享您的模型。当系统提示时，请输入您的令牌以完成登录：

>>> from huggingface_hub import notebook_login

>>> notebook_login()

加载 ELI5 数据集

首先，使用 🤗 Datasets 库加载 ELI5-Category 数据集的前 5000 个样本。这样你可以先进行实验，确保一切正常运行，然后再花更多时间在完整数据集上进行训练。

>>> from datasets import load_dataset

>>> eli5 = load_dataset("dany0407/eli5_category", split="train[:5000]")

使用 train_test_split 方法将数据集的 train 子集拆分为训练集和测试集：

>>> eli5 = eli5.train_test_split(test_size=0.2)

然后来看一个示例：

>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
  'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
  'score': [21, 19, 5, 3],
  'text_urls': [[],
   [],
   [],
   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},
 'title_urls': ['url'],
 'selftext_urls': ['url']}

虽然看起来内容很多，但你真正关心的其实只有 text 字段。语言建模任务的妙处在于，你不需要标签（这也是一种无监督任务），因为下一个词就是标签。

预处理

对于掩码语言建模，下一步是加载 DistilRoBERTa 分词器以处理 text 子字段：

>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilroberta-base")

从上面的示例中可以看出，text 字段实际上嵌套在 answers 内部。这意味着你需要使用 flatten 方法从嵌套结构中提取 text 子字段：

>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
 'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
 'answers.score': [21, 19, 5, 3],
 'answers.text_urls': [[],
  [],
  [],
  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
 'title_urls': ['url'],
 'selftext_urls': ['url']}

现在，每个子字段都变成了独立的列，如 answers 前缀所示，并且 text 字段现在是一个列表。与其分别对每个句子进行分词，不如将该列表转换为字符串，以便对它们进行联合分词。

下面是一个初步的预处理函数，用于将每个示例中的字符串列表拼接起来，并对结果进行分词：

>>> def preprocess_function(examples):
...     return tokenizer([" ".join(x) for x in examples["answers.text"]])

要对整个数据集应用此预处理函数，请使用 🤗 Datasets 的 map 方法。您可以通过设置 batched=True 来一次处理数据集中的多个元素，从而加速 map 函数，并使用 num_proc 增加进程数量。删除您不需要的任何列：

>>> tokenized_eli5 = eli5.map(
...     preprocess_function,
...     batched=True,
...     num_proc=4,
...     remove_columns=eli5["train"].column_names,
... )

该数据集包含 token 序列，但其中一些序列的长度超过了模型的最大输入长度。

现在你可以使用第二个预处理函数来

将所有序列拼接在一起
将拼接后的序列拆分为由 block_size 定义的较短片段，该值应既小于最大输入长度，又足够短以适配您的 GPU 显存。

>>> block_size = 128


>>> def group_texts(examples):
...     # Concatenate all texts.
...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
...     total_length = len(concatenated_examples[list(examples.keys())[0]])
...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
...     # customize this part to your needs.
...     if total_length >= block_size:
...         total_length = (total_length // block_size) * block_size
...     # Split by chunks of block_size.
...     result = {
...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
...         for k, t in concatenated_examples.items()
...     }
...     return result

对整个数据集应用 group_texts 函数：

>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)

现在使用 DataCollatorForLanguageModeling 创建一个示例批次。在整理数据时，动态填充 句子至批次中最长句子的长度，比将整个数据集填充到最大长度更高效。

使用序列结束标记作为填充标记，并指定 mlm_probability，以便在每次遍历数据时随机遮盖标记：

>>> from transformers import DataCollatorForLanguageModeling

>>> tokenizer.pad_token = tokenizer.eos_token
>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)

训练

如果您不熟悉使用 Trainer 对模型进行微调，请查看此处的基础教程！

现在您可以开始训练模型了！使用 AutoModelForMaskedLM 加载 DistilRoBERTa：

>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("distilbert/distilroberta-base")

此时，只剩下三个步骤：

在 TrainingArguments 中定义训练超参数。唯一必需的参数是 output_dir，用于指定模型的保存位置。通过设置 push_to_hub=True，您可以将该模型推送到 Hub（上传模型需要登录 Hugging Face 账户）。
将训练参数与模型、数据集和数据整理器一起传递给 Trainer。
调用 train() 来微调你的模型。

>>> training_args = TrainingArguments(
...     output_dir="my_awesome_eli5_mlm_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=lm_dataset["train"],
...     eval_dataset=lm_dataset["test"],
...     data_collator=data_collator,
...     processing_class=tokenizer,
... )

>>> trainer.train()

训练完成后，使用 evaluate() 方法评估模型并获取其困惑度：

>>> import math

>>> eval_results = trainer.evaluate()
>>> print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")
Perplexity: 8.76

然后使用 push_to_hub() 方法将您的模型共享到 Hub，以便所有人都能使用您的模型：

>>> trainer.push_to_hub()

如需了解如何针对掩码语言建模微调模型的更详细示例，请参阅对应的 PyTorch notebook。

推理

很好，现在你已经微调了一个模型，可以使用它进行推理了！

想出一些你希望模型来补全空白的文本，并使用特殊的 <mask> 标记来表示空白：

>>> text = "The Milky Way is a <mask> galaxy."

尝试使用微调后的模型进行推理的最简单方法是将其用于 pipeline()。使用你的模型实例化一个用于 fill-mask 的 pipeline，并将你的文本传入。如果需要，你可以使用 top_k 参数来指定要返回多少个预测结果：

>>> from transformers import pipeline

>>> mask_filler = pipeline("fill-mask", "username/my_awesome_eli5_mlm_model")
>>> mask_filler(text, top_k=3)
[{'score': 0.5150994658470154,
  'token': 21300,
  'token_str': ' spiral',
  'sequence': 'The Milky Way is a spiral galaxy.'},
 {'score': 0.07087188959121704,
  'token': 2232,
  'token_str': ' massive',
  'sequence': 'The Milky Way is a massive galaxy.'},
 {'score': 0.06434620916843414,
  'token': 650,
  'token_str': ' small',
  'sequence': 'The Milky Way is a small galaxy.'}]

对文本进行分词，并将 input_ids 作为 PyTorch 张量返回。您还需要指定 <mask> 标记的位置：

>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> inputs = tokenizer(text, return_tensors="pt")
>>> mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]

将输入传递给模型，并返回被掩码 token 的 logits：

>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> logits = model(**inputs).logits
>>> mask_token_logits = logits[0, mask_token_index, :]

然后返回概率最高的三个被掩码的 token，并将其打印出来：

>>> top_3_tokens = torch.topk(mask_token_logits, 3, dim=1).indices[0].tolist()

>>> for token in top_3_tokens:
...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))
The Milky Way is a spiral galaxy.
The Milky Way is a massive galaxy.
The Milky Way is a small galaxy.

Update on GitHub

Transformers

掩码语言建模

加载 ELI5 数据集

预处理

训练

推理