用biobert标记基因和蛋白质_weixin_26704853的博客-CSDN博客

The dataset used in this example is a combination of the BioNLP09 Corpus, BioNLP11 Corpus, and BioNLP13 Corpus. Although we focus on genes and proteins, there are other entity types such as diseases, chemicals, and drugs. There are 74 tags included in these experiments, but for the sake of brevity, here is a peek at the BioNLP13 Corpus tags in the BIO schema:

本示例中使用的数据集是BioNLP09语料库 BioNLP11语料库和BioNLP13语料库的组合 。 尽管我们专注于基因和蛋白质 但还有其他实体类型 例如疾病 化学物质和药物。 这些实验中包含74个标签 但是为了简洁起见 下面是BIO模式中的BioNLP13 Corpus标签

 B-Anatomical_system ,
 B-Cancer ,
 B-Cell , 
 B-Cellular_component ,
 B-Developing_anatomical_structure ,
 B-Gene_or_gene_product , 
 B-Immaterial_anatomical_entity ,
 B-Multi-tissue_structure ,
 B-Organ ,
 B-Organism , 
 B-Organism_subdivision ,
 B-Organism_substance ,
 B-Pathological_formation , 
 B-Simple_chemical ,
 B-Tissue ,
 I-Amino_acid ,
 I-Anatomical_system ,
 I-Cancer , 
 I-Cell ,
 I-Cellular_component ,
 I-Developing_anatomical_structure ,
 I-Gene_or_gene_product , 
 I-Immaterial_anatomical_entity ,
 I-Multi-tissue_structure ,
 I-Organ ,
 I-Organism , 
 I-Organism_subdivision ,
 I-Organism_substance ,
 I-Pathological_formation ,
 I-Simple_chemical , 
 I-Tissue ,
 O 

IV。 实作 (IV. Implementation)

First, we will want to import BioBERT from the original GitHub and transfer the files to our Colab notebook. Here we are downloading the main BioBERT file, extracting the BioBERT weights, and converting them to be applicable in PyTorch so as to work with the HuggingFace API. We move the config file for simplicity, and now are good to go!

首先 我们要从原始GitHub导入BioBERT 然后将文件传输到我们的Colab笔记本中。 在这里 我们正在下载主要的BioBERT文件 提取BioBERT权重 并将其转换为适用于PyTorch的格式 以便与HuggingFace API一起使用 。 为了简单起见 我们移动了配置文件 现在一切顺利

You will need the transformers libraries from HuggingFace. For the full list of installs and imports, consult my notebook. We set the maximum length of text and batch size as constraints we will be working with later. We will also create a device that utilizes GPUs for computation in Colab. The BertTokenizer class will take in the vocab.txt from the BioBERT file we’ve previously set up.

您将需要HuggingFace的变压器库。 有关安装和导入的完整列表 请参阅我的笔记本 。 我们将文本的最大长度和批处理大小设置为稍后将要使用的约束。 我们还将创建一种利用GPU在Colab中进行计算的设备。 BertTokenizer类将从我们之前设置的BioBERT文件中获取vocab.txt。

The SentenceFetch class will take in our data, read the TSV files that the BIO schema text comes in, and organize the sentences and tags into workable data structures. We then have methods to retrieve the sentences and tags.

SentenceFetch类将接收我们的数据 读取BIO模式文本所包含的TSV文件 并将句子和标签组织成可行的数据结构。 然后 我们有了检索句子和标签的方法。

We search through all the subdirectories of our root directory. In Colab, I would suggest uploading your data from either Google Drive or your local drive. We use the SentenceFetch class and create a list of sentences and tags to use in our experiments.

我们搜索根目录的所有子目录。 在Colab中 建议您从Google云端硬盘或本地硬盘上传数据。 我们使用SentenceFetch类并创建要在实验中使用的句子和标签的列表。

We create a helper function to tokenize the text without losing the labels to each token. We need those labels intact for our model.

我们创建了一个辅助函数来标记文本 而不会丢失每个标记的标签。 我们需要完整的模型标签。

Now we can get our input data normalized via “pad_sequences” in Keras. This is used to keep the fixed-length consistent with sequences that are shorter than our maximum length.

现在 我们可以通过Keras中的“ pad_sequences”将输入数据标准化。 这用于使固定长度与比我们的最大长度短的序列一致。

Now we can finalize the data preparation for our modeling. Attention masks are used when batching sequences together to indicate which tokens should be observed. We split our inputs and masks between training and validation data. Then we convert our data as tensors to work properly with PyTorch. Afterward, we pass these tensors through the data utils in PyTorch and finally have the data ready for our model.

现在 我们可以完成建模的数据准备。 将序列批处理在一起时 使用注意掩码来指示应观察哪些标记。 我们在训练和验证数据之间划分了输入和掩码。 然后 我们将数据转换为张量以与PyTorch一起正常工作。 之后 我们将这些张量通过PyTorch中的数据工具传递 最后为我们的模型准备好数据。

In the modeling stage, we use the BertConfig class to use the configuration file for the model. We also use the pre-trained weights to help establish our “state_dict”. The “state_dict” is a dictionary that maps each layer to its parameter tensor, which heavily increases modularity to our models and optimizers in PyTorch.

在建模阶段 我们使用BertConfig类为模型使用配置文件。 我们还使用预先训练的权重来帮助建立“ state_dict”。 “ state_dict”是一个字典 将每个层映射到其参数张量 这大大提高了我们在PyTorch中的模型和优化器的模块化。

We create a simple BioBERT class for the NER model. Our attributes are the layers in our network along with a forward pass method.

我们为NER模型创建一个简单的BioBERT类。 我们的属性是网络中的层以及前进传递方法。

With our primary model created, we set up the optimization and learning rate scheduler. We also set other hyperparameters here.

创建主要模型后 我们设置了优化和学习率调度程序。 我们还在这里设置其他超参数。

A function was created that manages epoch training. Here we notify the layers that we are in train mode so batch norm and dropout layers work in training mode rather than in eval mode, which is required in PyTorch. Gradients are computed and model weights are updated.

创建了一个管理时代训练的功能。 在这里 我们通知各层我们处于训练模式 因此批处理规范和辍学层在训练模式下工作 而不是在PyTorch要求的评估模式下工作。 计算梯度并更新模型权重。

A second function was created to evaluate our model. As one could guess, we know need to inform layers that we are in eval mode. We also deactivate the autograd engine to reduce memory usage and speed up computations. In the future, it would be more concise to consolidate these two functions as methods in the BioBERT model class.

创建了第二个函数来评估我们的模型。 可以猜测 我们知道需要告知图层我们处于评估模式。 我们还停用了自动分级引擎 以减少内存使用量并加快计算速度。 将来 将这两个功能合并为BioBERT模型类中的方法会更加简洁。

Here we will loop through our epochs, use our two functions, and print the results for both the training and validation loss/accuracy.

在这里 我们将循环浏览各个时期 使用我们的两个功能 并打印训练和验证损失/准确性的结果。

Our results show high accuracy of 96% for training accuracy and 95% for validation accuracy. Given the time it takes (although here the model ran incredibly fast with the available GPU!), there is no reason to increase epochs.

我们的结果表明 训练精度为96 验证精度为95 。 给定所需的时间(尽管这里的模型使用可用的GPU运行得非常快 ) 没有理由增加时间。

 Epoch 1/3 
Train Loss: 0.20869970354739556 Train Accuracy: 0.9479462699822381
Val Loss: 0.1037805580667087 Val Accuracy: 0.9576587301587302
 Epoch 2/3 
Train Loss: 0.09325889256480109 Train Accuracy: 0.9650584665482536 Val Loss: 0.09049581730413059 Val Accuracy: 0.9589087301587302
 Epoch 3/3 
Train Loss: 0.0828356556263529 Train Accuracy: 0.9658170515097693 Val Loss: 0.08888424655038213 Val Accuracy: 0.9585449735449736CPU times: user 8min 41s, sys: 6min 12s, total: 14min 54s 
Wall time: 14min 58s

When we consider loss, the learning curve expresses a reasonable loss over the three epochs:

当我们考虑损失时 学习曲线表示三个时期的合理损失