如何使用spaCy在CRAFT corpus上针对生物医学NER建立命名实体识别（NER）模型？
[我很难将该语料库中的xml文件预处理为spacy所使用的任何格式，任何帮助将不胜感激。我首先将xml文件转换为json格式，但是spacy不接受。 spacy期望什么格式的训练数据？我什至试图建立自己的NER模型，但无法像本xml中所给的那样预处理article文件。
这是一个使用spacy训练NER模型的示例，其中包括训练数据的预期格式（来自spacy's docs：]
import random
import spacy
TRAIN_DATA = [
("Uber blew through $1 million a week", {"entities": [(0, 4, "ORG")]}),
("Google rebrands its business apps", {"entities": [(0, 6, "ORG")]})]
nlp = spacy.blank("en")
optimizer = nlp.begin_training()
for i in range(20):
random.shuffle(TRAIN_DATA)
for text, annotations in TRAIN_DATA:
nlp.update([text], [annotations], sgd=optimizer)
nlp.to_disk("/model")
我正在使用的XML文件可在线获得here。记录示例如下：
<passage>
<infon key="section_type">ABSTRACT</infon>
<infon key="type">abstract</infon>
<offset>141</offset>
<text>
Breast cancer is the most frequent tumor in women, and in nearly two-thirds of cases, the tumors express estrogen receptor alpha (ERalpha, encoded by ESR1). Here, we performed whole-exome sequencing of 16 breast cancer tissues classified according to ESR1 expression and 12 samples of whole blood, and detected 310 somatic mutations in cancer tissues with high levels of ESR1 expression. Of the somatic mutations validated by a different deep sequencer, a novel nonsense somatic mutation, c.2830 C>T; p.Gln944*, in transcriptional regulator switch-independent 3 family member A (SIN3A) was detected in breast cancer of a patient. Part of the mutant protein localized in the cytoplasm in contrast to the nuclear localization of ERalpha, and induced a significant increase in ESR1 mRNA. The SIN3A mutation obviously enhanced MCF7 cell proliferation. In tissue sections from the breast cancer patient with the SIN3A c.2830 C>T mutation, cytoplasmic SIN3A localization was detected within the tumor regions where nuclear enlargement was observed. The reduction in SIN3A mRNA correlates with the recurrence of ER-positive breast cancers on Kaplan-Meier plots. These observations reveal that the SIN3A mutation has lost its transcriptional repression function due to its cytoplasmic localization, and that this repression may contribute to the progression of breast cancer.
</text>
<annotation id="38">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="246" length="23"/>
<text>estrogen receptor alpha</text>
</annotation>
<annotation id="39">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="271" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="40">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="291" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="41">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="392" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="42">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="512" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="43">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="720" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="44">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="868" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="45">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="915" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="46">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="930" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="47">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1048" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="48">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1087" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="49">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1201" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="50">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1331" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="51">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="185" length="5"/>
<text>women</text>
</annotation>
<annotation id="52">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="762" length="7"/>
<text>patient</text>
</annotation>
<annotation id="53">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="1031" length="7"/>
<text>patient</text>
</annotation>
<annotation id="54">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="397" length="10"/>
<text>expression</text>
</annotation>
<annotation id="55">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="517" length="10"/>
<text>expression</text>
</annotation>
<annotation id="56">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="1054" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="57">
<infon key="identifier">CVCL:0031</infon>
<infon key="type">CellLine</infon>
<location offset="964" length="4"/>
<text>MCF7</text>
</annotation>
<annotation id="58">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1494" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="59">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="346" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="60">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="743" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="61">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1017" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="62">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="477" length="6"/>
<text>cancer</text>
</annotation>
<annotation id="63">
<infon key="identifier">p.Q944*</infon>
<infon key="type">ProteinMutation</infon>
<location offset="642" length="9"/>
<text>p.Gln944*</text>
</annotation>
<annotation id="64">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="1130" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="65">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="176" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="66">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="630" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="67">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1258" length="14"/>
<text>breast cancers</text>
</annotation>
<annotation id="68">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="231" length="6"/>
<text>tumors</text>
</annotation>
<annotation id="69">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="141" length="13"/>
<text>Breast cancer</text>
</annotation>
</passage>
这里有一些代码可以助您一臂之力。这不是一个完整的解决方案，但是您提出的问题非常困难，并且您没有任何入门代码。
它不跟踪identifier或NCBI Homologene属性，但我认为可以将它们分别存储在字典中。
import xml.etree.cElementTree as ET
import spacy
nlp = spacy.load('en_core_web_sm')
# this is one child of the XML doc
# https://www.ncbi.nlm.nih.gov/research/pubtator-api/publications/export/biocxml?pmcids=PMC6207735
passage_string = """
<passage>
<infon key="section_type">ABSTRACT</infon>
<infon key="type">abstract</infon>
<offset>141</offset>
<text>
Breast cancer is the most frequent tumor in women, and in nearly two-thirds of cases, the tumors express estrogen receptor alpha (ERalpha, encoded by ESR1). Here, we performed whole-exome sequencing of 16 breast cancer tissues classified according to ESR1 expression and 12 samples of whole blood, and detected 310 somatic mutations in cancer tissues with high levels of ESR1 expression. Of the somatic mutations validated by a different deep sequencer, a novel nonsense somatic mutation, c.2830 C>T; p.Gln944*, in transcriptional regulator switch-independent 3 family member A (SIN3A) was detected in breast cancer of a patient. Part of the mutant protein localized in the cytoplasm in contrast to the nuclear localization of ERalpha, and induced a significant increase in ESR1 mRNA. The SIN3A mutation obviously enhanced MCF7 cell proliferation. In tissue sections from the breast cancer patient with the SIN3A c.2830 C>T mutation, cytoplasmic SIN3A localization was detected within the tumor regions where nuclear enlargement was observed. The reduction in SIN3A mRNA correlates with the recurrence of ER-positive breast cancers on Kaplan-Meier plots. These observations reveal that the SIN3A mutation has lost its transcriptional repression function due to its cytoplasmic localization, and that this repression may contribute to the progression of breast cancer.
</text>
<annotation id="38">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="246" length="23"/>
<text>estrogen receptor alpha</text>
</annotation>
<annotation id="39">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="271" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="40">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="291" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="41">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="392" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="42">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="512" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="43">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="720" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="44">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="868" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="45">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="915" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="46">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="930" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="47">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1048" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="48">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1087" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="49">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1201" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="50">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1331" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="51">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="185" length="5"/>
<text>women</text>
</annotation>
<annotation id="52">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="762" length="7"/>
<text>patient</text>
</annotation>
<annotation id="53">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="1031" length="7"/>
<text>patient</text>
</annotation>
<annotation id="54">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="397" length="10"/>
<text>expression</text>
</annotation>
<annotation id="55">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="517" length="10"/>
<text>expression</text>
</annotation>
<annotation id="56">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="1054" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="57">
<infon key="identifier">CVCL:0031</infon>
<infon key="type">CellLine</infon>
<location offset="964" length="4"/>
<text>MCF7</text>
</annotation>
<annotation id="58">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1494" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="59">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="346" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="60">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="743" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="61">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1017" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="62">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="477" length="6"/>
<text>cancer</text>
</annotation>
<annotation id="63">
<infon key="identifier">p.Q944*</infon>
<infon key="type">ProteinMutation</infon>
<location offset="642" length="9"/>
<text>p.Gln944*</text>
</annotation>
<annotation id="64">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="1130" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="65">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="176" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="66">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="630" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="67">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1258" length="14"/>
<text>breast cancers</text>
</annotation>
<annotation id="68">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="231" length="6"/>
<text>tumors</text>
</annotation>
<annotation id="69">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="141" length="13"/>
<text>Breast cancer</text>
</annotation>
</passage>"""
# turn into an object
passage = ET.fromstring(passage_string)
# these 3 definitions are per-passage
passage_annotations = passage.findall('./annotation')
passage_offset = int(passage.find('offset').text)
passage_text = passage.find('text').text
def get_entity_offset(offset_dict, passage_offset):
"""
XML given offset_dict gives offset relative to the start of the document
So subtract the passage offset (where passage starts relative to document beginning)
"""
start = int(offset_dict['offset']) - passage_offset
end = int(offset_dict['offset']) + (int(offset_dict['length']) + 1) - passage_offset
return start, end
# collect entities as a list of tuples of the form
# (start, end, entitiy_type)
passage_entities = []
for ann in passage_annotations:
entity_type = ann.find('./infon[@key="type"]').text
od = ann.find('./location').attrib
start, end = get_entity_offset(od, passage_offset)
passage_entities.append((start, end, entity_type))
# this is one entry in the spacy NER format
# you would want many entries
spacyd_passage = (passage_text, {"entities": passage_entities})
# prove this worked
for ent in passage_entities:
print(ent, passage_text[ent[0]:ent[1]])
# prints:
# (105, 129, 'Gene') estrogen receptor alpha
# (130, 138, 'Gene') (ERalpha
# (150, 155, 'Gene') ESR1
# (251, 256, 'Gene') ESR1
# (371, 376, 'Gene') ESR1
# (579, 585, 'Gene') (SIN3A
# (727, 735, 'Gene') ERalpha
# (774, 779, 'Gene') ESR1
# (789, 795, 'Gene') SIN3A
# (907, 913, 'Gene') SIN3A
# (946, 952, 'Gene') SIN3A
# (1060, 1066, 'Gene') SIN3A
# (1190, 1196, 'Gene') SIN3A
# (44, 50, 'Species') women
# (621, 629, 'Species') patient
# (890, 898, 'Species') patient
# (256, 267, 'Species') expression
# (376, 387, 'Species') expression
# (913, 924, 'DNAMutation') c.2830 C>T
# (823, 828, 'CellLine') MCF7
# (1353, 1367, 'Disease') breast cancer
# (205, 219, 'Disease') breast cancer
# (602, 616, 'Disease') breast cancer
# (876, 890, 'Disease') breast cancer
# (336, 343, 'Disease') cancer
# (501, 511, 'ProteinMutation') p.Gln944*
# (989, 995, 'Disease') tumor
# (35, 41, 'Disease') tumor
# (489, 500, 'DNAMutation') c.2830 C>T
# (1117, 1132, 'Disease') breast cancers
# (90, 97, 'Disease') tumors
# (0, 14, 'Disease') Breast cancer
所以，我注意到的第一件事是，给定的一些偏移量略有下降，赶上了(。您可以寻找if passage_text[ent[0]] == "("，然后将实体的起始位置移1来清除它，或者手动清除它。
此外，此代码使用一个子节点，即链接文档的passage。您将需要在本地下载该文档，而不是passage = ET.fromstring(passage_string)，而是创建tree = ET.parse('path_to_file')：
类似
import xml.etree.cElementTree as ET
tree = ET.parse('path_to_file')
root = tree.getroot()
passages = root.findall('./passages')
spacy_data = []
for passage in passages:
passage_annotations = passage.findall('./annotation')
passage_offset = int(passage.find('offset').text)
passage_text = passage.find('text').text
passage_entities = []
for ann in passage_annotations:
entity_type = ann.find('./infon[@key="type"]').text
od = ann.find('./location').attrib
start, end = get_entity_offset(od, passage_offset)
passage_entities.append((start, end, entity_type))
spacyd_passage = (passage_text, {"entities": passage_entities})
spacy_data.append(spacyd_package)
仍然可以改进。您需要使用[]拆分这些passage.text段落
import spacy nlp = spacy.load('en_core_web_sm') doc = nlp(passage_text) sents = list(doc.sents)但是棘手的部分是您需要进行算术运算以保持偏移索引正确。而且，您还需要查看每个实体的开始和结尾，以确保其停留在一个句子之内-可以想象它可以被句子边界分割，尽管可能不会。