NLTK fcfg语法解析器超出索引

问题描述 投票:0回答:1

我是NLTK的新手。试图将“给我看电影”转换成一个简单的SQL SELECT语句“从电影中选择标题”。

我相信句子是(VP + NP)与VP(V + PRO)和NP(DET + N)。但是我毫不怀疑我设置的.fcfg语法是不正确的,我在“anwser = trees”上得到以下索引错误,树木为空。

如何更正.fcfg?

IndexError:列表索引超出范围

进程以退出代码1结束

分析器

% start S
S[SEM=(?np + WHERE + ?vp)] -> NP[SEM=?np] VP[SEM=?vp]
VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> Det[SEM=?det] N[SEM=?n]
Det[SEM=''] -> 'the'
PRO[SEM=''] -> 'me'
N[SEM='title FROM films'] -> 'movies'
V[SEM='SELECT'] -> 'show'

Python代码

from nltk import load_parser
cp = load_parser('parser3.fcfg')
query = 'show me the movies'
trees = list(cp.parse(query.split()))
print(trees)
answer = trees[0].label()['SEM']
answer = [s for s in answer if s]
q = ' '.join(answer)
print(q)
python nltk
1个回答
0
投票

要调试语法,请从小处开始并扩展规则。

Lets start from underspecified VP and feature-structured V and N

from nltk import grammar, parse
from nltk.parse.generate import generate

g = """
VP -> V N
V[SEM='SELECT'] -> 'show'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show movies'.split())
print (list(trees)) 

[OUT]:

[Tree(VP[], [Tree(V[SEM='SELECT'], ['show']), Tree(N[SEM='title FROM films'], ['movies'])])]

Now, lets add the determiner.

g = """
VP -> V NP
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]
DT[SEM=''] -> 'the'
V[SEM='SELECT'] -> 'show'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show the movies'.split())
print (list(trees)) 

[OUT]:

[Tree(VP[], [Tree(V[SEM='SELECT'], ['show']), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])]

Then we add the pronoun.

我们想解析句子“给我看电影”

S[ VP[show me] NP[the movie] ]

所以我们必须改变我们的TOP到S -> VP NP的方式。

g = """
S -> VP NP
VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]
V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

[OUT]:

[Tree(S[], [Tree(VP[SEM=(SELECT, )], [Tree(V[SEM='SELECT'], ['show']), Tree(PRO[SEM=''], ['me'])]), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])]

Here comes the mystery

我们的TOP规则目前尚未明确规定,但如果我们同时指定左侧(LHS)和右侧(RHS),我们会发现它不起作用:

g = """
S[SEM=(?vp + WHERE + ?np)] -> VP[SEM=?vp] NP[SEM=?np]

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

即使我们删除了WHERE语义,我们也看到它没有解析:

g = """
S[SEM=(?vp + ?np)] -> VP[SEM=?vp] NP[SEM=?np]

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

[OUT]:

[]

但是如果我们只指定RHS,它会解析:

g = """
S -> VP[SEM=?vp] NP[SEM=?np]

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

[OUT]:

[Tree(S[], [Tree(VP[SEM=(SELECT, )], [Tree(V[SEM='SELECT'], ['show']), Tree(PRO[SEM=''], ['me'])]), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])]

当我们仅指定LHS时,同样有效:

g = """
S[SEM=(?vp + WHERE + ?np)] -> VP NP

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

[OUT]:

[Tree(S[SEM=(?vp+WHERE+?np)], [Tree(VP[SEM=(SELECT, )], [Tree(V[SEM='SELECT'], ['show']), Tree(PRO[SEM=''], ['me'])]), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])]

Lets recap

我们可以像NP和VP那样指定非终端但是TOP(即S -> VP NP)的特殊之处是什么?

如果我们破解语法并且只是给一个一元分支怎么办?

g = """
S -> SP
SP[SEM=(?vp + WHERE + ?np)] -> VP[SEM=?vp] NP[SEM=?np]

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

It parsed!

[OUT]:

[Tree(S[], [Tree(SP[SEM=(SELECT, , WHERE, , title FROM films)], [Tree(VP[SEM=(SELECT, )], [Tree(V[SEM='SELECT'], ['show']), Tree(PRO[SEM=''], ['me'])]), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])])]

有人应该向NLTK github repo提出这个问题。看起来它可能是保护TOP规则或bug的特殊功能=)

© www.soinside.com 2019 - 2024. All rights reserved.