Unwanted length mismatch

[timvieira]

Why are the following parses considered to have different lengths? I'm guessing it has something to do with a punctuation filter.

GOLD=  
    (ROOT (S (CC And) (NP (NP (NNS rents)) (PP (IN on) (NP (NP (NNP Beverly) (NNP Hills)
    (POS ')) (NNP Rodeo) (NNP Drive)))) (ADVP (RB generally)) (VP (VBP do) (RB n't) 
    (VP (VB exceed)  (NP (NP (RB about) ($ $) (CD 125)) (NP (DT a) (JJ square) 
    (NN foot))))) (. .)))
TEST= 
    (ROOT (S (CC And) (NP (NP (NNS rents)) (PP (IN on) (NP (NNP Beverly) (NNP Hills)))) 
    ('' ')  (NP (NNP Rodeo) (NNP Drive)) (ADVP (RB generally)) (VP (VBP do) (RB n't) 
    (VP (VB exceed) (NP (NP (QP (IN about) ($ $) (CD 125))) (NP (DT a) (NN square) 
    (NN foot))))) (. .)))

In this case, I think the TEST parse drops the token ('' '), but the GOLD parse does not because it is has a possessive tag.

[jkkummerfeld]

That's right, the punctuation dropping is based on POS tags. The motivation was to follow evalb, which does it based on tags too.

For punctuation that can be deterministically mapped to a POS tag the code ensures consistency (see

berkeley-parser-analyser/src/nlp_util/treebanks.py

Line 15 in ee3cd64

word_to_POS_mapping = {

) but it's less clear what to do for something like quotes. The simplest solution may be to have an option to remove based on tokens instead of tags, though I don't have time to work on that right this minute (I'll keep this issue open though so I can get to it in future).