delphin.tokens¶
YY tokens and token lattices.
-
class
delphin.tokens.
YYToken
[source]¶ A tuple of token data in the YY format.
- Parameters
id – token identifier
start – start vertex
end – end vertex
lnk – <from:to> charspan (optional)
paths – path membership
form – surface token
surface – original token (optional; only if
form
was modified)ipos – length of
lrules
? always 0?lrules – something about lexical rules; always “null”?
pos – pairs of (POS, prob)