delphin.tokens

YY tokens and token lattices.

class delphin.tokens.YYToken(id, start, end, lnk=None, paths=(1,), form=None, surface=None, ipos=0, lrules=('null',), pos=())[source]

A tuple of token data in the YY format.

Parameters:
  • id – token identifier

  • start – start vertex

  • end – end vertex

  • lnk – <from:to> charspan (optional)

  • paths – path membership

  • form – surface token

  • surface – original token (optional; only if form was modified)

  • ipos – length of lrules? always 0?

  • lrules – something about lexical rules; always “null”?

  • pos – pairs of (POS, prob)

classmethod from_dict(d)[source]

Decode from a dictionary as from to_dict().

to_dict()[source]

Encode the token as a dictionary suitable for JSON serialization.

class delphin.tokens.YYTokenLattice(tokens)[source]

A lattice of YY Tokens.

Parameters:

tokens – a list of YYToken objects

classmethod from_list(toks)[source]

Decode from a list as from to_list().

classmethod from_string(s)[source]

Decode from the YY token lattice format.

to_list()[source]

Encode the token lattice as a list suitable for JSON serialization.