Elementary Dependency Matching

Elementary Dependency Matching (EDM; Dridan and Oepen, 2011) is a metric for comparing two semantic dependency graphs that annotate the same sentence. It requires that each node is aligned to a character span in the original sentence.

See also

The delphin.edm module is the programmatic interface for the EDM functionality, while this guide describes the command-line interface.

Tip

The smatch metric (Cai and Knight, 2013) is essentially the same except that instead of relying on surface-aligned nodes it finds a mapping of nodes that optimizes the number of matching triples. The search uses stochastic hill-climbing, whereas EDM gives deterministic results. EDS and DMRS representations can be used with the smatch tool if they have been serialized to the PENMAN format (see delphin.codecs.edspenman and delphin.codecs.dmrspenman).

Command-line Usage

The edm subcommand provides a simple interface for computing EDM for EDS, DMRS, or MRS representations. The basic usage is:

$ delphin edm GOLD TEST

GOLD and TEST may be files containing serialized semantic representations or [incr tsdb()] test suites containing parsed analyses.

For example:

$ delphin edm gold.eds test.eds
Precision:   0.9344262295081968
   Recall:   0.9193548387096774
  F-score:   0.9268292682926829

Per-item information can be printed by increasing the logging verbosity to the INFO level (-vv). Weights for the different classes of triples can be adjusted with -A for argument structure, -N for node names, -P for node properties, -C for constants, and -T for graph tops. Try delphin edm --help for more information.

Differences from Dridan and Oepen, 2011

Following the mtool implementation, delphin.edm treats constant arguments (CARG) as independent triples, however, unlike mtool, they get their own category and weight. This implementation also follows mtool in checking if the graph tops are the same, also with their own category and weight. One can therefore get the same results as Dridan and Oepen, 2011 by setting the weights for top-triples and constant-triples to 0:

$ delphin edm -C0 -T0 GOLD TEST

Sometimes it helps to ignore missing items on the gold side, the test side, or both. Missing items can occur when GOLD or TEST are files with different numbers of representations, or when they are [incr tsdb()] test suites with different numbers of analyses per item. For example, to ignore pairs where the gold representation is missing, do the following:

$ delphin edm --ignore-missing=gold GOLD TEST

Relevance to non-EDS Semantic Representations

While EDM was designed for the semantic dependencies extracted from Elementary Dependency Structures (EDS), it can be used for other representations as long as they have surface alignments for the nodes. This implementation can natively work with a variety of DELPH-IN representations and formats via the --format option, including those for Minimal Recursion Semantics (MRS) and Dependency Minimal Recursion Semantics (DMRS). Non-DELPH-IN representations are also possible as long as they can be serialized into one of these formats.

Other Implementations

  1. Rebecca Dridan’s original Perl version (see the wiki):

  2. mtool: created for the 2019 CoNLL shared task on Meaning Representation Parsing

  3. As part of [incr tsdb()]

  4. As part of DeepDeepParser