Assess

This command assesses the performance and accuracy of an Eduction grammar against a set of pre-tagged examples.

You must supply a text file with one phrase on each line; the Assess feature checks whether each line contains a match.

You must specify at least one input file, using the -v parameter or the -w parameter. If required, you can specify both of these parameters.

The following table describes the parameters for this command.

-l <licensefile>

The file containing a valid license key for Eduction.

If you do not specify a license key, edktool attempts to load the license licensekey.dat in its current working directory. You must specify the license parameter if your license is in a different location.

-c <configfile>

A configuration file to control the assessment. See Eduction Configuration File.

You can specify one or more grammar files and one or more entities in place of a configuration file. Specifying a configuration file overrides the grammar (-g) or entity (-e) parameters.

-g <grammarfile>

A grammar file to use. Edktool ignores this option if you set a configuration file with -c.

If you provide a grammar file but you do not specify any entities with -e, Eduction extracts all entities in the grammar file.

You can use wildcard expressions in this parameter. See Wildcard Expressions in edktool.

-e <entity>

The entities to extract. Separate multiple entities with a comma. Edktool ignores this option if you set a configuration file with -c.

You can use wildcard expressions in this parameter. See Wildcard Expressions in edktool.

-x (Optional) Modifies the behavior so that Assess checks for exact matches.
-m <matched entities> (Optional) This parameter does not change the extraction behavior, but enables you to check which entities are producing the matches.
-v <valid_input> A file of phrases where a match would be valid.
-w <invalid_input> A file of phrases where a match would be invalid.
-a (Optional) Display additional output, including the results for every phrase in your input files. By default, the output includes explanations of each failure, and statistics such as recall, precision, and F1 (depending on the type of input file you provide).
-o <outputfile>

(Optional) Send the output to a file. By default, Eduction sends output to the console.

The output is a list of all phrases that failed. For valid input this would be a phrase that contained no match; for invalid input this would be a phrase that contained a match.

-q

(Optional) Run in quiet mode. In this case, edktool removes all descriptive messages from the output and shows only a list of examples that failed, in the form "FAIL: "text" is matched by "entity"" or similar, depending on the test specifications. If you also set the -a parameter, examples that pass are also included in the output.

For more information on how to use the Assess feature to check the effectiveness and performance of your grammar files, see Assess and Measure Eduction Grammars.

Example

edktool a -l <license> -c <configuration_file> [-a] [-o <output_file>]

Run several assessments from a single Eduction configuration file.

The configuration file must contain a numbered [assessmentN] section for each assessment you want to run. You must specify the input files, the entities to match, and whether to require exact matches. For example:

[assessment0]
valid=data.txt
[assessment1]
entities=entity1,entity2
valid=match.txt
invalid=should_not_match.txt
exact=true

You can specify multiple entities by separating them with commas, or by using wildcard expressions (see Wildcard Expressions in edktool).