Scorer for SF submissions
The scorer is available at this link: http://www.nist.gov/tac/2013/KBP/SentimentSF/tools.html.
- Typical usage: java SFScore submission-file assessment-file slots=KBP2013_English_SF_slot-list.txt trace
- Slot file for SF: KBP2013_English_SF_slot-list.txt
- Description of output: The scorer output file first lists a line for each of the responses that you submitted, prefixed with the judgment for that response.
In the case of a non-NIL response, the judgments are:
- C = Correct
- R = Redundant with reference KB
- r = redundant with another slot filler in your submission
- X = ineXact
- W = Wrong
- I = Ignored (i.e., Wrong) due to excessively long justification
In the case of a NIL response, the judgments are:
- M = Missing filler that is not Redundant with reference KB
- m = missing filler that is Redundant with the reference KB
- c = Correct (but no credit given)
The final evaluation score is given at the end of the file and is F1,
giving equal weight to Precision and Recall, and requiring that slot fillers
be returned even if they are already in the reference KB.
The official evaluation does not require that all of the judgments for
the 3 sets of justification offsets be Correct - only that the
justifications are sufficiently concise and accurate enough to determine
that the slot filler is correct. A subsequent version of the scorer will
provide the option of producing diagnostic scores based on the individual
judgments for the justification offsets.
Validator for SF submissions (v2.2, July 12, 2013)
This script checks that your SF output is formatted correctly.
Use this before submitting your SF results!
Validator for TSF submissions (v1.0, July 12, 2013)
This script checks that your TSF output is formatted correctly.
Use this before submitting your TSF results!
Snippet Extractor for SF and TSF submissions (v1.2, July 25, 2013)
This Java program extracts the text snippets corresponding to the offsets in a
SF or TSF submission. The extracted strings are append as tab-separated strings at
the end of each submission line, and the expanded file is saved with the
We highly recommend using this program to verify that the offsets you
reported in the submission file are correct. The strings extracted by this
software is what the LDC evaluators will see.
- Code: ShowTextForOffsets.java (compile with: javac ShowTextForOffsets.java)
java -Xmx1g ShowTextForOffsets SF|TSF path_to_doc_collection doc_offsets_file submision_file
java -Xmx1g ShowTextForOffsets SF ~/corpora/KBPCorpora/LDC2013E45_TAC_2013_KBP_Source_Corpus_disc_2/data/English/ doc_offsets_english.txt sample_submission_line.txt
data/Spanish in the collection path.
- Full doc offset list file for English: doc_offsets_english.txt.gz (unzip before using it)
- Full doc offset list file for Spanish: doc_offsets_spanish.txt.gz (unzip before using it)
- Sample submission line with valid offsets: sample_submission_line.txt