Scorer for SF submissions

The scorer is available at this link: http://www.nist.gov/tac/2013/KBP/SentimentSF/tools.html.

• Typical usage: java SFScore submission-file assessment-file slots=KBP2013_English_SF_slot-list.txt trace
• Slot file for SF: KBP2013_English_SF_slot-list.txt
• Description of output: The scorer output file first lists a line for each of the responses that you submitted, prefixed with the judgment for that response. In the case of a non-NIL response, the judgments are:
  • C = Correct
  • R = Redundant with reference KB
  • r = redundant with another slot filler in your submission
  • X = ineXact
  • W = Wrong
  • I = Ignored (i.e., Wrong) due to excessively long justification
  
  In the case of a NIL response, the judgments are:
  • M = Missing filler that is not Redundant with reference KB
  • m = missing filler that is Redundant with the reference KB
  • c = Correct (but no credit given)
  
  The final evaluation score is given at the end of the file and is F1, giving equal weight to Precision and Recall, and requiring that slot fillers be returned even if they are already in the reference KB.
  
  The official evaluation does not require that all of the judgments for the 3 sets of justification offsets be Correct - only that the justifications are sufficiently concise and accurate enough to determine that the slot filler is correct. A subsequent version of the scorer will provide the option of producing diagnostic scores based on the individual judgments for the justification offsets.

Validator for SF submissions (v2.2, July 12, 2013)

This script checks that your SF output is formatted correctly. Use this before submitting your SF results!

• Code: check_kbp_slot-filling.txt (change the extension to .pl after saving it)
• Usage: check_kbp_slot-filling.pl docid-list queries-xml-file submission_file
• Full docid list file for English: doc_ids_english.txt.gz (unzip before using it)
• Full docid list file for Spanish: doc_ids_spanish.txt.gz (unzip before using it)
• Sample SF queries file: tac_2013_kbp_spanish_slot_filling_sample_queries.xml
• Sample SF submission file: tac_2013_kbp_spanish_slot_filling_sample_submission.txt

Validator for TSF submissions (v1.0, July 12, 2013)

This script checks that your TSF output is formatted correctly. Use this before submitting your TSF results!

• Code: check_temporal-kbp_slot-filling.txt (change the extension to .pl after saving it)
• Usage: check_kbp_temporal-slot-filling.pl doclist_file queries_file results_file
• Full docid list file for English: doc_ids_english.txt.gz (unzip before using it)
• Sample TSF queries file: tac_2013_tsf_sample_queries.txt
• Sample TSF submission file: tac_2013_tsf_sample_submission.txt

Snippet Extractor for SF and TSF submissions (v1.2, July 25, 2013)

This Java program extracts the text snippets corresponding to the offsets in a SF or TSF submission. The extracted strings are append as tab-separated strings at the end of each submission line, and the expanded file is saved with the .expanded extension.

We highly recommend using this program to verify that the offsets you reported in the submission file are correct. The strings extracted by this software is what the LDC evaluators will see.

• Code: ShowTextForOffsets.java (compile with: javac ShowTextForOffsets.java)
• Usage: java -Xmx1g ShowTextForOffsets SF|TSF path_to_doc_collection doc_offsets_file submision_file
  For example:
  java -Xmx1g ShowTextForOffsets SF ~/corpora/KBPCorpora/LDC2013E45_TAC_2013_KBP_Source_Corpus_disc_2/data/English/ doc_offsets_english.txt sample_submission_line.txt
  Please include data/English or data/Spanish in the collection path.
• Full doc offset list file for English: doc_offsets_english.txt.gz (unzip before using it)
• Full doc offset list file for Spanish: doc_offsets_spanish.txt.gz (unzip before using it)
• Sample submission line with valid offsets: sample_submission_line.txt