Function Word Data Release

Released the data associated with Probing What Different NLP Tasks Teach Machines
about Function Word Comprehension (Kim et al. StarSem 2019).

The data can be found at https://github.com/decompositional-semantics-initiative/DNC/tree/master/function_words

Gender Parity Score

Added a python script to compute a Gender Parity Score for the recast WinoGender data. Description of the metric can be found in the README as well as in the updated draft of the paper

Removed incorrect not-entailed examples in VerbNet

The method to recast VerbNet annotations/examples into NLI created an issue in some examples. Here we provide a brief example and below we include statistics about the change.

Example

Consider the sentence Jackie chased the thief . which comes from the example in VN class chase-51.6. The semantics for this example from VN are motion(during(E), Agent) motion(during(E), Theme) indicating that both the Agent, i.e. Jackie, and the Theme, i.e. the thief moved. To create negative, i.e. not-entailed examples, we swapped thematic roles in our templates.

Consequently, this example sentence would then incorrectly label the example where the hypothesis would be The thief moved as not-entailed. In this version/release of the dataset, we removed such examples

Stats

	train	dev	test
unique contexts	23	6	2
all unique pairs	1386	143	160
bad pairs	43	11	4
% bad pairs	3.10	7.69	2.50
------------------	-------	------	------
# v0.1 examples	1441	154	164
# v0.1.1 examples	1398	143	160

Data Release for EMNLP 2018

This version corresponds to the data release for EMNLP.