Improve duplicate detection

[stefan-kolb]

• strict comparison does not care about entry type. Is this intended?

    @Test
    public void noStrictDuplicateForDifferentTypes() {
        BibEntry e1 = new BibEntry("1", "article");
        BibEntry e2 = new BibEntry("2", "journal");
        assertEquals(0, DuplicateCheck.compareEntriesStrictly(e1, e2), 0.01);
    }

• Correlate by words has a strange algorithm. Only works good for words appended at the end of the String.

     * Compare two strings on the basis of word-by-word correlation analysis.
     * TODO: strange algorithm as when there are only words inserted this gives a bad value, e.g.,
     * a test -> this a test (0.0)
     * characterization -> characterization of me (1.0)

• We need a small benchmark for the duplicate testing, i.e., a DB that has all kinds of expected duplicates.

[oscargus]

Is it worth having this build dependency as it basically does what the removed code below does and adds quite a bit of other unused methods? (I know @koppor was also skeptic.)

[stefan-kolb]

We already have this dependency inside the code that's why I reused it. For the general dependecy question, I'm not 100% sure as we might use more distance measures and the library is probably not very large. So why should I reinvent the wheel here.

[oscargus]

Yes, from that perspective it is OK for sure and we can of course revive the old code if required later. Oh, here I found it: koppor#131

[koppor]

Just go ahead. No worries. We migrate to another library if the functionality is the same. If not, we just keep the dependency. Reason for migrating is only koppor#135

[koppor]

Since JabRef 3.6 is close to be available in Debian/unstable, there is no need for a library migration.

[oscargus]

Good that you give it a go!

[koppor]

Are we really returning null? I think, we ommitted null, thus also line 79 is not covered by tests at all.

[koppor]

Strict comparison: We should also consider the type.

Correlate by words: I think, you did not find any other algorithm? Can't an existing algorithm be used? Something like counting the similar and the different words and dividing by max(len1, len2)?

Small benchmark: Can we do that as a separate issue? Maybe in the koppor-repo? 😇

[koppor]

Can we also add a test where the authors are written in a different serialization? Firstname Lastname?

[tobiasdiez]

What is the state of this PR?

Another improvement to the duplicate algorithm concerns how DOI, ISBN and edition fields should be weighted:

• Entries with the same DOI or ISBN are almost surely duplicates although they might have different titles or authors (Citavi is doing something similar)
• Entries which are almost identical but for edition or volume are probably no duplicates (but just a different version of the same book/article)

[lenhard]

And again: @stefan-kolb Is there an advance here? I think at some point, we should either try to proceed with the old PRs or scrap them.

[stefan-kolb]

No time right now. There are some point that we should fix however. You can close if you want tho.