Mention detection with Bert

.github/workflows/build.yml

@@ -7,13 +7,15 @@ on:
             - main
     pull_request:
         branches: [ main ]
+    workflow_dispatch:
+
 
 jobs:
     build:
         runs-on: ubuntu-latest
         strategy:
             matrix:
-                python-version: [3.7, 3.8]
+                python-version: [3.9]
 
         steps:
             - uses: actions/checkout@v3
@@ -40,4 +42,4 @@ jobs:
 
             - name: Test with pytest
               run: |
-                  pytest -W ignore
+                  pytest tests

.gitignore

@@ -106,6 +106,7 @@ celerybeat.pid
 .venv
 env/
 venv/
+venv3/
 ENV/
 env.bak/
 venv.bak/
@@ -133,3 +134,5 @@ dmypy.json
 
 # Project specific
 /data
+data
+000README

README.md

@@ -5,6 +5,20 @@
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/radboud-el)](https://pypi.org/project/radboud-el/) 
 [![PyPI](https://img.shields.io/pypi/v/radboud-el.svg?style=flat)](https://pypi.org/project/radboud-el/)
 
+---
+
+Example tests:
+
+* Flair: `python3 scripts/efficiency_test.py --process_sentences` 
+* Bert: `python3 scripts/efficiency_test.py --use_bert_base_uncased --split_docs_value 500`
+* Server (slower):
+    * `python3 src/REL/server.py --use_bert_base_uncased --split_docs_value 500 --ed-model ed-wiki-2019 data wiki_2019`
+    * `python3 scripts/efficiency_test.py --use_server`
+
+Needs installation of REL documents in directory `doc` (`ed-wiki-2019`,  `generic`  and `wiki_2019`)
+
+---
+
 REL is a modular Entity Linking package that is provided as a Python package as well as a web API. REL has various meanings -  one might first notice that it stands for relation, which is a suiting name for the problems that can be tackled with this package. Additionally, in Dutch a 'rel' means a disturbance of the public order, which is exactly what we aim to achieve with the release of this package.
 
 REL utilizes *English* Wikipedia as a knowledge base and can be used for the following tasks:

conftest.py

@@ -0,0 +1,11 @@
+import os
+import pytest
+
+
+def pytest_addoption(parser):
+    parser.addoption("--base_url", action="store", default=os.path.dirname(__file__) + "/src/data/")
+
+
+@pytest.fixture
+def base_url(request):
+    return request.config.getoption("--base_url")

docs/tutorials/custom_models.md

@@ -20,7 +20,7 @@ model, you can only use a local filepath.
 
 NER and ED models that we provide as part of REL can be loaded easily using
 aliases.  Available models are listed
-[on the REL repository](https://github.com/informagi/REL/tree/master/REL/models/models.json).
+[on the REL repository](https://github.com/informagi/REL/tree/master/src/REL/models/models.json).
 All models that need to be downloaded from the web are cached for subsequent
 use.
 

docs/tutorials/index.md

@@ -9,9 +9,10 @@ The remainder of the tutorials are optional and for users who wish to e.g. train
 
 1. [How to get started (project folder and structure).](how_to_get_started/)
 2. [End-to-End Entity Linking.](e2e_entity_linking/)
-3. [Evaluate on GERBIL.](evaluate_gerbil/)
-4. [Deploy REL for a new Wikipedia corpus](deploy_REL_new_wiki/):
-5. [Reproducing our results](reproducing_our_results/)
-6. [REL server](server/)
-7. [Notes on using custom models](custom_models/)
-7. [Conversational entity linking](conversations/)
+3. [Mention Detection models.](ner/)
+4. [Evaluate on GERBIL.](evaluate_gerbil/)
+5. [Deploy REL for a new Wikipedia corpus](deploy_REL_new_wiki/):
+6. [Reproducing our results](reproducing_our_results/)
+7. [REL server](server/)
+8. [Notes on using custom models](custom_models/)
+9. [Conversational entity linking](conversations/)

docs/tutorials/ner.md

@@ -0,0 +1,24 @@
+# Mention Detection models
+
+REL offers different named entity models for mention detection:
+
+- `flair`: named model for English, expects upper and lower case text (default)
+- `bert_base_cased`: basic name model for English, expects upper and lower case text
+- `bert_base_uncased`: basic name model for English, expects lower case text
+- `bert_large_cased`: extensive name model for English, expects upper and lower case text
+- `bert_large_uncased`: extensive name model for English, expects lower case text
+- `bert_multilingual`: multilingual name model, expects upper and lower case text
+
+To change the default Flair model, specify the required model with the `--tagger_ner_name` option, for example when calling the server:
+
+```bash
+python src/REL/server.py --tagger_ner_name bert_base_cased
+```
+
+or specify the model in the `tagger_name` parameter of a mention detection call:
+
+```python
+mentions_dataset, n_mentions = mention_detection.find_mentions(docs, tagger_ner="bert_base_cased")
+```
+
+The available named entity models are specified in the file `src/REL/ner/set_tagger_ner.py`. The file names refer to locations on the website huggingface.co, for example https://huggingface.co/flair/ner-english-fast . The file can be extended with new models, for example for other languages.

requirements.txt

@@ -8,3 +8,4 @@ pydantic
 segtok
 torch
 uvicorn
+scipy<=1.10

setup.cfg

@@ -43,17 +43,17 @@ package_dir =
     = src
 include_package_data = True
 install_requires =
-    anyascii
     colorama
-    fastapi
-    flair>=0.11
     konoha
-    nltk
-    pydantic
+    flair>=0.11
     segtok
-    spacy
     torch
-    uvicorn
+    nltk
+    anyascii
+    termcolor
+    syntok
+    spacy
+    scipy<=1.12.0
 
 [options.extras_require]
 develop =
@@ -80,3 +80,4 @@ where = src
 
 # [options.entry_points]
 # console_scripts =
+

src/REL/crel/s2e_pe/data.py

@@ -210,7 +210,7 @@ def pad_batch(self, batch, max_length):
                 example[0],
                 use_fast=False,
                 add_special_tokens=True,
-                pad_to_max_length=True,
+                padding='longest',
                 max_length=max_length,
                 return_attention_mask=True,
                 return_tensors="pt",

src/REL/db/base.py

@@ -186,9 +186,15 @@ def lookup_wik(self, w, table_name, column):
                 "select {} from {} where word = :word".format(column, table_name),
                 {"word": w},
             ).fetchone()
-        res = (
-            e if e is None else json.loads(e[0].decode()) if column == "p_e_m" else e[0]
-        )
+        if not e:
+            res = None
+        elif column == "p_e_m":
+            try:
+                res = json.loads(e[0].decode())
+            except AttributeError:
+                res = json.loads("".join(chr(int(x, 2)) for x in e[0].split()))
+        else:
+            res = e[0]
 
         return res