Merge pull request #152 from informagi/update-api

Update API to support multiple models

README.md

@@ -164,7 +164,7 @@ The remainder of the tutorials are optional and for users who wish to e.g. train
 3. [Evaluate on GERBIL.](https://rel.readthedocs.io/en/latest/tutorials/evaluate_gerbil/)
 4. [Deploy REL for a new Wikipedia corpus](https://rel.readthedocs.io/en/latest/tutorials/deploy_REL_new_wiki/):
 5. [Reproducing our results](https://rel.readthedocs.io/en/latest/tutorials/reproducing_our_results/)
-6. [REL as systemd service](https://rel.readthedocs.io/en/latest/tutorials/systemd_instructions/)
+6. [REL server](https://rel.readthedocs.io/en/latest/tutorials/server/)
 7. [Notes on using custom models](https://rel.readthedocs.io/en/latest/tutorials/custom_models/)
 
 ## REL variants

docs/server_api.md

@@ -0,0 +1,221 @@
+# REL server API docs
+
+This page documents usage for the [REL server](https://rel.cs.ru.nl/docs). The live, up-to-date api can be found either [here](https://rel.cs.ru.nl/api/docs) or [here](https://rel.cs.ru.nl/api/redocs).
+
+Scroll down for code samples, example requests and responses.
+
+## Server status
+
+`GET /`
+
+Returns server status.
+
+### Example
+
+> Response
+
+```json
+{
+  "schemaVersion": 1,
+  "label": "status",
+  "message": "up",
+  "color": "green"
+}
+```
+
+> Code
+
+```python
+>>> import requests
+>>> requests.get("https://rel.cs.ru.nl/api/").json()
+{'schemaVersion': 1, 'label': 'status', 'message': 'up', 'color': 'green'}
+```
+
+## Named Entity Linking
+
+`POST /`  
+`POST /ne`  
+
+Submit your text here for entity disambiguation or linking.
+
+The REL annotation mode can be selected by changing the path.
+use `/` or `/ne` for annotating regular text with named
+entities (default), `/ne_concept` for regular text with both concepts and
+named entities, and `/conv` for conversations with both concepts and
+named entities.
+
+> Schema 
+
+`text` (string)
+: Text for entity linking or disambiguation.
+
+`spans` (list)
+
+: For EL: the spans field needs to be set to an empty list.
+
+: For ED: spans should consist of a list of tuples, where each tuple refers to the start position (int) and length of a mention (int).
+
+: This is used when mentions are already identified and disambiguation is only needed. Each tuple represents start position and length of mention (in characters); e.g., [(0, 8), (15,11)] for mentions 'Nijmegen' and 'Netherlands' in text 'Nijmegen is in the Netherlands'.
+
+`tagger` (string)
+: NER tagger to use. Must be one of `ner-fast`, `ner-fast-with-lowercase`. Default: `ner-fast`.
+
+### Example
+
+> Request body
+
+```json
+{
+  "text": "If you're going to try, go all the way - Charles Bukowski.",
+  "spans": [
+    [
+      41,
+      16
+    ]
+  ],
+  "tagger": "ner-fast"
+}
+```
+
+> Response
+
+The 7 values of the annotation represent the start index, end index, the annotated word, prediction, ED confidence, MD confidence, and tag.
+
+```json
+[
+
+    [
+        41,
+        16,
+        "Charles Bukowski",
+        "Charles_Bukowski",
+        0,
+        0,
+        "NULL"
+    ]
+
+]
+```
+
+> Code
+
+```python
+>>> import requests
+>>> myjson = {
+  "text": "REL is a modular Entity Linking package that can both be integrated in existing pipelines or be used as an API.",
+  "tagger": "ner-fast"
+}
+>>> requests.post("https://rel.cs.ru.nl/api/ne", json=myjson).json()
+[[0, 3, 'REL', 'Category_of_relations', 0, 0, 'ORG'], [107, 3, 'API', 'Application_programming_interface', 0, 0, 'MISC']]
+```
+
+## Conversational entity linking
+
+`POST /conv`
+
+Submit your text here for conversational entity linking.
+
+> Schema
+
+`text` (list)
+: Text is specified as a list of turns between two speakers.
+
+    `speaker` (string)
+    : Speaker for this turn, must be one of `USER` or `SYSTEM`
+
+    `utterance` (string)
+    : Input utterance to be annotated.
+
+`tagger` (string)
+: NER tagger to use. Choices: `default`.
+
+
+### Example
+
+> Request body
+
+```json
+{
+  "text": [
+    {
+      "speaker": "USER",
+      "utterance": "I am allergic to tomatoes but we have a lot of famous Italian restaurants here in London."
+    },
+    {
+      "speaker": "SYSTEM",
+      "utterance": "Some people are allergic to histamine in tomatoes."
+    },
+    {
+      "speaker": "USER",
+      "utterance": "Talking of food, can you recommend me a restaurant in my city for our anniversary?"
+    }
+  ],
+  "tagger": "default"
+}
+```
+
+> Response
+
+The 7 values of the annotation represent the start index, end index, the annotated word, prediction, ED confidence, MD confidence, and tag.
+
+```json
+[
+  {
+    "speaker": "USER",
+    "utterance": "I am allergic to tomatoes but we have a lot of famous Italian restaurants here in London.",
+    "annotations": [
+      [17, 8, "tomatoes", "Tomato"],
+      [54, 19, "Italian restaurants", "Italian_cuisine"],
+      [82, 6, "London", "London"]
+    ]
+  },
+  ...
+]
+```
+
+> Code
+
+```python
+>>> import requests
+>>> myjson = {
+  "text": [...],
+  "tagger": "default"
+}
+>>> requests.post("https://rel.cs.ru.nl/api/conv", json=myjson).json()
+[{...}]
+```
+
+
+## Conceptual entity linking
+
+`POST /ne_concept`
+
+Submit your text here for conceptual entity disambiguation or linking.
+
+### Example
+
+> Request body
+
+```json
+{}
+```
+
+> Response
+
+Not implemented.
+
+```json
+{}
+```
+
+> Code
+
+```python
+>>> import requests
+>>> myjson = {
+  "text": ...,
+}
+>>> requests.post("https://rel.cs.ru.nl/api/ne_concept", json=myjson).json()
+{...}
+```
+

docs/tutorials/index.md

@@ -12,6 +12,6 @@ The remainder of the tutorials are optional and for users who wish to e.g. train
 3. [Evaluate on GERBIL.](evaluate_gerbil/)
 4. [Deploy REL for a new Wikipedia corpus](deploy_REL_new_wiki/):
 5. [Reproducing our results](reproducing_our_results/)
-6. [REL as systemd service](systemd_instructions/)
+6. [REL server](server/)
 7. [Notes on using custom models](custom_models/)
 7. [Conversational entity linking](conversations/)

docs/tutorials/server.md

@@ -0,0 +1,86 @@
+# REL server
+
+This section describes how to set up and use the REL server.
+
+## Running the server
+
+The server uses [fastapi](https://fastapi.tiangolo.com/) as the web framework. 
+FastAPI is a modern, fast (high-performance), web framework for building APIs bases on standard Python type hints. 
+When combined with [pydantic](https://docs.pydantic.dev/) this makes it very straightforward to set up a web API with minimal coding.
+
+```bash
+python ./src/REL/server.py \
+    $REL_BASE_URL \
+    wiki_2019 \
+    --ner-model ner-fast ner-fast-with-lowercase
+```
+
+This will open the API at the default `host`/`port`: <http://localhost:5555>.
+
+One of the advantage of using fastapi is its automated docs by adding `/docs` or `/redoc` to the end of the url:
+
+- <http://localhost:5555/docs>
+- <http://localhost:5555/redoc>
+
+You can use `python ./src/scripts/test_server.py` for some examples of the queries and to test the server.
+
+### Setup
+
+Set `$REL_BASE_URL` to the path where your data are stored (`base_url`).
+
+For mention detection and entity linking, the `base_url` must contain all the files specified [here](../how_to_get_started).
+
+In addition, for conversational entity linking, additonal files are needed as specified [here](../conversations)
+
+In summary, these paths must exist:
+
+ - `$REL_BASE_URL/wiki_2019` or `$REL_BASE_URL/wiki_2014`
+ - `$REL_BASE_URL/bert_conv`  for conversational EL)
+ - `$REL_BASE_URL/s2e_ast_onto ` for conversational EL)
+
+## Running REL as a systemd service
+
+In this tutorial we provide some instructions on how to run REL as a systemd
+service. This is a fairly simple setup, and allows for e.g. automatic restarts
+after crashes or machine reboots.
+
+### Create `rel.service`
+
+For a basic systemd service file for REL, put the following content into
+`/etc/systemd/system/rel.service`:
+
+```ini
+[Unit]
+Description=My REL service
+
+[Service]
+Type=simple
+ExecStart=/bin/bash -c "python src/REL/server.py"
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Note that you may have to alter the code in `server.py` to reflect
+necessary address/port changes.
+
+This is the simplest way to write a service file for REL; it could be more
+complicated depending on any additional needs you may have. For further
+instructions, see e.g. [here](https://wiki.debian.org/systemd/Services) or `man
+5 systemd.service`.
+
+### Enable the service
+
+In order to enable the service, run the following commands in your shell:
+
+```bash
+systemctl daemon-reload
+
+# For systemd >= 220:
+systemctl enable --now rel.service
+
+# For earlier versions:
+systemctl enable rel.service
+reboot
+```

mkdocs.yml

@@ -2,13 +2,14 @@ site_name: "REL: Radboud Entity Linker"
 
 nav:
   - Home: index.md
+  - API: server_api.md
   - Tutorials:
     - tutorials/how_to_get_started.md
     - tutorials/e2e_entity_linking.md
     - tutorials/evaluate_gerbil.md
     - tutorials/deploy_REL_new_wiki.md
     - tutorials/reproducing_our_results.md
-    - tutorials/systemd_instructions.md
+    - tutorials/server.md
     - tutorials/custom_models.md
     - tutorials/conversations.md
   - Python API reference: