Merge pull request #2 from microsoft/release-0.2.0

Release 0.2.0

README.md

@@ -1,24 +1,149 @@
-QLibServer is the assorted server system for QLib, which utilizes QLib for basic calculations and provides extensive server system and cache mechanism. With QLibServer, the data provided for QLib can be managed in a centralized manner.
+`Qlib-Server` is the data server system for [`Qlib`](https://github.com/microsoft/qlib). It enable `Qlib` to run in `online` mode. Under online mode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too.
 
+To sum up, `Qlib-Server` is designed to solve the following problems:
+
+* Manage the data in a centralized way, which makes data management (including cache management, date updating) much easier.
+* Reduce the amount of cache to be generated.
+* Make the client light-weighted and leverage the powerful computing resources of remote server
 
 
 - [Framework of qlib-erver](#framework-of-qlib-server)
 - [Quick start](#quick-start)
-  - [Installation](#installation)
+  - [Deployment](#deployment)
+    - [One-click Deployment](#one-click-deployment)
+    - [Step-by-Step Deployment](#step-by-step-deployment)
+  - [Using Qlib in Online Mode](#using-qlib-in-online-mode)
 - [More About Qlib](#more-about-qlib)
+- [Contributing](#contributing)
+
 
+# Framework of Qlib-Server
 
-# Framework of qlib-server
+<div style="align: center">
+<img src="docs/_static/img/framework.png" />
+</div>
 
+The `Client/Server` framework of `Qlib` is based on `WebSocket` considering its capability of **bidirectional communication** between client and server in **async** mode.
+
+`Qlib-Server` is based on [Flask](http://flask.pocoo.org/), which is a micro-framework for Python and here [Flask-SocketIO](https://flask-socketio.readthedocs.io) is used for websocket connection. 
 
 # Quick start
 
 
-## Installation
+## Deployment
+
+### One-click Deployment
+
+One-click deployment of `Qlib-Server` is supported, you can choose either of the following two methods for one-click deployment:
+
+- Deployment with `docker-compose`
+- Deployment in `Azure`
+
+#### One-click Deployment with `docker-compose`
+
+Deploy `Qlib-Server` with `docker-compose` according to the following processes:
+
+* Install `docker`, please refer to [Docker Installation](https://docs.docker.com/engine/install).
+* Install `docker-compose`, please refer to [Docker-compose Installation](https://docs.docker.com/compose/install/).
+* Run the following command to deploy `Qlib-Server`:
+
+    ```bash
+      git clone https://github.com/microsoft/qlib-server
+      cd qlib-server
+      sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env build
+      sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env up -d
+      # Use the following command to track the log
+      sudo docker-compose -f docker_support/docker-compose.yaml logs -f
+    ```
+
+#### One-click Deployment in `Azure`
+
+Firstly, You need to have an `Azure` account to deploy `Qlib-Server` in `Azure`. Then you can deploy `Qlib-Server` in `Azure` according to the following processes:
+
+* Install `azure-cli`, please refer to [install-azure-cli](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest).
+
+* Add the `Azure` account to the configuration file `azure_conf.yaml`
+
+    ```yaml
+        sub_id: Your Subscription ID
+        username: azure user name
+        password: azure password
+        # The resource group where the VM is located
+        resource_group: Resource group name
+    ```
+* Execute the deployment script
+    Run the following command:
+
+    ```bash
+
+        git clone https://github.com/microsoft/qlib-server
+        cd qlib-server/scripts
+        python azure_manager.py create_qlib_cs_vm \
+            --qlib_server_name test_server01 \
+            --qlib_client_names test_client01 \
+            --admin_username test_user \
+            --ssh_key_value ~/.ssh/id_rsa.pub \
+            --size standard_NV6_Promo\
+            --conf_path azure_conf.yaml
+    ```
+
+To know more about one-click Deployment, please refer to [Qlib-Server One-click Deployment](https://qlib-server.readthedocs.io/en/latest/build.html#one-click-deployment).
+
+### Step-by-step Deployment
+
+To know more about step-by-step Deployment, please refer to [Qlib-Server Step-by-step Deplyment]https://qlib-server.readthedocs.io/en/latest/build.html#step-by-step-deployment).
+
+
+## Using `Qlib` in `Online` Mode
+
+In the [Qlib Document](https://qlib.readthedocs.io/en/latest), the `Offline` mode has been introduced. 
+
+With `Qlib-Server`, you can use `Qlib` in `Online` mode, please initialize `Qlib` with the following code:
+
+```python
+import qlib
+ONLINE_CONFIG = {
+    # data provider config
+    "calendar_provider": {"class": "LocalCalendarProvider", "kwargs": {"remote": True}},
+    "instrument_provider": "ClientInstrumentProvider",
+    "feature_provider": {"class": "LocalFeatureProvider", "kwargs": {"remote": True}},
+    "expression_provider": "LocalExpressionProvider",
+    "dataset_provider": "ClientDatasetProvider",
+    "provider": "ClientProvider",
+    # config it in user's own code
+    "provider_uri": "127.0.0.1:/",
+    # cache
+    # Using parameter 'remote' to announce the client is using server_cache, and the writing access will be disabled.
+    "expression_cache": None,
+    "dataset_cache": None,
+    "calendar_cache": None,
+    "mount_path": "/data/stock_data/qlib_data",
+    "auto_mount": True,  # The nfs is already mounted on our server[auto_mount: False].
+    "flask_server": "127.0.0.1",
+    "flask_port": 9710,
+    "region": "cn",
+}
+
+qlib.init(**client_config)
+ins = D.list_instruments(D.instrumetns("all"), as_list=True)
+
+```
+
+For more details, please refer to [Qlib-Server Client](https://qlib-server.readthedocs.io/en/latest/client.html).
 
+# More About Qlib-Server
 
-# More About qlib-server
+The detailed documents are organized in docs. Sphinx and the readthedocs theme is required to build the documentation in html formats.
 
+```bash
+cd docs/
+conda install sphinx sphinx_rtd_theme -y
+# Otherwise, you can install them with pip
+# pip install sphinx sphinx_rtd_theme
+make html
+```
+
+You can also view the [latest document](https://qlib-server.readthedocs.io/en/latest/) online directly.
 
 # Contributing
 

docker_support/Dockerfile

@@ -1,24 +1,29 @@
 FROM continuumio/miniconda3:4.8.2
-ARG QLIB_DATA
-ARG QUEUE_HOST
-ARG REDIS_HOST
-ARG QUEUE_USER
-ARG QUEUE_PASS
-ARG FLASK_SERVER_HOST
-ENV HOME=root
-WORKDIR $HOME/qlib_server
+
+ARG DEFAULT_HOST=127.0.0.1
+ARG QLIB_DATA=/data/stock_data/qlib_data
+ARG QUEUE_HOST=$DEFAULT_HOST
+ARG REDIS_HOST=$DEFAULT_HOST
+ARG QUEUE_USER=guest
+ARG QUEUE_PASS=guest
+ARG FLASK_SERVER_HOST=$DEFAULT_HOST
+ARG QLIB_CODE=/code
+ARG SERVER_DIR=$QLIB_CODE/qlib-server
+
+WORKDIR $SERVER_DIR
 COPY . .
-VOLUME $QLIB_DATA
+
 RUN apt-get update\
     && apt-get install -y g++\
     && apt-get clean\
     && rm -rf /var/lib/apt/lists/*\
+    && cd ..\
     && git clone https://github.com/Microsoft/qlib\
     && cd qlib\
     && pip install --upgrade pip\
     && pip install --upgrade cython packaging numpy\
     && python setup.py install\
-    && cd ..\
+    && cd $SERVER_DIR\
     && python setup.py install\
     && /bin/sed -i "s@<QLIB_DATA>@$QLIB_DATA@g" config_template.yaml\
     && /bin/sed -i "s@<QUEUE_HOST>@$QUEUE_HOST@g" config_template.yaml\
@@ -29,4 +34,4 @@ RUN apt-get update\
     && git clone https://github.com/vishnubob/wait-for-it.git
 
 EXPOSE 9710
-CMD ["python", "main.py", "-c", "config_template.yaml"]
+CMD ["python", "main.py", "-c", "config_template.yaml"]

docker_support/.env → docker_support/docker-compose.env

@@ -5,3 +5,6 @@ RABBITMQ_DEFAULT_PASS=guest
 # qlib data
 QLIB_DATA=/data/stock_data/qlib_data
 FLASK_SERVER_HOST=0.0.0.0
+
+# qlib code dir
+QLIB_CODE=/code

docker_support/docker-compose.yml → docker_support/docker-compose.yaml

@@ -23,7 +23,7 @@ services:
     restart: always
   qlib-server:
     build: 
-      context: .
+      context: ../
       dockerfile: ./docker_support/Dockerfile
       args:
         - QUEUE_HOST=rabbitmq
@@ -32,6 +32,7 @@ services:
         - QUEUE_PASS=${RABBITMQ_DEFAULT_PASS}
         - QLIB_DATA=${QLIB_DATA}
         - FLASK_SERVER_HOST=${FLASK_SERVER_HOST}
+        - QLIB_CODE=${QLIB_CODE}
     volumes:
       - ${QLIB_DATA}:${QLIB_DATA}
     depends_on:

docs/build.rst

@@ -1,6 +1,6 @@
 .. _build:
 ==============================
-``Qlib-Server`` Building
+``Qlib-Server`` Deployment
 ==============================
 
 Introduction
@@ -27,15 +27,16 @@ Deploy ``Qlib-Server`` with docker-compose according to the following processes:
 
 - Install ``docker``, please refer to `Docker Installation <https://docs.docker.com/engine/install>`_.
 - Install ``docker-compose``, please refer to `Docker-compose Installation <https://docs.docker.com/compose/install/>`_.
-- Run the following command to deploy `Qlib-Server`:
+- Run the following command to deploy ``Qlib-Server``:
 
     .. code-block:: bash
 
         git clone https://github.com/microsoft/qlib-server
         cd qlib-server
-        sudo docker-compose up -d -f docker_support/docker_compose.yaml
+        sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env build
+        sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env up -d
         # Use the following command to track the log
-        sudo docker-compose -f docker_support/docker_compose.yaml logs -f
+        sudo docker-compose -f docker_support/docker-compose.yaml logs -f
 
 
 One-click Deployment in ``Azure``
@@ -376,6 +377,7 @@ Build ``Qlib-Server`` with Dockerfile according to the following processes:
                 REDIS_HOST=redis_server \
                 QUEUE_USER=rabbitmq_user \
                 QUEUE_PASS=rebbitmq_password \
-                FLASK_SERVER_HOST=127.0.0.1
+                FLASK_SERVER_HOST=127.0.0.1 \
+                QLIB_CODE=/code
         sudo docker run qlib-server
 

docs/client.rst

@@ -1,21 +1,21 @@
 
 =================================================
-Using ``Qlib`` in ``online`` Mode
+Using ``Qlib`` in ``Online`` Mode
 =================================================
 
 Introduction
 ================
-In the `Qlib document <TODO_URL>`_, the ``offline`` mode has been introduced. In addition to ``offline`` mode, users can use ``Qlib`` in ``online`` mode.
+In the `Qlib Document <https://qlib.readthedocs.io/en/latest>`_, the ``Offline`` mode has been introduced. In addition to ``offline`` mode, users can use ``Qlib`` in ``Online`` mode.
 
-The ``online`` mode is designed to solve the following problems:
+The ``Online`` mode is designed to solve the following problems:
 
 - Manage the data in a centralized way. Users don't have to manage data of different versions.
 - Reduce the amount of cache to be generated.
 - Make the data can be accessed in a remote way.
 
-In ``online`` mode, the data provided for ``Qlib`` will be managed in a centralized manner by ``Qlib-Server``.
+In ``Online`` mode, the data provided for ``Qlib`` will be managed in a centralized manner by ``Qlib-Server``.
 
-Using ``Qlib`` in ``online`` Mode
+Using ``Qlib`` in ``Online`` Mode
 =========================================
 
 Use ``Qlib`` in ``online`` mode according to the following steps:

docs/index.rst

@@ -5,12 +5,36 @@
 
 .. _user_guide:
 
+Document Structure
 ====================
 
+
+.. toctree::
+   :hidden:
+
+   Home <self>
+
+.. toctree::
+   :maxdepth: 3
+   :caption: INTRODUCTION:
+
+   Qlib-Server <server.rst>
+
+
+.. toctree::
+    :maxdepth: 3
+    :caption: BUILDING:
+
+    Qlib-Server Deployment <build.rst>
+
+.. toctree::
+    :maxdepth: 3
+    :caption: USAGE:
+
+    Qlib Online Mode <client.rst>
+
 .. toctree::
     :maxdepth: 3
+    :caption: CHANGELOG:
 
-    server
-    build
-    client
-    changelog
+    Changelog <changelog.rst>

docs/server.rst

@@ -1,28 +1,40 @@
 .. _server:
-==========================
-``Qlib-Server`` Structure
-==========================
-
+===========================================
+``Qlib-Server``: Quant Library Data Server
+===========================================
 .. currentmodule:: qlib_server
 
-The ``Client/Server structure`` of ``Qlib`` is based on WebSocket considering its capability of **bidirectional communication** between client and server in **async** mode.
+Introduction
+==================
+``Qlib-Server`` is the assorted server system for ``Qlib``, which utilizes ``Qlib`` for basic calculations and provides extensive server system and cache mechanism. With ``Qlib-Server``, the data provided for ``Qlib`` can be managed in a centralized manner.
+
+
+Framework
+==================
+
+.. image:: ./_static/img/framework.png
+    :align: center
+
+
+The ``Client/Server`` framework of ``Qlib`` is based on ``WebSocket`` considering its capability of **bidirectional communication** between client and server in **async** mode.
+
+
 
-Server
-================
+``Qlib-Server`` is based on `Flask <http://flask.pocoo.org/>`_, which is a micro-framework for Python and here `Flask-SocketIO <https://flask-socketio.readthedocs.io>`_ is used for websocket connection. 
 
-The server is based on `flask<http://flask.pocoo.org/>`_. Flask is a micro-framework for Python and here we use `flask-SocketIO<https://flask-socketio.readthedocs.io>`_ for websocket connection. The server provides the following functions:
+``Qlib-Server`` provides the following procedures:
 
 Listening to incoming request from client
 --------------------------------------------
 
-The clients will propose several types of requests to server. The server will parse the requests, collect the identical requests from different clients, record their session-ids, and submit these parsed tasks to a pipe. ``Qlib`` use `RabbitMQ<https://www.rabbitmq.com>`_ as this pipe. The tasks will be published to a channel `task_queue`.
+The clients will propose several types of requests to server. The server will parse the requests, collect the identical requests from different clients, record their session-ids, and submit these parsed tasks to a pipe. ``Qlib`` use `RabbitMQ <https://www.rabbitmq.com>`_ as this pipe. The tasks will be published to a channel `task_queue`.
 
 **RequestListener** is used for this function:
 
 .. autoclass:: qlib_server.request_handler.RequestListener
 
 
-After receiving these requests, the server will check whether different clients are asking for the same data. If so, to prevent repeated generation of data or repeated generation of cache files, the server will use `redis<https://redis.io/>`_ to maintain the session-ids of those clients. These session-ids will be deleted once this task is finished. To avoid IO conflicts, `redis_lock<https://pypi.org/project/python-redis-lock/>`_ is imported to make sure no tasks in redis will be read and written at the same time.
+After receiving these requests, the server will check whether different clients are asking for the same data. If so, to prevent repeated generation of data or repeated generation of cache files, the server will use `Redis <https://redis.io/>`_ to maintain the session-ids of those clients. These session-ids will be deleted once this task is finished. To avoid IO conflicts, `Redis_Lock <https://pypi.org/project/python-redis-lock/>`_ is imported to make sure no tasks in redis will be read and written at the same time.
 
 Responding clients with data
 -------------------------------

qlib_server/__init__.py

@@ -1,4 +1,3 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
-
-__version__ = "0.1.5.dev"
+__version__ = "0.2.0"

qlib_server/request_handler.py

@@ -58,6 +58,8 @@ def on_disconnect(self):
     @staticmethod
     def check_version(v):
         ver = C.client_version
+        if v.lower().endswith(".dev"):
+            v = v[: -4]
         if version.parse(v) not in SpecifierSet(ver):
             raise Exception("Client version mismatch, please upgrade your qlib client ({})".format(ver))