diff --git a/README.md b/README.md index 8f15323..43a1987 100644 --- a/README.md +++ b/README.md @@ -1,24 +1,149 @@ -QLibServer is the assorted server system for QLib, which utilizes QLib for basic calculations and provides extensive server system and cache mechanism. With QLibServer, the data provided for QLib can be managed in a centralized manner. +`Qlib-Server` is the data server system for [`Qlib`](https://github.com/microsoft/qlib). It enable `Qlib` to run in `online` mode. Under online mode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too. +To sum up, `Qlib-Server` is designed to solve the following problems: + +* Manage the data in a centralized way, which makes data management (including cache management, date updating) much easier. +* Reduce the amount of cache to be generated. +* Make the client light-weighted and leverage the powerful computing resources of remote server - [Framework of qlib-erver](#framework-of-qlib-server) - [Quick start](#quick-start) - - [Installation](#installation) + - [Deployment](#deployment) + - [One-click Deployment](#one-click-deployment) + - [Step-by-Step Deployment](#step-by-step-deployment) + - [Using Qlib in Online Mode](#using-qlib-in-online-mode) - [More About Qlib](#more-about-qlib) +- [Contributing](#contributing) + +# Framework of Qlib-Server -# Framework of qlib-server +
+ +
+The `Client/Server` framework of `Qlib` is based on `WebSocket` considering its capability of **bidirectional communication** between client and server in **async** mode. + +`Qlib-Server` is based on [Flask](http://flask.pocoo.org/), which is a micro-framework for Python and here [Flask-SocketIO](https://flask-socketio.readthedocs.io) is used for websocket connection. # Quick start -## Installation +## Deployment + +### One-click Deployment + +One-click deployment of `Qlib-Server` is supported, you can choose either of the following two methods for one-click deployment: + +- Deployment with `docker-compose` +- Deployment in `Azure` + +#### One-click Deployment with `docker-compose` + +Deploy `Qlib-Server` with `docker-compose` according to the following processes: + +* Install `docker`, please refer to [Docker Installation](https://docs.docker.com/engine/install). +* Install `docker-compose`, please refer to [Docker-compose Installation](https://docs.docker.com/compose/install/). +* Run the following command to deploy `Qlib-Server`: + + ```bash + git clone https://github.com/microsoft/qlib-server + cd qlib-server + sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env build + sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env up -d + # Use the following command to track the log + sudo docker-compose -f docker_support/docker-compose.yaml logs -f + ``` + +#### One-click Deployment in `Azure` + +Firstly, You need to have an `Azure` account to deploy `Qlib-Server` in `Azure`. Then you can deploy `Qlib-Server` in `Azure` according to the following processes: + +* Install `azure-cli`, please refer to [install-azure-cli](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest). + +* Add the `Azure` account to the configuration file `azure_conf.yaml` + + ```yaml + sub_id: Your Subscription ID + username: azure user name + password: azure password + # The resource group where the VM is located + resource_group: Resource group name + ``` +* Execute the deployment script + Run the following command: + + ```bash + + git clone https://github.com/microsoft/qlib-server + cd qlib-server/scripts + python azure_manager.py create_qlib_cs_vm \ + --qlib_server_name test_server01 \ + --qlib_client_names test_client01 \ + --admin_username test_user \ + --ssh_key_value ~/.ssh/id_rsa.pub \ + --size standard_NV6_Promo\ + --conf_path azure_conf.yaml + ``` + +To know more about one-click Deployment, please refer to [Qlib-Server One-click Deployment](https://qlib-server.readthedocs.io/en/latest/build.html#one-click-deployment). + +### Step-by-step Deployment + +To know more about step-by-step Deployment, please refer to [Qlib-Server Step-by-step Deplyment]https://qlib-server.readthedocs.io/en/latest/build.html#step-by-step-deployment). + + +## Using `Qlib` in `Online` Mode + +In the [Qlib Document](https://qlib.readthedocs.io/en/latest), the `Offline` mode has been introduced. + +With `Qlib-Server`, you can use `Qlib` in `Online` mode, please initialize `Qlib` with the following code: + +```python +import qlib +ONLINE_CONFIG = { + # data provider config + "calendar_provider": {"class": "LocalCalendarProvider", "kwargs": {"remote": True}}, + "instrument_provider": "ClientInstrumentProvider", + "feature_provider": {"class": "LocalFeatureProvider", "kwargs": {"remote": True}}, + "expression_provider": "LocalExpressionProvider", + "dataset_provider": "ClientDatasetProvider", + "provider": "ClientProvider", + # config it in user's own code + "provider_uri": "127.0.0.1:/", + # cache + # Using parameter 'remote' to announce the client is using server_cache, and the writing access will be disabled. + "expression_cache": None, + "dataset_cache": None, + "calendar_cache": None, + "mount_path": "/data/stock_data/qlib_data", + "auto_mount": True, # The nfs is already mounted on our server[auto_mount: False]. + "flask_server": "127.0.0.1", + "flask_port": 9710, + "region": "cn", +} + +qlib.init(**client_config) +ins = D.list_instruments(D.instrumetns("all"), as_list=True) + +``` + +For more details, please refer to [Qlib-Server Client](https://qlib-server.readthedocs.io/en/latest/client.html). +# More About Qlib-Server -# More About qlib-server +The detailed documents are organized in docs. Sphinx and the readthedocs theme is required to build the documentation in html formats. +```bash +cd docs/ +conda install sphinx sphinx_rtd_theme -y +# Otherwise, you can install them with pip +# pip install sphinx sphinx_rtd_theme +make html +``` + +You can also view the [latest document](https://qlib-server.readthedocs.io/en/latest/) online directly. # Contributing diff --git a/docker_support/Dockerfile b/docker_support/Dockerfile index 6723c25..a349104 100644 --- a/docker_support/Dockerfile +++ b/docker_support/Dockerfile @@ -1,24 +1,29 @@ FROM continuumio/miniconda3:4.8.2 -ARG QLIB_DATA -ARG QUEUE_HOST -ARG REDIS_HOST -ARG QUEUE_USER -ARG QUEUE_PASS -ARG FLASK_SERVER_HOST -ENV HOME=root -WORKDIR $HOME/qlib_server + +ARG DEFAULT_HOST=127.0.0.1 +ARG QLIB_DATA=/data/stock_data/qlib_data +ARG QUEUE_HOST=$DEFAULT_HOST +ARG REDIS_HOST=$DEFAULT_HOST +ARG QUEUE_USER=guest +ARG QUEUE_PASS=guest +ARG FLASK_SERVER_HOST=$DEFAULT_HOST +ARG QLIB_CODE=/code +ARG SERVER_DIR=$QLIB_CODE/qlib-server + +WORKDIR $SERVER_DIR COPY . . -VOLUME $QLIB_DATA + RUN apt-get update\ && apt-get install -y g++\ && apt-get clean\ && rm -rf /var/lib/apt/lists/*\ + && cd ..\ && git clone https://github.com/Microsoft/qlib\ && cd qlib\ && pip install --upgrade pip\ && pip install --upgrade cython packaging numpy\ && python setup.py install\ - && cd ..\ + && cd $SERVER_DIR\ && python setup.py install\ && /bin/sed -i "s@@$QLIB_DATA@g" config_template.yaml\ && /bin/sed -i "s@@$QUEUE_HOST@g" config_template.yaml\ @@ -29,4 +34,4 @@ RUN apt-get update\ && git clone https://github.com/vishnubob/wait-for-it.git EXPOSE 9710 -CMD ["python", "main.py", "-c", "config_template.yaml"] \ No newline at end of file +CMD ["python", "main.py", "-c", "config_template.yaml"] diff --git a/docker_support/.env b/docker_support/docker-compose.env similarity index 81% rename from docker_support/.env rename to docker_support/docker-compose.env index 8d86fde..ac63373 100644 --- a/docker_support/.env +++ b/docker_support/docker-compose.env @@ -5,3 +5,6 @@ RABBITMQ_DEFAULT_PASS=guest # qlib data QLIB_DATA=/data/stock_data/qlib_data FLASK_SERVER_HOST=0.0.0.0 + +# qlib code dir +QLIB_CODE=/code diff --git a/docker_support/docker-compose.yml b/docker_support/docker-compose.yaml similarity index 95% rename from docker_support/docker-compose.yml rename to docker_support/docker-compose.yaml index 5bffe1e..133cb38 100644 --- a/docker_support/docker-compose.yml +++ b/docker_support/docker-compose.yaml @@ -23,7 +23,7 @@ services: restart: always qlib-server: build: - context: . + context: ../ dockerfile: ./docker_support/Dockerfile args: - QUEUE_HOST=rabbitmq @@ -32,6 +32,7 @@ services: - QUEUE_PASS=${RABBITMQ_DEFAULT_PASS} - QLIB_DATA=${QLIB_DATA} - FLASK_SERVER_HOST=${FLASK_SERVER_HOST} + - QLIB_CODE=${QLIB_CODE} volumes: - ${QLIB_DATA}:${QLIB_DATA} depends_on: diff --git a/docs/_static/img/framework.png b/docs/_static/img/framework.png new file mode 100644 index 0000000..98c8c29 Binary files /dev/null and b/docs/_static/img/framework.png differ diff --git a/docs/build.rst b/docs/build.rst index a83e755..842cdc3 100644 --- a/docs/build.rst +++ b/docs/build.rst @@ -1,6 +1,6 @@ .. _build: ============================== -``Qlib-Server`` Building +``Qlib-Server`` Deployment ============================== Introduction @@ -27,15 +27,16 @@ Deploy ``Qlib-Server`` with docker-compose according to the following processes: - Install ``docker``, please refer to `Docker Installation `_. - Install ``docker-compose``, please refer to `Docker-compose Installation `_. -- Run the following command to deploy `Qlib-Server`: +- Run the following command to deploy ``Qlib-Server``: .. code-block:: bash git clone https://github.com/microsoft/qlib-server cd qlib-server - sudo docker-compose up -d -f docker_support/docker_compose.yaml + sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env build + sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env up -d # Use the following command to track the log - sudo docker-compose -f docker_support/docker_compose.yaml logs -f + sudo docker-compose -f docker_support/docker-compose.yaml logs -f One-click Deployment in ``Azure`` @@ -376,6 +377,7 @@ Build ``Qlib-Server`` with Dockerfile according to the following processes: REDIS_HOST=redis_server \ QUEUE_USER=rabbitmq_user \ QUEUE_PASS=rebbitmq_password \ - FLASK_SERVER_HOST=127.0.0.1 + FLASK_SERVER_HOST=127.0.0.1 \ + QLIB_CODE=/code sudo docker run qlib-server diff --git a/docs/client.rst b/docs/client.rst index a927c60..9f03777 100644 --- a/docs/client.rst +++ b/docs/client.rst @@ -1,21 +1,21 @@ ================================================= -Using ``Qlib`` in ``online`` Mode +Using ``Qlib`` in ``Online`` Mode ================================================= Introduction ================ -In the `Qlib document `_, the ``offline`` mode has been introduced. In addition to ``offline`` mode, users can use ``Qlib`` in ``online`` mode. +In the `Qlib Document `_, the ``Offline`` mode has been introduced. In addition to ``offline`` mode, users can use ``Qlib`` in ``Online`` mode. -The ``online`` mode is designed to solve the following problems: +The ``Online`` mode is designed to solve the following problems: - Manage the data in a centralized way. Users don't have to manage data of different versions. - Reduce the amount of cache to be generated. - Make the data can be accessed in a remote way. -In ``online`` mode, the data provided for ``Qlib`` will be managed in a centralized manner by ``Qlib-Server``. +In ``Online`` mode, the data provided for ``Qlib`` will be managed in a centralized manner by ``Qlib-Server``. -Using ``Qlib`` in ``online`` Mode +Using ``Qlib`` in ``Online`` Mode ========================================= Use ``Qlib`` in ``online`` mode according to the following steps: diff --git a/docs/index.rst b/docs/index.rst index f90e8b2..2d09642 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -5,12 +5,36 @@ .. _user_guide: +Document Structure ==================== + +.. toctree:: + :hidden: + + Home + +.. toctree:: + :maxdepth: 3 + :caption: INTRODUCTION: + + Qlib-Server + + +.. toctree:: + :maxdepth: 3 + :caption: BUILDING: + + Qlib-Server Deployment + +.. toctree:: + :maxdepth: 3 + :caption: USAGE: + + Qlib Online Mode + .. toctree:: :maxdepth: 3 + :caption: CHANGELOG: - server - build - client - changelog \ No newline at end of file + Changelog diff --git a/docs/server.rst b/docs/server.rst index 732c277..17febac 100644 --- a/docs/server.rst +++ b/docs/server.rst @@ -1,28 +1,40 @@ .. _server: -========================== -``Qlib-Server`` Structure -========================== - +=========================================== +``Qlib-Server``: Quant Library Data Server +=========================================== .. currentmodule:: qlib_server -The ``Client/Server structure`` of ``Qlib`` is based on WebSocket considering its capability of **bidirectional communication** between client and server in **async** mode. +Introduction +================== +``Qlib-Server`` is the assorted server system for ``Qlib``, which utilizes ``Qlib`` for basic calculations and provides extensive server system and cache mechanism. With ``Qlib-Server``, the data provided for ``Qlib`` can be managed in a centralized manner. + + +Framework +================== + +.. image:: ./_static/img/framework.png + :align: center + + +The ``Client/Server`` framework of ``Qlib`` is based on ``WebSocket`` considering its capability of **bidirectional communication** between client and server in **async** mode. + + -Server -================ +``Qlib-Server`` is based on `Flask `_, which is a micro-framework for Python and here `Flask-SocketIO `_ is used for websocket connection. -The server is based on `flask`_. Flask is a micro-framework for Python and here we use `flask-SocketIO`_ for websocket connection. The server provides the following functions: +``Qlib-Server`` provides the following procedures: Listening to incoming request from client -------------------------------------------- -The clients will propose several types of requests to server. The server will parse the requests, collect the identical requests from different clients, record their session-ids, and submit these parsed tasks to a pipe. ``Qlib`` use `RabbitMQ`_ as this pipe. The tasks will be published to a channel `task_queue`. +The clients will propose several types of requests to server. The server will parse the requests, collect the identical requests from different clients, record their session-ids, and submit these parsed tasks to a pipe. ``Qlib`` use `RabbitMQ `_ as this pipe. The tasks will be published to a channel `task_queue`. **RequestListener** is used for this function: .. autoclass:: qlib_server.request_handler.RequestListener -After receiving these requests, the server will check whether different clients are asking for the same data. If so, to prevent repeated generation of data or repeated generation of cache files, the server will use `redis`_ to maintain the session-ids of those clients. These session-ids will be deleted once this task is finished. To avoid IO conflicts, `redis_lock`_ is imported to make sure no tasks in redis will be read and written at the same time. +After receiving these requests, the server will check whether different clients are asking for the same data. If so, to prevent repeated generation of data or repeated generation of cache files, the server will use `Redis `_ to maintain the session-ids of those clients. These session-ids will be deleted once this task is finished. To avoid IO conflicts, `Redis_Lock `_ is imported to make sure no tasks in redis will be read and written at the same time. Responding clients with data ------------------------------- diff --git a/qlib_server/__init__.py b/qlib_server/__init__.py index acb8d32..4040b7e 100644 --- a/qlib_server/__init__.py +++ b/qlib_server/__init__.py @@ -1,4 +1,3 @@ # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. - -__version__ = "0.1.5.dev" +__version__ = "0.2.0" \ No newline at end of file diff --git a/qlib_server/request_handler.py b/qlib_server/request_handler.py index d03dcb4..2cdc2ef 100644 --- a/qlib_server/request_handler.py +++ b/qlib_server/request_handler.py @@ -58,6 +58,8 @@ def on_disconnect(self): @staticmethod def check_version(v): ver = C.client_version + if v.lower().endswith(".dev"): + v = v[: -4] if version.parse(v) not in SpecifierSet(ver): raise Exception("Client version mismatch, please upgrade your qlib client ({})".format(ver)) diff --git a/scripts/azure_manager.py b/scripts/azure_manager.py index f4cf304..0a11449 100644 --- a/scripts/azure_manager.py +++ b/scripts/azure_manager.py @@ -98,6 +98,8 @@ def __init__(self, host, user, password=None, ssh_private_key=None): password (str): password, default None; If password is None, use ssh_key. ssh_private_key (Path, str): ssh public key path, default None; If ssh_key is None, use password. """ + self._host = host + self._user = user if password is None and ssh_private_key is None: logger.warning("ssh_private_key and password are both none, ssh_private_key will use `~/.ssh/id_rsa`!") ssh_private_key = str(Path("~/.ssh/id_rsa").expanduser().resolve()) @@ -105,11 +107,11 @@ def __init__(self, host, user, password=None, ssh_private_key=None): if ssh_private_key is not None: ssh_key = str(Path(ssh_private_key).expanduser().resolve()) connect_kwargs = {"key_filename": ssh_key} - self.conn = Connection(host=host, user=user, connect_kwargs=connect_kwargs) + self.conn = Connection(host=self._host, user=self._user, connect_kwargs=connect_kwargs) else: config = Config(overrides={"sudo": {"password": password}}) connect_kwargs = {"password": password} - self.conn = Connection(host=host, user=user, connect_kwargs=connect_kwargs, config=config) + self.conn = Connection(host=self._host, user=self._user, connect_kwargs=connect_kwargs, config=config) self._home_dir = None @@ -122,12 +124,12 @@ def home_dir(self): def _execute_remote_shell(self, files, remote_dir=None): if remote_dir is None: remote_dir = self.home_dir - self.run(f"if [ ! -d {remote_dir} ]; then sudo mkdir -p {remote_dir}; fi") + self.mkdir_remote(remote_dir) if not isinstance(files, Iterable): files = [files] for file_path in map(lambda x: Path(x), files): - self.conn.put(str(file_path), remote_dir) + self.put(str(file_path), remote_dir) # NOTE: This place may require a password # Consider using `self.conn.sudo(f"/bin/bash /home/{user}/{shell_path.name}")` instead self.run(f"/bin/bash {remote_dir}/{file_path.name}") @@ -143,8 +145,8 @@ def _dump_remote_yaml(self, obj, remote_file_path="tmp.yaml"): with temp_client_path.open("w") as fp: yaml.dump(obj, fp) # create remote dir - self.run(f"if [ ! -d {r_dir} ]; then sudo mkdir -p {r_dir}; fi") - self.conn.put(str(temp_client_path), r_dir) + self.mkdir_remote(r_dir) + self.put(str(temp_client_path), r_dir) # delete tmp file temp_client_path.unlink() @@ -182,7 +184,7 @@ def deploy_qlib_server(self, client_config=None): client_config["auto_mount"] = False self._dump_remote_yaml(client_config, "~/qlib_client_config.yaml") - def run(self, command, hide=True): + def run(self, command, hide=None): """run command in remote server Parameters @@ -190,10 +192,24 @@ def run(self, command, hide=True): command : str command hide : bool, str - hide shell stdout or stderr, value from stdout/stderr/True, default True + hide shell stdout or stderr, value from stdout/stderr/True(stdout and stderr)/None(False), default None """ + logger.info(f"remote {self._user}@{self._host} running command: {command}") return self.conn.run(command, hide=hide) + def put(self, local_path, remote_dir): + """put file to remote + + Parameters + ---------- + local_path: str + local file path + remote_dir: str + remote dir + """ + logger.info(f"putting {local_path} ot {self._user}@{self._host}:{remote_dir}") + self.conn.put(local_path, remote_dir) + def run_python(self, command, hide=True, sudo=False): """run python command @@ -215,6 +231,21 @@ def run_python(self, command, hide=True, sudo=False): command = f"sudo {command}" return self.run(command, hide=hide) + def mkdir_remote(self, dir_path, sudo=True): + """mkdir dir in remote + + Parameters + ---------- + dir_path : str + remote dir path + sudo : bool + use sudo, value from True or False, by default True + """ + if sudo: + self.run(f"if [ ! -d {dir_path} ]; then sudo mkdir -p {dir_path}; fi") + else: + self.run(f"if [ ! -d {dir_path} ]; then mkdir -p {dir_path}; fi") + class CommandManager: def __init__(self, conf_path="azure_conf.yaml", region=VMManager.REG_OTHER): @@ -253,13 +284,26 @@ def __init__(self, conf_path="azure_conf.yaml", region=VMManager.REG_OTHER): # NOTE: China needs to be set `AzureChinaCloud` to login successfully subprocess.run("az cloud set -n AzureChinaCloud", shell=True) - login_cmd = f"az login -u '{self.username}' -p '{self.password}'" - if subprocess.check_call(login_cmd, shell=True, stdout=subprocess.DEVNULL) != 0: - raise ValueError("Invalid credential") + if subprocess.run("az account show", shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL).returncode != 0: + login_cmd = f"az login -u '{self.username}' -p '{self.password}'" + subprocess.check_call(login_cmd, shell=True, stdout=subprocess.DEVNULL) set_subscription_cmd = f"az account set --subscription {self.sub_id}" - if subprocess.check_call(set_subscription_cmd, shell=True, stdout=subprocess.DEVNULL) != 0: - raise ValueError("Invalid set subscription") + subprocess.check_call(set_subscription_cmd, shell=True, stdout=subprocess.DEVNULL) + + def _azure_run(self, command, output=False, stdout=None, stderr=None): + """run command in azure-cli + + Parameters + ---------- + command : str + shell command + """ + if not command.startswith("az "): + command = f"az {command}" + command = f"{command} -g {self.resource_group}" + logger.info(f"running: {command}") + return subprocess.check_output(command, shell=True) if output else subprocess.run(command, shell=True, stdout=stdout, stderr=stderr) def start(self, *vm_names): """start vm @@ -280,7 +324,7 @@ def start(self, *vm_names): """ logger.info(vm_names) for name in vm_names: - subprocess.run(f"az vm start -n {name} -g {self.resource_group}", shell=True) + self._azure_run(f"vm start -n {name}") def stop(self, *vm_names): """stop vm @@ -302,21 +346,22 @@ def stop(self, *vm_names): logger.info(vm_names) for name in vm_names: - subprocess.run(f"az vm deallocate -n {name} -g {self.resource_group}", shell=True) + self._azure_run(f"vm deallocate -n {name}") def _list(self, only_running=False): - output = subprocess.check_output(f"az vm list -d -g {self.resource_group}", shell=True) + output = self._azure_run(f"vm list -d", output=True) ret = json.loads(output) vmm = VMManager(ret) for vm in vmm.list(only_running=only_running): yield vm def _list_ip(self, vm_name=None, ip_type="private"): - cmd_str = "az vm list-ip-addresses" + logger.info(f"ip_type={ip_type}") + cmd_str = "vm list-ip-addresses" if vm_name is not None: cmd_str = f"{cmd_str} --name {vm_name}" - output = subprocess.check_output(cmd_str, shell=True) + output = self._azure_run(cmd_str, output=True) ret = json.loads(output) vmm = VMManager(ret) for vm in vmm.list_ip(ip_type=ip_type): @@ -366,7 +411,7 @@ def view(self): $ python azure_manager.py view --only_running --config_path azure_conf.yaml --region cn """ - subprocess.run("az vm list -d", shell=True) + self._azure_run("vm list -d") def create_vm( self, @@ -419,12 +464,15 @@ def create_vm( if ssh_key_value is None and admin_password is None: logger.error("ssh_key_value and admin_password cannot be none at the same time!") return - shell = f"az vm create -g {self.resource_group} --image {image} --size {size} --admin-username {admin_username} --public-ip-address-allocation {public_ip_address_allocation} " + shell = f"vm create --image {image} --size {size} --admin-username {admin_username} --public-ip-address-allocation {public_ip_address_allocation} " if vnet_name: shell += f"--vnet-name {vnet_name} " if nsg: shell += f"--nsg {nsg} " if ssh_key_value: + ssh_key_path = Path(ssh_key_value).expanduser().resolve() + if ssh_key_path.exists(): + ssh_key_value = str(ssh_key_path) shell += f"--ssh-key-value {ssh_key_value} " if admin_password: shell += f"--admin-password {admin_password} " @@ -432,7 +480,7 @@ def create_vm( for name in vm_names.split(","): name = name.strip() logger.info(f"create vm: {name}...") - subprocess.run(shell + f"-n {name}", shell=True) + self._azure_run(shell + f"-n {name}") logger.info(f"create vm finished: {name}") def create_qlib_cs_vm( diff --git a/scripts/install_qlib_client.sh b/scripts/install_qlib_client.sh index 82696e3..9ccf2e7 100644 --- a/scripts/install_qlib_client.sh +++ b/scripts/install_qlib_client.sh @@ -10,22 +10,25 @@ CODE_DIR=$HOME/"code" DOWNLOADS_DIR=$HOME/"downloads" CONDA_DIR=$HOME/"miniconda3" -# create dir -if [ ! -d "$CONDA_DIR" ]; then - mkdir $CONDA_DIR -fi - -if [ ! -d "$CODE_DIR" ]; then - mkdir $CODE_DIR -fi -if [ ! -d "$DOWNLOADS_DIR" ]; then - mkdir $DOWNLOADS_DIR -fi - -if [ ! -d "$STOCK_DATA_DIR" ]; then - sudo mkdir -p $STOCK_DATA_DIR -fi +# create dir +function create_dir_by_sudo() { + if [ ! -d $1 ]; then + sudo mkdir -p $1 + fi +} + +function create_dir() { + if [ ! -d $1 ]; then + mkdir $1 + fi +} + + +create_dir $CONDA_DIR +create_dir $CODE_DIR +create_dir $DOWNLOADS_DIR +create_dir_by_sudo $STOCK_DATA_DIR # install miniconda3 wget $MINICONDA -O $DOWNLOADS_DIR/"miniconda3.sh" diff --git a/scripts/install_qlib_server.sh b/scripts/install_qlib_server.sh index ab42835..d162155 100644 --- a/scripts/install_qlib_server.sh +++ b/scripts/install_qlib_server.sh @@ -8,16 +8,17 @@ STOCK_DATA_DIR=/data/stock_data/qlib_data CODE_DIR=$HOME/"code" +# install docker function install_docker_in_ubuntu() { - sudo apt-get update; - sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common; - curl -fsSL https://download.docker.com/linux/ubuntu/gpg |sudo apt-key add -; + sudo apt-get update + sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common + curl -fsSL https://download.docker.com/linux/ubuntu/gpg |sudo apt-key add - sudo apt-key fingerprint 0EBFCD88 sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ - stable"; - sudo apt-get install -y docker-ce docker-ce-cli containerd.io; + stable" + sudo apt-get install -y docker-ce docker-ce-cli containerd.io } @@ -27,10 +28,25 @@ function install_docker_compose() { sudo chmod +x /usr/local/bin/docker-compose } +# create dir +function create_dir_by_sudo() { + if [ ! -d $1 ]; then + sudo mkdir -p $1 + fi +} + +function create_dir() { + if [ ! -d $1 ]; then + mkdir $1 + fi +} + + install_docker_in_ubuntu install_docker_compose - +create_dir $CODE_DIR +create_dir_by_sudo $STOCK_DATA_DIR # create dir if [ ! -d "$CODE_DIR" ]; then @@ -45,7 +61,7 @@ fi # install qlib server cd $CODE_DIR git clone $QLIB_SERVER -cd qlib_server -sudo docker-compose build -f docker_support/docker_compose.yaml -sudo docker-compose up -d -f docker_support/docker_compose.yaml +cd qlib-server +sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env build +sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env up -d