Dev changes (#56)

* Restructured simulation analysis code
made log extraction a single-binary operation

* Small fixes to make remote trials work well with simulations and VM

* fix

* added metadata in tag of worker image

* changed to cloud logger and some auto-format changes

* more logging changes

* Refinements

* dsub local changes

* proposal propose action outcome JSONified

* sight.silent change and upgraded dsub

* cloud log fix

* incremental

* formatter changes

* more formatter changes

* added setuptools explicit version

* added changes in requirements.txt for different usecases

* updated client dependencies with instruction of formatter

* more

* remove prints

* Merged more

* more merging

* fix for dsub_local after stable merge

* local  setting for yapf formatter

* handle ports related logic

---------

Co-authored-by: Greg Bronevetsky <bronevet@google.com>
Co-authored-by: hrushikeshm-g <hrushikeshm@google.com>
Co-authored-by: Abhinav Rao <abhinavprao@google.com>
Co-authored-by: bronevet-abc <75458629+bronevet-abc@users.noreply.github.com>

Author: 4 people

.config/.isort.cfg

@@ -0,0 +1,5 @@
+[settings]
+profile = google
+use_parentheses = true
+line_length = 80
+multi_line_output = 3

.config/.style.yapf

@@ -0,0 +1,4 @@
+[style]
+based_on_style = google
+indent_width = 2
+column_limit = 80

.config/setup.txt

@@ -0,0 +1,2 @@
+yapf -ir -vv --style .config/.style.yapf .
+isort . --settings-path .config/ -v

.gitignore

@@ -1,7 +1,8 @@
 __pycache__
 *.pyc
 .env
-.vscode/
+.vscode/launch.json
 */.cache_local_data/*
 */.cache_redis_data/*
 extra/*
+**/*.log

.pre-commit-config.yaml

@@ -0,0 +1,23 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.0.1
+    hooks:
+      - id: trailing-whitespace
+        exclude:  ^.*\.patch$
+
+      - id: end-of-file-fixer
+        exclude:  ^.*\.patch$
+
+      - id: check-yaml
+
+  - repo: https://github.com/pre-commit/mirrors-yapf
+    rev: v0.32.0
+    hooks:
+      - id: yapf
+        args: ['--style', '.config/.style.yapf']
+
+  - repo: https://github.com/pre-commit/mirrors-isort
+    rev: v5.10.1
+    hooks:
+      - id: isort
+        args: ['--settings-path', '.config/.isort.cfg']

.vscode/settings.json

@@ -0,0 +1,11 @@
+{
+  "[python]": {
+    "editor.formatOnSaveMode": "file",
+    "editor.formatOnSave": true,
+    "editor.formatOnType": true,
+    "editor.defaultFormatter": "eeyore.yapf",
+  },
+  "python.analysis.typeCheckingMode": "off",
+  "yapf.args": ["--style", "{based_on_style: Google, indent_width: 2, column_limit: 80}"],
+}
+

README.md

@@ -141,7 +141,8 @@ cd venvs
 virtualenv sight_env --python=python3.10
 source ~/venvs/sight_env/bin/activate
 
-# Install necessary dependencies from requirement.txt file
+# Install setup.py compatible setuptools and necessary dependencies from requirement.txt file
+pip install setuptools==58.2.0
 pip install -r ~/x-sight/py/sight/requirements.txt
 ```
 Note : if error ```ModuleNotFoundError: No module named 'virtualenv'``` occurs, try installing virtualenv using pip,
@@ -155,6 +156,64 @@ source ~/venvs/sight_env/bin/activate
 cd ~/x-sight
 ```
 
+#### Setup yapf in vscode
+
+add extention eeyore.yapf in vscode.
+add .vscode folder in the repo and create settings.json file which will override the defauls settings of the vscode
+add following code snippet there or change accordingly if you already have custom setting
+
+```
+{
+  "[python]": {
+    "editor.formatOnSaveMode": "file",
+    "editor.formatOnSave": true,
+    "editor.formatOnType": true,
+    "editor.defaultFormatter": "eeyore.yapf",
+  },
+  "python.analysis.typeCheckingMode": "off",
+  "yapf.args": ["--style", "{based_on_style: Google, indent_width: 2, column_limit: 80}"],
+}
+```
+you might need to restart vscode to see the changes
+
+If you are setting up this first time and want to apply this style to existing repo
+create .config folder in the repo
+
+1.  Create .style.yapf file with following content in .config folder
+```
+[style]
+based_on_style = google
+indent_width = 2
+column_limit = 80
+```
+
+2.  Create .isort.cfg file with following content in .config folder
+```
+[settings]
+profile = google
+use_parentheses = true
+line_length = 80
+multi_line_output = 3
+```
+
+run the following commands from root folder of the repo to apply those style changes
+```
+yapf -ir -vv --style .config/.style.yapf .
+isort . --settings-path .config/ -v
+```
+#### setup pre-commit hook
+
+
+this pre-commit hook checks for the same formatting style we just setup locally
+```
+pip install pre-commit
+```
+
+-   make sure you created .style.yapf and .isort.cfg file in .config folder from the previous step.
+-   make sure your repo contains .pre-commit-config.yaml file in root directory of repo
+
+
+
 ### User Permissions:
 
 Note : all the follow up commands using $PROJECT_ID assumes you have it already set to your gcp project id. If not, set it via
@@ -247,11 +306,9 @@ account can have required permissions.
 2) Create service image from the code and host it on [gcr.io](http://gcr.io)
 
     ```bash
-    docker build --tag gcr.io/$PROJECT_ID/sight-default -f sight_service/Dockerfile .
-
-    gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://gcr.io
-
-    docker push gcr.io/$PROJECT_ID/sight-default
+    docker build --tag gcr.io/$PROJECT_ID/sight-default:$(git rev-parse --abbrev-ref HEAD)-$(git rev-parse --short HEAD) -f sight_service/Dockerfile . && \
+    gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://gcr.io && \
+    docker push gcr.io/$PROJECT_ID/sight-default:$(git rev-parse --abbrev-ref HEAD)-$(git rev-parse --short HEAD)
     ```
 
 3) With the help of the image, launch cloud run service
@@ -266,11 +323,9 @@ Host the worker image in a cloud which will be used as default image by the
 workers spawned using sight unless specified otherwise.
 
 ```bash
-docker build --tag gcr.io/$PROJECT_ID/sight-worker -f py/Dockerfile .
-
-gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://gcr.io
-
-docker push gcr.io/$PROJECT_ID/sight-worker
+docker build --tag gcr.io/$PROJECT_ID/sight-worker:$(git rev-parse --abbrev-ref HEAD)-$(git rev-parse --short HEAD) -f py/Dockerfile . && \
+gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://gcr.io && \
+docker push gcr.io/$PROJECT_ID/sight-worker:$(git rev-parse --abbrev-ref HEAD)-$(git rev-parse --short HEAD)
 ```
 
 ## Logging API
@@ -763,3 +818,5 @@ cd ~/x-sight
 python sight_service/service_root.py
 ```
 And from another terminal session, User can run any valid command from [this](#example-training-invocation-commands) section and change the flag ```--deployment_mode=local``` to indicate that sight_service is running locally.
+
+

fvs_sight/fvs_api.py

@@ -1,4 +1,5 @@
 from typing import Any
+
 from sight.proto import sight_pb2
 
 FVS_PARAMS = {
@@ -26,8 +27,8 @@ def create_attr_props(
   """
   return {
       key: (sight_pb2.DecisionConfigurationStart.AttrProps() if value
-            is not None else sight_pb2.DecisionConfigurationStart.AttrProps())
-      for key, value in config_dict.items()
+            is not None else sight_pb2.DecisionConfigurationStart.AttrProps()
+           ) for key, value in config_dict.items()
   }
 
 
@@ -58,6 +59,7 @@ def get_action_attrs():
   action_config.update(expand_params_for_cycles(fvs_params=FVS_PARAMS))
   return create_attr_props(action_config)
 
+
 # action_attrs = {
 #     "a1":
 #     sight_pb2.DecisionConfigurationStart.AttrProps(
@@ -77,13 +79,13 @@ def get_action_attrs():
 # }
 
 
-
 def get_outcome_attrs():
   """Returns the outcome attributes for the FVS outcome.
   """
   outcome_config = {'time_series': None}
   return create_attr_props(outcome_config)
 
+
 # outcome_attrs = {
 #     "time_series":
 #     sight_pb2.DecisionConfigurationStart.AttrProps(

fvs_sight/fvs_sight_worker.py

@@ -1,29 +1,29 @@
-
-import yaml
+import json
+import os
+from typing import Sequence
 
 from absl import app
 from absl import flags
-from sight.widgets.decision import decision_episode_fn
 # from fvs_sight.fvs_api import action_attrs, outcome_attrs
 from fvs_sight import fvs_api
-
+import pandas as pd
 from sight import data_structures
 from sight.proto import sight_pb2
 from sight.sight import Sight
 from sight.widgets.decision import decision
-from typing import Sequence
-import os
-import pandas as pd
-import json
+from sight.widgets.decision import decision_episode_fn
 import yaml
 
 
-def simulate_fvs(sight,params_dict):
-    print('here params_dict is :', params_dict)
-    mitigation_list = [227.6, 273.4, 273.3, 248.6, 165.3, 130.6, 106.4, 92.1, 81.7, 62.8]
-    sim_stream = pd.Series(mitigation_list)
-    # print(sim_stream)
-    return sim_stream
+def simulate_fvs(sight, params_dict):
+  print('here params_dict is :', params_dict)
+  mitigation_list = [
+      227.6, 273.4, 273.3, 248.6, 165.3, 130.6, 106.4, 92.1, 81.7, 62.8
+  ]
+  sim_stream = pd.Series(mitigation_list)
+  # print(sim_stream)
+  return sim_stream
+
 
 def driver_fn(sight):
 
@@ -34,9 +34,9 @@ def driver_fn(sight):
   #    return None
   # raise SystemError
 
-  sim_stream = simulate_fvs(sight,params_dict)
+  sim_stream = simulate_fvs(sight, params_dict)
 
-  outcome = {'time_series' : sim_stream}
+  outcome = {'time_series': sim_stream}
   print("outcome : ", outcome)
 
   decision.decision_outcome('outcome_label', sight, reward=0, outcome=outcome)
@@ -45,32 +45,24 @@ def driver_fn(sight):
 
 #temporary created
 def get_sight_instance():
-    params = sight_pb2.Params(
-        label="kokua_experiment",
-        bucket_name=f'{os.environ["PROJECT_ID"]}-sight',
-    )
-    sight_obj = Sight(params)
-    return sight_obj
+  params = sight_pb2.Params(
+      label="kokua_experiment",
+      bucket_name=f'{os.environ["PROJECT_ID"]}-sight',
+  )
+  sight_obj = Sight(params)
+  return sight_obj
 
 
 def main(argv: Sequence[str]) -> None:
-    if len(argv) > 1:
-        raise app.UsageError("Too many command-line arguments.")
-
-    with get_sight_instance() as sight:
-      decision.run(
-            driver_fn=driver_fn,
-            sight=sight,
-            action_attrs=fvs_api.get_action_attrs(),
-            outcome_attrs=fvs_api.get_outcome_attrs()
-        )
-
-if __name__ == "__main__":
-    app.run(main)
-
-
-
-
+  if len(argv) > 1:
+    raise app.UsageError("Too many command-line arguments.")
 
+  with get_sight_instance() as sight:
+    decision.run(driver_fn=driver_fn,
+                 sight=sight,
+                 action_attrs=fvs_api.get_action_attrs(),
+                 outcome_attrs=fvs_api.get_outcome_attrs())
 
 
+if __name__ == "__main__":
+  app.run(main)