RL backtest with simulator #1299

lihuoran · 2022-09-19T07:08:16Z

Now support running RL backtest with SAOE simulator.

Description

Motivation and Context

How Has This Been Tested?

Pass the test by running: pytest qlib/tests/test_all_pipeline.py under upper directory of qlib.
If you are adding a new feature, test on your own test scripts.

Screenshots of Test Results (if appropriate):

Pipeline test:
Your own tests:

Types of changes

Fix bugs
Add new feature
Update documentation

you-n-g · 2022-09-29T13:35:33Z

qlib/rl/contrib/backtest.py

+    split: str = "stock",
+    cash_limit: float = None,
+    generate_report: bool = False,
+) -> Union[Tuple[pd.DataFrame, dict], pd.DataFrame]:


There should be some docstring for this function.

you-n-g · 2022-09-29T13:38:49Z

qlib/rl/contrib/backtest.py

@@ -90,26 +92,109 @@ def _convert_indicator_to_dataframe(indicator: dict) -> Optional[pd.DataFrame]:
    return records


-def _generate_report(decisions: list, report_dict: dict) -> dict:
+def _generate_report(decisions: list, report_dicts: List[dict]) -> dict:


I think there should be richer annotation for the input (e.g. report) and the returned report

For example, @ dataclass with typed fields and detailed docstrings

I did some research on this issue and I found that this is related to the entire collect_data_loop() / backtest() lifecycle, so we need a lot of efforts to optimize it. Since it is not about the core function of RL backtest, I suggest we leave this in future PRs.

Maybe we can leave a TODO here...

you-n-g · 2022-09-29T13:40:41Z

qlib/rl/contrib/backtest.py

+            }
+        )
+
+        simulator = SingleAssetOrderExecution(


How does this simulator generate reports?
I didn't find how the step is called.

The simulator will execute one hidden step when it is created. When it is used in training, it will pause at the first yield of the internal strategies. However, in backtest, there will (should) not be any pauses, so the simulator will run until it stops.

you-n-g · 2022-09-29T13:44:53Z

qlib/rl/contrib/backtest.py

+    stock_pool = stock_pool
+
+    single = single_with_simulator if with_simulator else single_with_collect_data_loop
+    if parallel_mode:


I don't think parallel_mode is a required parameter.
Joblib will fall to single process when n_jobs == 1

you-n-g · 2022-09-29T14:08:19Z

qlib/rl/order_execution/simulator_qlib.py

    executor_config
        Executor configuration
    exchange_config
        Exchange configuration
    qlib_config
        Configuration used to initialize Qlib. If it is None, Qlib will not be initialized.
+    cash_limit:
+        Cash limit.
+    backtest_mode


backtest_mode is not a necessary parameter if we carefully design it.
It should disappear with CollectDataEnvWrapper in the future.

Please add doc for it.

Agree. backtest_mode looks ugly in the init signature.

you-n-g · 2022-09-29T14:08:47Z

qlib/rl/order_execution/simulator_qlib.py

+                    executor.inner_strategy.set_env(CollectDataEnvWrapper())
+                executor = executor.inner_executor
+
+        self.step(action=None)


Why should it call step in the reset phase?

Call step() with None is to "activate" the internal generator.

Please Add comments about it.

you-n-g · 2022-09-29T14:10:14Z

qlib/rl/order_execution/simulator_qlib.py

        )
        assert isinstance(self._collect_data_loop, Generator)

-        self._last_yielded_saoe_strategy = self._iter_strategy(action=None)
+        if backtest_mode:
+            executor: BaseExecutor = self._executor


Add comments that it should be removed in the future

qlib/rl/order_execution/state.py

qlib/rl/data/native.py

matluster · 2022-10-08T14:36:31Z

qlib/rl/order_execution/state.py

+    fill_val = fill_method(original_data)
+    return np.array([tmp.get(t, fill_val) for t in total_time_list])
+
+
 class SAOEStateAdapter:


I suggest moving this adapter to state_adapter.py or simulator_qlib.py. If I'm using the simple simluator, there is no reason I will be interested in all these adapter-related logics.

Can be left as TODO. :)

matluster · 2022-10-08T14:37:46Z

qlib/rl/contrib/backtest.py

-    for key in ["1minute", "5minute", "30minute", "1day"]:
-        if key not in report_dict["indicator"]:
+    decision_details = pd.concat([getattr(d, "details") for d in decisions if hasattr(d, "details")])
+    for key in ["1min", "5min", "30min", "1day"]:


I think I hard-coded this to quickly run through the experiments.
For open source version, it's worth making it more general.

This is part of the following issue mentioned by you-n-g before. I will redesign the entire logic in later PRs.

I think there should be richer annotation for the input (e.g. report) and the returned report
For example, @ dataclass with typed fields and detailed docstrings

matluster · 2022-10-08T14:38:15Z

qlib/backtest/decision.py

@@ -576,3 +576,16 @@ def __repr__(self) -> str:
            f"trade_range: {self.trade_range}; "
            f"order_list[{len(self.order_list)}]"
        )
+
+
+class TradeDecisionWithDetails(TradeDecisionWO):


Add some explanations on why (in what scenarios) we need this.

qlib/rl/order_execution/policy.py

matluster · 2022-10-08T14:41:36Z

qlib/rl/order_execution/simulator_qlib.py

    executor_config
        Executor configuration
    exchange_config
        Exchange configuration
    qlib_config
        Configuration used to initialize Qlib. If it is None, Qlib will not be initialized.
+    cash_limit:
+        Cash limit.
+    backtest_mode


Agree. backtest_mode looks ugly in the init signature.

ChiahungTai · 2022-10-14T00:45:29Z

Why this PR is merged even break the CI???
It make other PRs fail the checks........

you-n-g · 2022-10-14T01:39:32Z

The CI will be fixed in this PR soon #1314

* RL backtest with simulator * Minor modification in init_qlib * Cherry pick PR 1302 * Resolve PR comments * Fix missing data processing * Minor bugfix * Add TODOs and docs * Add a comment

RL backtest with simulator

08f725c

lihuoran requested review from you-n-g and ultmaster September 19, 2022 07:08

lihuoran added 2 commits September 27, 2022 14:12

Minor modification in init_qlib

1907372

Cherry pick PR 1302

22cb8ee

you-n-g reviewed Sep 29, 2022

View reviewed changes

lihuoran added 3 commits October 6, 2022 12:11

Resolve PR comments

321691d

Fix missing data processing

35a19aa

Minor bugfix

773188b

you-n-g reviewed Oct 8, 2022

View reviewed changes

qlib/rl/order_execution/state.py Show resolved Hide resolved

you-n-g reviewed Oct 8, 2022

View reviewed changes

qlib/rl/data/native.py Show resolved Hide resolved

matluster reviewed Oct 8, 2022

View reviewed changes

lihuoran and others added 2 commits October 9, 2022 10:13

Add TODOs and docs

e8b4165

Merge branch 'main' into huoran/migrate_amc4th

2a1d838

ultmaster approved these changes Oct 12, 2022

View reviewed changes

Add a comment

c82d05e

you-n-g merged commit 216a8ec into main Oct 12, 2022

you-n-g deleted the huoran/migrate_amc4th branch October 12, 2022 08:44

you-n-g added the enhancement New feature or request label Dec 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RL backtest with simulator #1299

RL backtest with simulator #1299

lihuoran commented Sep 19, 2022 •

edited

Loading

you-n-g Sep 29, 2022

you-n-g Sep 29, 2022

lihuoran Oct 6, 2022

you-n-g Oct 8, 2022

you-n-g Sep 29, 2022

lihuoran Oct 6, 2022

you-n-g Sep 29, 2022

lihuoran Oct 6, 2022

you-n-g Sep 29, 2022

lihuoran Oct 6, 2022

matluster Oct 8, 2022

you-n-g Sep 29, 2022

lihuoran Oct 6, 2022

you-n-g Oct 12, 2022

you-n-g Sep 29, 2022

lihuoran Oct 6, 2022

matluster Oct 8, 2022

matluster Oct 8, 2022

lihuoran Oct 9, 2022

matluster Oct 8, 2022

lihuoran Oct 9, 2022

matluster Oct 8, 2022

ChiahungTai commented Oct 14, 2022

you-n-g commented Oct 14, 2022

RL backtest with simulator #1299

RL backtest with simulator #1299

Conversation

lihuoran commented Sep 19, 2022 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Screenshots of Test Results (if appropriate):

Types of changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChiahungTai commented Oct 14, 2022

you-n-g commented Oct 14, 2022

lihuoran commented Sep 19, 2022 •

edited

Loading