Resolve issues while running Automatic update of daily frequency data (from yahoo finance) for US region #1358

HyeongminMoon · 2022-11-15T07:07:42Z

Description

According to README.md, I tried automatic update with US region.

python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir <user data dir> --trading_date <start date> --end_date <end date> --region us

Current implements are based on CN index, so I entered some issues:

There is no method name for US index normalize, YahooNormalizeUS1dExtend
(AttributeError: module 'collector' has no attribute 'YahooNormalizeUS1dExtend' while running update_data_to_bin #1196)
-> resolved at commit b2a76df
While downloading qlib 1d data, exists_qlib_data always return False
-> Not resolved. I resolved this by removing "PRN" from the qlib_data/us_index/instruments/all.txt, but I have no idea how can I integrate this in code. Anyway this is not critial issue(no error, just takes ~2min time additionally)
While normalizing, some symbols cause error in pd.read_csv because of their names
-> resolved at commit a9cb66b
While parsing index, it always tries cn_index/collector
-> resolved at commit 0be6b99

I am totally ready for accepting your guidance if there is. Thanks.

Motivation and Context

Motivated by #1196, making automatic update available for US region.
We will be able to use automatic update(with crontab!) for US region after adopting this request.

How Has This Been Tested?

Pass the test by running: pytest qlib/tests/test_all_pipeline.py under upper directory of qlib.
Pass the test by running: python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir <user data dir> --trading_date <start date> --end_date <end date> --region us.

Screenshots of Test Results (if appropriate):

Types of changes

Fix bugs
Add new feature
Update documentation

…S region

you-n-g · 2022-11-27T11:30:47Z

Please merge master to fix the CI error. @HyeongminMoon

scripts/data_collector/base.py

SunsetWolf · 2022-11-30T05:36:55Z

Regarding the second point of your description, my guess is that it is not possible to determine whether qlib_dir exists or not, which causes GetData to be executed every time update_to_bin is run, which doesn't affect anything, but takes some unnecessary time. Is my guess correct?

HyeongminMoon · 2022-11-30T07:21:25Z

Regarding the second point of your description, my guess is that it is not possible to determine whether qlib_dir exists or not, which causes GetData to be executed every time update_to_bin is run, which doesn't affect anything, but takes some unnecessary time. Is my guess correct?

Correct.

SunsetWolf · 2022-11-30T07:28:55Z

Regarding the second point of your description, my guess is that it is not possible to determine whether qlib_dir exists or not, which causes GetData to be executed every time update_to_bin is run, which doesn't affect anything, but takes some unnecessary time. Is my guess correct?

Correct.

If that's the case, I think changing the not in this judgment statement might solve the problem.

HyeongminMoon · 2022-11-30T07:38:13Z

Regarding the second point of your description, my guess is that it is not possible to determine whether qlib_dir exists or not, which causes GetData to be executed every time update_to_bin is run, which doesn't affect anything, but takes some unnecessary time. Is my guess correct?

Correct.

If that's the case, I think changing the not in this judgment statement might solve the problem.

No, I think this is not from exists_qlib_data, but from a bug during the download sessions. I found there is a symbol name 'PRN' in the qlib_data/us_index/instruments/all.txt while PRN.csv does not exist in downloaded source directory.

SunsetWolf · 2022-12-01T05:50:11Z

Regarding the second point of your description, my guess is that it is not possible to determine whether qlib_dir exists or not, which causes GetData to be executed every time update_to_bin is run, which doesn't affect anything, but takes some unnecessary time. Is my guess correct?

Correct.

If that's the case, I think changing the not in this judgment statement might solve the problem.

No, I think this is not from exists_qlib_data, but from a bug during the download sessions. I found there is a symbol name 'PRN' in the qlib_data/us_index/instruments/all.txt while PRN.csv does not exist in downloaded source directory.

Since windows cannot create some special name folders, "prn" is one of them, in order to ensure that users of various systems can also use qlib, we add "_qlib_" prefix to these special names, so the "prn" seen in instruments/all.txt, and features/_qlib_prn/ are matched , because in exists_qlib_data, there is no remove prefix for the names in features, so returns False, we provide a method to remove the prefix, I think this method can be used in the exists_qlib_data code, to solve the problem.

HyeongminMoon · 2022-12-01T08:57:21Z

Regarding the second point of your description, my guess is that it is not possible to determine whether qlib_dir exists or not, which causes GetData to be executed every time update_to_bin is run, which doesn't affect anything, but takes some unnecessary time. Is my guess correct?

Correct.

If that's the case, I think changing the not in this judgment statement might solve the problem.

No, I think this is not from exists_qlib_data, but from a bug during the download sessions. I found there is a symbol name 'PRN' in the qlib_data/us_index/instruments/all.txt while PRN.csv does not exist in downloaded source directory.

Since windows cannot create some special name folders, "prn" is one of them, in order to ensure that users of various systems can also use qlib, we add "qlib" prefix to these special names, so the "prn" seen in instruments/all.txt, and features/_qlib_prn/ are matched , because in exists_qlib_data, there is no remove prefix for the names in features, so returns False, we provide a method to remove the prefix, I think this method can be used in the exists_qlib_data code, to solve the problem.

I'll try this too. Thanks.

HyeongminMoon · 2022-12-01T09:44:09Z

Now all issues resolved, I checked the script once again too.

you-n-g · 2022-12-02T09:06:44Z

scripts/data_collector/base.py

@@ -289,7 +289,18 @@ def __init__(

    def _executor(self, file_path: Path):
        file_path = Path(file_path)
-        df = pd.read_csv(file_path)
+
+        default_na = pd._libs.parsers.STR_NA_VALUES


It would be great if we could add more context and comments here.
It is hard to understand the code without an explanation of motivation.

Thank you for review! I added explanation comments:)

you-n-g · 2022-12-05T06:50:23Z

It looks perfect now.
Thanks!

… (from yahoo finance) for US region (microsoft#1358) * Update YahooNormalizeUS1dExtend(microsoft#1196) * Prevent pandas read_csv errors while running update_data_to_bin for US region * Fix parse_index error while running update_data_to_bin for US region * prevent pandas.read_csv error on specific symbol names * Reordering parameters for better rendering * removes prefix during feature_dir existence checking * add explanation comments

HyeongminMoon added 3 commits November 15, 2022 11:38

Update YahooNormalizeUS1dExtend(microsoft#1196)

b2a76df

Prevent pandas read_csv errors while running update_data_to_bin for U…

a9cb66b

…S region

Fix parse_index error while running update_data_to_bin for US region

0be6b99

you-n-g requested a review from SunsetWolf November 27, 2022 11:31

you-n-g assigned SunsetWolf Nov 27, 2022

Merge branch 'microsoft:main' into main

fec31d6

SunsetWolf reviewed Nov 30, 2022

View reviewed changes

scripts/data_collector/base.py Show resolved Hide resolved

HyeongminMoon added 3 commits December 1, 2022 17:15

Merge branch 'microsoft:main' into main

6fe818d

prevent pandas.read_csv error on specific symbol names

04cfbc5

Reordering parameters for better rendering

4380271

removes prefix during feature_dir existence checking

749c2a0

you-n-g reviewed Dec 2, 2022

View reviewed changes

HyeongminMoon added 2 commits December 5, 2022 10:58

add explanation comments

50d83bb

Merge branch 'microsoft:main' into main

53ad2ff

you-n-g merged commit 9d8a8c6 into microsoft:main Dec 5, 2022

you-n-g added the bug Something isn't working label Dec 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolve issues while running Automatic update of daily frequency data (from yahoo finance) for US region #1358

Resolve issues while running Automatic update of daily frequency data (from yahoo finance) for US region #1358

HyeongminMoon commented Nov 15, 2022 •

edited

Loading

you-n-g commented Nov 27, 2022

SunsetWolf commented Nov 30, 2022

HyeongminMoon commented Nov 30, 2022

SunsetWolf commented Nov 30, 2022

HyeongminMoon commented Nov 30, 2022

SunsetWolf commented Dec 1, 2022

HyeongminMoon commented Dec 1, 2022

HyeongminMoon commented Dec 1, 2022

you-n-g Dec 2, 2022

HyeongminMoon Dec 5, 2022

you-n-g commented Dec 5, 2022

Resolve issues while running Automatic update of daily frequency data (from yahoo finance) for US region #1358

Resolve issues while running Automatic update of daily frequency data (from yahoo finance) for US region #1358

Conversation

HyeongminMoon commented Nov 15, 2022 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Screenshots of Test Results (if appropriate):

Types of changes

you-n-g commented Nov 27, 2022

SunsetWolf commented Nov 30, 2022

HyeongminMoon commented Nov 30, 2022

SunsetWolf commented Nov 30, 2022

HyeongminMoon commented Nov 30, 2022

SunsetWolf commented Dec 1, 2022

HyeongminMoon commented Dec 1, 2022

HyeongminMoon commented Dec 1, 2022

you-n-g Dec 2, 2022

Choose a reason for hiding this comment

HyeongminMoon Dec 5, 2022

Choose a reason for hiding this comment

you-n-g commented Dec 5, 2022

HyeongminMoon commented Nov 15, 2022 •

edited

Loading