Overriding Duplicate and Ignore Lists

[emretezel]

Hi Everyone,

I would like to override the following settings in my system config, whereby I deliberately don't want to specify any duplicate instrument, nor would I like to pick the ones from defaults. The same for trading restrictions and bad markets. I do want to override ignore_instruments.

I thought using empty list and dictionary notation would avoid picking defaults for duplicate instruments, trading restrictions and bad markets, but it is still picking. Any tips, other than having at least one item in these lists?

duplicate_instruments:
  include: {}
  exclude: {}

exclude_instrument_lists:
  ignore_instruments:
    - ETHANOL
    - EURIBOR
    - KRWUSD
  trading_restrictions: []
  bad_markets: []

[robcarver17]

I think this is caused 'deliberately' by the behaviour of:https://github.com/robcarver17/pysystemtrade/blob/master/sysdata/config/fill_config_dict_with_defaults.py

On balance the business of what is duplicate, excluded or included is probably a private matter. My gut feeling is to replace these with blanks in the defaults.yaml or remove them entirely (need to check this will not cause a problem), and then people need to add them back to their private config. Thoughts?

[robcarver17]

Okay so I've removed these from the defaults.yaml (or rather replace them with a skeleton example of non existent markets).

[tgibson11]

I already did this for include/excluded in my repo. (Made them empty dictionaries in defaults.yaml).

I didn't have to for bad_markets, because they are lists, and lists appear to be completely replaced if specified in the private config. Dictionaries aren't replaced, more like "merged". This seems like an inconsistency, but I can see the reason for it in the code. It makes some things easier, but it can be surprising if you're not aware of the difference.

Edit: Emre describes different behavior for lists than what I remember...it was a while ago, so maybe I'm misremember. Or maybe it's a difference between empty and non-empty lists

[tgibson11]

One more thing that might be related: I noticed that I have to specify included/excluded and bad_markets in BOTH my private_config (for production) and my system config (for backtests).

It would be nice if that duplication wasn't necessary.

[robcarver17]

The duplication issue is now live for me, since I've had to copy and paste long lists of duplicates / bad markets.

It makes most sense for this to live in private config, since the downstream code in production that uses these lists can't see individual backest .yamls.

In fact this is a long overdue change really, the private config.yaml should be able to modify the backtesting environment and not just production.

So globally now the code will work like this:
Configuration comes from: defaults, overriden by private_config.yaml, overriden by the system backtest .yaml.

I need to test and document, but that's the general idea. Results in a lot of simplification but hopefully won't have any unintended consequences.

[robcarver17]

I've pushed this change now.

[robcarver17]

Created an issue.