Issue with running strategies based on trading_calendars

[erykml]

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: MacOS Mojave 10.14.5
• Python Version: Python 3.5.6 :: Anaconda, Inc.
• Python Bitness: 64
• How did you install Zipline: conda
• Python packages: alembic==0.7.7
  appnope==0.1.0
  asn1crypto==0.24.0
  backcall==0.1.0
  bcolz==0.12.1
  beautifulsoup4==4.7.1
  bleach==2.1.4
  Bottleneck==1.2.1
  certifi==2018.8.24
  cffi==1.11.5
  chardet==3.0.4
  click==6.7
  contextlib2==0.5.5
  cryptography==2.3.1
  cycler==0.10.0
  cyordereddict==0.2.2
  Cython==0.28.5
  decorator==4.4.0
  defusedxml==0.6.0
  empyrical==0.5.0
  entrypoints==0.2.3
  html5lib==1.0.1
  idna==2.7
  intervaltree==2.1.0
  ipykernel==4.10.0
  ipython==6.5.0
  ipython-genutils==0.2.0
  ipywidgets==7.4.1
  jedi==0.12.1
  Jinja2==2.10
  jsonschema==2.6.0
  jupyter==1.0.0
  jupyter-client==5.3.1
  jupyter-console==5.2.0
  jupyter-core==4.5.0
  kiwisolver==1.0.1
  Logbook==0.12.5
  lru-dict==1.1.4
  lxml==4.2.5
  Mako==1.0.10
  MarkupSafe==1.0
  matplotlib==3.0.0
  mistune==0.8.3
  mkl-fft==1.0.6
  mkl-random==1.0.1
  multipledispatch==0.6.0
  nbconvert==5.5.0
  nbformat==4.4.0
  networkx==1.11
  notebook==5.6.0
  numexpr==2.6.1
  numpy==1.14.6
  pandas==0.22.0
  pandas-datareader==0.6.0
  pandocfilters==1.4.2
  parso==0.5.0
  patsy==0.5.0
  pexpect==4.6.0
  pickleshare==0.7.4
  prometheus-client==0.7.1
  prompt-toolkit==1.0.15
  ptyprocess==0.6.0
  pycparser==2.19
  Pygments==2.4.2
  pyOpenSSL==18.0.0
  pyparsing==2.4.0
  PySocks==1.6.8
  python-dateutil==2.7.3
  pytz==2019.1
  pyzmq==17.1.2
  qtconsole==4.5.1
  requests==2.19.1
  requests-file==1.4.3
  requests-ftp==0.3.1
  scipy==1.1.0
  Send2Trash==1.5.0
  simplegeneric==0.8.1
  six==1.11.0
  sortedcontainers==2.0.5
  soupsieve==1.9.2
  SQLAlchemy==1.2.11
  statsmodels==0.9.0
  TA-Lib==0.4.17
  tables==3.4.4
  terminado==0.8.1
  testpath==0.3.1
  toolz==0.10.0
  tornado==5.1.1
  trading-calendars==1.8.1
  traitlets==4.3.2
  urllib3==1.23
  watermark==1.8.1
  wcwidth==0.1.7
  webencodings==0.5.1
  widgetsnbextension==3.4.1
  wrapt==1.10.11
  yahoofinancials==1.5
  zipline==1.3.0

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

• I wanted to import European Stocks data into zipline and run strategies
• I'm getting the following error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1483315200000000000

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-33-9eaf08e4c73f> in <module>()
----> 1 get_ipython().run_cell_magic('zipline', '--start 2017-1-2 --end 2019-6-28 --capital-base 1050.0 --bundle eu_stocks -o out.pkl', "\ndef initialize(context):\n    set_benchmark('aex')\n    context.asset = symbol('abn')\n\ndef handle_data(context, data):\n    order_target_percent(context.asset, 0.5)")

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2165             magic_arg_s = self.var_expand(line, stack_depth)
   2166             with self.builtin_trap:
-> 2167                 result = fn(magic_arg_s, cell)
   2168             return result
   2169 

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/zipline/__main__.py in zipline_magic(line, cell)
    309             '%s%%zipline' % ((cell or '') and '%'),
    310             # don't use system exit and propogate errors to the caller
--> 311             standalone_mode=False,
    312         )
    313     except SystemExit as e:

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/click/core.py in main(self, args, prog_name, complete_var, standalone_mode, **extra)
    695             try:
    696                 with self.make_context(prog_name, args, **extra) as ctx:
--> 697                     rv = self.invoke(ctx)
    698                     if not standalone_mode:
    699                         return rv

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/click/core.py in invoke(self, ctx)
    893         """
    894         if self.callback is not None:
--> 895             return ctx.invoke(self.callback, **ctx.params)
    896 
    897 

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/click/core.py in invoke(*args, **kwargs)
    533         with augment_usage_errors(self):
    534             with ctx:
--> 535                 return callback(*args, **kwargs)
    536 
    537     def forward(*args, **kwargs):

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/click/decorators.py in new_func(*args, **kwargs)
     15     """
     16     def new_func(*args, **kwargs):
---> 17         return f(get_current_context(), *args, **kwargs)
     18     return update_wrapper(new_func, f)
     19 

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/zipline/__main__.py in run(ctx, algofile, algotext, define, data_frequency, capital_base, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, blotter)
    274         local_namespace=local_namespace,
    275         environ=os.environ,
--> 276         blotter=blotter,
    277     )
    278 

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/zipline/utils/run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
    167             equity_minute_reader=bundle_data.equity_minute_bar_reader,
    168             equity_daily_reader=bundle_data.equity_daily_bar_reader,
--> 169             adjustment_reader=bundle_data.adjustment_reader,
    170         )
    171 

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/zipline/data/data_portal.py in __init__(self, asset_finder, trading_calendar, first_trading_day, equity_daily_reader, equity_minute_reader, future_daily_reader, future_minute_reader, adjustment_reader, last_available_session, last_available_minute, minute_history_prefetch_length, daily_history_prefetch_length)
    289                 self._first_trading_day
    290             )
--> 291             if self._first_trading_day is not None else (None, None)
    292         )
    293 

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/trading_calendars/trading_calendar.py in open_and_close_for_session(self, session_label)
    758         # http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#datetime-with-tz  # noqa
    759         return (
--> 760             sched.at[session_label, 'market_open'].tz_localize(UTC),
    761             sched.at[session_label, 'market_close'].tz_localize(UTC),
    762         )

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/pandas/core/indexing.py in __getitem__(self, key)
   1867 
   1868         key = self._convert_key(key)
-> 1869         return self.obj._get_value(*key, takeable=self._takeable)
   1870 
   1871     def __setitem__(self, key, value):

~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/pandas/core/frame.py in _get_value(self, index, col, takeable)
   1983 
   1984         try:
-> 1985             return engine.get_value(series._values, index)
   1986         except (TypeError, ValueError):
   1987 

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

KeyError: Timestamp('2017-01-02 00:00:00+0000', tz='UTC')

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. I downloaded data into CSV files (two files names abn.csv and aex.csv). The head of one looks like this:

date,open,high,low,close,volume,dividend,split
2017-01-02,21.100000381469727,21.295000076293945,20.94499969482422,21.274999618530273,407919,0,0
2017-01-03,21.334999084472656,22.06999969482422,21.334999084472656,21.950000762939453,1453872,0,0
2017-01-04,21.989999771118164,22.135000228881836,21.934999465942383,22.125,1169976,0,0
2017-01-05,22.114999771118164,22.280000686645508,22.0,22.125,1102774,0,0
2017-01-06,22.100000381469727,22.25,22.045000076293945,22.229999542236328,766804,0,0
2017-01-09,22.21500015258789,22.219999313354492,21.93000030517578,21.989999771118164,1006061,0,0
2017-01-10,22.024999618530273,22.165000915527344,21.875,22.079999923706055,954217,0,0
2017-01-11,22.100000381469727,22.25,21.950000762939453,22.049999237060547,1140509,0,0
2017-01-12,21.90999984741211,22.104999542236328,21.829999923706055,21.850000381469727,1231540,0,0

2. I modified the extension.py file to contain:

import pandas as pd

from zipline.data.bundles import register
from zipline.data.bundles.csvdir import csvdir_equities

start_session = pd.Timestamp('2017-01-02', tz='utc')
end_session = pd.Timestamp('2019-06-28', tz='utc')

# register the bundle
register(
    'eu_stocks',  
    csvdir_equities(
        ['daily'],
        '/path/',
    ),
    calendar_name='XAMS',  # Euronext Amsterdam
    start_session=start_session,
    end_session=end_session
)

3. I ingested the data
4. I ran the following strategy

def initialize(context):
    set_benchmark('aex')
    context.asset = symbol('abn')

def handle_data(context, data):
    order_target_percent(context.asset, 0.5)

What steps have you taken to resolve this already?

I tried following this issue but with no success, as I already have the latest version.

Sincerely,
erykml

[trevor-richardson]

@erykml I have the exact same issue and I am trying to work through your example medium post. How did you fix this issue? Is it something with the pandas version? I'm using conda on fresh ubuntu 18.04. I'm getting this error walking through -- https://towardsdatascience.com/backtesting-trading-strategies-using-custom-data-in-zipline-e6fd65eeaca0. Thanks!

[erykml]

Hi @trevor-richardson, did you try following all the steps described in this Notebook, while naturally adjusting the paths?