Yahoo URL Error

[merl-dev]

When the following is run

%matplotlib inline
import pyfolio as pf
import pandas as pd
import numpy as np
stock_rets = pf.utils.get_symbol_rets('AGG')

I get a RemoteDataError for the baked-in URL. Apparently Yahoo has made some recent changes to their API's

---------------------------------------------------------------------------
RemoteDataError                           Traceback (most recent call last)
<ipython-input-3-fd2a04977723> in <module>()
      3 import pandas as pd
      4 import numpy as np
----> 5 stock_rets = pf.utils.get_symbol_rets('AGG')

/home/ubuntu/anaconda3/envs/idpf/lib/python3.5/site-packages/pyfolio/utils.py in get_symbol_rets(symbol, start, end)
    473     return SETTINGS['returns_func'](symbol,
    474                                     start=start,
--> 475                                     end=end)
    476 
    477 

/home/ubuntu/anaconda3/envs/idpf/lib/python3.5/site-packages/pyfolio/utils.py in default_returns_func(symbol, start, end)
    259         rets = rets[start:end]
    260     else:
--> 261         rets = get_symbol_from_yahoo(symbol, start=start, end=end)
    262 
    263     return rets[symbol]

/home/ubuntu/anaconda3/envs/idpf/lib/python3.5/site-packages/pyfolio/utils.py in get_symbol_from_yahoo(symbol, start, end)
    212         Returns of symbol in requested period.
    213     """
--> 214     px = web.get_data_yahoo(symbol, start=start, end=end)
    215     rets = px[['Adj Close']].pct_change().dropna()
    216     rets.index = rets.index.tz_localize("UTC")

/home/ubuntu/anaconda3/envs/idpf/lib/python3.5/site-packages/pandas_datareader/data.py in get_data_yahoo(*args, **kwargs)
     31 
     32 def get_data_yahoo(*args, **kwargs):
---> 33     return YahooDailyReader(*args, **kwargs).read()
     34 
     35 def get_data_yahoo_actions(*args, **kwargs):

/home/ubuntu/anaconda3/envs/idpf/lib/python3.5/site-packages/pandas_datareader/yahoo/daily.py in read(self)
     75     def read(self):
     76         """ read one data from specified URL """
---> 77         df = super(YahooDailyReader, self).read()
     78         if self.ret_index:
     79             df['Ret_Index'] = _calc_return_index(df['Adj Close'])

/home/ubuntu/anaconda3/envs/idpf/lib/python3.5/site-packages/pandas_datareader/base.py in read(self)
    171         # If a single symbol, (e.g., 'GOOG')
    172         if isinstance(self.symbols, (compat.string_types, int)):
--> 173             df = self._read_one_data(self.url, params=self._get_params(self.symbols))
    174         # Or multiple symbols, (e.g., ['GOOG', 'AAPL', 'MSFT'])
    175         elif isinstance(self.symbols, DataFrame):

/home/ubuntu/anaconda3/envs/idpf/lib/python3.5/site-packages/pandas_datareader/base.py in _read_one_data(self, url, params)
     78         """ read one data from specified URL """
     79         if self._format == 'string':
---> 80             out = self._read_url_as_StringIO(url, params=params)
     81         elif self._format == 'json':
     82             out = self._get_response(url, params=params).json()

/home/ubuntu/anaconda3/envs/idpf/lib/python3.5/site-packages/pandas_datareader/base.py in _read_url_as_StringIO(self, url, params)
     89         Open url (and retry)
     90         """
---> 91         response = self._get_response(url, params=params)
     92         out = StringIO()
     93         if isinstance(response.content, compat.binary_type):

/home/ubuntu/anaconda3/envs/idpf/lib/python3.5/site-packages/pandas_datareader/base.py in _get_response(self, url, params)
    115             time.sleep(self.pause)
    116 
--> 117         raise RemoteDataError('Unable to read URL: {0}'.format(url))
    118 
    119     def _read_lines(self, out):

RemoteDataError: Unable to read URL: http://ichart.finance.yahoo.com/table.csv

[twiecki]

That seems like a bug in pandas-datareader.

[freddiev4]

Hi @merl-dev we're also seeing this over in quantopian/zipline#1776. It looks like Yahoo has made some changes to their API and is expecting some sort of cookie/session tracking

[merl-dev]

Yes it would appear so, the download link on a history request at yahoo finance has changed url, query params, and now includes a session token. Do they ever document these changes?

[closedLoop]

I've created a workaround that substitutes out yahoo for google data. I'm sure there are some ticker mis-matches that could cause issues, but for my purposes tonight it solves it. Pull request coming shortly

%matplotlib inline
import pyfolio as pf
import pandas as pd
import numpy as np
stock_rets = pf.utils.get_symbol_rets('AGG')

print stock_rets.tail()

returns

Date
2017-05-11 00:00:00+00:00   -0.000092
2017-05-12 00:00:00+00:00    0.003686
2017-05-15 00:00:00+00:00   -0.000367
2017-05-16 00:00:00+00:00    0.001286
2017-05-17 00:00:00+00:00    0.005137
Name: AGG, dtype: float64

[gusgordon]

There's an unmerged fix in pandas-datareader: pydata/pandas-datareader#331 I suppose we should wait until they push this and do a release.

After that's implemented, we can bump the required version of datareader. We could also additionally put in the Google fallback in #386.

@twiecki thoughts?

[twiecki]

Yes agreed. And I like the idea of a fallback.

…

On May 24, 2017 4:20 PM, "Gus Gordon" ***@***.***> wrote: There's an unmerged fix in pandas-datareader: pydata/pandas-datareader#331 <pydata/pandas-datareader#331> I suppose we should wait until they push this and do a release. After that's implemented, we can bump the required version of datareader. We could also additionally put in the Google fallback in #386 <#386>. @twiecki <https://github.com/twiecki> thoughts? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#385 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApJmN5KYbg6U_9wI0NTox2G6TwB7Espks5r9DypgaJpZM4NdwbP> .

[citynorman]

My super quick hack to get at least something going:

import pandas as pd
from datetime import datetime
import pytz
import pyfolio as pf
from zipline import run_algorithm

# just create a series with 0 returns
start = datetime(2011, 1, 1, 0, 0, 0, 0, pytz.utc)
end = datetime(2012, 1, 1, 0, 0, 0, 0, pytz.utc)
benchmark_rets=pd.Series(0,index=pd.date_range(start,end))

# do stuff
perf = run_algorithm(start, end, initialize, [...])
returns, positions, transactions = pf.utils.extract_rets_pos_txn_from_zipline(perf)

# use 0 return benchmark
pf.show_perf_stats(returns,benchmark_rets)
pf.create_returns_tear_sheet(returns, benchmark_rets=benchmark_rets)

As in #380 be nice to have options to disable those benchmarks to prevent those kinds of issues (while the underlying problem gets fixed).

[dartdog]

Just ran into this as I am trying to get started,,
I tried just using the pandas datareader for Google but it seems that the various functions in Pyfolio still refer to the now broken Yahoo in pf.create_returns_tear_sheet in the single stock example, so even though I did the initial read of data from Google and put it in Stock_rets, I can't complete the example.. Is there a global work around?

As it seems Yahoo is out of the business or at the least unstable, and Pandas_datareader can handle Google and other datasources.. at the moment..

Seems like it would be great to have a variable or separate io module that can be swapped out, to be set as to what data source to use for all of the pyfolio data sources, with yahoo and data reader so baked it, there seems no easy fix...? TIA If anyone has suggestions I'm interested!

[chrism2671]

I've just run into the same issue. It would be great to remove the dependence on external data and just make it an option.

[JP-MRPhys]

Same, issue, agree with chrism, maybe users can provide both returns (stocks/portfolio) and benchmark as citynorm mentioned!