MAINT Make long/short positions match gross leverage

[humdings]

This makes pos.get_long_short_pos return a dataframe
that matches the gross leverage of the portfolio.

I removed the gross_lev argument since the leverage is
dependent on positions dataframe.

This PR is addressing #30

[twiecki]

Am I correct in assuming that this does not yet fix the issue with there still being a cash allocation incorrectly inferred when there is shorting?

[twiecki]

Otherwise looks great! Thanks for the quick turn-around.

[humdings]

Yes, I do think you are correct, the problem is that normalizing the data throws everything out of whack when shorting is involved. We can't use the absolute value of the short positions and still have it make sense.

Consider a portfolio with 1x leverage that is equally long and short. They sell 50% of the account value and get the cash proceeds, then buy assets with the other 50%. That means the account has 100% of the account value sitting in cash still, but some of it is collateral for the short sales. That cash earns interest and is an important component of a long/short strategy, especially in a high interest environment.

We lose that when normalizing the long/short/cash components because they should equal 200% of the account value if using absolute values. 100% in cash, then 50/50 in long/short stocks.

Having this return the dataframe defined on line 59 before the normalizing would be more correct IMO.

I don't think the area plot is the best thing for this either. Here are a few plots of a long/short strategy without normalizing the data, the area plot is not very nice, but the density plot, box plot, and histogram capture the distribution of allocations nicely.

This area plot is not so intuitive as to what I'm looking at IMO.

A density plot captures the allocation distribution well

Histograms work too, but it's a bit more cluttered.

Box plots are not that pretty, but the information is at least meaningful.

[justinlent]

@humdings thanks for that in depth overview. I really do like that boxplot version at the end but I agree it's a bit confusing and overkill for this visualization usecase.
I think this is a case where the most important takeaway that we are trying to convey is literally the long exposure , short exposure, and "left over cash" (cash excluding proceeds from shorting). 
I agree the interest earned on cash vs cash held as collateral is important if rates are high, but I think we can exclude that nuanced level of detail here.  I totally think we should eventually have an entire "Margin profile" section in the tearsheet down the road where this additional breakdown would be awesome, especially the additional p/l earned every month for interest on cash, short interest paid of hard to borrow stocks, % of cash held as collateral, % of total margin used, etc--all as timeseries showing how they change over the lifetime of the algo's history 
how would you feel with this proposed approach?

Sent from iPhone

On Sun, Sep 13, 2015 at 10:08 AM -0700, "David Edwards" notifications@github.com wrote:

Yes, I do think you are correct, the problem is that normalizing the data throws everything out of whack when shorting is involved. We can't use the absolute value of the short positions and still have it make sense.

Consider a portfolio with 1x leverage that is equally long and short. They sell 50% of the account value and get the cash proceeds, then buy assets with the other 50%. That means the account has 100% of the account value sitting in cash still, but some of it is collateral for the short sales. That cash earns interest and is an important component of a long/short strategy, especially in a high interest environment.

I we lose that when normalizing the long/short/cash components because they should equal 200% of the account value if using absolute values. 100% in cash, then 50/50 in long/short stocks.

Having this return the dataframe defined on line 59 before the normalizing would be more correct IMO. I don't think the area plot is the best thing for this either.

Here are a few plots of a long/short strategy without normalizing the data, the area plot is not very nice, but the density plot, box plot, and histogram capture the distribution of allocations nicely.

This area plot is not so intuitive as to what I'm looking at IMO.

A density plot captures the allocation distribution well

Histograms work too, but it's a bit more cluttered.

Box plots are not that pretty, but the information is a least meaningful.

—
Reply to this email directly or view it on GitHub.

[humdings]

I doesn't feel right to me to mess with the cash level honestly. In this particular case if we want the area plot to match up with the gross leverage, we want to exclude the cash altogether. The cash can be inferred from the long/short allocations.

Starting with a positions dataframe here is a script that demos what I mean.

nocash = positions.drop('cash', axis=1)
longs = nocash[nocash > 0].sum(axis=1)
shorts = nocash[nocash < 0].abs().sum(axis=1)
cash = positions.cash

leverage = (longs + shorts) / (longs - shorts + cash)
net_liquidation = longs - shorts + cash

allocations = pd.DataFrame({'long': longs,
                            'short': shorts}) / net_liquidation

print np.allclose(allocations.sum(axis=1) - leverage, 0.0)

# Produces True

So the gross leverage matches up when the cash position is excluded. If we want to compare long vs. short allocations I'd say we don't worry about the cash component since it can be solved for if the leverage and long/short allocations are known.

[twiecki]

@humdings Thanks for the detailed answer. Your last suggestion makes a lot of sense to me. Exclude the cash as it can be inferred from the leverage and have this analysis focus only on long vs. short exposure.

In addition, how do you feel about dropping the leverage plot which now is redundant with this one?

[twiecki]

Also interested in what @justinlent thinks about dropping cash.

[justinlent]

+1 @humdings and Thomas. remove cash from the exposure plot since it can be inferred, and yep we can kill off the current Leverage plot in favor of this new Gross Exposure plot

Sent from iPhone

On Mon, Sep 14, 2015 at 2:43 AM -0700, "Thomas Wiecki" notifications@github.com wrote:

Also interested in what @justinlent thinks about dropping cash.

—
Reply to this email directly or view it on GitHub.

[humdings]

That all sounds good to me, I'll update get_long_short_pos to drop the cash component.

[twiecki]

Running the zipline example NB I get this behavior:

[humdings]

Strange, I'll look into it now

[humdings]

That is odd because it seems to work fine for me, it looks like the long portion is not plotting for you.

[twiecki]

Something seems to have gone wrong with the uploading of the image.

[twiecki]

@humdings can you run the NBs and commit the updated ones?

[twiecki]

@humdings Can you force pull? I resolved the conflicts. But I still get that issue of an empty plot. Perhaps it's a python 3 issue?

[humdings]

What exactly do you need me to do? Where am I pulling from and to where? I'm not sure what commands to run.

[twiecki]

git checkout long_short_cash_investigation
git pull -f origin long_short_cash_investigation

[humdings]

I'm still getting conflicts, I'll just scrap the repo and clone again

[humdings]

I'm all up to date but the notebook is barfing now. I get this error when opening the notebook

Error loading notebook
Unreadable Notebook: /Users/dedwards/repos/pyfolio/pyfolio/examples/zipline_algo_example.ipynb NotJSONError('Notebook does not appear to be JSON: u\'{\\n "cells": [\\n {\\n "cell_type": "...',)

[twiecki]

try git checkout master -- /Users/dedwards/repos/pyfolio/pyfolio/examples/zipline_algo_example.ipynb to restore the ipynb.

[twiecki]

Closing in favor of #147