BUG fix _Unstacker int32 limit in dataframe sizes (pandas-dev#26314)

[KaonToPion]

• closes Pivot / unstack on large data frame does not work int32 overflow #26314
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

I have tested it with :
df = pd.DataFrame(np.random.randint(low=0, high=1500000, size=(90000, 2)), columns=['a', 'b'])
df.set_index(['a', 'b']).unstack()

I am not sure if I should add it as a test, it requires quite some memory. I am also hesitant about where the test should be located in case it's added.

[jreback]

pls always add a test and make sure it fails first before adding a path

[jreback]

I am not sure if I should add it as a test, it requires quite some memory. I am also hesitant about where the test should be located in case it's added.

how long does this take / how much memory.?

you can mark tests as well.

[jreback]

can u merge master and see if can get passing

[simonjayhawkins]

@KaonToPion closing as stale. please LMK if you want to continue and the PR will be reopened.

[KaonToPion]

@simonjayhawkins could you re-open it? I can take it again

[KaonToPion]

I am having trouble with the fix. This is the code

# Bug fix GH 20601
# If the data frame is too big, the number of unique index combination
# will cause int32 overflow on windows environments.
# We want to check and raise an error before this happens

num_rows = np.max([index_level.size for index_level in self.new_index_levels])
num_columns = self.removed_level.size

# GH20601: This forces an overflow if the number of cells is too high.

num_cells = np.multiply(num_rows, num_columns, dtype=np.int32)
if num_rows > 0 and num_columns > 0 and num_cells <= 0:
    raise ValueError("Unstacked DataFrame is too big, causing int32 overflow")

The issue right now is that "raise ValueError" generates troubles in all the other systems so there are two ways out of this:

1. Checking with int64
   num_cells = np.multiply(num_rows, num_columns, dtype=np.int64)
   This solves the issue, but I am not sure how to test this, because It doesn't matter the size of my Ram (I have even tried with a 500 Gb RAM machine) , that I have a memoryError way before I am able to raise the ValueError exception using num_cells

2. Just deleting the num_cells, num_rows, and num_columns and the raise_exception so go forward with the MemoryErrors that a lot of people are going to have when doing the df.unstack() because their data will unwrapp.

My first commit tried the first option but trying to test it I find that the second option is probably the best option.

Any opinion on this @jreback @simonjayhawkins ?