Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LogisticRegression cannot train from Dask DataFrame #84

Open
julioasotodv opened this issue Nov 4, 2017 · 33 comments
Open

LogisticRegression cannot train from Dask DataFrame #84

julioasotodv opened this issue Nov 4, 2017 · 33 comments

Comments

@julioasotodv
Copy link

A simple example:

from dask import dataframe as dd
from dask_glm.datasets import make_classification
from dask_ml.linear_model import LogisticRegression

X, y = make_classification(n_samples=10000, n_features=2)

X = dd.from_dask_array(X, columns=["a","b"])
y = dd.from_array(y)

lr = LogisticRegression()
lr.fit(X, y)

Returns KeyError: (<class 'dask.dataframe.core.DataFrame'>,)

I did not have time to try if it is also the case for other models.
@TomAugspurger
Copy link
Member

TomAugspurger commented Nov 6, 2017 •
edited
Loading

Thanks. At the moment the dask_glm based estimators just work with dask arrays, not dataframes. You can use .values to get the array.

I'm hoping to put in some helpers for handling all the extra DataFrame metadata sometime soon, so this will be more consistent across estimators.

@julioasotodv
Copy link
Author

Thank you so much for the quick response!

The problem is that when fitting a glm with intercept (which is usually the case), the dask array containing the features needs to have defined the chunk size, which I believe it is not possible when the array comes from a dataframe.

Anyways, I will reach out to the main dask issue page and ask there.

Thank you!

@TomAugspurger
Copy link
Member

@julioasotodv, yes I forgot about that case. Let me put something together quick.

@julioasotodv
Copy link
Author

Do you think there is a way to achieve this without making changes to dask's engine itself?

@TomAugspurger
Copy link
Member

TomAugspurger commented Nov 7, 2017 via email

@julioasotodv
Copy link
Author

I see. Would it work with that fix, even if chunksize is not defined for the underlying dask array?

@TomAugspurger
Copy link
Member

Yes, that should work. The solvers only require that the shape along the second axis is known:

from dask_ml.linear_model import LinearRegression
from dask_ml.datasets import make_regression

X, y = make_regression(chunks=50)

df = dd.from_dask_array(X)
X2 = df.values  # dask.array with unknown chunks along first dim

lm = LinearRegression(fit_intercept=False)
lm.fit(X2, y)

Note that fit_intercept does not currently work with unknown chunks. But when dask/dask-glm@master...TomAugspurger:add-intercept-dd is merged, you'd just do

lm = LinearRegression()  # fit_intercept=True
lm.fit(df)

And the intercept is added during the fit.

@julioasotodv
Copy link
Author

That's awesome!

But let me be just a little picky with that change (dask/dask-glm@master...TomAugspurger:add-intercept-dd):

In theory, if using either L1 or L2 regularization (or Elastic Net), the penalty term should not affect the intercept (this is, the "ones" column that works as the intercept should not be multiplied by the Lagrange multipliers that perform the actual regularization).

However, it would still be better than not having intercept. What do you think about this?

@TomAugspurger
Copy link
Member

Thanks, I'll take a look at how other packages handle regularization of the intercept, but I think your correct. cc @moody-marlin thoughts on that?

@cicdw
Copy link

cicdw commented Nov 13, 2017

Yea, I agree that the intercept should not be included in the regularization; I believe this is recommended best practice, and also not regularizing the intercept ensures that all regularizers still produce estimates which satisfy that the residuals have mean 0, which preserves the standard interpretation of things like R^2, etc.

@TomAugspurger TomAugspurger mentioned this issue Nov 14, 2017
Open
@TomAugspurger
Copy link
Member

Opened dask/dask-glm#65 to track that.

I'll deprecate the estimators in dask_glm and move them over here later today.

@jakirkham
Copy link
Member

See there is PR ( dask/dask-glm#66 ) to deprecate the dask-glm estimators and PR ( #94 ), which seems to have migrated the bulk of that content to dask-ml. Is this still the plan?

@TomAugspurger
Copy link
Member

TomAugspurger commented Jun 6, 2018 via email

@asifali22
Copy link

asifali22 commented Sep 5, 2018 •
edited
Loading

I'm facing the same issue.

Traceback (most recent call last):
  File "diya_libs/alog_main.py", line 20, in <module>
    clf.fit(X, y)
  File "/Users/asifali/workspace/pythonProjects/ML-engine-DataX/pre-processing/diya_libs/lib/algorithms/diya_logit.py", line 67, in fit
    self.estimator.fit(X, y)
  File "/anaconda3/lib/python3.6/site-packages/dask_ml/linear_model/glm.py", line 153, in fit
    X = self._check_array(X)
  File "/anaconda3/lib/python3.6/site-packages/dask_ml/linear_model/glm.py", line 167, in _check_array
    X = add_intercept(X)
  File "/anaconda3/lib/python3.6/site-packages/multipledispatch/dispatcher.py", line 164, in __call__
    return func(*args, **kwargs)
  File "/anaconda3/lib/python3.6/site-packages/dask_glm/utils.py", line 147, in add_intercept
    raise NotImplementedError("Can not add intercept to array with "
NotImplementedError: Can not add intercept to array with unknown chunk shape

Initially I tried with Dask DataFrame, later changed to Dask Array using
X = X.values #resulted in nan chunks which is causing the above error.
What am I supposed to do now? How do I install the fix, mentioned above? As it is not present in the version available on pip.

@TomAugspurger
Copy link
Member

@asifali22 that looks strange. Can you provide a full example? Does the following work for you?

from dask import dataframe as dd
from dask_glm.datasets import make_classification
from dask_ml.linear_model import LogisticRegression

X, y = make_classification(n_samples=10000, n_features=2)

X = dd.from_dask_array(X, columns=["a","b"])
y = dd.from_array(y)

lr = LogisticRegression()
lr.fit(X.values, y.values)

@TomAugspurger TomAugspurger mentioned this issue Sep 5, 2018
Closed
@thebeancounter
Copy link

Having a similar issue with dask array @TomAugspurger see my SO question, Any idea?

@TomAugspurger
Copy link
Member

@thebeancounter do you have a minimal example? http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

@thebeancounter
Copy link

@TomAugspurger
Hi. The code is in the SO question, do you mean copy it here?

@TomAugspurger
Copy link
Member

TomAugspurger commented Jun 14, 2019 via email

@thebeancounter
Copy link

@TomAugspurger

Data is defined
It's regular cifar10 data, passed via a pre trained resnet 50 for feature extraction. Trains well with sklearn. I can't guarantee that there are no zero variance columns but those should not prevent learning anyway! Only waste some processing time.

Here is the data zipped (read it from folder with generator just for preventing memory from exploding)

i = ImageDataGenerator(preprocessing_function=preprocess_input)

train_flow = i.flow_from_directory(directory=test_dir, target_size=(224, 224), class_mode="sparse", batch_size=1024, shuffle=True)

pre_model = ResNet50(weights="imagenet", include_top=False)
pre_model.compile(optimizer=Adam(), loss=categorical_crossentropy)

labels = []
data = []
for i in range(len(train_flow)):
    imgs, l = next(train_flow)
    data.append(pre_model.predict(imgs))
    labels.append(l)

labels = np.concatenate(labels)
data = np.concatenate(data, axis=0)
data = data.reshape(-1, np.prod(data.shape[1:]))

Data is under
github.com/thebeancounter/data

@TomAugspurger
Copy link
Member

TomAugspurger commented Jun 16, 2019 via email

@thebeancounter
Copy link

@TomAugspurger

Hi, I posted the code and the data. It's a solid example :-)

Anyhow, Can you maybe post a working example for using numpy array for logistic regression in dask?

@TomAugspurger
Copy link
Member

TomAugspurger commented Jun 16, 2019 via email

@thebeancounter
Copy link

@TomAugspurger
my data originally comes from a numpy array, I need to convert it to some form that dask can learn on. Can't find any example for that in the tutorial, maybe that's the issue, can you point me to something of that kind?

@TomAugspurger
Copy link
Member

TomAugspurger commented Jun 17, 2019 via email

@xiaozhongtian
Copy link

xiaozhongtian commented Jun 19, 2019 •
edited
Loading

@TomAugspurger

  • Unknown chunksize

I have seen above and there is the case:

X2 = df.values  # dask.array with unknown chunks along first dim

For me if i use .values, I will not know the chunksize for this array

x= df_train.values
dask.array<values, shape=(nan, 11), dtype=float64, chunksize=(nan, 11)>

And will this influence the distributed computation?
Like the managing the memory, the speed?

  • fit_intercept:
    The same question with the block above:
m_dkl.fit(df_train.values,df["target"])

NotImplementedError: Can not add intercept to array with unknown chunk shape

Will i need to use fit_intercept = False? will the performance be the same as sci-kit learn?

  • The difference between dask-ml glm and sci-kit learn glm
import dask_ml.linear_model as dkl  
import sklearn.linear_model as skl 
m_skl = skl.LogisticRegression(C=0.01, penalty='l1', n_jobs=-1,random_state=0)
m_dkl = dkl.LogisticRegression(C=0.01, penalty='l1', n_jobs=-1,random_state=0)

m_skl.fit(df_train,df["target"])
m_dkl.fit(df_train.values,df["target"])

In my case, I find that the sci-kit learn estimator accept the dask data fomat(array, dataframe),so, what is the big difference between these?
Is the dask-glm just fitting better in the case "big data" with the specific chunksize ? If we don't know the chunksize above, dask-ml.glm will do it as sci-kit learn or we will have a auto chunksize for distribution?

@thebeancounter
Copy link

@TomAugspurger

Scikit learn will not utilize the machines cores, and takes way way way too long to run...
Looking for a multithreaded solution.

@thebeancounter
Copy link

@xiaozhongtian can you please clarify? are you asking a question? Not sure I see the connection to this thread.

@xiaozhongtian
Copy link

@TomAugspurger
I'm asking a question with the same confusion in the above.

@xiaozhongtian
Copy link

@thebeancounter

Scikit learn will not utilize the machines cores, and takes way way way too long to run...

With the n_job = -1 in sci-kit learn, it uses the multi-process to fit. no?

But here, I want to know the manage of the memory for scikit learn and dask-ml.
If we don't use the chunk to divise the dataset, there will be no different with sci-ket learn in my opinion.

@carloszanella
Copy link

I'm having the same problem by building a dataframe from dask arrays, then calling .values just before passing it to a dask_ml.LinearRegression model. Anyone figured this out?

@stsievert
Copy link
Member

I'm having the same problem

I presume you mean an NotImplementedError: Can not add intercept to array with unknown chunk shape from #84 (comment). Try dask.DataFrame.to_dask_array(lengths=True) https://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.DataFrame.to_dask_array

This will compute the chunk sizes and the length of the array.

@stsievert stsievert mentioned this issue Apr 26, 2020
Merged
@Abhishekdutt9
Copy link

Use lr.fit(X.values, y.values) instead

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@TomAugspurger @stsievert @jakirkham @cicdw @thebeancounter @asifali22 @julioasotodv @xiaozhongtian @carloszanella @Abhishekdutt9