How to give sample weights?

[kgullikson88]

Is there a way to give sample weights to the fit method? I see that the metrics can take it as an argument, but the fit method doesn't.

[mfeurer]

Could you please describe your use case? In case the data is unbalanced, Auto-sklearn configures whether to activate the class_weight feature.

[kgullikson88]

I have some samples that I really need to get right, so want to assign a larger penalty for getting those ones wrong. Many of the sklearn classifiers take a sample_weight keyword argument in the fit method for this purpose (logistic regression does, for example)

[mfeurer]

I see your point. Sample weights would then have to be passed to each method in auto-sklearn, especially the final ensemble building procedure, right? However, I think that this would contradict the principle of auto-sklearn which tries to optimize a user-given metric. Do you think it would be possible that you create a custom metric which penalizes a solution for missing your important data points?

[kgullikson88]

That is sort of what I'm asking how to do. I tried making a custom metric (f3-score) that takes sample weights:

from sklearn.metrics import precision_score, recall_score
from autosklearn.metrics make_scorer

def score_func(y_true, y_pred, beta=3, sample_weight=None):
    if sample_weight is not None:
        prec = precision_score(y_true=y_true, y_pred=y_pred, sample_weight=sample_weight)
        rec = recall_score(y_true=y_true, y_pred=y_pred, sample_weight=sample_weight)
    else:
        prec = precision_score(y_true=y_true, y_pred=y_pred)
        rec = recall_score(y_true=y_true, y_pred=y_pred)
    if prec == 0 and rec == 0:
        return 0.0
    return (1 + beta**2) * prec * rec / (beta**2 * prec + rec)

scorer = make_scorer('f3_score', score_func, sample_weight=weights)

However, when I fit with that, I get errors about incompatible sizes because the metric doesn't know which samples are in the holdout set. The sample weights has shape (full_sample_size,), while the y_true and y_pred values have smaller sizes due to cross validation.

[mfeurer]

Thanks for pointing that out. It would be great to have auto-sklearn accept sample weights for the scoring functions. However, I will not be able to implement this feature in the next weeks. If you want to contribute this feature, I would be happy to assist with that.

[kgullikson88]

Sure, I could give it a shot if you could point me in the direction of where I would need to make the changes.

[mfeurer]

Great. Here's a brief tour:

auto-sklearn stores the data in a class AbstractDataManager, from which XYDataManager is derived. The data managers are used to persist the data on disk and are loaded by the evaluation module which takes care about restricting the runtime and memory usage of the target algorithm. Weights of the data points would have to be persisted, too.

It is then used in the evaluator class where the optimization loss is calculated. I think these are the code pieces where changes need to be done in order to influence the optimization procedure.

Furthermore, you would need to change the call to the scoring function in the ensemble builder and ensemble selection. Those two will be a bit trickier, as they rely on the correct sorting of the data (as the sorting will change due to the resampling strategy). You can have a look here how the targets are built in order to accommodate for the change of order.

I hope this is not too complicated and gives a good overview of where the code needs to be changed. In general, a search for call of calculate_score would be a good idea in case I missed one.

One more note: I will probably have no time to reply tomorrow and will be out of office for a few days afterwards. Therefore, I might not reply immediately until next Wednesday.

[xiangning-chen]

Hey, have the sample_weight problem be solved now? Thanks

[forest-jiang]

It would be great if we can add sample_weight that representing the confidence of each data point.

[kaiweiang]

@mfeurer Hi, regarding passing the right sample weights, I'm thinking we can leverage the index from pandas.DataFrame or pandas.Series. Currently, it can be done in sklearn's GridSearchCV (see https://stackoverflow.com/questions/49581104/sklearn-gridsearchcv-not-using-sample-weight-in-score-function?answertab=active#tab-top

I do know that autosklearn converts the X and y into numpy array by applying sklearn.utils.check_array although the pandas data frame are passed to the fit method. Any specific reason you enforce this?

[mfeurer]

We convert pandas dataframes to numpy arrays because we never found the time to update our packages to accept pandas. I'd be happy about pull requests overcoming this issue.

[krey]

Just throwing in another use case: I'm doing multioutput regression and there are some missing values y

[simonprovost]

Any luck some folks would have create the PR for the sample_weight addition while searching for the best configuration? Cheers.