getting error in orange - cannot create Silhouette score > 5000 records

[arunado]

Please help me understand this error? cannot create Silhouette score > 5000 records

WHat should be done here?

[markotoplak]

Silhuette score is O(n^2) algorithm and it is thus slow (and uses lots of memory) for big data sets. Users often tried to use it with data sets that were too big, and their Orange therefore crashed.

Because it is used with K-Means, which you can use on big data sets precisely because it is fast, we limited silhouette to 5000 instances, otherwise silhuette score would make k-means useless in those cases.

[ajdapretnar]

@arunado Do you have a suggestion for a better error?

[arunado]

@ajdapretnar Sure, Lets do a warning when uploading a data set > 5000 saying certain functions may not work.

Thanks

[ajdapretnar]

But there already is a warning. I was referring to a better wording of the warning.

[irgolic]

I'm with @arunado on this one.
We do this around a couple places in Orange, just to avoid crashes. Even if we're preventing the user from doing something 'stupid', it can be extremely frustrating for the user, having their agency taken away like this.

Crashing should be fixed by widget multiprocessing. Until then, can we just make widgets which would take a long time only run manually? As in, apply automatically is disabled, and a warning is thrown asking the user to press 'Apply changes' for processing to begin, knowing full well it may take a long time.

[ajdapretnar]

As far as I understand, k-Means can still be run, but without Silhouette. Alternatively, we could just run silhouette on a sample of 5000 instances (suboptimal).

If we let the user run Silhouette on an arbitrarily large data set (without multiprocessing), it would often happen that the user will be forced to force quite Orange, because it will be useless for ages.

[irgolic]

I know it will take ages for Orange to become responsive once more, but taking away the user's agency is not the answer.

What about showing a checkbox in the extended description of the warning itself, allowing users to enable silhouettes past 5k? This will include an explanation that it may be slow.

[markotoplak]

This has not that much to do about responsiveness than it does with memory needed for that particular operation. K-means already operates in a different thread.

In principle, I very much agree with allowing the users to do anything they wish so a do-it-anyway button is welcome. But for K-means users would not gain much. Sihuette computation uses O(n^2) memory (at least that implementation used to do it that way when we added the check), so the class of problems where the override would be applicable is quite small.

On the other hand, silhuette score approximation should work very well on large data sets, where the number of clusters is much lower than the size of data. Therefore we could compute approximate silhuette scores on big data sets and show a warning that these are approximate.