WIP: Classification based on subject and preview text

[st3iny]

Supersedes #7918

[st3iny]

I ran the meta estimator 100 times on the same data set, extracted all parameters of all individual estimators and did some number crunching with python and plotting with gnuplot.

Parameters of the best configuration for my personal mails (considering variance and f1 score):

k	weighted	kernel
15	true	Manhattan

The data shows a significant improvement of the f1 score while showing acceptable variance. The lowest recorded f1 score is still higher than the vanilla f1 score.

Vanilla extractor and vanilla estimator (GaussianNB)

$ occ mail:account:train -vvv 4
[debug] found 1 incoming mailbox(es)
[debug] found 1 outgoing mailbox(es)
[debug] found 350 messages of which 141 are important
[debug] data set split into 280 (i: 112) training and 70 (i: 29) validation sets with 4 dimensions
[debug] classification report: {"recall":0.3448275862068966,"precision":1,"f1Score":0.5128205128205129}
[debug] classifier validated: recall(important)=0.3448275862069, precision(important)=1 f1(important)=0.51282051282051
[debug] classifier 71 persisted
42MB of memory used

New extractor and new estimator (KNN)

$ occ mail:account:train -vvv --new 4
[debug] found 1 incoming mailbox(es)
[debug] found 1 outgoing mailbox(es)
[debug] found 350 messages of which 141 are important
[debug] data set split into 280 (i: 112) training and 70 (i: 29) validation sets with 14 dimensions
[debug] classification report: {"recall":1,"precision":0.7631578947368421,"f1Score":0.8656716417910448}
[debug] classifier validated: recall(important)=1, precision(important)=0.76315789473684 f1(important)=0.86567164179104
[debug] classifier 73 persisted
82MB of memory used

Variance across 100 experiments

[ChristophWurst]

Personal account

old

[debug] found 1 incoming mailbox(es)
[debug] found 1 outgoing mailbox(es)
[debug] found 350 messages of which 174 are important
[debug] data set split into 280 (i: 121) training and 70 (i: 53) validation sets with 4 dimensions
[debug] classification report: {"recall":0.49056603773584906,"precision":0.96296296296296291,"f1Score":0.65000000000000002}
[debug] classifier validated: recall(important)=0.49056603773585, precision(important)=0.96296296296296 f1(important)=0.65
[debug] classifier 3255 persisted

new

[debug] found 1 incoming mailbox(es)
[debug] found 1 outgoing mailbox(es)
[debug] found 350 messages of which 174 are important
[debug] data set split into 280 (i: 121) training and 70 (i: 53) validation sets with 14 dimensions
[debug] classification report: {"recall":0.81132075471698117,"precision":0.87755102040816324,"f1Score":0.84313725490196079}
[debug] classifier validated: recall(important)=0.81132075471698, precision(important)=0.87755102040816 f1(important)=0.84313725490196
[debug] classifier 3256 persisted

Work account

old

[debug] found 1 incoming mailbox(es)
[debug] found 1 outgoing mailbox(es)
[debug] found 350 messages of which 187 are important
[debug] data set split into 280 (i: 154) training and 70 (i: 33) validation sets with 4 dimensions
[debug] classification report: {"recall":1,"precision":1,"f1Score":1}
[debug] classifier validated: recall(important)=1, precision(important)=1 f1(important)=1
[debug] classifier 3258 persisted

new

[debug] found 1 incoming mailbox(es)
[debug] found 1 outgoing mailbox(es)
[debug] found 350 messages of which 187 are important
[debug] data set split into 280 (i: 154) training and 70 (i: 33) validation sets with 14 dimensions
[debug] classification report: {"recall":0.87878787878787878,"precision":1,"f1Score":0.93548387096774188}
[debug] classifier validated: recall(important)=0.87878787878788, precision(important)=1 f1(important)=0.93548387096774
[debug] classifier 3259 persisted

[ChristophWurst]

Performance analysis

[ChristophWurst]

Current finding is that the input data looks skewed. The dimensional reduction produces tiny numbers. Normalization brings them back into a reasonable range but the values have almost no variance across the feature vectors of all messages.

Word count vectorization works. There are lots of 0s and occasional 1s. It is rare to have a word more than once in a single subject. Dimensional reduction will still make sense. Applying https://docs.rubixml.com/2.0/transformers/tf-idf-transformer.html after the WCV would be worth a try.