Fixed stratified splitting with Dask #1883
Conversation
jppgks
force-pushed
the
stratified-sampling
branch
from
2d8c7b8 to
037fd80
Compare
|
For a partitioned dataset, we can stratify split each partition individually to obtain a global stratified split. Proof. |
Assumes every partition fits in memory
Given $P$ the set of partitions of dataset $D$, and $S$ the set of unique values in the stratify column. When stratified splitting every partition individually, $\forall s \in S:$
$$
\begin{align}
&\sum_{p \in P} \\%\_{train} \times | p\_{S = s} | & \\%\_{train} \texttt{ of every partition's } s
\texttt{ values will be in that partition's train} \\
= &\\%\_{train} \times \sum_{p \in P} | p\_{S = s} | & \texttt{which is } \\%\_{train} \texttt{ of the sum of all } s \texttt{ value counts over all partitions} \\
= &\\%\_{train} \times | D\_{S = s} | & \texttt{which is } \\%\_{train} \texttt{ of the total number of } s \texttt{ values} \\
\end{align}
$$
$\\%\_{train}$ of records in $D$ with value $s$ in the stratify column will land in the training set. Equivalent proof for the valid and test split.
|
From @justinxzhao: Move |
|
From @ShreyaR: might want to repartition after splitting into train/valid/test. Determine when to make the repartition call |
Assumes every partition fits in memory
Given $P$ the set of partitions of dataset $D$, and $S$ the set of unique values in the stratify column. When stratified splitting every partition individually, $\forall s \in S:$
$$
\begin{align}
&\sum_{p \in P} \\%\_{train} \times | p\_{S = s} | & \\%\_{train} \texttt{ of every partition's } s
\texttt{ values will be in that partition's train} \\
= &\\%\_{train} \times \sum_{p \in P} | p\_{S = s} | & \texttt{which is } \\%\_{train} \texttt{ of the sum of all } s \texttt{ value counts over all partitions} \\
= &\\%\_{train} \times | D\_{S = s} | & \texttt{which is } \\%\_{train} \texttt{ of the total number of } s \texttt{ values} \\
\end{align}
$$
$\\%\_{train}$ of records in $D$ with value $s$ in the stratify column will land in the training set. Equivalent proof for the valid and test split.
No description provided.