Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
with pl.StringCache():
    s = pl.Series('s', ['a'], pl.Categorical(ordering = 'lexical'))
    
s.to_frame()._row_encode([(False, False, False)])

Log output

thread '<unnamed>' panicked at crates/polars-core/src/chunked_array/ops/row_encode.rs:122:29:
assertion failed: payloads.len() >= num_known_categories as usize
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Issue description

Panic occurs when it shouldn't, although I am not really sure how to handle this situation since the categories are no longer available.

Expected behavior

No panic.

Installed versions

Replace this line with the output of pl.show_versions(). Leave the backticks in place.