Skip to content

Improve performance of RunEndEncoded cast #8707

@vegarsti

Description

@vegarsti

Once #8709 is done, we'll have a benchmark we can use to try to improve the REE casting performance. @alamb and @tustvold have a neat suggestion to use the filter kernel:

I think what @tustvold is getting at is that the filter kernel is (very) fast and thus using it rather than the take kernel is likely to be faster.

So I think the idea here is instead of building up value_indexes it would be to implement some way to turn the result of Partition into a boolean array and pass to the filter kernel

    let partitions = partition(&[Arc::clone(cast_array)])?;
    // .... 
    let filter: BooleanArray = partitions.into_inner(); // not sure about this API
    // call filter instead of take
    let values_array = filter(&cast_array, &filter)?;

Originally posted by @alamb in #8589 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions