Tracking issue for Vec::extract_if and LinkedList::extract_if

[Gankra]

Feature gate: #![feature(extract_if)] (previously drain_filter)

This is a tracking issue for Vec::extract_if and LinkedList::extract_if, which can be used for random deletes using iterators.

Public API

pub mod alloc {
    pub mod vec {
        impl<T, A: Allocator> Vec<T, A> {
            pub fn extract_if<F>(&mut self, filter: F) -> ExtractIf<'_, T, F, A>
            where
                F: FnMut(&mut T) -> bool,
            {
            }
        }

        #[derive(Debug)]
        pub struct ExtractIf<'a, T, F, #[unstable(feature = "allocator_api", issue = "32838")] A: Allocator = Global>
        where
            F: FnMut(&mut T) -> bool, {}

        impl<T, F, A: Allocator> Iterator for ExtractIf<'_, T, F, A>
        where
            F: FnMut(&mut T) -> bool,
        {
            type Item = T;
            fn next(&mut self) -> Option<T> {}
            fn size_hint(&self) -> (usize, Option<usize>) {}
        }

        impl<T, F, A: Allocator> Drop for ExtractIf<'_, T, F, A>
        where
            F: FnMut(&mut T) -> bool,
        {
            fn drop(&mut self) {}
        }
    }

    pub mod collections {
        pub mod linked_list {
            impl<T> LinkedList<T> {
                pub fn extract_if<F>(&mut self, filter: F) -> ExtractIf<'_, T, F>
                where
                    F: FnMut(&mut T) -> bool,
                {
                }
            }

            pub struct ExtractIf<'a, T: 'a, F: 'a>
            where
                F: FnMut(&mut T) -> bool, {}

            impl<T, F> Iterator for ExtractIf<'_, T, F>
            where
                F: FnMut(&mut T) -> bool,
            {
                type Item = T;
                fn next(&mut self) -> Option<T> {}
                fn size_hint(&self) -> (usize, Option<usize>) {}
            }

            impl<T, F> Drop for ExtractIf<'_, T, F>
            where
                F: FnMut(&mut T) -> bool,
            {
                fn drop(&mut self) {}
            }

            impl<T: fmt::Debug, F> fmt::Debug for ExtractIf<'_, T, F>
            where
                F: FnMut(&mut T) -> bool,
            {
                fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {}
            }
        }
    }
}

Steps / History

• Implementation: Add Vec::drain_filter #43245
• removed drain-on-drop behavior, renamed to extract_if
  • Don't drain-on-drop in DrainFilter impls of various collections. #104455
  • Remove drain-on-drop behavior from various DrainFilter iterators libs-team#136
• Stabilization PR

Unresolved Questions

• What should the method be named?
• Should extract_if accept a Range argument?
• Missing Send+Sync impls on linked list's ExtractIf, see comment

See #43244 (comment) for a more detailed summary of open issues.

[RalfJung]

Related issues:
#25477
#34265

[bluss]

Maybe this doesn't need to include the kitchen sink, but it could have a range parameter, so that it's like a superset of drain. Any drawbacks to that? I guess adding bounds checking for the range is a drawback, it's another thing that can panic. But drain_filter(.., f) can not.

[rustonaut]

Is there any chance this will stabilize in some form in the not to far future?

[rustonaut]

If the compiler is clever enough to eliminate the bounds checks
in the drain_filter(.., f) case I would opt for doing this.

( And I'm pretty sure you can implement it in a way
which makes the compiler clever eneugh, in the worst
case you could have a "in function specialization",
basically something like if Type::of::<R>() == Type::of::<RangeFull>() { dont;do;type;checks; return } )

[jonhoo]

I know this is bikeshedding to some extent, but what was the reasoning behind naming this drain_filter rather than drain_where? To me, the former implies that the whole Vec will be drained, but that we also run a filter over the results (when I first saw it, I thought: "how is this not just .drain(..).filter()?"). The former on the other hand indicates that we only drain elements where some condition holds.

[rustonaut]

No idea, but drain_where sounds much better and is much more intuitive.
Is there still a chance to change it?

[bluss]

.remove_if has been a prior suggestion too

[rustonaut]

I think drain_where does explains it the best. Like drain it returns values, but it does not drain/remove all values but just such where a given condition is true.

remove_if sounds a lot like a conditional version of remove which just removes a single item by index if a condition is true e.g. letters.remove_if(3, |n| n < 10); removes the letter at index 3 if it's < 10.

drain_filter on the other hand is slightly ambiguous, does it drain then filter in a more optimized way (like filter_map) or does if drain so that a iterator is returned comparble to the iterator filter would return,
and if so shouldn't it be called filtered_drain as the filter get logically used before...

[Gankra]

There is no precedent for using _where or _if anywhere in the standard library.

[jonhoo]

@gankro is there a precedent for using _filter anywhere? I also don't know that that is that a reason for not using the less ambiguous terminology? Other places in the standard library already use a variety of suffixes such as _until and _while.

[crlf0710]

The "said equivalent" code in the comment is not correct... you have to minus one from i at the "your code here" site, or bad things happens.

[thegranddesign]

IMO it's not filter that's the issue. Having just searched for this (and being a newbie), drain seems to be fairly non-standard compared to other languages.

Again, just from a newbie perspective, the things I would search for if trying to find something to do what this issue proposes would be delete (as in delete_if), remove, filter or reject.

I actually searched for filter, saw drain_filter and kept searching without reading because drain didn't seem to represent the simple thing that I wanted to do.

It seems like a simple function named filter or reject would be much more intuitive.

[thegranddesign]

On a separate note, I don't feel as though this should mutate the vector it's called on. It prevents chaining. In an ideal scenario one would want to be able to do something like:

        vec![
            "",
            "something",
            a_variable,
            function_call(),
            "etc",
        ]
            .reject(|i| { i.is_empty() })
            .join("/")

With the current implementation, what it would be joining on would be the rejected values.

I'd like to see both an accept and a reject. Neither of which mutate the original value.

[rpjohnst]

You can already do the chaining thing with filter alone. The entire point of drain_filter is to mutate the vector.

[thegranddesign]

@rpjohnst so I searched here, am I missing filter somewhere?

[rpjohnst]

Yes, it's a member of Iterator, not Vec.

[Gankra]

Drain is novel terminology because it represented a fourth kind of ownership in Rust that only applies to containers, while also generally being a meaningless distinction in almost any other language (in the absence of move semantics, there is no need to combine iteration and removal into a single ""atomic"" operation).

Although drain_filter moves the drain terminology into a space that other languages would care about (since avoiding backshifts is relevant in all languages).

[polarathene]

I came across drain_filter in docs as a google result for rust consume vec. I know that due to immutability by default in rust, filter doesn't consume the data, just couldn't recall how to approach it so I could manage memory better.

drain_where is nice, but as long as the user is aware of what drain and filter do, I think it's clear that the method drains the data based on a predicate filter.

[jonhoo]

I still feel as though drain_filter implies that it drains (i.e., empties) and then filters. drain_where on the other hand sounds like it drains the elements where the given condition holds (which is what the proposed function does).

[tmccombs]

Shouldn't linked_list::DrainFilter implement Drop as well, to remove any remaining elements that match the predicate?

[Gankra]

Yes

[photino]

I would prefer drain_while, since we already have drain, skip_while, take_while, wait_while.

And the documentation also says that using this method is equivalent to the following code:

let mut i = 0;
while i < vec.len() {
    if some_predicate(&mut vec[i]) {
        let val = vec.remove(i);
        // your code here
    } else {
        i += 1;
    }
}

[tguichaoua]

I would prefer drain_while, since we already have drain, skip_while, take_while, wait_while.

And the documentation also says that using this method is equivalent to the following code:

let mut i = 0;
while i < vec.len() {
    if some_predicate(&mut vec[i]) {
        let val = vec.remove(i);
        // your code here
    } else {
        i += 1;
    }
}

From take_while and skip_while documentation :

After false is returned, take_while()’s job is over,

After false is returned, skip_while()’s job is over,

The _while suffix implies it's job is over after the first time the predicate returns false.
something like this

while predicate(...) {
    // do job
}
// stop job

But as you mention the equivalent code of extract_if/drain_filter/drain_if is different: it doesn't use the predicate in the loop condition.

[photino]

This is a subtle difference. I prefer drain_if now.

The _while suffix implies it's job is over after the first time the predicate returns false. something like this

while predicate(...) {
    // do job
}
// stop job

But as you mention the equivalent code of extract_if/drain_filter/drain_if is different: it doesn't use the predicate in the loop condition.

[rsalmei]

I also understand this func in terms of "drain". I think this word should appear in its name somehow.

[cybersoulK]

drain/extract already conveys a continuous iteration.
drain_if or extract_if
drain_mapped or extract_mapped (#43244 (comment))

[the8472]

The methods were intentionally renamed because they behave differently compared to drain. Drain keeps draining if you drop the Drain struct. I.e. unlike most iterators it is not lazy.
extract_if on the other hand is lazy and only removes elements as long as the iterator gets spun.

[rsalmei]

The methods were intentionally renamed because they behave differently compared to drain. Drain keeps draining if you drop the Drain struct. I.e. unlike most iterators it is not lazy.

extract_if on the other hand is lazy and only removes elements as long as the iterator gets spun.

I wasn't aware of that, thanks. It makes sense to be different then.

[Emilgardis]

Maybe the history section should be updated to highlight the name change and the reason a bit better

[NexusXe]

Is it possible to point to this feature when a user is trying to use the drain_filter method? This seems like it could cause some confusion.

[rsalmei]

I'm eagerly anticipating the addition of this feature to the standard library. Could it be stabilized this year?

[ebkalderon]

Quick question for anyone familiar with the internals of the unstable extract_if() methods: I noticed that all existing ExtractIf<'a, ...> iterators don't implement the FusedIterator trait. Since these iterators supposedly visit each element exactly once (unless the iterator is dropped partway through iteration, of course), it would seem that they are suitable candidates for implementing FusedIterator. Is this understanding correct? Or is there a reason these iterators aren't fused? Thanks!

[the8472]

They shouldn't suddenly restart, no. So it most likely is an oversight.

[joshtriplett]

@rfcbot merge

@rfcbot concern vec-needs-a-range-argument

We discussed this in today's @rust-lang/libs-api meeting. We were in favor of stabilizing these two, and we have consensus on the current names. However, we would like to have a range parameter to Vec::extract_if.

[rfcbot]

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Amanieu
• @BurntSushi
• @dtolnay
• @joshtriplett
• @m-ou-se

Concerns:

• vec-needs-a-range-argument (Tracking issue for Vec::extract_if and LinkedList::extract_if #43244 (comment))

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[cuviper]

Not necessarily a blocking concern, but can we remove the where F: FnMut constraint on struct ExtractIf and its Drop and Debug? The drain-on-drop behavior was removed a while ago, so now we only need to be callable for Iterator.

[the8472]

Sure I'll add that to my PR.

[Amanieu]

Also another note from the libs-api meeting is that we are still interested in solving more exotic use cases such as ExtractMapped proposed in #43244 (comment). However this may be better done via some form of cursor API rather than an iterator: you would need to manually increment the cursor, but it would give you precise control at each point whether to mutate or remove the value currently being pointed to by the cursor.

Such an API should be proposed as a separate ACP, with inspiration from linked list cursors (#58533) and B-Tree cursors (#107540).