Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: IntoIterator for Box<[T]> + method dispatch mitigation for editions < 2024 #116607

Closed
wants to merge 2 commits into from

Conversation

clarfonthey
Copy link
Contributor

@clarfonthey clarfonthey commented Oct 10, 2023

ACP: rust-lang/libs-team#263
References #59878

Recommendation per ACP: this should receive a crater run to gauge impact. If there's no impact, it can be merged as-is, but otherwise it will need a similar edition-based workaround to the array implementation.

In addition to what was proposed by the ACP, this also adds IntoIterator for &Box<[T]> and IntoIterator for &mut Box<[T]> to ensure that those work as expected. I also already had to change at least one line in the compiler to account for this change, which isn't a good sign toward whether edition-specific mitigations may be needed, but we'll see.

@rustbot
Copy link
Collaborator

rustbot commented Oct 10, 2023

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Oct 10, 2023
@rust-log-analyzer

This comment has been minimized.

@clarfonthey clarfonthey changed the title [NEEDS CRATER] IntoIterator for Box<[T]> IntoIterator for Box<[T]> Oct 10, 2023
@clarfonthey
Copy link
Contributor Author

It appears that Clippy is explicitly testing for this, so, I'll have to take a look at that. But it should at least be ready for a Crater run if we're looking to just judge the impact of the initial impl.

@fee1-dead
Copy link
Member

@bors try

@bors
Copy link
Contributor

bors commented Oct 11, 2023

⌛ Trying commit b4c7843 with merge ec8d137...

bors added a commit to rust-lang-ci/rust that referenced this pull request Oct 11, 2023
IntoIterator for Box<[T]>

ACP: rust-lang/libs-team#263

Recommendation per ACP: this should receive a crater run to gauge impact. If there's no impact, it can be merged as-is, but otherwise it will need a similar edition-based workaround to the array implementation.

In addition to what was proposed by the ACP, this also adds `IntoIterator for &Box<[T]>` and `IntoIterator for &mut Box<[T]>` to ensure that those work as expected. I also already had to change at least one line in the compiler to account for this change, which isn't a good sign toward whether edition-specific mitigations may be needed, but we'll see.
}

#[stable(feature = "boxed_slice_into_iter", since = "CURRENT_RUSTC_VERSION")]
impl<'a, I, A: Allocator> !Iterator for &'a Box<[I], A> {}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this needed? is the !Iterator impl for [_] not enough?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately not. :(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment here for why they're needed would be very 🆒

@bors
Copy link
Contributor

bors commented Oct 11, 2023

☀️ Try build successful - checks-actions
Build commit: ec8d137 (ec8d137b933d39ddf0a2878617a0335f0f9d7116)

@rust-log-analyzer

This comment has been minimized.

@Mark-Simulacrum
Copy link
Member

@craterbot check

@craterbot
Copy link
Collaborator

👌 Experiment pr-116607 created and queued.
🤖 Automatically detected try build ec8d137
⚠️ Try build based on commit b4c7843, but latest commit is f31acaa. Did you forget to make a new try build?
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

@craterbot craterbot added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 14, 2023
@craterbot
Copy link
Collaborator

🚧 Experiment pr-116607 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

@craterbot
Copy link
Collaborator

🎉 Experiment pr-116607 is completed!
📊 600 regressed and 5 fixed (378027 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the blacklist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

@craterbot craterbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Oct 19, 2023
@clarfonthey
Copy link
Contributor Author

I'm not 100% clear on the results (it looks like more than 9 root crates are actually failing), but it does seem to me like some sort of edition mechanism would be good to include. Is that the kind of thing that should have an RFC, or would it be reasonable to say that the ACP is enough?

@Mark-Simulacrum Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 21, 2023
@clarfonthey
Copy link
Contributor Author

clarfonthey commented Nov 3, 2023

After taking a look at what was done for 2021 edition, it appears that there wasn't any particular RFC for it, just, a PR was made and that was the solution. So, I'm going to follow suit and start work on the override, hopefully not having to tread too deeply in compiler internals.

After a quick search, it shouldn't be too difficult to modify the rustc_skip_array_during_method_dispatch attribute to work with boxed slices, and to create a boxed_slice_into_iter edition lint in addition to the array_into_iter edition lint. My current plan is to replace this with a more generic rustc_skip_during_method_dispatch attribute which can accept a list of items, and for now it'll be array and boxed_slice. Hopefully nothing else will be added, but you can never be too careful. This is mostly just to avoid having separate rustc_skip_array_during_method_dispatch and rustc_skip_boxed_slice_during_method_dispatch attributes.

The only major questions I have here are:

  • It appears that the method dispatch code is replicated inside the rust-analyzer code base, and I'm going to assume I shouldn't touch that for now, and rust-analyzer can be fixed later.
  • The edition guides exist inside the repo but the 2024 edition guide hasn't been made yet, so, I'm not sure if I should write up something for this similar to that for the 2021 edition on arrays, or if I should hold off for now.
  • I don't think that this should need an FCP assuming that there are tests that verify that the behaviour is the same on the 2021 edition, although I'm assuming that once everything is done, we'll want to do a crater run again to verify that.

EDIT: I'm going to push the code I have right now even though I know it's not done, to give an idea of what I've changed so far. Will mark the PR accordingly once it's finished.

@clarfonthey clarfonthey changed the title IntoIterator for Box<[T]> WIP: IntoIterator for Box<[T]> Nov 3, 2023
@clarfonthey clarfonthey changed the title WIP: IntoIterator for Box<[T]> WIP: IntoIterator for Box<[T]> + method dispatch mitigation for editions < 2024 Nov 3, 2023
@clarfonthey
Copy link
Contributor Author

Will try and return to this PR this week. I would like to see this in the 2024 edition.

@rust-log-analyzer
Copy link
Collaborator

The job x86_64-gnu-llvm-17 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
GITHUB_ACTION=__run_7
GITHUB_ACTIONS=true
GITHUB_ACTION_REF=
GITHUB_ACTION_REPOSITORY=
GITHUB_ACTOR=clarfonthey
GITHUB_API_URL=https://api.github.com
GITHUB_BASE_REF=master
GITHUB_ENV=/home/runner/work/_temp/_runner_file_commands/set_env_1332b52b-04d9-40f6-a0df-8035b7437968
GITHUB_EVENT_NAME=pull_request
---
GITHUB_SERVER_URL=https://github.com
GITHUB_SHA=f2a3b1eeca713cc2f653519808271c30d2b2b56a
GITHUB_STATE=/home/runner/work/_temp/_runner_file_commands/save_state_1332b52b-04d9-40f6-a0df-8035b7437968
GITHUB_STEP_SUMMARY=/home/runner/work/_temp/_runner_file_commands/step_summary_1332b52b-04d9-40f6-a0df-8035b7437968
GITHUB_TRIGGERING_ACTOR=clarfonthey
GITHUB_WORKFLOW_REF=rust-lang/rust/.github/workflows/ci.yml@refs/pull/116607/merge
GITHUB_WORKFLOW_SHA=f2a3b1eeca713cc2f653519808271c30d2b2b56a
GITHUB_WORKSPACE=/home/runner/work/rust/rust
GOROOT_1_20_X64=/opt/hostedtoolcache/go/1.20.14/x64
---
#17 exporting to docker image format
#17 sending tarball 45.0s done
#17 DONE 55.4s
##[endgroup]
Setting extra environment values for docker:  --env ENABLE_GCC_CODEGEN=1 --env GCC_EXEC_PREFIX=/usr/lib/gcc/
[CI_JOB_NAME=x86_64-gnu-llvm-17]
##[group]Clock drift check
  local time: Thu Mar 21 02:40:37 UTC 2024
  network time: Thu, 21 Mar 2024 02:40:37 GMT
  network time: Thu, 21 Mar 2024 02:40:37 GMT
##[endgroup]
sccache: Starting the server...
##[group]Configure the build
configure: processing command line
configure: 
configure: build.configure-args := ['--build=x86_64-unknown-linux-gnu', '--llvm-root=/usr/lib/llvm-17', '--enable-llvm-link-shared', '--set', 'rust.thin-lto-import-instr-limit=10', '--set', 'change-id=99999999', '--enable-verbose-configure', '--enable-sccache', '--disable-manage-submodules', '--enable-locked-deps', '--enable-cargo-native-static', '--set', 'rust.codegen-units-std=1', '--set', 'dist.compression-profile=balanced', '--dist-compression-formats=xz', '--disable-dist-src', '--release-channel=nightly', '--enable-debug-assertions', '--enable-overflow-checks', '--enable-llvm-assertions', '--set', 'rust.verify-llvm-ir', '--set', 'rust.codegen-backends=llvm,cranelift,gcc', '--set', 'llvm.static-libstdcpp', '--enable-new-symbol-mangling']
configure: target.x86_64-unknown-linux-gnu.llvm-config := /usr/lib/llvm-17/bin/llvm-config
configure: llvm.link-shared     := True
configure: rust.thin-lto-import-instr-limit := 10
configure: change-id            := 99999999
---
    |
209 | macro_rules! rustc_attr {
    | ----------------------- when calling this macro
...
898 |         editions < 2021 (array) or editions < 2024 (boxed_slice)."
    |                                                                   ^ missing tokens in macro arguments
note: while trying to match `,`
   --> compiler/rustc_feature/src/builtin_attrs.rs:225:83
    |
    |
225 |     ($attr:ident, $typ:expr, $tpl:expr, $duplicates:expr, $encode_cross_crate:expr, $msg:expr $(,)?) => {

   Compiling rustc_type_ir v0.0.0 (/checkout/compiler/rustc_type_ir)
error: could not compile `rustc_feature` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

@clarfonthey
Copy link
Contributor Author

(did a very crude rebase, and the previous code wasn't even working, so. gonna poke at this more later)

// so those calls will still resolve to the slice implementation, by reference.
#[stable(feature = "boxed_slice_into_iter", since = "CURRENT_RUSTC_VERSION")]
impl<I, A: Allocator> IntoIterator for Box<[I], A> {
type IntoIter = vec::IntoIter<I, A>;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder, should this really be publically the same type? (can't be changed later)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's hard to imagine why we would want something different.

vec::IntoIter is

  • ptr + cap + alloc (needed to free allocation)
  • ptr to first not yielded element
  • ptr one past the last not yielded element

You need all the same things for a box. Since the size of initialized elements changes when iterating, there is no difference with a vec -- you need to save the allocation and you need to save the initialized range in it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also worth pointing out that not making them the same type is also something we can't revert after the fact, so, it feels like we should just pick the logical option here and make them the same.

@cuviper
Copy link
Member

cuviper commented Mar 24, 2024

Should we also do this for Box<[T; N]>? That could possibly deserve a separate iterator type, since the capacity is hard-coded in N, but I think it would also be fine to just use vec::IntoIter as well.

@clarfonthey
Copy link
Contributor Author

I feel like Box<[T; N]> is actually a bit more controversial for the type since it might be valuable to keep the N as part of the iterator type, but we also don't want to move it off the heap when deconstructing.

That one I feel like we could potentially copy array::IntoIter and make a proper version, which then makes me question whether we should just do the same for regular boxes too just to be consistent.

@the8472 the8472 added the T-lang Relevant to the language team, which will review and decide on the PR/issue. label Apr 16, 2024
@m-ou-se
Copy link
Member

m-ou-se commented Apr 16, 2024

We discussed this in today's libs-api meeting. We'd be happy to see this in the next edition.

We didn't think it is worth making separate IntoIter types, and would prefer simply reusing Vec's IntoIter (for both box-of-slice and box-of-array). Keeping the N part of the type doesn't seem to have a lot of benefits, as it only represents an upper bound, which has very limited use.

@the8472
Copy link
Member

the8472 commented Apr 16, 2024

Marking as T-lang since it affects method dispatch.

@clarfonthey
Copy link
Contributor Author

I also have been meaning to get back to this but kept running into unrelated life issues. Will hopefully be able to do a proper dive soon-- the deadline to get an initial implementation in two weeks seems pretty doable; my plan, as demonstrated by the existing mess, is to just piggyback off the 2021 edition implementation.

@tmandry tmandry added T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. and removed T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Apr 17, 2024
@clarfonthey
Copy link
Contributor Author

Okay, I thought about it and I simply cannot justify putting in this much work with this little help given the other things I have going on in life. Would love to see this in edition 2024, and I'm sure there are folks knowledgeable in this code who could pick up the slack, but if it's pushed to a future edition, that's fine too.

I don't want this enough to abandon other priorities at the moment, especially as I'm unemployed and this effort means nothing to any job I apply to. Apologies but also, not sorry for my decision. Will close this and folks can decide what to do.

@compiler-errors
Copy link
Member

I'm sorry to hear that @clarfonthey. I'll make sure to pick this up for the edition, and make sure to keep the commits with you as the author, unless you don't want to -- ping me on Zulip if so. Thanks for all the effort you put into this up til now!

@clarfonthey
Copy link
Contributor Author

Whether you keep me on the commits doesn't really matter much, since these records still exist, although I do appreciate the gesture.

bors added a commit to rust-lang-ci/rust that referenced this pull request Apr 18, 2024
…=<try>

[crate] Add `Box<[T; N]>: IntoIterator` without any method dispatch hacks

**Unlike** `Box<[T]>` (rust-lang#116607 (comment)), there's a much higher chance that this will not conflict with existing usages since it produces an iterator with the same type before/after this change, but let's test that theory with crater.

Ideally we have fewer migrations that are tangled up in hacks like `rustc_skip_during_method_dispatch`, so if this crater comes back clean, I'd strongly suggest landing this as-is.

As for the rationale for having this impl at all, I agree (as `@clarfonthey` pointed out in rust-lang#124097 (comment)) that it is generally better for any user to not require moving the array *out* of the box just to turn it into an iterator.
bors added a commit to rust-lang-ci/rust that referenced this pull request May 21, 2024
…fleLapkin

Add `IntoIterator` for `Box<[T]>` + edition 2024-specific lints

* Adds a similar method probe opt-out mechanism to the `[T;N]: IntoIterator` implementation for edition 2021.
* Adjusts the relevant lints (shadowed `.into_iter()` calls, new source of method ambiguity).
* Adds some tests.
* Took the liberty to rework the logic in the `ARRAY_INTO_ITER` lint, since it was kind of confusing.

Based mostly off of rust-lang#116607.

ACP: rust-lang/libs-team#263
References rust-lang#59878
Tracking for Rust 2024: rust-lang#123759

Crater run was done here: rust-lang#116607 (comment)
Consensus afaict was that there is too much breakage, so let's do this in an edition-dependent way much like `[T; N]: IntoIterator`.
RalfJung pushed a commit to RalfJung/miri that referenced this pull request May 22, 2024
Add `IntoIterator` for `Box<[T]>` + edition 2024-specific lints

* Adds a similar method probe opt-out mechanism to the `[T;N]: IntoIterator` implementation for edition 2021.
* Adjusts the relevant lints (shadowed `.into_iter()` calls, new source of method ambiguity).
* Adds some tests.
* Took the liberty to rework the logic in the `ARRAY_INTO_ITER` lint, since it was kind of confusing.

Based mostly off of #116607.

ACP: rust-lang/libs-team#263
References #59878
Tracking for Rust 2024: rust-lang/rust#123759

Crater run was done here: rust-lang/rust#116607 (comment)
Consensus afaict was that there is too much breakage, so let's do this in an edition-dependent way much like `[T; N]: IntoIterator`.
flip1995 pushed a commit to flip1995/rust-clippy that referenced this pull request May 24, 2024
Add `IntoIterator` for `Box<[T]>` + edition 2024-specific lints

* Adds a similar method probe opt-out mechanism to the `[T;N]: IntoIterator` implementation for edition 2021.
* Adjusts the relevant lints (shadowed `.into_iter()` calls, new source of method ambiguity).
* Adds some tests.
* Took the liberty to rework the logic in the `ARRAY_INTO_ITER` lint, since it was kind of confusing.

Based mostly off of #116607.

ACP: rust-lang/libs-team#263
References #59878
Tracking for Rust 2024: rust-lang/rust#123759

Crater run was done here: rust-lang/rust#116607 (comment)
Consensus afaict was that there is too much breakage, so let's do this in an edition-dependent way much like `[T; N]: IntoIterator`.
@clarfonthey clarfonthey deleted the box-into-iter branch May 28, 2024 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-lang Relevant to the language team, which will review and decide on the PR/issue. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.