net: be more specific about winapi features

[gftea]

Motivation

be more specific about winapi features (#4737)

Solution

add winapi/std dependency in net feature, remove features in dependencies.winapi.
Below run all OK on my windows 10 PC

cargo build --all-features
cargo check --all-features
cargo test --all-features

[Darksonn]

Can you try running this test (from the tokio subdirectory):

cargo hack check --feature-powerset --no-dev-deps

It will compile Tokio with every possible combination of feature flags. I think there are around 2000 combinations, so it might take an hour or two (it doesn't recompile every dependency every time).

I can't run it myself unfortunately as I don't have a windows computer. You will need cargo hack to do it.

[gftea]

running it now, while waiting, can you help explain why it works even though we have below imports?
It seems the docsrs is not cfg by default. e.g. handleapi is not explictly added in feature's dependency, why compilation pass?

#[cfg(not(docsrs))]
mod doc {
    pub(super) use crate::os::windows::ffi::OsStrExt;
    pub(super) use crate::winapi::shared::minwindef::{DWORD, FALSE};
    pub(super) use crate::winapi::um::fileapi;
    pub(super) use crate::winapi::um::handleapi;
    pub(super) use crate::winapi::um::namedpipeapi;
    pub(super) use crate::winapi::um::winbase;
    pub(super) use crate::winapi::um::winnt;

    pub(super) use mio::windows as mio_windows;
}

[Darksonn]

Which version of mio are you building with? (This should be visible in your Cargo.lock.)

[Darksonn]

Can you give me the output of this command as well?

cargo tree --features full -i winapi -e all

[gftea]

Now I hit a error during running cargo hack as below. But why cargo check --all-features pass? Is it because --all-features has includes some default features which also include those module?

info: running `cargo check --no-default-features --features process` on tokio (67/7176)
   Compiling winapi v0.3.9
    Checking tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio)
error[E0432]: unresolved import `winapi::um::handleapi`
  --> tokio\src\process\windows.rs:35:17
   |
35 | use winapi::um::handleapi::{DuplicateHandle, INVALID_HANDLE_VALUE};
   |                 ^^^^^^^^^ could not find `handleapi` in `um`

error[E0432]: unresolved import `winapi::um::processthreadsapi`
  --> tokio\src\process\windows.rs:36:17
   |
36 | use winapi::um::processthreadsapi::GetCurrentProcess;
   |                 ^^^^^^^^^^^^^^^^^ could not find `processthreadsapi` in `um`

error[E0432]: unresolved import `winapi::um::winbase`
  --> tokio\src\process\windows.rs:38:17
   |
38 | use winapi::um::winbase::{RegisterWaitForSingleObject, INFINITE};
   |                 ^^^^^^^ could not find `winbase` in `um`

[gftea]

$ cargo tree --features full -i winapi -e all
winapi v0.3.9
└── tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio)
    ├── tokio feature "bytes"
    │   ├── tokio feature "io-util"
    │   │   └── tokio feature "full" (command-line)
    │   └── tokio feature "process"
    │       └── tokio feature "full" (command-line)
    ├── tokio feature "default" (command-line)
    │   ├── tokio-stream v0.1.9 (C:\Users\kenny\Documents\repo\tokio\tokio-stream)
    │   │   ├── tokio-stream feature "default"
    │   │   │   └── tokio-test v0.4.2 (C:\Users\kenny\Documents\repo\tokio\tokio-test)
    │   │   │       └── tokio-test feature "default"
    │   │   │           [dev-dependencies]
    │   │   │           └── tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio) (*)
    │   │   │   [dev-dependencies]
    │   │   │   └── tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio) (*)
    │   │   └── tokio-stream feature "time"
    │   │       └── tokio-stream feature "default" (*)
    │   └── tokio-test v0.4.2 (C:\Users\kenny\Documents\repo\tokio\tokio-test) (*)
    ├── tokio feature "fs"
    │   └── tokio feature "full" (command-line)
    ├── tokio feature "full" (command-line)
    ├── tokio feature "io-std"
    │   └── tokio feature "full" (command-line)
    ├── tokio feature "io-util" (*)
    ├── tokio feature "libc"
    │   ├── tokio feature "net"
    │   │   └── tokio feature "full" (command-line)
    │   ├── tokio feature "process" (*)
    │   └── tokio feature "signal"
    │       └── tokio feature "full" (command-line)
    ├── tokio feature "macros"
    │   └── tokio feature "full" (command-line)
    ├── tokio feature "memchr"
    │   └── tokio feature "io-util" (*)
    ├── tokio feature "mio"
    │   ├── tokio feature "net" (*)
    │   ├── tokio feature "process" (*)
    │   └── tokio feature "signal" (*)
    ├── tokio feature "net" (*)
    ├── tokio feature "num_cpus"
    │   └── tokio feature "rt-multi-thread"
    │       └── tokio feature "full" (command-line)
    ├── tokio feature "once_cell"
    │   ├── tokio feature "process" (*)
    │   ├── tokio feature "rt"
    │   │   └── tokio-test v0.4.2 (C:\Users\kenny\Documents\repo\tokio\tokio-test) (*)
    │   │   ├── tokio feature "full" (command-line)
    │   │   ├── tokio feature "rt-multi-thread" (*)
    │   │   └── tokio feature "test-util"
    │   │       └── tokio-test v0.4.2 (C:\Users\kenny\Documents\repo\tokio\tokio-test) (*)
    │   └── tokio feature "signal" (*)
    ├── tokio feature "parking_lot"
    │   └── tokio feature "full" (command-line)
    ├── tokio feature "process" (*)
    ├── tokio feature "rt" (*)
    ├── tokio feature "rt-multi-thread" (*)
    ├── tokio feature "signal" (*)
    ├── tokio feature "signal-hook-registry"
    │   ├── tokio feature "process" (*)
    │   └── tokio feature "signal" (*)
    ├── tokio feature "socket2"
    │   └── tokio feature "net" (*)
    ├── tokio feature "sync"
    │   ├── tokio-stream v0.1.9 (C:\Users\kenny\Documents\repo\tokio\tokio-stream) (*)
    │   └── tokio-test v0.4.2 (C:\Users\kenny\Documents\repo\tokio\tokio-test) (*)
    │   ├── tokio feature "full" (command-line)
    │   └── tokio feature "test-util" (*)
    ├── tokio feature "test-util" (*)
    ├── tokio feature "time"
    │   └── tokio-test v0.4.2 (C:\Users\kenny\Documents\repo\tokio\tokio-test) (*)
    │   ├── tokio feature "full" (command-line)
    │   ├── tokio feature "test-util" (*)
    │   └── tokio-stream feature "time" (*)
    ├── tokio feature "tokio-macros"
    │   └── tokio feature "macros" (*)
    └── tokio feature "winapi"
        ├── tokio feature "net" (*)
        ├── tokio feature "process" (*)
        └── tokio feature "signal" (*)
├── winapi feature "cfg"
│   └── ntapi v0.3.7
│       ├── ntapi feature "default"
│       │   [dev-dependencies]
│       │   └── tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio) (*)
│       └── ntapi feature "user"
│           └── ntapi feature "default" (*)
├── winapi feature "consoleapi"
│   └── tokio feature "signal" (*)
├── winapi feature "default"
│   ├── ntapi v0.3.7 (*)
│   ├── remove_dir_all v0.5.3
│   │   └── remove_dir_all feature "default"
│   │       └── tempfile v3.3.0
│   │           └── tempfile feature "default"
│   │               ├── proptest v1.0.0
│   │               │   ├── proptest feature "bit-set"
│   │               │   │   └── proptest feature "default"
│   │               │   │       [dev-dependencies]
│   │               │   │       └── tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio) (*)
│   │               │   ├── proptest feature "break-dead-code"
│   │               │   │   └── proptest feature "default" (*)
│   │               │   ├── proptest feature "default" (*)
│   │               │   ├── proptest feature "fork"
│   │               │   │   ├── proptest feature "default" (*)
│   │               │   │   └── proptest feature "timeout"
│   │               │   │       └── proptest feature "default" (*)
│   │               │   ├── proptest feature "lazy_static"
│   │               │   │   └── proptest feature "std"
│   │               │   │       ├── proptest feature "default" (*)
│   │               │   │       └── proptest feature "fork" (*)
│   │               │   ├── proptest feature "quick-error"
│   │               │   │   └── proptest feature "std" (*)
│   │               │   ├── proptest feature "regex-syntax"
│   │               │   │   └── proptest feature "std" (*)
│   │               │   ├── proptest feature "rusty-fork"
│   │               │   │   ├── proptest feature "fork" (*)
│   │               │   │   └── proptest feature "timeout" (*)
│   │               │   ├── proptest feature "std" (*)
│   │               │   ├── proptest feature "tempfile"
│   │               │   │   └── proptest feature "fork" (*)
│   │               │   └── proptest feature "timeout" (*)
│   │               └── rusty-fork v0.3.0
│   │                   └── proptest v1.0.0 (*)
│   │                   ├── rusty-fork feature "timeout"
│   │                   │   └── proptest feature "timeout" (*)
│   │                   └── rusty-fork feature "wait-timeout"
│   │                       └── rusty-fork feature "timeout" (*)
│   │               [dev-dependencies]
│   │               └── tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio) (*)
│   ├── socket2 v0.4.4
│   │   ├── socket2 feature "all"
│   │   │   └── tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio) (*)
│   │   └── socket2 feature "default"
│   │       └── tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio) (*)
│   │       [dev-dependencies]
│   │       └── tokio v1.19.2 (C:\Users\kenny\Documents\repo\tokio\tokio) (*)
│   └── tempfile v3.3.0 (*)
├── winapi feature "errhandlingapi"
│   └── remove_dir_all v0.5.3 (*)
├── winapi feature "evntrace"
│   └── ntapi v0.3.7 (*)
├── winapi feature "fileapi"
│   ├── remove_dir_all v0.5.3 (*)
│   └── tempfile v3.3.0 (*)
├── winapi feature "handleapi"
│   ├── socket2 v0.4.4 (*)
│   └── tempfile v3.3.0 (*)
├── winapi feature "in6addr"
│   └── ntapi v0.3.7 (*)
├── winapi feature "inaddr"
│   └── ntapi v0.3.7 (*)
├── winapi feature "minwinbase"
│   └── ntapi v0.3.7 (*)
├── winapi feature "namedpipeapi"
│   └── tokio feature "net" (*)
├── winapi feature "ntsecapi"
│   └── ntapi v0.3.7 (*)
├── winapi feature "std"
│   └── remove_dir_all v0.5.3 (*)
│   └── tokio feature "net" (*)
├── winapi feature "threadpoollegacyapiset"
│   └── tokio feature "process" (*)
├── winapi feature "winbase"
│   ├── remove_dir_all v0.5.3 (*)
│   └── tempfile v3.3.0 (*)
├── winapi feature "windef"
│   └── ntapi v0.3.7 (*)
├── winapi feature "winerror"
│   └── remove_dir_all v0.5.3 (*)
├── winapi feature "winioctl"
│   └── ntapi v0.3.7 (*)
├── winapi feature "ws2ipdef"
│   └── socket2 v0.4.4 (*)
└── winapi feature "ws2tcpip"
    └── socket2 v0.4.4 (*)

[gftea]

$ grep mio -C 2 ../Cargo.lock

[[package]]
name = "mio"
version = "0.7.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
--
 "libc",
 "log",
 "miow",
 "ntapi",
 "winapi",
--

[[package]]
name = "mio"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
--

[[package]]
name = "mio-aio"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a316ed8ed5bc3989660d8333ad91abe9c165f6347aa876cfdfbddb0b906fafc"
dependencies = [
 "mio 0.7.14",
 "nix 0.22.3",
]

[[package]]
name = "miow"
version = "0.3.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
--
 "loom",
 "memchr",
 "mio 0.8.3",
 "mio-aio",
 "mockall",
 "nix 0.24.1",

[Darksonn]

Okay, so let's take handleapi as an example. We see this output:

├── winapi feature "handleapi"
│   ├── socket2 v0.4.4 (*)
│   └── tempfile v3.3.0 (*)

This means that both socket2 and tempfile enable the winapi/handleapi feature. The tempfile crate is only used in tests, so we can ignore it.

Since features are combined from all dependencies, this means that the feature is also available to Tokio if socket2 requires it. However, we only add socket2 when tokio/net is enabled, so if you are compiling with --features process like in the failing example, then we are not enabling net, and therefore socket2 does not get enabled, and therefore we don't get winapi/handleapi available in Tokio.

Furthermore, it is possible that socket2 stops requiring the winapi/handleapi feature in a future release. For example, this could happen if socket2 switches to the windows-sys crate instead. In that case, Tokio would break because we use the handleapi feature, but there is no longer any crates that list the feature as required.

What we are looking for here is that Tokio should list all of the winapi features that Tokio needs such that even if socket2 (or another dependency) switches to windows-sys, this does not result in Tokio breaking. This is what happened in #4662 where the mio crate and socket2 both switched to windows-sys simultaneously. It seems like socket2 has reverted this change since then, but we need to be prepared in case this happens again.

[gftea]

@Darksonn thanks! Now I understand better.
by the way there 7176 combination of feature flags when running cargo hack check --feature-powerset --no-dev-deps
Now so far only +50% has been run, taking more quite a long time.

[Darksonn]

Since I saw that cargo tree output, I understand the situation a bit better myself as well. You can stop the cargo hack run. It has become clear that it can't really tell us if we've added the flags everywhere we need them because socket2 already enables most of the features for us.

Can you figure out which winapi features listed in the cargo tree output are used by Tokio, and list them under the appropriate Tokio features?

[gftea]

please check latest comitted changes.
I searched all use of winapi::um, and add them to corresponding features

[Darksonn]

I think the below is everything else we are missing.

[Darksonn]

Assuming CI passes, then this looks good to me.