std: Properly handle interior NULs in std::process #31056

kamalmarhubi · 2016-01-20T16:56:13Z

This reports an error at the point of calling Command::spawn() or one of
its equivalents.

rust-highfive · 2016-01-20T16:56:20Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @aturon (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

kamalmarhubi · 2016-01-20T16:57:23Z

r? @alexcrichton @nagisa

kamalmarhubi · 2016-01-20T16:59:12Z

I think both issues will manifest on Windows as well, but I made no changes there. The added regression tests will likely cause failures on Windows.

alexcrichton · 2016-01-20T17:32:41Z

src/libstd/sys/unix/process.rs

+// to that point and report an error via a `Result` rather than by
+// panicking.
+#[derive(Debug, Clone)]
+pub enum ValidatedCString {


In the case that an invalid string is passed down, we don't necessarily need to store the original data, so perhaps this could just store a flag in Command as to whether something with a nul byte in it has been passed in? That way once we hit spawn we can just check the flag and return an error appropriately.

I mostly kept it so the Debug impl on std::process::Command would display something. I can change this though.

Ah that's a good point. The Debug impl would probably be fine to start including something like <string-with-nul> or something like that instead of the actual value, however. It seems relatively complicated to define a type like this just for a Debug implementation that's unlikely to ever be seen.

Maybe (ab)use Option for this?

pub struct Command { program: Option<CString>, args: Vec<Optrion<ValidatedCString>>, env: Option<HashMap<Option<CString>, Option<ValidatedCString>>>, cwd: Option<Option<CString>>, uid: Option<uid_t>, gid: Option<gid_t>, session_leader: bool, saw_nul: bool, }

The Debug impl could then use unwrap_or("<string-with-nul>").

This is a bit weird for env, as the hash map would lose context of how many vars were nulful. The cwd would be a bit unwieldy with the double Option...

@alexcrichton any thoughts on how best to handle this?

Oh I wouldn't change the internal structure of Command much, just add a flag indicating that something with a nul byte was passed in. The Debug implementation only shows the program and its arguments, so if we really want we can store CString::new("<string-with-nul>").unwrap() in those places, but other than that we don't need to track the information elsewhere.

Ok, so that means dropping this whole ValidatedCString thing, right? Shall I close this and open an new PR skipping that altogether?

I also don't have a good grasp on what Windows does here. I think my tests on std::process will fail there, but without seeing the failures I'm not sure what to do to fix it.

Nah keeping this PR is fine (just pushing the changes to it).

I do think that the tests will fail on Windows as well, but it should be pretty straightforward (e.g. checking encode_wide() for zeros) and relatively the same implementation as Unix.

I've changed the unix version. I'll try to make changes for Windows, but it'll be kind of fumbling in the dark.

brson · 2016-01-20T23:33:26Z

I marked this as a regression since it changes behavior, but I'm in favor.

kamalmarhubi · 2016-01-21T16:21:50Z

Argh I missed the warnings and didn't realise they were errors in the build. Just pushed something that should pass. There are still a couple of open discussion points though.

alexcrichton · 2016-01-22T20:44:44Z

src/libstd/sys/unix/process.rs

 }
 fn init_env_map(&mut self) {
 if self.env.is_none() {
- self.env = Some(env::vars_os().collect());
+ // Will not add NULs to env: preexisting environment will not contain any.
+ self.env = Some(env::vars_os().map(|(k, v)| (k, v)).collect());


I don't think that this needs to change, right?

erm yeah that's a non-perfect refactor of an intermediate stage I had.... time to apply map id => id...

alexcrichton · 2016-01-22T20:47:50Z

Looks good to me! I think the Windows implementation will just need to be tweaked and this should be good to go. If you want to test things out it and you're on Unix you may be able to use this helper script I have to at least ensure the standard library itself compiles, I suspect this is a situation where "when it compiles it works" :)

kamalmarhubi · 2016-01-22T21:04:58Z

Thanks for the script!

kamalmarhubi · 2016-02-02T21:50:35Z

Sorry for letting this go for so long. I kept putting off learning about OsStr on windows. This should be done now!

alexcrichton · 2016-02-03T01:52:59Z

src/libstd/sys/windows/process.rs

@@ -43,13 +43,25 @@ fn mk_key(s: &OsStr) -> OsString {
 })
 }

+fn ensure_no_nuls<T: AsRef<OsStr>>(str: T) -> io::Result<T> {
+ let has_nul = {
+ let bytes = str.as_ref().as_inner().inner.as_inner();


Ah can this actually use the .encode_wide() method? That's what'll be passed down to the OS anyway, and is a little more reliably than looking at the internal bytes.

The environment block is built up using .extend(), and I wanted to avoid the extra allocation and copy of the keys and values that would be necessary to check the .encode_wide() output. See https://github.com/rust-lang/rust/pull/31056/files#diff-9a0a769432651d9c59644e0a8c7f887eR353

We can avoid the repeated allocation by reusing a buffer for the conversion, but we'd still pay the extra copy cost. I'm happy to do this however you prefer, though.

Oh I see what you mean. Ignore above!

alexcrichton · 2016-02-03T01:53:59Z

Thanks @kamalmarhubi! Looks good to me modulo one nit and I'd be fine sending to bors after.

kamalmarhubi · 2016-02-03T02:22:26Z

Yay! I'm really excited to see what the bors experience is like, so I'll try and keep this moving. :-)

alexcrichton · 2016-02-03T02:31:26Z

Thanks! Can you squash the commits down into one as well? (sorry forgot to check that last time)

kamalmarhubi · 2016-02-03T02:43:29Z

Done, and rebased.

alexcrichton · 2016-02-03T03:32:35Z

@bors: r+ 59d070c0d684926de73fe0400096b63091798de7

bors · 2016-02-03T05:23:04Z

⌛ Testing commit 59d070c with merge 28c6780...

bors · 2016-02-03T06:38:00Z

💔 Test failed - auto-win-gnu-64-nopt-t

This reports an error at the point of calling `Command::spawn()` or one of its equivalents. Fixes rust-lang#30858 Fixes rust-lang#30862

kamalmarhubi · 2016-02-03T15:55:46Z

Fixed test and squashed. Diff: kamalmarhubi/rust@59d070c...7c64bf1

kamalmarhubi · 2016-02-03T15:56:26Z

Urg pushed the wrong thing.

kamalmarhubi · 2016-02-03T16:09:22Z

Actually no, I pushed the right thing I just can't get the compare url to display as I'd like. Here's the diff:

diff --git a/src/libstd/sys/windows/process.rs b/src/libstd/sys/windows/process.rs
index 758044c..61cf28b 100644
--- a/src/libstd/sys/windows/process.rs
+++ b/src/libstd/sys/windows/process.rs
@@ -419,11 +419,12 @@ mod tests {
     #[test]
     fn test_make_command_line() {
         fn test_wrapper(prog: &str, args: &[&str]) -> String {
-            String::from_utf16(
-                &make_command_line(OsStr::new(prog),
-                                   &args.iter()
-                                        .map(|a| OsString::from(a))
-                                        .collect::<Vec<OsString>>())).unwrap()
+            let command_line = &make_command_line(OsStr::new(prog),
+                                                  &args.iter()
+                                                       .map(|a| OsString::from(a))
+                                                       .collect::<Vec<OsString>>())
+                                    .unwrap();
+            String::from_utf16(command_line).unwrap()
         }

         assert_eq!(

alexcrichton · 2016-02-03T17:11:11Z

@bors: r+ 7c64bf1

bors · 2016-02-03T17:19:11Z

⌛ Testing commit 7c64bf1 with merge 8fc73c7...

…hton This reports an error at the point of calling `Command::spawn()` or one of its equivalents. Fixes #30858 Fixes #30862

bors · 2016-02-03T19:26:03Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-mac-ios-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt

rust-highfive assigned aturon Jan 20, 2016

rust-highfive assigned alexcrichton and unassigned aturon Jan 20, 2016

alexcrichton reviewed Jan 20, 2016
View reviewed changes

brson added relnotes Marks issues that should be documented in the release notes of the next release. regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. labels Jan 20, 2016

alexcrichton reviewed Jan 22, 2016
View reviewed changes

alexcrichton reviewed Feb 3, 2016
View reviewed changes

kamalmarhubi force-pushed the std-process-nul-chars branch from b3d4b4e to 59d070c Compare February 3, 2016 02:42

kamalmarhubi changed the title ~~std: Properly handle interior NULs in std::process on unix~~ std: Properly handle interior NULs in std::process Feb 3, 2016

std: Properly handle interior NULs in std::process

7c64bf1

This reports an error at the point of calling `Command::spawn()` or one of its equivalents. Fixes rust-lang#30858 Fixes rust-lang#30862

kamalmarhubi force-pushed the std-process-nul-chars branch from 59d070c to 7c64bf1 Compare February 3, 2016 15:55

bors added a commit that referenced this pull request Feb 3, 2016

Auto merge of #31056 - kamalmarhubi:std-process-nul-chars, r=alexcric…

8fc73c7

…hton This reports an error at the point of calling `Command::spawn()` or one of its equivalents. Fixes #30858 Fixes #30862

bors merged commit 7c64bf1 into rust-lang:master Feb 3, 2016

std: Properly handle interior NULs in std::process #31056

std: Properly handle interior NULs in std::process #31056

Conversation

kamalmarhubi commented Jan 20, 2016

rust-highfive commented Jan 20, 2016

kamalmarhubi commented Jan 20, 2016

kamalmarhubi commented Jan 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brson commented Jan 20, 2016

kamalmarhubi commented Jan 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Jan 22, 2016

kamalmarhubi commented Jan 22, 2016

kamalmarhubi commented Feb 2, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Feb 3, 2016

kamalmarhubi commented Feb 3, 2016

alexcrichton commented Feb 3, 2016

kamalmarhubi commented Feb 3, 2016

alexcrichton commented Feb 3, 2016

bors commented Feb 3, 2016

bors commented Feb 3, 2016

kamalmarhubi commented Feb 3, 2016

kamalmarhubi commented Feb 3, 2016

kamalmarhubi commented Feb 3, 2016

alexcrichton commented Feb 3, 2016

bors commented Feb 3, 2016

bors commented Feb 3, 2016