Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: skip terminfo parsing in Miri #100206

Merged
merged 1 commit into from
Aug 8, 2022
Merged

Conversation

RalfJung
Copy link
Member

@RalfJung RalfJung commented Aug 6, 2022

Terminfo parsing takes a significant amount of time in Miri, making libtest startup very slow. To work around that Miri in fact unsets the TERM variable. However, this means we don't get colors in cargo miri test.

So I propose we add some logic in libtest that skips parsing terminfo files under Miri, and just uses the regular basic coloring commands (taken from the colored crate).

As far as I can see, these two commands are all that libtest ever needs from terminfo, so Miri doesn't even lose any functionality through this. If you want I can entirely remove the terminfo parsing code and just use these commands instead.

Cc rust-lang/miri#2292 @saethlin

@rustbot rustbot added the T-libs Relevant to the library team, which will review and decide on the PR/issue. label Aug 6, 2022
@rustbot
Copy link
Collaborator

rustbot commented Aug 6, 2022

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

  • Stabilizing library features
  • Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
  • Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
  • Changing public documentation in ways that create new stability guarantees
  • Changing observable runtime behavior of library APIs

@rust-highfive
Copy link
Collaborator

r? @thomcc

(rust-highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 6, 2022
@@ -119,13 +130,22 @@ pub(crate) struct TerminfoTerminal<T> {
impl<T: Write + Send> Terminal for TerminfoTerminal<T> {
fn fg(&mut self, color: color::Color) -> io::Result<bool> {
let color = self.dim_if_necessary(color);
if cfg!(miri) && color < 8 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not obvious to me what happens if color = 9. The logic for num_colors isn't totally straightforward, but I think in Miri we end up with num_colors = 0, right? Could we explain this in a comment?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah num_colors is always 0 in Miri. I added a comment.

@saethlin
Copy link
Member

saethlin commented Aug 6, 2022

Wow this is a lot less code than I expected!

// The Miri logic for this only works for the most basic 8 colors, which we just assume
// the terminal will support. (`num_colors` is always 0 in Miri, so higher colors will
// just fail. But libtest doesn't use any higher colors anyway.)
let s = format!("\x1B[3{color}m");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, slightly surprised this is more efficient under miri than something like write!(self.out, "\x1B[3{color}m").

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it is more efficient. But Miri isn't that slow, a bit of formatting is fine.

The terminfo parser is doing a lot of work and constructing a huge data structure to represent the parsed result, that's why this is noticeable.

But your code looks nicer anyway. :D So, happy to change it accordingly.

@thomcc
Copy link
Member

thomcc commented Aug 7, 2022

@bors r+

(P.S. Let me know if you need review on any unix terminal or windows console stuff inside miri, terminals are a bit of an obsession of mine, so I'm pretty familiar with how this all works. I'm guessing this is probably enough for now though)

@bors
Copy link
Contributor

bors commented Aug 7, 2022

📌 Commit 771df125f18c5c562744b21718dce10825530b58 has been approved by thomcc

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 7, 2022
@RalfJung
Copy link
Member Author

RalfJung commented Aug 7, 2022

@bors r-

I'll fix that nit you mentioned

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 7, 2022
@RalfJung
Copy link
Member Author

RalfJung commented Aug 7, 2022

(P.S. Let me know if you need review on any unix terminal or windows console stuff inside miri, terminals are a bit of an obsession of mine, so I'm pretty familiar with how this all works. I'm guessing this is probably enough for now though)

Thanks! Miri does not really do anything with the terminal, it just forwards the bytes it gets from the interpreted program. Though #98070 will be nice for Miri since it means we can implement the isatty shim in a better way. :)

However you might then be the right person for the other question I had in this PR: given that, from what I can see, these two command codes are literally everything libtest ever uses -- do you think it would make sense to remove all that terminfo parsing and just use these commands? (E.g. the colored crate doesn't do any terminfo parsing and seems to work fine, though maybe it just hasn't been tested enough.) In particular I don't know if that will work on Windows.

@RalfJung
Copy link
Member Author

RalfJung commented Aug 7, 2022

@bors r=thomcc

@bors
Copy link
Contributor

bors commented Aug 7, 2022

📌 Commit 5076c90 has been approved by thomcc

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 7, 2022
@thomcc
Copy link
Member

thomcc commented Aug 7, 2022

do you think it would make sense to remove all that terminfo parsing and just use these commands? (E.g. the colored crate doesn't do any terminfo parsing and seems to work fine, though maybe it just hasn't been tested enough.)

I'd have to look what all it uses. If it's just 8 colors and reset, this uses a very consistent set of escape sequences across a wide range of terminals -- you need to start worrying about terminfo mostly when you care about advanced features, especially events handling, since the sequences terminals send for events are often very different, even among modern terminals which in most other ways have converged on doing things like more or less xterm.

That said, removing this would probably break compatibility with certain retro terminals though (which currently we probably support... But really, those terminals should just disabling color anyway, I suspect.

If we're only interested in basic coloring, removing this is probably a good idea, as it will speed things up for everybody to not query the terminfo DB. It will also avoid doing weird things if users have $TERM set to a goofy value (like TERM=vt100, which I've seen before).

That said, I'd like to look through terminfo.src to see if there's anything people are likely to still use that have other values for these (I know often reset (sgr0) will often reset the charset too, but we don't change this, so it's fine).

In particular I don't know if that will work on Windows.

Windows doesn't have a terminfo db, and uses the console apis (e.g. winapi functions which probably turn into syscalls) to change the color. I believe we already do this in libtest, so windows isn't a concern.

On modern windows (Win10 and later) the escape sequences are supported, although you may need to explicitly ask the console to ENABLE_VIRTUAL_TERMINAL_PROCESSING. This is actually on by default on many consoles, but has the benefit of returning an error on older windows and if the terminal can't support it.

That said, this is fiddley and for simple stuff like we want, so I wouldn't bother changing away from the console apis (besides, we still support win7 and win8 at the moment).

bors added a commit to rust-lang-ci/rust that referenced this pull request Aug 7, 2022
…iaskrgr

Rollup of 6 pull requests

Successful merges:

 - rust-lang#100019 (Revive suggestions for boxed trait objects instead of impl Trait)
 - rust-lang#100038 (Document the `no-std` target option in config.toml.example)
 - rust-lang#100194 (Remove even more box syntax uses from src/test)
 - rust-lang#100206 (test: skip terminfo parsing in Miri)
 - rust-lang#100230 (Use start_point instead of next_point to point to elided lifetime amp…)
 - rust-lang#100244 (Add armv4t-none-eabi take2)

Failed merges:

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 07315fe into rust-lang:master Aug 8, 2022
@rustbot rustbot added this to the 1.65.0 milestone Aug 8, 2022
bors added a commit to rust-lang/miri that referenced this pull request Aug 8, 2022
stop excluding TERM env var on Unix

Effectively reverts #2018.
Needs rust-lang/rust#100206 to not be terribly slow.
Fixes #2292.
@RalfJung RalfJung deleted the miri-terminfo branch August 8, 2022 13:03
bors added a commit to rust-lang/miri that referenced this pull request Sep 20, 2022
remove Windows TERM env var hack and -Zmiri-env-exclude

The hack should not be needed any more since rust-lang/rust#100206. And that also mostly removes the need for `-Zmiri-env-exclude` -- if needed, users can still achieve the same by running `(unset VAR; cargo miri test)`.

I am keeping `-Zmiri-env-forward` since it is still useful, e.g. to have RUST_BACKTRACE=full set in an otherwise deterministic execution.

`@rust-lang/miri` any objections to removing `-Zmiri-env-exclude`?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-libs Relevant to the library team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants