OSX: can't run test framework - sometimes

[ccoupe]

Running the Tests/ tests_video_vlc.rb and tests_svg.rb as written fails on OSX unless its done with a $ ./cshoes -w Tests/...rb command line invocation. When run **without** the -w switch (or from a Splash screen pick) then the null osx $stdout has been _captured_ in a test/unit internal variable (io) in therequire`s or setup of minitest before the call to Shoes.terminal (or deprecated Shoes.show_console).

Although this seems related to issue #224 and #236 (which are mostly fixed), it's different. The nascent test framework GUI from @passenger94 doesn't account for the idea that stdout is defined later. I suspect test/unit might have an option to change that io variable. Or it should have.

[passenger94]

test/unit do provide an interface to hook a gui, i choose to go the console route so that i could use shoes terminal, was more interesting to me :-)

[ccoupe]

I may try to move the stdout/stderr notification center loops (if you will) into regular Shoes start up on osx and unless their is a terminal window, they wouldn't update anything just flush. That way $stdout is setup earlier as it is when run with -w on the commandline.

[ccoupe]

There's a curiosity or four with test-unit and Shoes/OSX. On OSX the xterm-256color is not being found in ENV['TERM'] or it's not being passed to the right places. That looks like a test-unit bug.

There are options to TestRunner.new that might help with setting output to $stderr -but I've yet to find any doc about where run() is located and what options it supports and what I have seen say's no options.

[IanTrudel]

test/unit do provide an interface to hook a gui, i choose to go the console route so that i could use shoes terminal, was more interesting to me :-)

Shoes users would benefit from a GUI version. we should always consider the long term benefits of extending Shoes ecosystem, encouraging users to use its features and report issues.

[ccoupe]

@backorder , so far the tests are only of value to people willing to fix shoes internals. Those folks can handle the terminal output and there is GUI to select which tests and button to not display details. I'm perfectly happy running them -w on the osx command line. The problem is they should also run loaded from the splash screen on osx. It behaves badly because Shoes/OSX doesn't have working ruby STDOUT at the time test/unit is loaded.

[ccoupe]

Some progress. I updated the test/unit gem to 3.2.0 (osx was 3.0.8, linux: 3.1.7) - no change on the colorization. I think I've found the bug that causes it to fail when run w/o -w. It's likely the monkey patch on IO.puts. Completely unsure why it works with -w.

One more clarification for @backorder, the GUI tests in test/unit are Gtk2, Tk, or Fox. You have to 'gem install test-unit-full to get them. Not part of normal Ruby. Shoes could be added to the list but would have to be co-ordinated/distributed by test/unit developers. Not much gain for Shoes since @passenger94's code is pretty and works (mostly). Shoes Scripters who want tests are either working on a non GUI code base or they want visual testing - button press -> something on screen. The latter is very difficult cross platform/cross-gtk-theme. The former was always available to anyone who wanted to spend sometime learning tests frameworks and Shoes which @Passenger has done.

Perhaps someone will be inspired to do a gui for mini-test since Ruby delivers both and evidently opinions about which is better runs deep.

[ccoupe]

It gets deeper and deeper. So many moving parts if you can't depend on STDOUT or $stdout.
This is ./cshoes -w and looks almost OK until you look at the what the test_svg.rb has been modified to produce.. Compare to this . Look carefully - Both are wrong. Yet It's the same Shoes.terminal code run from the same script. The difference is when Shoes.terminal is called.

My current thinking is that Shoes OSX needs to create it's stdout fd from an NSPipe before initializing Ruby, (world.c, shoes_init() and gems on the paths) and store the NSFileHandles and fds globally, Then when the event loop is started add an NSNotifcationCenter observer that basically just empties the pipe so it doesn't hang Shoes with a pipe full condition. IF a script calls Shoes.terminal (or Shoes.show_console) then cocoa-term.m would replace that Observer sink with a new one that puts the chars into the NSTextView like is done now.

Would it be easier to hook into minitest as an plugin sort of thing that interprets test results and appends to a Shoes edit_box? Maybe, Maybe not.

None of that explains why bugs/bug236.rb can work (-w or not) when displaying colors.

[passenger94]

following from a bit too far away
what a journey ... !

[ccoupe]

Before I could really get into what osx is doing I decided I need a better launch script for cshoes on OSX. This doesn't fix the terminal issues on OSX but if you comment out the Shoes.terminal in the tests, they do run and display colors on the launch terminal and brings back the double script bug so shoes.rb is not osx correct yet and the optsparse handling needs a little rewrite which affects all platforms - a whole lot of testing, more delays.

[ccoupe]

Now that cshoes can run the test using the launch terminal or the Shoes.terminal, I picked Tests/tests_color.rb to use for further testing of all the strange visual results. It's just a matter of commenting out the Shoes.terminal line. First I'll show linux and osx when using the native terminal.

CT-term-linux.png

CT-term-osx.png

Which is better? Which is more Correct? I'm guessing xterm-256color is not the same as whatever Linux announces and test/unit cares. There could be version differences for test/unit as well.
Let's uncomment the Shoes.terminal call and use it instead.

'CT-Shoes-linux.png`

Whoa! Shoes got the wrong color and dropped some of the colored characters. Not good. Also word wrapped. That a hint for dropped chars.

CT-Shoes-osx.png

No colors. Dropped a line of dashes and maybe more.

Shoes linux and Shoes OSX are both wrong. So many moving parts and bugs! This may never get fixed properly (I haven't tried on Windows yet which has both DOS and Msys consoles (4 more tests). OSX users can also change their emulation to xterm and xterm-16color (and many more) - another hint and series of tests to show that Shoes.terminal is not a replacement for the launch terminal.

Clear eyes might notice that OSX doesn't have a vertical scroll bar. You can scroll with the wheel though.

[IanTrudel]

I like the last screenshot the best. You still can use the colours to highlight successful and failed tests, or have an option to turn it on/off.

[ccoupe]

@backorder - you missed the point. All four of them should be similar (colored). Three are are, one incorrectly with cyan instead of green and the one you like, should have color - it's the default setting for test/unit. Shoes is not handling default color settings for two of them properly.

[ccoupe]

If you look closer, OSX native terminal sets a background of green with black text and Linux native terminal draws green dots. That's nothing to do with Shoes or the script settings (and both test/units are 3.20.0 now). Shoes.terminal ought to be close. It's not.

It could be that test/unit is sending a identify yourself escape sequence to Shoes.terminal and Shoes is dumping it on the floor and then ruby/test/unit picks something from ENV['TERM'] as a fallback and then sends half baked color setttings. That is something I can test for and supply if needed.

[ccoupe]

I modified the terminal code to capture the data stream before it's interpreted for esc sequences. On Linux it's sending an escape sequence I didn't handle for setting colors and it would be a mess to implement it properly - Any one want to type in an 256 element table of RGB values in C?

On OSX + Shoes.terminal there are no escape sequences being sent. zip, nada, none but the capture does have other controls chars so it is getting the raw stream from test/unit. debugging test/unit in Shoes is probably easier on OSX but I'm not looking forward to it.

[ccoupe]

OSX color diagnosed. test/unit checks if the output is a tty? before checking for ENV["TERM"] and that tty test will fail on Shoes/OSX in Shoes.terminal. No way to fix that in Shoes. Tests need to explicitly ask for color.

@test_result = Test::Unit::UI::Console::TestRunner.run(tests, {use_color: true})

you can set to it false if you don't like colors. The shade of green is a little different between native xterm-256color and Shoes.terminal but I consider that very small problem unlikely to be fixed or even noticed.

[ccoupe]

Tests/tests_video.rb has been a real head spinner of escape parsing because test/unit sets fg and bg in one ESC sequence sometimes. On my Linux system the audio test fails (known problem, nothing to do with Shoes or vlc) First the Linux without Shoes.terminal

Then what Shoes.terminal does (after the commit).

If you look closely, you'll find some oddities that only matter if you care how the tests look.

[ccoupe]

For consistency I'm going to add the osx screen captures for Tests/tests_video.rb showing the error they find - a different error from linux but the point is to show the text formatting and colors.

There are color differences which will be important to someone I suppose. The solution is too
implement the full 256 entry table of rgb instead of my hack. There's probably a copy in libvte if I can find it and it isn't too macro-ed up. Shorter term, there is a {:color_scheme => something} option to use 8 colors if anybody can figure what the name is (a class?, a constant? ). @passenger94 ?

I think I'll create a new github issue about the color256 problem and close this one since one can run tests with or without color. The colors may be wrong in some peoples opinion but since this a very unlikely to be used part of Shoes.

I'm also going to remove any cursor positioning code. Bad idea and who cares?

[ccoupe]

One mystery solved.The Gtk theme determines how/which test/unit escapes are sent. on the pi2, it looks like it does on OSX - green background, black character whereas on my Ubuntu theme it's green characters on a white background. I don't know whether to laugh, cry or sigh.

[ccoupe]

Closed in Shoes 3.3.2