morestack on x86 32-bit does not save edx nor floating point stack

[nikomatsakis]

Right now, we return 64-bit values and doubles using the native convention, which is to use EDX:EAX and the floating stack respectively. morestack only saves/restores EAX, however. I've observed failures as a result, in particular the src/test/run-pass/auto-encode.rs test when executed with the default options (the failure goes away with RUST_MIN_STACK set to something appropriately high). Due to the unpredictable nature of stack switching, though, the failures tend to come and go.

(cc @brson)

[graydon]

Yeah, my best guess at the end of yesterday's adventure with auto-encode debugging was someone clobbering the register holding the high half of the v variable in a logging / io function. Calling any io function after loading v caused the failure. I'd believe stack switching / morestack is the culprit.

[nikomatsakis]

This is currently manifesting on my system as extra::time::run_tests failing an assert_eq even though left and right are equal. I have a patch for EDX, not sure about the matter of double returns.

[nikomatsakis]

Ah, I see that wikipedia states: "The x87 floating point registers ST0 to ST7 must be empty (popped or freed) when calling a new function, and ST1 to ST7 must be empty on exiting a function." We are definitely doing this wrong, but I'm not 100% sure of the correct X86 assembly to save the stack iff it is non-empty.

[nikomatsakis]

I believe the correct thing to do is to use fstp, which will set a flag if st(0) had a value. We then must push the flags register too, I guess, and then conditionally do a fldp. Alternatively, maybe we can just use outpointers to return doubles :)

[nikomatsakis]

I should add: the code for doubles subtle enough that I would like a test that reproduces it. I have not been able to produce one, although I do have a test that uses the #[fixed_stack_segment] annotation to ensure a stack switch when returning a double. This leads me to think that while we are technically violating the cdecl ABI, it may in fact be harmless for doubles (though I thought I saw errors that seems to be specifically caused by the floating point stack being corrupted, so maybe I just haven't found the right code path to trigger an error).

[emberian]

I think this is still relevant, @nikomatsakis?

[alexcrichton]

In today's meeting we have decided to jettison segmented stacks. Due to this no longer being used for switching stacks, I believe that this issue is moot.