Compiler infinite loop with memory allocation

[novacrazy]

EDIT:

This issue is with recursively imported modules like this:

pub mod myMod {
    pub use super::*;
}

where if crate A defines a module like that, attempting to use it in crate B will cause rustc and rustdoc to choke.

---

I'm sorry I can't provide any compiler output for this, because it technically did not crash, but I did get the thread stack using Process Explorer.

Here is the process info when I decided to kill the process:

And here is the stack trace where it seems to copy new memory:

rustc_resolve seems to crop up often as I refreshed the stack view.

I can't reproduce this with a short piece of code, because the resolution stuff is tied into the project, but I can show you exactly what happened in my code to produce this effect.

Basically, I'm writing an OpenAL wrapper, and attempting to refer to GLuint rather than ALuint causes the bug.

For example:

unsafe fn run() -> ALResult<()> {
    let sample_rate: GLuint = 44_100;
    let freq: GLuint = 600;
}

fn main() {
    unsafe {
        run().unwrap();
    }
}

should give an error that GLuint is not found, because it was a typo on my part, but instead it seems to soft-fail exhaustively resolving GLuint

If I correct my typo to ALuint, it compiles instantly with minimal memory usage.

Meta

Version:

rustc 1.15.0-nightly (71c06a56a 2016-12-18)
binary: rustc
commit-hash: 71c06a56a120a0d5e3b224105ee3e6754f83e5fa
commit-date: 2016-12-18
host: x86_64-pc-windows-msvc
release: 1.15.0-nightly
LLVM version: 3.9

[novacrazy]

Getting this exact behavior very often now with types that don't exist in scope.

[novacrazy]

It might not be related, but rustdoc is hanging even when rustc successfully compiles. It has the exact same memory and CPU behavior, but I can't get a good stack trace.

EDIT: After about 6GB of memory usage by rustdoc, it's stack overflowed. So this is most definitely a recursion issue. Visual Studio was unable to give an exact function name where the overflow occurred, unfortunately.

[novacrazy]

I'm letting rustc overflow (eventually, after eating all my RAM), and I notice this memory behavior.

Could this be a Vec reallocating?

[novacrazy]

Another note: All this memory is either empty or very, very redundant. Although it has allocated over 16GB now, Windows 10 memory compression is keeping it around 6GB actual RAM usage.

EDIT: 4.5GB actual usage after a periodic recompression by the OS. I'm kind of impressed.

[TimNN]

I'm trying to reproduce this, so a few questions:

• does the problem occur on the stable or beta channel? Or do you know of a nightly version in which the problem does not occur?

• in run you have -> ALResult<()> however run doesn't return anything, which should produce another error. Does the problem reproduce without the -> ALResult?

• How and where (another module, extern crate, ...) are GLUint, ALUint and ALResult defined? How are they imported into the current module?

[novacrazy]

After working around this for more than a few hours I can safely say this only occurs with types that are not in scope. Any and all other errors don't impact it. After I correct GLuint to ALuint, I did in fact get the missing return value error.

The OpenAL-Soft bindings look like this:

plus the build script.

With about 200 types, over 500 constants (including over a 100 EFX presets that are whole structures), and about 150 extern FFI function declarations, and so forth. Nothing particularly complex, though, just a lot of stuff. ALuint and the others are all defined there. By itself, all that works fine, and builds and links fine.

The wrapper library I'm writing looks like:

The only other dependencies it has is a custom fork of nalgebra that I've used before and works perfectly fine.

However, I just remembered I do something odd with the types and constants. I've separated them out into those multiple types.rs and consts.rs modules, and then in the lib.rs file I declare:

pub use types::*;

pub mod all {
    pub use super::*;
    pub use super::efx::*;
    pub use super::ext::*;

    pub use super::consts::*;
    pub use super::efx::consts::*;
    pub use super::ext::consts::*;
}

And then extern crate openalsoft_sys as als, then use als::all::* wherever I need the OpenAL FFI functions.

ALResult is defined in al_error.rs, and is just a relatively simple enum with OpenAL errors and things like IO Error, NulError, etc. It's a pattern I've used before with no issues.

I can't say for certain if older nightlies are affected, since I've only started on this project in the last day. I might be able to get it running on Beta soon... It doesn't use any nightly features specifically, I think.

Ten minutes later, I can confirm this issue exists in the beta channel.

Stable actually gives an interesting set of errors I need to investigate. It's saying in my all mod, that things have already been imported...

I'll see if I can get it compiling on stable soon. Family stuff in a few hours and still need sleep, but I'll try soon.

[novacrazy]

Ah, I think I see.

in mod all, I import efx and ext, both of which define a types mod. So it's importing two separate types modules into the same module scope. In fact, I'm surprised it compiled at all before. It seems the beta and nightly channels just merge the two modules together or ignore them entirely.

The only reason I could use the types defined in those modules, is because in each efx and ext mod.rs files, I had pub use self::types::*;

I'm renaming the files and modules and trying everything again.

[novacrazy]

Nope, that didn't fix it. Problem exists on stable, too. If anything, it's allocating memory leaks even faster than the nightly.

EDIT: Even rustdoc hanging when rustc succeeds still exists on stable.

I should note I've been testing this by intentionally switching some AL type with a non-existent GL type.

[novacrazy]

Repos are here:
openalsoft-sys
openalsoft-rs

Of course it's all still very WIP, so apologies in advance. I'm not sure if it even works correctly outside of MSVC yet.

[novacrazy]

I figured out how to create a minimal test case: https://github.com/novacrazy/rust-test-case-38585

[novacrazy]

Okay, nevermind the test case. I know exactly what the issue is.

pub mod myMod {
    pub use super::*;
}

Is recursive. myMod imports myMod, causing type/symbol resolution to choke.

EDIT: But it only causes issues from other crates, as far as I can tell so far.

So if crate A does the above, using use A::myMod::* in crate B will trigger the issue.

[TimNN]

Awesome work @novacrazy!

cc @jseyfried, this is your are of expertise, I think.

This is a (minor) regression from 1.7.0 to 1.8.0.

[novacrazy]

Some questions:

• How should Rust handle "recursive" modules like what happened here?
• Why did stable Rust (1.14) complain about the two types modules imported into the same namespace while the beta or nightly did not? And is that intentional? It is an ambiguity, after all.

Things to note just in case they overlooked it in my ramblings:

• This only seems to occur when the "recursive" module is defined in another crate.
• The infinite resolve recursion only occurs when the resolve logic is searching for types that do not exist in scope. Easily found types just return early, I presume, as they should.
• For the above issue, rustdoc recurses even when rustc does not, such as when all types are valid and exist.

[jseyfried]

@novacrazy Thanks for the report! This is #34324, fixed in #38539 (just merged).

How should Rust handle "recursive" modules like what happened here?

Recursive imports should "just work". Let me know if you have further questions about how we should handle them in the language or in the compiler.

Why did stable Rust (1.14) complain about the two types modules imported into the same namespace while the beta or nightly did not? And is that intentional?

Yes, this is due to the recent stabilization of RFC 1560 (specifically this rule and this rule).

rustdoc recurses even when rustc does not, such as when all types are valid and exist

Interesting, this might be is a separate issue in rustdoc. Would you mind opening another issue for problems with recursion in rustdoc, ideally using a nightly that includes #38539?