representable check fails if an enum references another enum that is cyclic

[nikomatsakis]

Here is a test case:

// Compiler fails here because it starts to check foo which references
// bar which is cyclic.
enum foo { foo(bar) }
enum bar { bar_none, bar_some(bar) }

// As before, but with order reversed, so that should we happen to
// start checking from end of file forward, we'd still have the same
// problem.
enum bar1 { bar1_none, bar1_some(bar1) }
enum foo1 { foo1(bar1) }

fn main() {}

I think the right approach is to replace ty_structurally_contains() with something like walk_interior_tys(). then we have to keeep a stack of def-ids and when we enter a new nominal type check that we do not already have it on the stack. i started this then realized it was a big distraction from what I'm trying to do, so filing a bug instead. Marking as A-typesystem as this check is performed in middle/typeck/check.rs

[pnkfelix]

Just to make this concrete: both of the cases that Niko describes yield ICE's. The first (the foo and bar items) yields an ICE with no useful error information. The second (bar1 and foo1) yields an error describing the illegal recursive enum type (followed by an ICE :).

[pcwalton]

I don't believe this is backwards incompatible, renominating.

[graydon]

accepted for production-ready milestone

[bblum]

appears to be properly classified

[pnkfelix]

Accepting for P-high.

[pnkfelix]

Tagging as mentorship bug, with @nikomatsakis as mentor

[isabelmu]

This code triggers the same infinite recursion in type_structurally_contains:

enum E { A, B(Bad) }
struct Bad { x: Bad }

[isabelmu]

I have a fix for this written, in case anyone is wondering whether to start working on it. I'll submit a pull request soon.