Method "List#orElse(PartialFunction[A, B])" has malformed generic signature

[scabug]

What steps will reproduce the problem (please be specific and use > wikiformatting)?

Calling .getGenericReturnType or .getGenericParameterTypes on that specific method object fails with a jl.reflect.GenericSignatureFormatError

It is possible that the Scala compiler produces invalid classfiles for other cases as well, but I only had problems with this method for now.

  classOf[List[_]].getMethods.find(_.getName == "orElse").get.getGenericReturnType

What is the expected behavior?
a return value of type jl.reflect.Type

What do you see instead?
a java.lang.reflect.GenericSignatureFormatError

What versions of the following are you using?
Scala:
I was using Scala 2.8.1

Java:
$$ java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.2) (6b20-1.9.2-0ubuntu1)
OpenJDK Client VM (build 19.0-b09, mixed mode, sharing)

Operating system:
Ubuntu Linux 10.10

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-4067?orig=1
Reporter: AlexK (alexk)
Attachments:

• badsigs.txt (created on Dec 8, 2010 3:27:11 AM UTC, 34947 bytes)

[scabug]

@paulp said:
In the interests of a more valid generic future, I checked in a -Yverify-generics option. I attached file badsigs.txt with the output from current trunk.

[scabug]

@nilskp said:
This bug means that you cannot use Scala libraries in Java projects in Eclipse.
This is a huge problem for interoperability and is hopefully treated as a very high priority item and I would expect to be reason enough, if nothing else, for a 2.8.2 release.

Example of Eclipse failure:

package foo;

import scala.runtime.AbstractFunction1;

public class MyFunction1 extends AbstractFunction1<Double, Void> {
    public Void apply(Double d) {
        System.out.println(d);
        return null;
    }
}

This leads to Eclipse vomiting this:

Inconsistent classfile encountered: The undefined type parameter A is referenced from within Function1<T1,R>.$$anonfun$$compose$$1

[scabug]

@paulp said:
As of r24059 I am suppressing the invalid signatures. However we should consider this a very short term measure. If we don't get this fixed quite soon I'm going to have to at least start issuing a warning for each suppressed signature (as is done if you give -Yverify-generics right now.) This should certainly be a blocker for 2.9.

[scabug]

@odersky said:
I could use some help with this one. Fact if the matter is, I can't clone myself. I can either spend my time on working on problems like these, or on getting an Eclipse IDE that works. Right now it's the Eclipse IDE.

[scabug]

@paulp said:
Replying to [comment:11 odersky]:

I could use some help with this one. Fact if the matter is, I can't clone myself. I can either spend my time on working on problems like these, or on getting an Eclipse IDE that works. Right now it's the Eclipse IDE.

I know that cloning issue. Actually I fixed this all up today before I saw your comment. Three commits to appear soon.

[scabug]

@paulp said:
(In r24063) Taking a ride from 106 invalid generic signatures to zero.
Closes #4067.

Numbers 1-72:
Problem: Primitive types appearing in generic bounds.
Response: Use the boxed types.

Numbers 73-105:
Problem: Doubly nested classes can become Outer.Inner.Inner2
Response: Don't allow dots except immediately following angle brackets.

Number 106:
Problem: Array may be being used as a higher kinded type,
and in such cases should not receive the special Array treatment.
Solution: args.nonEmpty

Observe my claim is that 106/106 signatures have been made
"valid", not "correct" or even "more nearly correct." As to that
I would like a second opinion. And.... review by moors!

[scabug]

@odersky said:
Fantastic! That resolves a huge pile of problems.

[scabug]

Marko Mikulicic (ithkuil) said:
For the inpatients, I've created a jar post-processor which cleans up the signatures of the 2.8.1 scala-library.jar (and possibly others) in such a way that eclipse doesn't throw the exception during completion.

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:15 ithkuil]:

For the inpatients, I've created a jar post-processor which cleans up the signatures of the 2.8.1 scala-library.jar (and possibly others) in such a way that eclipse doesn't throw the exception during completion.

forgot the link: http://www.coldcode.net/2011/01/scala-generics-and-eclipse.html

[scabug]

@nilskp said:
Any chance of seeing this released anytime soon? I would hate to wait for 2.9 whenever that may arrive.

[scabug]

bryan hunt (sentimental) said:
Replying to [comment:17 nilskp]:

Any chance of seeing this released anytime soon?
I would hate to wait for 2.9 whenever that may arrive.

I second that.

A year down the line, I'm starting to ask myself if I should have taken the Scala road at all.

Sometimes this whole ecosystem seems too flaky to believe.

The proffered solution is either:

A) Upgrade everything to nightly

OR

B) Download and install a tool from someone I've never heard of, which updates itself from god only knows where and does something that I don't understand to a load of my production binary files.
Last time that happened it was a present from the Cult of the Dead Cow.

Both of the above are extremely lame. Extremely lame for a project that's trying to be taken seriously. And that writes a ton of books about how great it is.

Here's how I read the situation:

2 months ago we released a STABLE release of our much vaunted, non-binary compatible between releases project.

Unfortunately, (especially as one of our main selling points is binary compatibility with Java), our project is not binary compatible with Java as it generates invalid class files.

We've fixed it in Version Control and will bundle it with our next release, which will occur at some indeterminate time in the future.

In the meantime you can all just sit on your hands because you can't integrate your Scala libraries with your Java systems.

Oh, alternatively you could download all your Scala dependencies and recompile them to the (unreleased) 2.9.0.

Except they break.

Right now we're too busy writing research papers, inventing new words, writing books about how great the system is or doing consultancy to bother with this, the actual business of writing software.

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:19 sentimental]:

Replying to [comment:17 nilskp]:

Any chance of seeing this released anytime soon?
I would hate to wait for 2.9 whenever that may arrive.

I second that.

A year down the line, I'm starting to ask myself if I should have taken the Scala road at all.

Sometimes this whole ecosystem seems too flaky to believe.

The proffered solution is either:

A) Upgrade everything to nightly

OR

B) Download and install a tool from someone I've never heard of, which updates itself from god only knows where and does something that I don't understand to a load of my production binary files.
Last time that happened it was a present from the Cult of the Dead Cow.

Both of the above are extremely lame. Extremely lame for a project that's trying to be taken seriously. And that writes a ton of books about how great it is.

Here's how I read the situation:

2 months ago we released a STABLE release of our much vaunted, non-binary compatible between releases project.

Unfortunately, (especially as one of our main selling points is binary compatibility with Java), our project is not binary compatible with Java as it generates invalid class files.

We've fixed it in Version Control and will bundle it with our next release, which will occur at some indeterminate time in the future.

In the meantime you can all just sit on your hands because you can't integrate your Scala libraries with your Java systems.

I'm very exited with scala and I'm eager to use it more. I understand your emotions, but still I don't feel this is a blocker issue.

How does this issue really affect integration with java?

I only noticed a problem in eclipse auto completion, the eclipse java compiler is happy otherwise, javac too; are there other issues?

Instead of moving to the head build of the scala compiler, please consider using my jar postprocessing tool at:
http://www.coldcode.net/2011/01/scala-generics-and-eclipse.html

[scabug]

bryan hunt (sentimental) said:
Replying to [comment:20 ithkuil]:

Replying to [comment:19 sentimental]:

Replying to [comment:17 nilskp]:

Any chance of seeing this released anytime soon?
I would hate to wait for 2.9 whenever that may arrive.
I only noticed a problem in eclipse auto completion, the eclipse java compiler is happy otherwise, javac too; are there other issues?

Instead of moving to the head build of the scala compiler, please consider using my jar postprocessing tool at:
http://www.coldcode.net/2011/01/scala-generics-and-eclipse.html

Jar postprocessing tool.

Firstly I did download it, I don't know who you are so I ran it under an zero privilege user account. It didn't work. I forget the exact error message.

I'm glad it didn't work. Because if it had worked I would have just patched our production code-base with unknown modifications provided by someone I've never heard of via the Internet.

Lets be honest, if I had an employee and they did that, I'd fire their ass.

If the tool is legitimate why does the Scala project not run it upon the libraries which they release?

When I download from Ubuntu, Redhat, Microsoft, Oracle or any other provider the product works. I don't need to download a binary modifier from an unknown third party to get them to work.

I don't even know why I'm entertaining the idea. It's so far off of what's considered responsible or normal project behaviour. I know I'd be laughed out of the office if I told them I'd done it.

So seriously, when will I be able to compile Scala files and use them in Eclipse without sending my workspace red?

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:21 sentimental]:

Replying to [comment:20 ithkuil]:

Replying to [comment:19 sentimental]:

Replying to [comment:17 nilskp]:

Any chance of seeing this released anytime soon?
I would hate to wait for 2.9 whenever that may arrive.
I only noticed a problem in eclipse auto completion, the eclipse java compiler is happy otherwise, javac too; are there other issues?

Instead of moving to the head build of the scala compiler, please consider using my jar postprocessing tool at:
http://www.coldcode.net/2011/01/scala-generics-and-eclipse.html

Jar postprocessing tool.

Firstly I did download it, I don't know who you are so I ran it under an zero privilege user account. It didn't work. I forget the exact error message.

Thank you for reporting. github has an issue tracker, I would be glad to hear about any issues. I use the patched scala-library.jar jar myself in a product I use at work. Did you try it on some other jar besides the scala-library?

If you are unable to run the tool, you can find the patched scala-library jar at:

https://github.com/downloads/mmikulicic/scala-fix-generics/scala-library.jar

I don't expect you to trust it. The question is, does it solve your problem?

The issues my postprocessor fixes are primitive types declared as type parameter bounds in generic signatures. There might be a bug in my code, and some corner cases were not considered. But it could also be that your problem is different and you spotted another scenario which would be worth reporting.

I'm glad it didn't work. Because if it had worked I would have just patched our production code-base with unknown modifications provided by someone I've never heard of via the Internet.

Lets be honest, if I had an employee and they did that, I'd fire their ass.

I just share my work. I don't expect people blindly updating their production env with that code. I would happy to find a solution to a problem I have, if anything to just know that a fix is possible, then I would double check it. In this case, it's about 304 lines of code; I'm sure that if you really have interest in using scala for real work you can find time to contribute.

If the tool is legitimate why does the Scala project not run it upon the libraries which they release?

1. because it's a hack
2. because it's a new hack
3. because it's a new unreviewed hack

I'd be interested to know if there are enough interested people who run into this problem
and would like to see it solved in a pragmatic way until the new scala release is there.

In the meantime there is a fix in eclipse which is available in the current milestone and in april it should be released. I don't know if it fixes all of your problems, but it does fix the code completion exception. (which was the only issue I had; see my last question below)

When I download from Ubuntu, Redhat, Microsoft, Oracle or any other provider the product works. I don't need to download a binary modifier from an unknown third party to get them to work.

really? I recently downloaded a libjna deb patch from "unknown" sources at some PPA in order to run cassandra on ubuntu 10.04. I tested it, reviewed it and now I'm using it on devel machines. I happens a lot of time when you are using bleeding edge software.

Why use bleeding edge software? This is another issue.

So seriously, when will I be able to compile Scala files and use them in Eclipse without sending my workspace red?

So you actually have some java code which breaks eclipse other than it's code completion?
I'd be interesting to see some examples. Would you be so kind to send a stripped down example? I would be extremely useful for me to know in advance about potential issues I may have in future.

[scabug]

bryan hunt (sentimental) said:
Marko,

I don't want to hurt your feelings here or detract from the work that you've done.

Stability appears time and time again to be lacking in this system.

At this stage I don't expect Scala to be 'bleeding edge'. By version 2.8.1 I could accept some rough edges but I'd expect them to be capable of generating valid Java class files.

While you seem to be deeply involved in this issue, albeit peripherally as you are not directly associated with the project - I would rather that the core team rolled out those fixes as part of the standard distribution.

really? I recently downloaded a libjna deb patch from >"unknown" sources at some PPA in order to run cassandra on >ubuntu 10.04. I tested it, reviewed it and now I'm using it >on devel machines. I happens a lot of time when you are >using bleeding edge software.

Good for you.

Bryan

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:23 sentimental]:

Marko,

I don't want to hurt your feelings here or detract from the work that you've done.

Stability appears time and time again to be lacking in this system.

At this stage I don't expect Scala to be 'bleeding edge'. By version 2.8.1 I could accept some rough edges but I'd expect them to be capable of generating valid Java class files.

Bryan,

I don't want to put down your complaints on stability of Scala; it's a rather widespread feeling.

Despites all that I still choose to invest in it, since I find that the advantages might outweight the costs of fighting with rough edges.

Hence, I would really grateful if you find some time to provide more details on the kind of issue you are having. You mentioned eclipse projects issuing build errors. Can you please provide some sample code that breaks the compiling of mixed scala and java?

[scabug]

bryan hunt (sentimental) said:
@serializable
case class Topic(@BeanProperty var pk: Int) {
@BeanProperty var tag: String = ""
@BeanProperty var count: Int = -1

def this(_pk: Int, _tag: String, _count: Int) = {
this (_pk)
this.tag = _tag
this.count = _count
}

def this(_pk: Int, _tag: String) = {
this(_pk)
this.tag = _tag
this.count = -1
}

override def toString() = {
"pk:[" + pk + "] tag:[" + tag + "] count:[" + count + "]"
}
}
case class User(val id: Int, val ts: List[Topic])
trait ICallback {
def respond(u: User, ts: List[Topic]);
}

As far as I can recall, this code (as part of a jar file) was (as per the Eclipse.org bug report) breaking code completion across my Eclipse workspace.

Further, I seem to remember, specifically it was the class, User that was causing the problem.

Like I said, I'm operating on 2.9.0 now, my project is already a month behind schedule but this seemed to be to ridiculous to let pass without comment.

To any Scala team members who may be listening - You guys need to watch out for things like this if you want to be taken seriously a year from now.

[scabug]

bryan hunt (sentimental) said:
Sorry code formatting went a bit ugly there.

[scabug]

@nilskp said:
Try implementing Function1 or extending AbstractFunction1. Good luck.

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:27 nilskp]:

Try implementing Function1 or extending AbstractFunction1. Good luck.

Not sure I understood you comment. Do you mean that there are issues implementing Function1 or extending AbstractFunction1 in java? Any issues would be related to this particular generic signature bug or other causes?

[scabug]

bryan hunt (sentimental) said:
Function1... you are kidding, right?

It's counted at least 76 methods to implement...

Not quite as easy as a callback interface...

[scabug]

@nilskp said:
Try implementing any generic trait in a pure Java project, or extending any generic class. I suggested AbstractFunction1.

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:25 sentimental]:

case class User(val id: Int, val ts: List[Topic])
I tried your code and as expected it broke the eclipse build:

!MESSAGE Error in JDT Core during reconcile
!STACK 1
Java Model Exception: java.lang.ClassCastException
        at org.eclipse.jdt.internal.core.CompilationUnitProblemFinder.process_aroundBody0(CompilationUnitProblemFinder.java:233)
        at org.eclipse.jdt.internal.core.CompilationUnitProblemFinder.process_aroundBody1$$advice(CompilationUnitProblemFinder.java:246)
        at org.eclipse.jdt.internal.core.CompilationUnitProblemFinder.process_aroundBody2(CompilationUnitProblemFinder.java:1)
        at org.eclipse.jdt.internal.core.CompilationUnitProblemFinder$$AjcClosure3.run(CompilationUnitProblemFinder.java:1)

It seems that scala List causes the problem. I tried to change to Seq[Topic] and it worked fine.

I drilled down and I found a minimal test case which exhibits the same problem:

public class Test {
	public void simpleTest() {
		scala.collection.immutable.List l = null;
	}
}

I built the code with javac without problems; eclipse breaks though.

I tried my patched scala library file and eclipse manages to build this test code.
I deduce that this problem was indeed caused by the generic declarations and not another thing, despites the radically different eclipse stack trace.

Here is the git repo which reproduces the problem. You can see that:

1. javac compiles the code with an unpatched scala-library.jar
2. eclipse builds without errors when using the patched scala library

https://github.com/mmikulicic/scala-test-generics

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:31 nilskp]:

Try implementing any generic trait in a pure Java project, or extending any generic class. I suggested AbstractFunction1.

Thanks. Eclipse spits:

Multiple markers at this line
	- Inconsistent classfile encountered: The undefined type parameter A is referenced from within Function1<T1,R>.$$anonfun
	 $$compose$$1
	- Inconsistent classfile encountered: The undefined type parameter A is referenced from within Function1<T1,R>.$$anonfun
	 $$compose$$1

Ant (javac) compiles though.

I tried with the patched jar and the problem is still there, so I guess some other signature declaration issue.

(https://github.com/mmikulicic/scala-test-generics, branch "function1")

---

Here's the dump of the generic signatures for class Function1.$$anonfun$$compose$$1

visiting class scala/Function1$$$$anonfun$$compose$$1, signature null
method <clinit> signature: null 
method apply signature: (TA;)TR; 
method <init> signature: (Lscala/Function1<TT1;TR;>;)V 

I tried to encode the similar idea in plain java:

public abstract class Test<T1,R> {
    public abstract R apply(T1 v);
    
    public <A> Test<A, R> compose(final Test<A, T1> g) {
        final Test<T1, R> self = this;
        return new Test<A, R>() {
            public R apply(A x) {
                return self.apply(g.apply(x));
            }
        };
    }
}

visiting class Test$$1, signature LTest<TA;TR;>;
method <init> signature: null 
method apply signature: (TA;)TR; 

visiting class Test, signature <T1:Ljava/lang/Object;R:Ljava/lang/Object;>Ljava/lang/Object;
method <init> signature: null 
method apply signature: (TT1;)TR; 
method compose signature: <A:Ljava/lang/Object;>(LTest<TA;TT1;>;)LTest<TA;TR;>

Where Test is the equivalent of AbstractFunction1 and Test$$1 is the equivalent of scala/Function1$$$$anonfun$$compose$$1.

So I suspect that the scala compiler is not emitting the generic declaration for the anonymous class
representing the anonymous function expanded from (I omitted the @specialized anno for readability):

trait Function1[-T1, +R] extends AnyRef { self =>
  def apply(v1:T1): R
  override def toString() = "<function1>"

  /** (f compose g)(x) ==  f(g(x))
   */
  def compose[A](g: A => T1): A => R = { x => apply(g(x)) }

Does this always happen? or only when there is some recursive type definition like this one?
(Actually the (A => R) return type should mean Function1[A, R])

Perhaps it's better to handle this in another ticket, since it seems that it's not caused by a malformed signature but a missing signature.

[scabug]

bryan hunt (sentimental) said:
Ithkuil,

It's been fixed since Changeset 24063.

It would be cool is if they would backport and release a version 2.8.2 with these fixes.

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:34 sentimental]:

Ithkuil,

It's been fixed since Changeset 24063.

great news

It would be cool is if they would backport and release a version 2.8.2 with these fixes.

Yes, I think I wouldn't feel comfortable moving to unreleased 2.9.0 yet in order to benefit from this. I also don't think it would be easy to monkey patch the signatures in this case, except if it happens only in a couple of well known places.

[scabug]

bryan hunt (sentimental) said:
That's exactly my point.

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:36 sentimental]:

That's exactly my point.

Yes sorry I was confused because we have two distinct issues in this thread, one for the missing decls of Function1, and the other of the incorrect type bounds of orElse.

[scabug]

bryan hunt (sentimental) said:
Remote communication is always difficult.

It is often the case whereby one finds oneself taking at cross-purposes.

So are we now in agreement about the problem and it's solution?

[scabug]

Marko Mikulicic (ithkuil) said:
Replying to [comment:38 sentimental]:

Remote communication is always difficult.

It is often the case whereby one finds oneself taking at cross-purposes.

So are we now in agreement about the problem and it's solution?

I was always in agreement on the solution; I was just proposing a quick hack to solve one of the two problems, but a hack is not a solution :). But if I were affected by the second issue, I wouldn't mind to have a hack also for it, if there nobody from the Scala maintainers can invest effort in performing a backport and maintenance release. Just a pragmatic approach :)

Perhaps we should open a distinct ticket requesting for backporting this fixes to the 2.8.x series ?

[scabug]

bryan hunt (sentimental) said:
Yes, I think that is a very good idea.

[scabug]

bryan hunt (sentimental) said:
Request to create a new release with the fix committed in Changeset 24063

[https://lampsvn.epfl.ch/trac/scala/ticket/4259]