Infer method calls on known types and warn about nonexistent methods #11399

radeusgd · 2024-10-24T15:15:09Z

Pull Request Description

Implements Inferring types of method calls on Atoms #9812
Allows us to propagate type inference through method calls, at least in a basic way.
Adds a 'lint warning': when calling a method on an object of a known specific (non-Any) type that does not exist on that type, a warning is reported, because such a call would always result in No_Such_Method error at runtime.
- Any has special behaviour - if x : Any we don't know what x might be, so we allow all methods on it and defer to the runtime to see if they will be valid or not.
- This check is not proving correctness, but instead it's only triggering for 'provably wrong' situations.
Includes changes from Simplify BindingsMap by instead relying on StaticModuleScope for atom type resolution #11955:
- removing obsolete AtomTypeInterface,
- simplifying TypeRepresentation to be a plain data structure,
- and removing a cycle from IR in BindingsMap in favour of relying on StaticModuleScope.

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

… hypothesis

…tion no longer in outer layer

…ce tests Ideally, the static analysis tests should be runnable without the Graal runtime context. We'd like to be able to run these static passes without Graal runtime, e.g. inside of code editors. But refactoring import resolution and PackageRepository to work without Graal is out of scope of the current types work. It can be planned for later, when needed. For now the focus is to make type checking provide actual value - before that any editor integration wouldn't be useful anyway.

radeusgd · 2024-12-31T13:42:22Z

I think I tried addressing all of the comments. I will very much appreciate your reviews on this next iteration of the PR. Sorry it is happening after such a long time - I will try to schedule time to iterate on this PR faster this time.

@Akirathan

@Akirathan suggested to remove his `validateConsistency` check, but I've found I could fix the asserts by caching the returned value in persisted Map (if there was no caching, then on every ir.passData().get(...) a new instance of metadata was recreated, so the `validateConsistency` was failing because it requires object _identity_ and not only equality). We can either keep this 'cache' or remove Pavel's check.

This reverts commit 9d5deaa.

# Conflicts: # engine/runtime/src/main/scala/org/enso/interpreter/runtime/IrToTruffle.scala

radeusgd · 2025-01-02T14:41:55Z

I'm getting weird test failures in ModuleManagementTest, removing the assertion as @Akirathan suggested has changed the error but it is still failing at deserialization in a weird way:

[info] - should allow importing literal-source modules *** FAILED *** (1 second, 195 milliseconds)
[info]   org.graalvm.polyglot.PolyglotException: java.lang.ArrayIndexOutOfBoundsException: Index 158052 out of bounds for length 1
[info]   at org.enso.persistance/org.enso.persist.PerInputImpl$InputCache.getRef(PerInputImpl.java:351)
[info]   at org.enso.persistance/org.enso.persist.PerInputImpl.findReference(PerInputImpl.java:58)
[info]   at org.enso.persistance/org.enso.persist.PerReferencePeristance.readObject(PerReferencePersistance.java:33)
[info]   at org.enso.persistance/org.enso.persist.PerReferencePeristance.readObject(PerReferencePersistance.java:8)
[info]   at org.enso.persistance/org.enso.persist.Persistance.readWith(Persistance.java:162)
[info]   at org.enso.persistance/org.enso.persist.PerInputImpl.readInline(PerInputImpl.java:67)
[info]   at org.enso.runtime/org.enso.interpreter.caches.PersistBindingsMap_Argument.readObject(PersistBindingsMap_Argument.java:14)
[info]   at org.enso.runtime/org.enso.interpreter.caches.PersistBindingsMap_Argument.readObject(PersistBindingsMap_Argument.java:4)
[info]   at org.enso.persistance/org.enso.persist.Persistance.readWith(Persistance.java:162)
[info]   at org.enso.persistance/org.enso.persist.PerInputImpl.readIndirect(PerInputImpl.java:218)
[info]   ...

Interestingly, it seems that my followup PR #11955 that removes the BindingsMap -> Expression cycle from the IR seems to not have these failures.

@JaroslavTulach do you think we can merge both PRs at once (I need to finish the followup but I'm working on that) OR would you like to help me investigating this serialization issue? Seems like there is still some issue in the persistance when cycles are around.

IMO it will be easier to just merge the followup, although we can also try investigating the persistance to make it more robust as such failures probably shouldn't be happening - but I don't think fixing persistance should be the focus of this PR in more extent than necesary.

…s (no parameter type mismatch checks yet, instead testing method mismatch)

radeusgd · 2025-01-02T19:07:47Z

I wanted to keep these PRs separate but it initially (I'm waiting for full CI suite to pass to verify) seems that the followup PR #11955 is fixing the persistance issues that are causing test failures in this PR. So I think ideally I'd merge #11955 into this PR and then merge the whole changeset as a single change. I'm not merging yet so that you can review more easily - first my answers to your requests in this PR and then the followup PR that builds on top of it.

JaroslavTulach

I believe the abstraction of algorithms into singletonizers and their usage is correct. I don't feel completely qualified to talk about the static analysis, but I left few comments inline. If the CI is green, then I am for integrating these changes.

engine/common/src/main/java/org/enso/common/CompilationStage.java

...e/runtime-compiler/src/main/java/org/enso/compiler/common/BuildScopeFromModuleAlgorithm.java

engine/runtime-compiler/src/main/java/org/enso/compiler/pass/analyse/types/BuiltinTypes.java

...me-compiler/src/main/java/org/enso/compiler/pass/analyse/types/TypeInferencePropagation.java

engine/runtime-compiler/src/main/java/org/enso/compiler/pass/analyse/types/BuiltinTypes.java

JaroslavTulach · 2025-01-07T07:10:03Z

engine/runtime-compiler/src/main/scala/org/enso/compiler/Compiler.scala

    * @param useGlobalCacheLocations whether or not the compilation result should
    *                                  be written to the global cache
    * @return future to track subsequent serialization of the library
    */
  def compile(
    shouldCompileDependencies: Boolean,
+    shouldWriteCache: Boolean,


I assume this is a reasonable way to avoid the static type information to be stored in the caches.

Yes, plus IIRC it fixes a weird behaviour - that even if caches were disabled, they would still be written after compilation or something.

Now, if we run --no-ir-cache, we neither read nor write the caches, which I think is correct.

The reason why we are seeing failures like https://github.com/enso-org/enso/actions/runs/12680614808/job/35342759884?pr=12023#step:7:2875 is probably because of this nor write behavior. Previously there was one failing run and the bad caches would get overwritten. Now the bad caches stick there forever.

That's rational, but looks like the

enso/engine/runtime-benchmarks/src/main/java/org/enso/compiler/benchmarks/module/ImportStandardLibrariesBenchmark.java

Line 129 in cb72487

throw new AssertionError("Unexpected output (errors?) from the compiler: " + out.toString());

may not be ready for that!

Oh...

Can't we just run the engine with --compile to write the new caches, before running this bench?

engine/runtime-compiler/src/main/scala/org/enso/compiler/context/InlineContext.scala

JaroslavTulach · 2025-01-07T07:18:19Z

engine/runtime-integration-tests/src/test/java/org/enso/compiler/test/StaticAnalysisTest.java

+   *
+   * <p>Ideally, the tests for the static analysis capabilities of the compiler should _not_ depend
+   * on the Graal runtime context, as they should be runnable in other contexts - i.e. in a Visual
+   * Studio Code language server.


That would indeed be ideal! The integration of the runtime-compiler into VSCode is blocked right now because we have no way to run the runtime-compiler without TruffleCompilerContext right now. If only these tests could be moved to runtime-compiler project test directory! Then:

they wouldn't depend on TruffleCompilerContext

they would need an independent compiler context implementation

such an implementation could be used in VSCode as well

I remember @Akirathan was once dreaming about using runtime-compiler in VSCode and this was a huge "known unknown".

Yes I think ideally we should move them there - it seems like the right direction.

But implementing the independent compiler context is definitely a task for a separate PR.

engine/runtime-integration-tests/src/test/java/org/enso/compiler/test/StaticAnalysisTest.java

…atom type resolution (#11955)

radeusgd · 2025-01-07T18:15:44Z

In 6b0ce25 I merged the followup PR so that we can merge them both to develop in one try.

I will proceed with merging once CI gets green.

radeusgd · 2025-01-08T15:40:07Z

I've ran the benchmarks and tried comparing against develop:
benchs.zip

It's really quite hard to tell in each benchmark if any deviations are just due to benchmark instability or real regression (how are you guys managing to live without seeing the standard deviation error bars on the measurements? 😮).

Here are a few 'more concerning' results I saw:

Still they don't seem bad, so I think I will proceed with merging.

radeusgd added 30 commits August 13, 2024 19:47

rename pass

2715738

WIP: find types from checked signatures

c1d4cc1

rename CompilerTest -> ParserTest

b284ec5

refactor: move some common methods to separate class to allow for reuse

7968622

add a test for the type extraction

f21b783

fix test

a1bf4f7

WIP tests

28626fa

tests for methods

9b5a87d

add a test verifying return-type check on member methods - to test my…

f626367

… hypothesis

add a test that finds the cause of issue: FORCE node makes the ascrip…

8aeb690

…tion no longer in outer layer

fix resolution of return types in type signatures

5c61beb

adding tests for member method calls

f5ca4af

find the right place to resolve calls

435cd0e

javafmt

218efcd

update tests

7594894

fix test

9a40731

WIP

215a91f

fix compilation (stub)

f2b3dd7

better message in test

273687d

WIP: StaticModuleScope

ea15ebc

WIP: StaticModuleScope 2

cafc96d

WIP

730e3b9

checkpoint

2aa0377

wiring the StaticModuleScopeAnalysis pass with rest of the system

e39f248

formatting

da82e47

debugging

52ba2dd

WIP: workaround for static methods on module?

5152227

debugging

b7d0fef

debug

94d7323

radeusgd added 5 commits January 2, 2025 13:36

avoid NoSuchElementException in assert

4f0eb2a

Revert "try fixing failing asserts validateConsistency;"

81f9c51

This reverts commit 9d5deaa.

remove assert as requested by @Akirathan

4b08ada

Merge branch 'develop' into wip/radeusgd/9812-infer-method-calls

5d09c01

# Conflicts: # engine/runtime/src/main/scala/org/enso/interpreter/runtime/IrToTruffle.scala

add example test from @Akirathan but adapted to this PR's capabilitie…

c30f7f0

…s (no parameter type mismatch checks yet, instead testing method mismatch)

JaroslavTulach approved these changes Jan 7, 2025

View reviewed changes

radeusgd added 9 commits January 7, 2025 11:56

Merge branch 'develop' into wip/radeusgd/9812-infer-method-calls

a1feacc

CR: naming, mark some classes final etc.

836a083

fix after merge - change how to get instance of TruffleFileSystem

1229611

CR: simplify how fallback scope is cached, make class static

0a3df8b

CR: rename materialized to resolved

66e93f7

CR: use ScalaConversions

97fa043

CR: remove wrong FQN

a348219

Merge branch 'develop' into wip/radeusgd/9812-infer-method-calls

a8b1505

Simplify BindingsMap by instead relying on StaticModuleScope for …

6b0ce25

…atom type resolution (#11955)

GregoryTravis approved these changes Jan 7, 2025

View reviewed changes

radeusgd added 2 commits January 8, 2025 10:08

Merge branch 'develop' into wip/radeusgd/9812-infer-method-calls

9717358

CR change var to interface type

22e6649

radeusgd added the CI: Ready to merge This PR is eligible for automatic merge label Jan 8, 2025

mergify bot merged commit 787372e into develop Jan 8, 2025
51 checks passed

mergify bot deleted the wip/radeusgd/9812-infer-method-calls branch January 8, 2025 15:41

radeusgd added a commit that referenced this pull request Jan 8, 2025

Fix a bad merge between #11926 and #11399

dcb6477

mergify bot pushed a commit that referenced this pull request Jan 8, 2025

Fix a bad merge between #11926 and #11399 (#12022)

bfd31f9

radeusgd mentioned this pull request Jan 8, 2025

Bump persistance version id #12023

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Infer method calls on known types and warn about nonexistent methods #11399

Infer method calls on known types and warn about nonexistent methods #11399

radeusgd commented Oct 24, 2024 •

edited

Loading

radeusgd commented Dec 31, 2024

radeusgd commented Jan 2, 2025

radeusgd commented Jan 2, 2025

JaroslavTulach left a comment

JaroslavTulach Jan 7, 2025

radeusgd Jan 7, 2025

JaroslavTulach Jan 9, 2025

radeusgd Jan 9, 2025

JaroslavTulach Jan 7, 2025

radeusgd Jan 7, 2025

radeusgd commented Jan 7, 2025

radeusgd commented Jan 8, 2025

Infer method calls on known types and warn about nonexistent methods #11399

Infer method calls on known types and warn about nonexistent methods #11399

Conversation

radeusgd commented Oct 24, 2024 • edited Loading

Pull Request Description

Important Notes

Checklist

radeusgd commented Dec 31, 2024

radeusgd commented Jan 2, 2025

radeusgd commented Jan 2, 2025

JaroslavTulach left a comment

Choose a reason for hiding this comment

JaroslavTulach Jan 7, 2025

Choose a reason for hiding this comment

radeusgd Jan 7, 2025

Choose a reason for hiding this comment

JaroslavTulach Jan 9, 2025

Choose a reason for hiding this comment

radeusgd Jan 9, 2025

Choose a reason for hiding this comment

JaroslavTulach Jan 7, 2025

Choose a reason for hiding this comment

radeusgd Jan 7, 2025

Choose a reason for hiding this comment

radeusgd commented Jan 7, 2025

radeusgd commented Jan 8, 2025

radeusgd commented Oct 24, 2024 •

edited

Loading