Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update expressions run in embedded python server error #5426

Closed
niloc132 opened this issue Apr 27, 2024 · 5 comments · Fixed by #5436
Closed

Update expressions run in embedded python server error #5426

niloc132 opened this issue Apr 27, 2024 · 5 comments · Fixed by #5436
Assignees
@nbauernfeind
Labels
bug Something isn't working core Core development tasks query engine
Milestone
2. April 2024

Comments

@niloc132
Copy link
Member

This appears to be a regression in 0.34.

Code to run in python, with deephaven-core and deephaven-embedded installed:

from deephaven_server import Server
s = Server().start()

from deephaven import time_table

table = time_table("PT1S").update(formulas=["X=i"])

The last line will fail with an error:
java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200

Full error message:

Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 109, in shared_lock
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 256, in auto_locking_ctx
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 764, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: io.deephaven.UncheckedDeephavenException: Compilation failed
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:172)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:141)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:73)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1518)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1500)
        at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3639)
        at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1499)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1477)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:100)
        at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
caused by java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:825)
        at io.deephaven.engine.context.QueryCompilerImpl.compileHelper(QueryCompilerImpl.java:563)
        at io.deephaven.engine.context.QueryCompilerImpl.compile(QueryCompilerImpl.java:276)
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:156)
        ... 10 more
caused by java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:134)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:171)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:976)
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:818)
        ... 13 more


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 763, in update
    with _query_scope_ctx(), auto_locking_ctx(self):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 255, in auto_locking_ctx
    with shared_lock(arg.update_graph):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 111, in shared_lock
    raise DHError(e, "exception raised in the enclosed code block.") from e
deephaven.dherror.DHError: exception raised in the enclosed code block. : java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 109, in shared_lock
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 256, in auto_locking_ctx
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 764, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: io.deephaven.UncheckedDeephavenException: Compilation failed
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:172)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:141)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:73)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1518)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1500)
        at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3639)
        at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1499)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1477)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:100)
        at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
caused by java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:825)
        at io.deephaven.engine.context.QueryCompilerImpl.compileHelper(QueryCompilerImpl.java:563)
        at io.deephaven.engine.context.QueryCompilerImpl.compile(QueryCompilerImpl.java:276)
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:156)
        ... 10 more
caused by java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:134)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:171)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:976)
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:818)
        ... 13 more



The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 766, in update
    raise DHError(e, "table update operation failed.") from e
deephaven.dherror.DHError: table update operation failed. : java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 109, in shared_lock
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 256, in auto_locking_ctx
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 764, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: io.deephaven.UncheckedDeephavenException: Compilation failed
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:172)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:141)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:73)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1518)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1500)
        at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3639)
        at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1499)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1477)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:100)
        at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
caused by java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:825)
        at io.deephaven.engine.context.QueryCompilerImpl.compileHelper(QueryCompilerImpl.java:563)
        at io.deephaven.engine.context.QueryCompilerImpl.compile(QueryCompilerImpl.java:276)
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:156)
        ... 10 more
caused by java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:134)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:171)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:976)
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:818)
        ... 13 more


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 763, in update
    with _query_scope_ctx(), auto_locking_ctx(self):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 255, in auto_locking_ctx
    with shared_lock(arg.update_graph):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 111, in shared_lock
    raise DHError(e, "exception raised in the enclosed code block.") from e
deephaven.dherror.DHError: exception raised in the enclosed code block. : java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 109, in shared_lock
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 256, in auto_locking_ctx
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 764, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: io.deephaven.UncheckedDeephavenException: Compilation failed
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:172)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:141)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:73)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1518)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1500)
        at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3639)
        at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1499)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1477)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:100)
        at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
caused by java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:825)
        at io.deephaven.engine.context.QueryCompilerImpl.compileHelper(QueryCompilerImpl.java:563)
        at io.deephaven.engine.context.QueryCompilerImpl.compile(QueryCompilerImpl.java:276)
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:156)
        ... 10 more
caused by java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:134)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:171)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:976)
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:818)
        ... 13 more
@niloc132 niloc132 added bug Something isn't working triage labels Apr 27, 2024
@niloc132
Copy link
Member Author

This regression appears to be caused by #5070 - the commit prior to that change does not have this issue, but that commit causes this.

@rcaudy rcaudy added query engine core Core development tasks and removed triage labels Apr 27, 2024
@rcaudy rcaudy added this to the 2. April 2024 milestone Apr 27, 2024
@devinrsmith
Copy link
Member

Our main community image does not exhibit this issue, appears to be specific to the embedded python server. For reference, running the same query formula in the community image produces something like:

$ find /cache -type f
/cache/.JsPluginsZipFilesystem13610993811594593474/deephaven-js-plugins.zip
/cache/script-session-classes/f4fcf678-7167-4e3e-b8af-5c4f30495c39/io/deephaven/temp/c_7f3e63ff2190de96d978de3d723129623aaa22226001a02b65dec404ca886cd6v65_0/Formula$FormulaFillContext.class
/cache/script-session-classes/f4fcf678-7167-4e3e-b8af-5c4f30495c39/io/deephaven/temp/c_7f3e63ff2190de96d978de3d723129623aaa22226001a02b65dec404ca886cd6v65_0/Formula.class

@jmao-denver
Copy link
Contributor

jmao-denver commented Apr 27, 2024 •
edited
Loading

Here is a somewhat similar/related case and I talked to both @nbauernfeind and @rcaudy about it.

deephaven-core/py/server/tests/test_barrage.py

Line 23 in c71506e

# the cacheDir env var is required to avoid a problem when the same cache dir is used by two server instances,

@chipkent chipkent added the devrel-watch DevRel team is watching label Apr 29, 2024
@devinrsmith
Copy link
Member

Here is a somewhat similar/related case and I talked to both @nbauernfeind and @rcaudy about it.

deephaven-core/py/server/tests/test_barrage.py

Line 23 in c71506e

# the cacheDir env var is required to avoid a problem when the same cache dir is used by two server instances,

When you are running two instances of DH on the same physical (or virtual) instance, it's probably better practice to set the application name:

https://deephaven.io/core/docs/how-to-guides/configuration/native-application/#application-name

which will ensure that the data, config, and cache dirs are unique.

@rcaudy rcaudy mentioned this issue May 2, 2024
Merged
@rcaudy
Copy link
Member

rcaudy commented May 2, 2024

Closed by #5436

@rcaudy rcaudy closed this as completed May 2, 2024
@chipkent chipkent removed the devrel-watch DevRel team is watching label May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nbauernfeind nbauernfeind

Labels
bug Something isn't working core Core development tasks query engine
Projects
None yet
Milestone
2. April 2024
Development

Successfully merging a pull request may close this issue.

Move Script Cache Clearing to Dagger nbauernfeind/deephaven-core
6 participants
@niloc132 @nbauernfeind @jmao-denver @chipkent @rcaudy @devinrsmith