Failure while calling fetch-ebook-metadata process

[hende101]

Hi all.

I'm running Jelu in Docker and getting this error when trying to fetch a book from an online source.
Docker is running on a Truenas Scale machine (x86).

It will hang until I restart the container and produces this in the log:

2025-01-22T21:57:06.957Z  INFO 1 --- [main] i.github.bayang.jelu.JeluApplicationKt   : Starting JeluApplicationKt v0.65.1 using Java 17.0.13 with PID 1 (/app/BOOT-INF/classes started by root in /app)
2025-01-22T21:57:06.958Z DEBUG 1 --- [main] i.github.bayang.jelu.JeluApplicationKt   : Running with Spring Boot v3.3.3, Spring v6.1.12
2025-01-22T21:57:06.959Z  INFO 1 --- [main] i.github.bayang.jelu.JeluApplicationKt   : No active profile set, falling back to 1 default profile: "default"
2025-01-22T21:57:07.812Z  INFO 1 --- [main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JDBC repositories in DEFAULT mode.
2025-01-22T21:57:07.849Z  INFO 1 --- [main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 33 ms. Found 0 JDBC repository interfaces.
2025-01-22T21:57:08.304Z  INFO 1 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 11111 (http)
2025-01-22T21:57:08.313Z  INFO 1 --- [main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2025-01-22T21:57:08.314Z  INFO 1 --- [main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.28]
2025-01-22T21:57:08.352Z  INFO 1 --- [main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2025-01-22T21:57:08.353Z  INFO 1 --- [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1358 ms
2025-01-22T21:57:08.563Z  INFO 1 --- [main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2025-01-22T21:57:08.634Z  INFO 1 --- [main] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.sqlite.jdbc4.JDBC4Connection@1fa9692b
2025-01-22T21:57:08.635Z  INFO 1 --- [main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2025-01-22T21:57:08.989Z  INFO 1 --- [main] liquibase.changelog                      : Reading from DATABASECHANGELOG
2025-01-22T21:57:09.042Z  INFO 1 --- [main] liquibase.ui                             : Database is up to date, no changesets to execute
2025-01-22T21:57:09.042Z  INFO 1 --- [main] liquibase.changelog                      : Reading from DATABASECHANGELOG
2025-01-22T21:57:09.051Z  INFO 1 --- [main] liquibase.util                           : UPDATE SUMMARY
2025-01-22T21:57:09.051Z  INFO 1 --- [main] liquibase.util                           : Run:                          0
2025-01-22T21:57:09.051Z  INFO 1 --- [main] liquibase.util                           : Previously run:              25
2025-01-22T21:57:09.051Z  INFO 1 --- [main] liquibase.util                           : Filtered out:                 0
2025-01-22T21:57:09.051Z  INFO 1 --- [main] liquibase.util                           : -------------------------------
2025-01-22T21:57:09.051Z  INFO 1 --- [main] liquibase.util                           : Total change sets:           25
2025-01-22T21:57:09.052Z  INFO 1 --- [main] liquibase.util                           : Update summary generated
2025-01-22T21:57:09.062Z  INFO 1 --- [main] liquibase.lockservice                    : Successfully released change log lock
2025-01-22T21:57:09.063Z  INFO 1 --- [main] liquibase.command                        : Command execution complete
2025-01-22T21:57:09.681Z  INFO 1 --- [main] r$InitializeUserDetailsManagerConfigurer : Global AuthenticationManager configured with UserDetailsService bean with name userService
2025-01-22T21:57:09.715Z  INFO 1 --- [main] o.s.b.a.w.s.WelcomePageHandlerMapping    : Adding welcome page: class path resource [public/index.html]
2025-01-22T21:57:10.118Z  INFO 1 --- [main] o.s.l.c.support.AbstractContextSource    : Property 'userDn' not set - anonymous context will be used for read-only operations
2025-01-22T21:57:10.167Z  INFO 1 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 11111 (http) with context path '/'
2025-01-22T21:57:10.183Z  INFO 1 --- [main] i.g.b.jelu.service.AppLifecycleAware     : Lucene index not found, trigger rebuild
2025-01-22T21:57:10.189Z  INFO 1 --- [main] i.g.b.jelu.service.SearchIndexService    : Rebuild index for: [book, author]
2025-01-22T21:57:10.190Z  INFO 1 --- [main] i.g.b.jelu.service.SearchIndexService    : Rebuilding index for Book
2025-01-22T21:57:10.279Z  INFO 1 --- [main] i.g.b.jelu.service.SearchIndexService    : Number of entities: 119 and pages : 1
2025-01-22T21:57:10.363Z  INFO 1 --- [main] i.g.b.jelu.service.SearchIndexService    : Processing page 1 of 1 (5000 elements)
2025-01-22T21:57:10.364Z  INFO 1 --- [http-nio-11111-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2025-01-22T21:57:10.364Z  INFO 1 --- [http-nio-11111-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2025-01-22T21:57:10.365Z  INFO 1 --- [http-nio-11111-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
2025-01-22T21:57:10.383Z  WARN 1 --- [http-nio-11111-exec-1] o.s.web.servlet.PageNotFound             : No mapping for GET /add-book
2025-01-22T21:57:10.682Z  INFO 1 --- [main] i.g.b.jelu.service.SearchIndexService    : Wrote Book index in 321.028428ms
2025-01-22T21:57:10.967Z DEBUG 1 --- [main] i.g.b.jelu.service.SearchIndexService    : no authors index yet
2025-01-22T21:57:11.039Z  INFO 1 --- [main] i.g.bayang.jelu.search.LuceneHelper      : Lucene index version: 3
2025-01-22T21:57:11.097Z  INFO 1 --- [main] i.github.bayang.jelu.JeluApplicationKt   : Started JeluApplicationKt in 4.443 seconds (process running for 4.783)
2025-01-22T21:59:20.800Z ERROR 1 --- [http-nio-11111-exec-4] i.g.b.j.s.m.p.CalibreMetadataProvider    : failure while calling fetch-ebook-metadata process

java.lang.InterruptedException: null

[hende101]

Manually running a metadata search from within the container results in the following output:

# /calibre/fetch-ebook-metadata -o -d 90 -t "Counter-clock world"
Failed to import PyQt module: PyQt6.QtWebEngineCore with error: libXcomposite.so.1: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 47, in <module>
  File "site.py", line 43, in main
  File "calibre/utils/ipc/worker.py", line 196, in main
  File "<string>", line 1, in <module>
  File "calibre/scraper/qt.py", line 292, in webengine_worker
  File "bypy-importer.py", line 279, in exec_module
  File "calibre/scraper/webengine_backend.py", line 16, in <module>
ImportError: cannot import name 'QWebEnginePage' from 'qt.webengine' (/calibre/lib/calibre-extensions/python-lib.bypy.frozen/qt/webengine.pyc)
Traceback (most recent call last):
  File "calibre/scraper/qt.py", line 234, in _dispatch
  File "json/__init__.py", line 346, in loads
  File "json/decoder.py", line 337, in decode
  File "json/decoder.py", line 355, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
<?xml version='1.0' encoding='utf-8'?>
<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="uuid_id" version="2.0">
    <metadata xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:opf="http://www.idpf.org/2007/opf">
        <dc:identifier opf:scheme="calibre" id="calibre_id">047220e3-b6c4-4222-8d15-cb9f6fba3610</dc:identifier>
        <dc:identifier opf:scheme="uuid" id="uuid_id">12e4e151-f55e-485f-b293-22638b47ba44</dc:identifier>
        <dc:title>Counter-Clock World</dc:title>
        <dc:creator opf:file-as="Unknown" opf:role="aut">Philip K. Dick</dc:creator>
        <dc:contributor opf:file-as="calibre" opf:role="bkp">calibre (7.24.0) [https://calibre-ebook.com]</dc:contributor>
        <dc:date>1967-02-01T08:00:00+00:00</dc:date>
        <dc:description>&lt;p&gt;Sebastian Hermes wishes his future was history ... In the Counter-Clock World time runs backwards. People are born in their graves, dug up when they can start speaking, and are reintroduced into society where they grow younger before returning to the womb to be unborn. The Hobart Phase has been reversing time since 1986 and nothing has quite been the same since. The library is the most powerful and most feared organization in the world, responsible for the eradication of records from history for events which have no longer happened. Anarch Peak, a black religious leader, is due to be old-born and the library wants him dead. With a feud already developing in relgious communities throughout the world, Seb Hermes is sent to retrieve Anarch from his captors. He must succeed before tensions trigger an all-out religous war.&lt;/p&gt;</dc:description>
        <dc:publisher>Vintage</dc:publisher>
        <dc:identifier opf:scheme="GOODREADS">156723</dc:identifier>
        <dc:identifier opf:scheme="AMAZON">0375719334</dc:identifier>
        <dc:identifier opf:scheme="ISBN">9780007127702</dc:identifier>
        <dc:identifier opf:scheme="GOOGLE">HMDTwMjeTa4C</dc:identifier>
        <dc:language>eng</dc:language>
        <dc:subject>Science Fiction</dc:subject>
        <dc:subject>Fantasy</dc:subject>
        <dc:subject>Religion</dc:subject>
        <meta name="calibre:rating" content="4"/>
    </metadata>
    <guide/>
</package>

It does seem to eventually fetch it after failing to import some python modules?

[bayang]

the fetch ebook metadata binary comes from calibre and is written in python.
Looks like it first complains that a shared library is missing and then eventually grabs the metadata online to then again complain about a formatting issue.
I'll see what I can find.

You can use the inventaire.io provider in the meantime.

[bayang]

can you check if your problem could be similar to the one explained here : #67

IE : could you check if libXcomposite.so.1 has ABI tags that could interfere ?

See here for more infos : https://forum.qt.io/topic/136058/libqt6core-so-6-cannot-open-shared-object-file-even-though-it-exists-seems-to-depend-on-docker-host-os/6

[ebiagi]

I have the same problem starting from the 0.55.1 version and trying the solution #67 is not working.
I've also tried to enable the inventaire.io fallback but it throws the following error

ERROR: The Compose file './compose.yaml' is invalid because:
Unsupported config option for services.jelu: 'metadataProviders'

here's my docker compose

 jelu:
     image: wabayang/jelu:latest
     container_name: jelu
     restart: always
     env_file:
         - /home/docker-run/dockerenv/jelu.env
     stdin_open: true
     tty: true
     ports:
         - 127.0.0.1:11111:11111
     volumes:
         - /srv/dockervolumes/jelu/config:/config
         - /srv/dockervolumes/jelu/database:/database
         - /srv/dockervolumes/jelu/files/images:/files/images
         - /srv/dockervolumes/jelu/files/imports:/files/imports
         - /etc/timezone:/etc/timezone:ro
     metadataProviders:
         - is-enabled: true
           apiKey: ""
           order: -100000
           name: "jelu-debug"
         - is-enabled: false
           apiKey: "YOUR_API_KEY_HERE"
           order: -10
           name: "google"
         - name: "inventaireio"
           is-enabled: true
           order: 200000
           config: "it"

[bayang]

@ebiagi as the error message says : your compose file is invalid.
The metadataProviders block you put in it is jelu config, not compose.
You need to either put it in your jelu config file or either convert it to env vars and put it in the env section of your compose file.

Also what was your initial issue ?
Is it the libXcomposite issue ?
Are you running on a NAS as well ?

[ebiagi]

Oh my, it's clearly explained in the first lines of the config page https://bayang.github.io/jelu-web/configuration/ , sorry, my bad!
Yes I have the same libXcomposite issue and I'm not running it on a NAS but on a Debian 12 VPS

[ebiagi]

I don't know, I'm not getting the libXcomposite error anymore

root@328310e0eb66:/app# /calibre/fetch-ebook-metadata -o -d 90 -t "Counter-clock world"
<?xml version='1.0' encoding='utf-8'?>
<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="uuid_id" version="2.0">
    <metadata xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:opf="http://www.idpf.org/2007/opf">
        <dc:identifier opf:scheme="calibre" id="calibre_id">d0a13c4c-8598-47e5-8957-8c933b2a4765</dc:identifier>
        <dc:identifier opf:scheme="uuid" id="uuid_id">6945edd1-849d-4fba-b10e-ef46b440138c</dc:identifier>
        <dc:title>Counter-Clock World</dc:title>
        <dc:creator opf:file-as="Unknown" opf:role="aut">Philip K. Dick</dc:creator>
        <dc:contributor opf:file-as="calibre" opf:role="bkp">calibre (7.24.0) [https://calibre-ebook.com]</dc:contributor>
        <dc:date>1967-02-01T08:00:00+00:00</dc:date>
        <dc:description>&lt;p&gt;Sebastian Hermes wishes his future was history ... In the Counter-Clock World time runs backwards. People are born in their graves, dug up when they can start speaking, and are reintroduced into society where they grow younger before returning to the womb to be unborn. The Hobart Phase has been reversing time since 1986 and nothing has quite been the same since. The library is the most powerful and most feared organization in the world, responsible for the eradication of records from history for events which have no longer happened. Anarch Peak, a black religious leader, is due to be old-born and the library wants him dead. With a feud already developing in relgious communities throughout the world, Seb Hermes is sent to retrieve Anarch from his captors. He must succeed before tensions trigger an all-out religous war.&lt;/p&gt;</dc:description>
        <dc:publisher>Vintage</dc:publisher>
        <dc:identifier opf:scheme="GOODREADS">156723</dc:identifier>
        <dc:identifier opf:scheme="AMAZON">0375719334</dc:identifier>
        <dc:identifier opf:scheme="ISBN">9780007127702</dc:identifier>
        <dc:identifier opf:scheme="GOOGLE">HMDTwMjeTa4C</dc:identifier>
        <dc:language>eng</dc:language>
        <dc:subject>Science Fiction</dc:subject>
        <dc:subject>Fantasy</dc:subject>
        <dc:subject>Religion</dc:subject>
        <meta name="calibre:rating" content="4"/>
    </metadata>
    <guide/>
</package>

but the web search is not working, keeps searching with no results

The following lines appears in the logs some minutes after I launched the search and then closed the search window

2025-01-28T22:48:32.868+01:00 DEBUG 1 --- [io-11111-exec-1] i.g.b.jelu.service.metadata.OpfParser    : parsed dto MetadataDto(title=Counter-Clock World, isbn10=null, isbn13=9780007127702, summary=<p>Sebastian Hermes wishes his future was history ... In the Counter-Clock World time runs backwards. People are born in their graves, dug up when they can start speaking, and are reintroduced into society where they grow younger before returning to the womb to be unborn. The Hobart Phase has been reversing time since 1986 and nothing has quite been the same since. The library is the most powerful and most feared organization in the world, responsible for the eradication of records from history for events which have no longer happened. Anarch Peak, a black religious leader, is due to be old-born and the library wants him dead. With a feud already developing in relgious communities throughout the world, Seb Hermes is sent to retrieve Anarch from his captors. He must succeed before tensions trigger an all-out religous war.</p>, image=null, publisher=Vintage, pageCount=null, publishedDate=1967-02-01T08:00:00+00:00, authors=[Philip K. Dick], tags=[Science Fiction, Fantasy, Religion], series=null, numberInSeries=null, language=eng, googleId=HMDTwMjeTa4C, amazonId=0375719334, goodreadsId=156723, librarythingId=null, isfdbId=null, openlibraryId=null, inventaireId=null, noosfereId=null)