Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot process any item anymore #1642

Closed
mirisbowring opened this issue Jul 8, 2022 · 7 comments
Closed

Cannot process any item anymore #1642

mirisbowring opened this issue Jul 8, 2022 · 7 comments

Comments

@mirisbowring
Copy link

mirisbowring commented Jul 8, 2022 •
edited
Loading

Hi,
today i uploaded a bulk of files (around 23) they started to get into "stuck".
First i waited but they failed.
I checked the logs and got:

Fri, July 8th, 2022, 18:03: ============ Start processing zweite Idee OG_Perspektive__Perspektive.pdf ============
Fri, July 8th, 2022, 18:03: Checking for duplicate files
Fri, July 8th, 2022, 18:03: Job A83YyiH2r.../Collective-Name/process-item/High execution failed. Retrying later.: ERROR: column i.inreplyto does not exist Position: 141
Fri, July 8th, 2022, 18:04: ============ Start processing zweite Idee OG_Perspektive__Perspektive.pdf ============
Fri, July 8th, 2022, 18:04: Job A83YyiH2r.../Collective-Name/process-item/High execution failed. Retrying later.: ERROR: column i.inreplyto does not exist Position: 141

Even a Server restart did not help -> seems like the database is corrupt?

Here are the Logs from Docker-Compose:


docspell-joex                 | 2022.07.08 20:04:38:0006 [io-comp...] [INFO ] docspell.scheduler.impl.LogSink.logInternal:41 - >>> 2022-07-08T18:04:38.369172Z Info 7jh12AbtN.../Collective-Name/process-item/High: ============ Start processing 2022_01_rechnung_5623982349_20220105.pdf ============ (jobId: "7jh12AbtNkN-4dNG5LFGXd7-tVRZrDw8WsU-4xt6jQSxZ4f", task: "process-item", group: "Collective-Name", jobInfo: "7jh12AbtN.../Collective-Name/process-item/High")
docspell-joex                 | 2022.07.08 20:04:38:0006 [io-comp...] [WARN ] docspell.store.impl.DoobieMeta - SQL Failure: ExecFailure(SELECT DISTINCT i.itemid, i.cid, i.name, i.itemdate, i.source, i.incoming, i.state, i.corrorg, i.corrperson, i.concperson, i.concequipment, i.inreplyto, i.duedate, i.created, i.updated, i.notes, i.folder_id FROM item i INNER JOIN attachment a ON a.itemid = i.itemid INNER JOIN attachment_source s ON s.id = a.attachid LEFT JOIN attachment_archive r ON r.id = a.attachid WHERE ((a.filemetaid IN (? ) OR s.file_id IN (? ) OR r.file_id IN (? )) AND i.state IN (? , ? )),List(Collective-Name/attachmentsource/23G9EFjCy8deM36Ux26NKM8aiLCmhNnHxHj68H85wCXv, Collective-Name/attachmentsource/23G9EFjCy8deM36Ux26NKM8aiLCmhNnHxHj68H85wCXv, Collective-Name/attachmentsource/23G9EFjCy8deM36Ux26NKM8aiLCmhNnHxHj68H85wCXv, Premature, Processing),1028119 nanoseconds,org.postgresql.util.PSQLException: ERROR: column i.inreplyto does not exist
docspell-joex                 |   Position: 141)
docspell-joex                 | 2022.07.08 20:04:38:0001 [io-comp...] [ERROR] docspell.scheduler.impl.SchedulerImpl.wrapTask:290 - Job 7jh12AbtN.../Collective-Name/process-item/High execution failed. Retries exceeded.
docspell-joex                 | org.postgresql.util.PSQLException: ERROR: column i.inreplyto does not exist
docspell-joex                 |   Position: 141
docspell-joex                 | 	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
docspell-joex                 | 	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
docspell-joex                 | 	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
docspell-joex                 | 	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
docspell-joex                 | 	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
docspell-joex                 | 	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:167)
docspell-joex                 | 	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:119)
docspell-joex                 | 	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
docspell-joex                 | 	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
docspell-joex                 | 	at doobie.free.KleisliInterpreter$PreparedStatementInterpreter.$anonfun$executeQuery$2(kleisliinterpreter.scala:874)
docspell-joex                 | 	at doobie.free.KleisliInterpreter.$anonfun$primitive$2(kleisliinterpreter.scala:113)
docspell-joex                 | 	at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at uncancelable @ doobie.WeakAsync$$anon$1.uncancelable(WeakAsync.scala:36)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at suspend @ doobie.WeakAsync$$anon$1.suspend(WeakAsync.scala:34)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
docspell-joex                 | 
docspell-joex                 | 2022.07.08 20:04:38:0005 [io-comp...] [ERROR] docspell.scheduler.impl.LogSink.logInternal:49 - >>> 2022-07-08T18:04:38.376977Z Error 7jh12AbtN.../Collective-Name/process-item/High: Job 7jh12AbtN.../Collective-Name/process-item/High execution failed. Retries exceeded.
docspell-joex                 | org.postgresql.util.PSQLException: ERROR: column i.inreplyto does not exist
docspell-joex                 |   Position: 141
docspell-joex                 | 	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
docspell-joex                 | 	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
docspell-joex                 | 	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
docspell-joex                 | 	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
docspell-joex                 | 	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
docspell-joex                 | 	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:167)
docspell-joex                 | 	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:119)
docspell-joex                 | 	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
docspell-joex                 | 	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
docspell-joex                 | 	at doobie.free.KleisliInterpreter$PreparedStatementInterpreter.$anonfun$executeQuery$2(kleisliinterpreter.scala:874)
docspell-joex                 | 	at doobie.free.KleisliInterpreter.$anonfun$primitive$2(kleisliinterpreter.scala:113)
docspell-joex                 | 	at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at uncancelable @ doobie.WeakAsync$$anon$1.uncancelable(WeakAsync.scala:36)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at suspend @ doobie.WeakAsync$$anon$1.suspend(WeakAsync.scala:34)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-joex                 | 	at $anonfun$tailRecM$1 @ doobie.util.transactor$Transactor$$anon$4.$anonfun$apply$4(transactor.scala:166)
docspell-joex                 | 	at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
@mirisbowring
Copy link
Author

Strange... Now i cannot even view the files anymore:

image

docspell-db                   | 2022-07-08 18:12:06.728 UTC [36] ERROR:  column i.inreplyto does not exist at character 975
docspell-db                   | 2022-07-08 18:12:06.728 UTC [36] STATEMENT:  SELECT i.itemid, i.cid, i.name, i.itemdate, i.source, i.incoming, i.state, i.corrorg, i.corrperson, i.concperson, i.concequipment, i.inreplyto, i.duedate, i.created, i.updated, i.notes, i.folder_id, o.oid, o.cid, o.name, o.street, o.zip, o.city, o.country, o.notes, o.created, o.updated, o.short_name, o.org_use, pers0.pid, pers0.cid, pers0.name, pers0.street, pers0.zip, pers0.city, pers0.country, pers0.notes, pers0.created, pers0.updated, pers0.oid, pers0.person_use, pers1.pid, pers1.cid, pers1.name, pers1.street, pers1.zip, pers1.city, pers1.country, pers1.notes, pers1.created, pers1.updated, pers1.oid, pers1.person_use, e.eid, e.cid, e.name, e.created, e.updated, e.notes, e.equip_use, ref.itemid, ref.name, f.id, f.name FROM item i LEFT JOIN organization o ON o.oid = i.corrorg LEFT JOIN person pers0 ON pers0.pid = i.corrperson LEFT JOIN person pers1 ON pers1.pid = i.concperson LEFT JOIN equipment e ON e.eid = i.concequipment LEFT JOIN item ref ON ref.itemid = i.inreplyto LEFT JOIN folder f ON f.id = i.folder_id WHERE i.itemid = $1 
docspell-restserver           | 2022.07.08 20:12:06:0002 [io-comp...] [WARN ] docspell.store.impl.DoobieMeta - SQL Failure: ExecFailure(SELECT i.itemid, i.cid, i.name, i.itemdate, i.source, i.incoming, i.state, i.corrorg, i.corrperson, i.concperson, i.concequipment, i.inreplyto, i.duedate, i.created, i.updated, i.notes, i.folder_id, o.oid, o.cid, o.name, o.street, o.zip, o.city, o.country, o.notes, o.created, o.updated, o.short_name, o.org_use, pers0.pid, pers0.cid, pers0.name, pers0.street, pers0.zip, pers0.city, pers0.country, pers0.notes, pers0.created, pers0.updated, pers0.oid, pers0.person_use, pers1.pid, pers1.cid, pers1.name, pers1.street, pers1.zip, pers1.city, pers1.country, pers1.notes, pers1.created, pers1.updated, pers1.oid, pers1.person_use, e.eid, e.cid, e.name, e.created, e.updated, e.notes, e.equip_use, ref.itemid, ref.name, f.id, f.name FROM item i LEFT JOIN organization o ON o.oid = i.corrorg LEFT JOIN person pers0 ON pers0.pid = i.corrperson LEFT JOIN person pers1 ON pers1.pid = i.concperson LEFT JOIN equipment e ON e.eid = i.concequipment LEFT JOIN item ref ON ref.itemid = i.inreplyto LEFT JOIN folder f ON f.id = i.folder_id WHERE i.itemid = ? ,List(Ident(ES2NkcSPPqY-vKVz8aR7so5-Px8wWU3Cm5S-SL5zw1LxQiK)),1290637 nanoseconds,org.postgresql.util.PSQLException: ERROR: column i.inreplyto does not exist
docspell-restserver           |   Position: 975)
docspell-restserver           | 2022.07.08 20:12:06:0003 [io-comp...] [DEBUG] com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc.PgConnection@62c6ac08
docspell-restserver           | 2022.07.08 20:12:06:0004 [io-comp...] [INFO ] org.http4s.server.middleware.Logger - HTTP/1.1 GET /api/v1/sec/item/ES2NkcSPPqY-vKVz8aR7so5-Px8wWU3Cm5S-SL5zw1LxQiK
docspell-restserver           | 2022.07.08 20:12:06:0005 [io-comp...] [INFO ] org.http4s.server.middleware.Logger - service raised an error: class org.postgresql.util.PSQLException
docspell-restserver           | 2022.07.08 20:12:06:0006 [io-comp...] [ERROR] org.http4s.server.service-errors - Error servicing request: GET /api/v1/sec/item/ES2NkcSPPqY-vKVz8aR7so5-Px8wWU3Cm5S-SL5zw1LxQiK from 10.100.10.1
docspell-restserver           | org.postgresql.util.PSQLException: ERROR: column i.inreplyto does not exist
docspell-restserver           |   Position: 975
docspell-restserver           | 	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
docspell-restserver           | 	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
docspell-restserver           | 	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
docspell-restserver           | 	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
docspell-restserver           | 	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
docspell-restserver           | 	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:167)
docspell-restserver           | 	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:119)
docspell-restserver           | 	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
docspell-restserver           | 	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
docspell-restserver           | 	at doobie.free.KleisliInterpreter$PreparedStatementInterpreter.$anonfun$executeQuery$2(kleisliinterpreter.scala:874)
docspell-restserver           | 	at doobie.free.KleisliInterpreter.$anonfun$primitive$2(kleisliinterpreter.scala:113)
docspell-restserver           | 	at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
docspell-restserver           | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-restserver           | 	at $anonfun$apply$10 @ fs2.Stream$.$anonfun$resourceWeak$3(Stream.scala:3631)
docspell-restserver           | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-restserver           | 	at $anonfun$apply$10 @ fs2.Stream$.$anonfun$resourceWeak$3(Stream.scala:3631)
docspell-restserver           | 	at uncancelable @ doobie.WeakAsync$$anon$1.uncancelable(WeakAsync.scala:36)
docspell-restserver           | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-restserver           | 	at $anonfun$apply$10 @ fs2.Stream$.$anonfun$resourceWeak$3(Stream.scala:3631)
docspell-restserver           | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-restserver           | 	at $anonfun$apply$10 @ fs2.Stream$.$anonfun$resourceWeak$3(Stream.scala:3631)
docspell-restserver           | 	at suspend @ doobie.WeakAsync$$anon$1.suspend(WeakAsync.scala:34)
docspell-restserver           | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-restserver           | 	at $anonfun$apply$10 @ fs2.Stream$.$anonfun$resourceWeak$3(Stream.scala:3631)
docspell-restserver           | 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
docspell-restserver           | 	at $anonfun$apply$10 @ fs2.Stream$.$anonfun$resourceWeak$3(Stream.scala:3631)
docspell-restserver           | 	at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
docspell-restserver           | 
docspell-restserver           | 2022.07.08 20:12:06:0003 [io-comp...] [DEBUG] com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc.PgConnection@912aa7d
docspell-restserver           | 2022.07.08 20:12:06:0004 [io-comp...] [INFO ] org.http4s.server.middleware.Logger - HTTP/1.1 GET /api/v1/sec/person?full=true&q=&sort=name
docspell-restserver           | 2022.07.08 20:12:06:0005 [io-comp...] [INFO ] org.http4s.server.middleware.Logger - HTTP/1.1 200 OK
docspell-restserver           | 2022.07.08 20:12:06:0006 [io-comp...] [DEBUG] com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Reset (autoCommit) on connection org.postgresql.jdbc.PgConnection@27c43e34
docspell-restserver           | 2022.07.08 20:12:06:0007 [io-comp...] [INFO ] org.http4s.server.middleware.Logger - HTTP/1.1 GET /api/v1/sec/folder?q=&sort=name
docspell-restserver           | 2022.07.08 20:12:06:0008 [io-comp...] [INFO ] org.http4s.server.middleware.Logger - HTTP/1.1 200 OK

@eikek
Copy link
Owner

eikek commented Jul 8, 2022

Did you maybe try out a nightly version and then went back to a previous version?

@mirisbowring
Copy link
Author

yep, i used the nightly build due to the OIDC Topic. But everythin was working fine until i uploaded those files via web some hours ago.

After, I tried to switch to latest but this does not improve anything. :)

Did you make any changes to the DB?

@mirisbowring
Copy link
Author

Well, I switched back to nightly and i am able to view the PDFs (at least). But uploading is still not possbile with the Error from the Thread Heads. :)

@mirisbowring
Copy link
Author

docspell/modules/store/src/main/resources/db/migration/postgresql/V1.38.1__remove_unused_column.sql

Line 1 in fe8d2fc

alter table "item" drop column "inreplyto";

So, you removed this column.
I migrated to nightly - probably my joex (which is still on latest) must be upgraded to nightly too right? :D

@mirisbowring
Copy link
Author

HaHaaa! 😄

This was indeed the case!
Switching the restserver to nightly only was definetely a bad idea. Had to change both (restserver AND joex) to nightly to have everything work again.

Thank you for this great tool and have a nice day! 😄 👍

@eikek
Copy link
Owner

eikek commented Jul 9, 2022

Great you found it! That is correct: downgrading is not supported. Once you use a nightly you need to stay, upgrade to the next release or do a dbdump before and load that back in. Also true, that you always should put joex + rrestserver to the same version.

I'm using postgres and there I sometimes do a create database docspell_temp with template docspell to copy my "production" database and then do the nightly experiments there :)

@daviddavo daviddavo mentioned this issue May 27, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@eikek @mirisbowring