Pre-receive Hook: Wait for file - software duplex

[raumneun]

Hi there,
this is more a question than an issue - maybe a feature request. In any case, thank you for the great work with paperless-ng!
I was using a self-cooked minimalistic paperless web-app which used folders (because it was easier to implement) successfully for quite some time now but decided to switch to paperless-ng because it is... well.. better in just about every way :)
With one exception to my usual workflow:
Our scanner has an ADF, but unfortunately scans only single-sided. So in my web-app, I used a python script that combines two PDFs from the front (odd page numbers) and backs (even page numbers) by interleaving the pages.
Obviously, this would not work with paperless-ng out of the box as the consumer would consume the first pdf before the second arrived.

So I was wondering if it was possible to write a pre-consumption script that if a file appears in the consumption directory, that contains the word "front", it waits in an infinite loop (with a timeout of some sort) until the second file arrives, then combines these two and exits afterwards, such that the consumption carries on as usual.
To prevent the second file to be consumed from a different worker, the pre-consumption script should also check for the word "back" and ignore this file altogether.

Do you see any problem with that approach? Or even better: do you have a better idea to solve this elegantly? Obviously, I could do all outside of paperless-ng with another service, but I would very much like to keep it all in one place as this is much easier to backup and transfer to other systems then having yet another service running.

thanks!

Cheers
Max

[jonaswinkler]

Hello! Comments below.

Hi there,
this is more a question than an issue - maybe a feature request. In any case, thank you for the great work with paperless-ng!
I was using a self-cooked minimalistic paperless web-app which used folders (because it was easier to implement) successfully for quite some time now but decided to switch to paperless-ng because it is... well.. better in just about every way :)

Thank you :)

With one exception to my usual workflow:
Our scanner has an ADF, but unfortunately scans only single-sided. So in my web-app, I used a python script that combines two PDFs from the front (odd page numbers) and backs (even page numbers) by interleaving the pages.

I'd check with the software of the scanner. I could imagine that is has some pseudo duplex options (Scan, rotate the pages, scan again, and the sofware does the splining.)

Obviously, this would not work with paperless-ng out of the box as the consumer would consume the first pdf before the second arrived.

Correct.

So I was wondering if it was possible to write a pre-consumption script that if a file appears in the consumption directory, that contains the word "front", it waits in an infinite loop (with a timeout of some sort) until the second file arrives, then combines these two and exits afterwards, such that the consumption carries on as usual.
To prevent the second file to be consumed from a different worker, the pre-consumption script should also check for the word "back" and ignore this file altogether.

Do you see any problem with that approach?

The only obvious issue I see is that the pre and post consumption scripts cannot cancel the consumption. Paperless does not evaluate the exit code of these scripts and continues anyway. However, this is an easy fix. Apart from that, this might be possible, but won't be very elegant. You will also get lots of errors in the logs, because you'll have to cancel consumption for either the front or back documents regularly.

Or even better: do you have a better idea to solve this elegantly? Obviously, I could do all outside of paperless-ng with another service, but I would very much like to keep it all in one place as this is much easier to backup and transfer to other systems then having yet another service running.

• I'd probably search for scanner software that does this for you. If it's a network scanner, well, that's not an option.
• Doing this in paperless is pretty difficult. The general assumption for the document consumption pipeline is that one file from any source will result in exactly one document in paperless. Trying to implement any sort of file merging or splitting breaks that assumption, and would require me to make lots of fundamental changes.
• I was actually thinking about Split single PDF into multiple ones using a delimiter page #317 when I read this. My comment over there also describes some issues related to this. The use case in that issue is different, but I'd propose the same solution as over there. The idea is as follows:
  1. Make a script that watches a folder just as paperless does.
  2. That script may implement any kind of custom file merging/splitting logic you want.
  3. The script would put processed files into the consumption folder of paperless (or use the rest API), and simply copy files that don't need any merging / splitting.
  4. (Optional) Make this a reusable docker image that can be easily integrated into any docker compose setup (see the referenced issue).
• This is an additional service, but I think it might be a good solution nonetheless.

thanks!

Cheers
Max

[Philmo67]

Maybe try NAPS2 ?

[raumneun]

Thanks for your comments.
I was not aware that the consumption scripts do not cancel consumption after an exit code unequal to 0. Maybe this would be a good idea anyway?
I do agree that generating lots of errors in the log is not very elegant and also that this is should not be a core feature of paperless-ng as you state in the documentation that it expects files to work with regardless where they come from and should not handle preprocessing of some sort.

I am going for a completely headless design (well, at least for the scanning part), so using the scanner software is not really an option. Imho, this whole thing works only if digitizing the documents is as easy and fast as possible - otherwise I just don't do it ;)
I will go the route you described and create a python watchdog for another folder (say, consume-duplex) script that does the magic of combining the PDF and move it to the consumption directory. I'll try to build a docker image from that which could be integrated in the docker-compose setup for paperless-ng. If this is successful, I'll publish it at docker-hub so everyone interested may use it.

[jonaswinkler]

You might want to look at src/documents/management/commands/document_consumer.py for details on how to watch a folder for changes with inotify / watchdog.

• With inotify, you can listen to CLOSE_WRITE events, but that's not available on every file system.
• With watchdog, you get a notification the instant a file is created, and you'll have to wait until the file is complete by periodically checking its size / time modified.

[frzb]

I created a headless service for providing software based duplex scanning: https://github.com/frzb/softduplex

It was created to overcome the lack of duplex scanning by our document scanner similar to the scenario that @raumneun described here.

[torbengb]

Hallo Frzb!
Would it be possible to let it monitor only 1 input folder?

• Wait until there are at least 2 files present.
• Then take the oldest (odd) and the second-oldest (even) and spline them into an output folder (or even straight to the paperless consume folder.).
• Any remaining files are ignored, and will be picked up by the next iteration of the above!

This way, I could ADF-scan several stacks of duplex pages, with the understanding that every set of 2 files comprise the odd and even pages of one stack. Much simpler for the user!

Obviously, non-duplex scans should not be saved to that input folder in the first place; the user should send them straight to the paperless consume folder.

[frzb]

@torbengb I got your point.
I think your approach for simplifying the workflow makes a lot of sense.
As the current approach just works for me and my use case it will take some time until I implement you proposal.
But I am open for PRs and as we have a test suite: Just give it a try until the checks are green.

PS: In general the whole procedure should be filesystem event driven via inotify.

[henfri]

Hello,

on a similar Issue I have created the script below:
My scanner (Brother ADS1100) creates the filenames like this:

Receipt_004942.jpg
Receipt_004942_2.jpg
Receipt_004942_3.jpg
Receipt_004946.jpg
Receipt_004946_2.jpg
Receipt_004946_3.jpg
Receipt_004946_4.jpg
Receipt_004946_5.jpg
Receipt_004946_6.jpg

I suspect all Brother scanners (ADS*) do it that way. Thus, it could be good to link this script in the docs...
The script watches the scanner folder and combines Receipt_004946*.jpg into one pdf, Receipt_004942*.jpg into the other pdf etc.
Then it moves the original files away and stores the *.pdf int the consume folder of paperless.

Maybe it is of help for anyone.

from fpdf import FPDF
import glob, os

debug=False

def dprint(s):
  if debug: print(s)

# imagelist is the list with all image filenames

os.chdir(source)
for series in glob.glob("Receipt_??????.jpg"):
    dprint("----next series------")
    series_list=[]
    for file in glob.glob(series[0:-4]+"*.jpg"):
      series_list.append(file)
    series_list.sort()
    dprint(series_list)
    try:
        pdf=FPDF()
        for image in series_list:
            dprint("add page")
            pdf.add_page()
            pdf.image(image,0,0,210,297)
        outfile=os.path.join(consume,series_list[0][0:-4]+".pdf")
        pdf.output(outfile, "F")
        for image in series_list: #new loop so that the complete series is only moved if we do not run into the except
            os.rename(os.path.join(source,image),os.path.join(dest,image))
    except Exception as e:
        dprint("Problem with {}. \n {}".format(outfile,e))
    dprint("Successfully written {}".format(outfile))

[torwag]

I see the challenge here, that you do not know when the last page will be added. (Actually, I do not see a folder watching at all). So at which point and how do you start the script? Esp. if you work on network scanners, it might also appear that the file itself is not yet completed. The script needs to check if the file writing is finished before processing that file. Otherwise, you might observe corrupted pages, which sometimes appear if for whatever reason the file upload takes extra time. Reference: I figured that out by pure pain debugging ;)

BTW: My brother ADS-2800w can be set-up to scan directly into a PDF

[henfri]

Hello,

thanks for your feedback. Indeed, there is that risk in my approach. I run the script every 10 minutes - so the risk is very low.
Thanks for the hint scanning directly to PDF. In fact, my ADS-1100w is able to do that as well... facepalm. I had forgotten that I intentionally set it up to create jpg only, as my previous "DMS" (OneNote) does not OCR PDFs but only jpg, IIRC.
I thought that the scanner was not able to create PDFs when scanning to network.

Thanks, I will try that.