HWP conversion fails on MacOS M1

[apyrgio]

HWP conversion was recently added in #460 and was tested across various platforms (Windows, MacOS, Linux) through our CI. However, when running our tests on a MacOS M1 platform (which is not available through any of our CI runners), we get the following error:

[DEBUG] Marking doc UWYs15 as 'converting'
[INFO ] > /usr/local/bin/docker run --network none -u dangerzone --security-opt=no-new-privileges:true --cap-drop all --rm -v /var/folders/7c/49675rcx2nb0lwjms_6wmdpr0000gs/T/tmp6rzrqzt1/unsafe/input_file:/tmp/input_file:Z -v /var/folders
/7c/49675rcx2nb0lwjms_6wmdpr0000gs/T/tmp6rzrqzt1/pixels:/tmp/dangerzone:Z -e ENABLE_TIMEOUTS=1 dangerzone.rocks/dangerzone /usr/bin/python3 -m dangerzone.conversion.doc_to_pixels
[INFO ] [doc UWYs15] 0% UNTRUSTED> Installing LibreOffice extension 'h2orestart.oxt'
[INFO ] [doc UWYs15] 0% UNTRUSTED> Converting to PDF using LibreOffice                                 
[INFO ] [doc UWYs15] 3% UNTRUSTED> Calculating number of pages
[ERROR] [doc UWYs15] 3% UNTRUSTED> PDF file is corrupted

(taken from the dangerzone-cli)

If one tries to convert the file to PDF via Libreoffice:

libreoffice --headless --infilter="Hwp2002_Reader" --convert-to pdf:writer_pdf_Export <file>

they will get the following generic error:

Error: source file could not be loaded

Note that the exact same operation in an Intel MacOS device works.

[deeplow]

Maybe we should add to this release a note in the changelog that this is not yet support on arm macs.

And this is yet another case where the error message is confusing because it looks like pdfinfo is the one failing (saying that the PDF was corrupted), where in reality it wasn't even produced since LibreOffice failed.

[OctopusET]

I will take a look at this issue. I can borrow the m1 macbook for testing soon.

[apyrgio]

One assumption is that H2Orestart, the plugin we rely on for HWP conversions, somehow fails on ARM platforms. This is further reinforced by the fact that of the 3 HWP-related test files that we have under tests/test_docs_exernal:

sample-hwp.hwp.b64
sample-hwp97.hwp.b64
sample-hwpx.hwpx.b64

only two of those fail (sample-hwp.hwp.b64 and sample-hwpx.hwpx.b64). The sample-hwp97.hwp.b64 file passes, probably because LibreOffice already had support for it.

There's one issue with this assumption though: H2Orestart is a Java plugin, meaning that it should work across architectures, bar some exceptions. These exceptions usually boil down to usage of JNI (see this SO answer), but that's something we can detect. If we do jar tf h2orestart.jar, we get back a list of .class files, which should be platform agnostic.

@OctopusET thanks a lot for taking a look at it. Tagging @ebandal as well, in case they are interested in this issue as well.

[deeplow]

Following some feedback, we have two options here:
a) not ship the hwp feature until it's fully working across the board (I guess we can ignore Qubes for now)
b) Inform users that choose hwp or hwpx file on arm Macs that it's not supported (via an alert, for example)
c) fix the bug, of course.

I'd rather go for option b) if c) is not possible. It should be relatively trivial to implement.

[OctopusET]

I can use the M1 macbook after August 8 for dangerzone development. Before that I can only try some conversion in M1 macbook with h2orestart.

[OctopusET]

I just started work on M1 macbook yesterday.
Seems like just conversion with H2Orestart is working.

Question: I just drop the commit that is "HWP/HWPX disable on MacOS Apple Silicon", is there any way to re-enable the HWP/HWPX conversion feature on MacOS Appli Silicon?

[OctopusET]

And with Dangerzone GUI, there's an issue with selecting the HWP/HWPX files. You can't select it. So, I tried with only CLI.

[deeplow]

Question: I just drop the commit that is "HWP/HWPX disable on MacOS Apple Silicon", is there any way to re-enable the HWP/HWPX conversion feature on MacOS Appli Silicon?

Thanks for looking into this. We can do this but only for the next release since we just shipped this one. However, @apyrgio did experience this issue, so it may still happen particular files.

And with Dangerzone GUI, there's an issue with selecting the HWP/HWPX files. You can't select it. So, I tried with only CLI.

We decided to remove the ability to add these files in platforms where they were considered not compatible. Not a perfect solution, but it leads to avoiding getting errors later.

[OctopusET]

@deeplow, Ah, I mean, is there any way to re-enable that feature other than dropping commits.

[deeplow]

I think it's no big deal. We can always do another commit that does the reverse.

[OctopusET]

With my short analyses, it fails when it's checking converted file is exist or not.
I think it's almost the same issue as the previous one. #460 (comment).
Libreoffice might work differently on MacOS.

I think it's not H2ORestart's issue. I tested with several files including the our test files.

[OctopusET]

I can't use my borrowed M1 Macbook everyday. It might take while for actual testing.

[OctopusET]

It's strange because the conversion worked for me, but the issue said it didn't even work.

So I restart my Macbook.
I found very strange behavior.

I tried the HWP->PDF, HWPX->PDF conversion on cold booted macbook.
It failed like you mentioned at first. But, after you open libreoffice GUI, it's working.

And if you kill libreoffice, like with killall soffice.
It fails again.

[OctopusET]

If you try with --safe-mode option, there are some additional error logs.

$ soffice --safe-mode --headless --infilter="Hwp2002_Reader" --convert-to pdf:writer_pdf_Export some.hwp
libc++abi: terminating due to uncaught exception of type com::sun::star::deployment::DeploymentException
Unspecified Application Error

[apyrgio]

@ebandal did a nice dig and probed the Alpine Linux devs: https://gitlab.alpinelinux.org/alpine/aports/-/issues/15212. Turns out that the LibreOffice package for aarch64 does not support Java. Hopefully this will be resolved upstream. Once it does, we should remove our GUI restrictions for this platform.

[OctopusET]

https://gitlab.alpinelinux.org/alpine/aports/-/commit/74d443f479df15fc57e6fde6ac02a36b24afdded

They enabled the java support for the aarch64

[OctopusET]

I think rebuilding image would fix it. I will test it soon.

[OctopusET]

I had to change the docker image to alpine:edge (20230901).
But it works!

[apyrgio]

Nice, thanks for staying on top of this. I'll take a look at your PR and associated issue and we'll fix this.

[apyrgio]

@OctopusET: Quick update now that we are about to cut the 0.5.0 release. We wanted to switch back to alpine:latest (see #542) before the release, provided that the required LibreOffice version is in the repos. This does not seem to be the case, so we had two options at hand: either release on alpine:edge, or stay with alpine:latest and not include the latest fixes.

Ultimately, we decided that we don't want to risk any last minute breaking changes from the edge repo. This means that we will revert your fixes for now, and wait until November, for the new release of Alpine Linux, to properly include them.

It's a bit of a tough decision, but we believe it's best to prioritize the stability of the software over new features. Thank you for your persistence on this feature, and rest assured that the next release will include it 🙂.

[OctopusET]

I agree with you about stability concerns.
And eventually the package will be updated, so no need to rush for this.

Thank you for letting me know about this

[OctopusET]

@apyrgio https://www.alpinelinux.org/posts/Alpine-3.19.0-released.html

[deeplow]

Thank you very much pinging us about this and submitting a PR. We are just about to push out a security release, which should indeed include alpine 3.19. However, since it's a security-focused release and we had already hit feature-freeze, we will have to include your PR in the next one.