Content-Transfter-Encoding not specified in headers

[keithveeva]

Hi Pillip,

Our project uses your as2-lib core library to handle our AS2 communication and we cannot seem to be able to send the Content-Transfter-Encoding header (CTE) in our message. We set it explicitly in our message and it looks to be picked up by your code, but digging into the underlying sdk call it looks like this header is on a list of restricted headers, can you confirm if this is the case or how can we enable this in your library? Looking at the sdk code this is controlled by a system property, ideally we would like to NOT turn this system property on, please let me know if there are any other options.

Thanks,
Keith

[phax]

Hi Keith,
I'm aware of this situation, as I myself was struggeling with that.
That is one of the reasons I didn't release 4.2.0 yet.
The Content-Transfer-Encoding (CTE) can basically be set for the payload as well as for the certificate part.
Unfortunately when using encryption it seems like the CTE is not properly honored from within BC (or at least I didn't find out how).
It would be great if you could confirm that this only affects encrypted messages.
Thanks, Philip

[keithveeva]

Hi Phillip, We have a requirement to use only encrypted messages so I didn't test unencrypted messages. Looking at the SDK, I'm not sure you can set this header without setting the system property, can you please confirm? Thanks, Keith

…

On Mon, Oct 1, 2018 at 11:51 AM Philip Helger ***@***.***> wrote: Hi Keith, I'm aware of this situation, as I myself was struggeling with that. That is one of the reasons I didn't release 4.2.0 yet. The Content-Transfer-Encoding (CTE) can basically be set for the payload as well as for the certificate part. Unfortunately when using encryption it seems like the CTE is not properly honored from within BC (or at least I didn't find out how). It would be great if you could confirm that this only affects encrypted messages. Thanks, Philip — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#71 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ApiHyGjtV_5Qb2Wfu4tShROaPUFfMYjrks5ugjoQgaJpZM4XCOPW> .

[keithveeva]

Also we're using version 3.1.0, let me know if the later versions would address this issue. Thanks, Keith

…

On Mon, Oct 1, 2018 at 11:56 AM Keith Chan ***@***.***> wrote: Hi Phillip, We have a requirement to use only encrypted messages so I didn't test unencrypted messages. Looking at the SDK, I'm not sure you can set this header without setting the system property, can you please confirm? Thanks, Keith On Mon, Oct 1, 2018 at 11:51 AM Philip Helger ***@***.***> wrote: > Hi Keith, > I'm aware of this situation, as I myself was struggeling with that. > That is one of the reasons I didn't release 4.2.0 yet. > The Content-Transfer-Encoding (CTE) can basically be set for the payload > as well as for the certificate part. > Unfortunately when using encryption it seems like the CTE is not properly > honored from within BC (or at least I didn't find out how). > It would be great if you could confirm that this only affects encrypted > messages. > Thanks, Philip > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#71 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ApiHyGjtV_5Qb2Wfu4tShROaPUFfMYjrks5ugjoQgaJpZM4XCOPW> > . >

[phax]

I assume you refer to AS2ClientRequest.setContentTransferEncoding which was introduced in 3.0.4.
And this is exactly what is not working properly in all cases :(

[keithveeva]

I'm looking at this stack HttpURLConnection#setRequestProperty *-> *if you look at this method, it checks if a header is restricted and CTE is on that list, which prevents it from getting set and sent. AS2HttpHeaderWrapperHttpURLConnection#setHttpHeader AS2SenderModule#updateHttpHeaders AS2SenderModule#_sendViaHTTP AS2SenderModule#handle AS2Client#sendSynchronous

…

On Mon, Oct 1, 2018 at 12:50 PM Philip Helger ***@***.***> wrote: I assume you refer to AS2ClientRequest.setContentTransferEncoding which was introduced in 3.0.4. And this is exactly what is not working properly in all cases :( — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#71 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ApiHyFMPmIig55VyvQhawymQ9H5fU0NYks5ugke0gaJpZM4XCOPW> .

[phax]

That sounds pretty okay to me :) The layout of a multipart message is like this (source):

 MIME-Version: 1.0
 Content-Type: multipart/mixed; boundary=frontier

 This is a multi-part message in MIME format.
 
 --frontier
 Content-Type: text/plain

 This is the body of the message.
 --frontier
 Content-Type: text/html
 Content-Transfer-Encoding: base64

 PGh0bWw+CiAgPGhlYWQ+CiAgPC9oZWFkPgogIDxib2R5PgogICAgPHA+VGhpcyBpcyB0aGUg
 Ym9keSBvZiB0aGUgbWVzc2FnZS48L3A+CiAgPC9ib2R5Pgo8L2h0bWw+Cg==
 --frontier--

As every part can have it's own CTE (as shown in the example), you cannot use it on top-level. When using AS2 the first part would be the payload (for which the CTE can be set) and the second part would be the signature (for which no CTE can be set).
My issue is, that I can't get the CTE working for the first block

[keithveeva]

Looking at the headers generated, since we encrypt the message, it's NOT generated as a multipart message, its content type is not multipart message.

…

On Tuesday, October 2, 2018, Philip Helger ***@***.***> wrote: That sounds pretty okay to me :) The layout of a multipart message is like this (source <https://de.wikipedia.org/wiki/Multipurpose_Internet_Mail_Extensions>): MIME-Version: 1.0 Content-Type: multipart/mixed; boundary=frontier This is a multi-part message in MIME format. --frontier Content-Type: text/plain This is the body of the message. --frontier Content-Type: text/html Content-Transfer-Encoding: base64 PGh0bWw+CiAgPGhlYWQ+CiAgPC9oZWFkPgogIDxib2R5PgogICAgPHA+VGhpcyBpcyB0aGUg Ym9keSBvZiB0aGUgbWVzc2FnZS48L3A+CiAgPC9ib2R5Pgo8L2h0bWw+Cg== --frontier-- As every part can have it's own CTE (as shown in the example), you cannot use it on top-level. When using AS2 the first part would be the payload (for which the CTE can be set) and the second part would be the signature (for which no CTE can be set). My issue is, that I can't get the CTE working for the first block — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#71 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ApiHyNLtHxc_B6y5NthaEAMR1QMVp6NPks5ugwZRgaJpZM4XCOPW> .

[phax]

Hi Keith,
I think I found the reason, it just takes a bit to implement everything.
The current situation is a bit of a mess, because the current version contains old "HttpUrlConnection" as well as mew "HttpClient" based implementations in parallel.
So I need to fix it for both, release 4.2.0 and than get rid of the old version.
// Philip

[keithveeva]

Hi Phillip, Sounds great! I was suspecting the version of HttpURLConnection maybe outdated, but wanted to consult you as you're the expert of this code. For release 4.2.0, is it possible to squeeze in the enhancement I logged in #70? If you can expose a way to specify the chunk length in the AS2ClientRequest or AS2ClientSettings object that'd be great. Currently we have to make a fork of your code and make this change on our end, if you can squeeze it in release 4.2.0 then we can get rid of our fork. Thanks, Keith

…

On Tue, Oct 2, 2018 at 9:56 AM Philip Helger ***@***.***> wrote: Hi Keith, I think I found the reason, it just takes a bit to implement everything. The current situation is a bit of a mess, because the current version contains old "HttpUrlConnection" as well as mew "HttpClient" based implementations in parallel. So I need to fix it for both, release 4.2.0 and than get rid of the old version. // Philip — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#71 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ApiHyBmhkEMQHk27LfAWMhg0v6qtjsqIks5ug3CWgaJpZM4XCOPW> .

[phax]

So it took some time but I finally managed to get the Content-Transfer-Encoding thing working. It works at least with the new "large file support" version based on Apache HTTP Client. I'm still struggeling with the old "HttpURLConnection" version, because of this "forbidden header" thingy.
The workaround according to https://bugs.openjdk.java.net/browse/JDK-6996110 is to specify the system property -Dsun.net.http.allowRestrictedHeaders=true on the commandline.

[phax]

Additionally "compress before signing" does not work with Content-Transfer-Encoding other than "binary". I'm investigating this

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[phax]

With #87 the HttpUrlConnection is gone

[phax]

Now the CTE can be set and the MIC calculation is consistent with the one from Mendelson