Escaping "(" and ")" using ^040 and & ^41 in gs1 datamatrix encoder ends up with AI syntax error

[Senshi26]

Hello ! @terryburton advised me to raise this issue, since it appears to be a bug.

Service throws error when you try to use parentheses as part of AI in any AI along with &parse and &parsefnc optional parameters.

(Example when "(" in 21 AI)
error - wrong syntax(Error: bwipp.GS1badCSET82character#2517: AI 21

Here's a full url with parameters(url encoded) :

http://bwipjs-api.metafloor.com/?bcid=gs1datamatrix&text=(01)03386460023344(21)5EK%5E040jlEhKVH%2CZ(91)EE08(92)xrmaCaDwfiJHiS2Z1Ad90tjmX5NwOQYMPqwAtpYE1s0%3D&parse&parsefnc

reproduced with and without url encoding
reproduced with and without optional parameters

STEPS TO REPRODUCE

1. Use gs1 datamatrix encoder in bwip-js
2. Use &parse and &parsefnc parameters
3. Add text with "(" or ")" as part of any AI in text parameter
4. Escape them as described in https://github.com/bwipp/postscriptbarcode/wiki/GS1-Application-Identifier-Standard-Format
5. Try to obtain gs1 datamatrix on bwipjs-api

Will be really appreciated if you can fix this.
The thing is: we are facing "(" and ")" in serial (21 AI) quite often. The example datamatrix I'd provided belongs to perfume.

[metafloor]

Currently on holiday in remote southern Utah with intermittent access to internet. Will investigate this issue later next week.

[metafloor]

BTW, what version of bwip-js are you using? Older versions handled the parse option separately. New versions do not and should work identically to BWIPP's parse handling.

[Senshi26]

Hey ! I'm using
BWIPJS_VERSION=3.4.2
BWIPP_VERSION=2023-04-03

[metafloor]

Definitely need to upgrade to the current version.

[Senshi26]

Could you please advise how to upgrade it ?

[metafloor]

Typically, to upgrade a dependency, use npm update in your project's directory. If you don't have bwip-js listed as a dependency in your project's package.json, then npm install bwip-js also works. You need version 4.1+.

[Senshi26]

Could you please answer , since I'm not Node.js guy, can't you just update it on bwipjs-api.metafloor.com ?
Wouldn't it be easier ?

[metafloor]

It's your choice how you want to maintain your software. You should explore using npm as it simplifies and automates updates. But using git or the zip file link on github works as well. The bwip-js github repo is always up to date with the latest version.

[Senshi26]

Ok, you're right. I was just raising the issue on specific place (bwipjs-api.metafloor.com). I thought, that since you are maintaining the web service - you can help, fixing it's sources. From I can tell I'm not the only one, who is using it, that's why may be you can consider fixing(updating) the source of bwipjs-api.metafloor.com. It will help me along with other users. Will be really appreciated !

[metafloor]

Apologies. I mis-understood what you were asking. The version level of the public API has been updated. It is supposed to automatically update on each new release, but apparently there is a problem with the release script.

[Senshi26]

Just before thanking you guys for such a great job you're doing - will justify a bit for those who may encounter same issue with parentheses while generating GS1 datamatrix on bwipjs-api.metafloor.com

here's the example where you have a ")" within 21 AI (Serial Number)
URL encoded :
http://bwipjs-api.metafloor.com/?bcid=gs1datamatrix&text=(01)03386460007092(21)r1O%3DuezE%29V%22Vb(91)EE07(92)Gr%2B9UO8qBZ4K0N6MVV%2BgOmyMX%2BzHixtFaIkrPj0AiWk%3D&parse&parsefnc

%29 is a ")" sign and the service is totally ok with it
You don't need to escape it using "^41" and "^42"

Thank you so much ! I'm sure that many people who are using your web service are happy 👍 Keep this service on ! Simple guys like me are relying on you !

[Senshi26]

So sorry to tell, but the issue occurred again.
here's an example datamatrix

http://bwipjs-api.metafloor.com/?bcid=gs1datamatrix&text=(01)02900059737460(21)BS7%27B%28e25P%2CC%22(91)EE07(92)frTu8qwL1HH2VyNmR%2BJA5NU7bTaNaThtkGBqFjrFUQk%3D&parse&parsefnc

I've tested and got

Error: bwipp.GS1unknownAI#3205: AI e25P,C"(91: Unrecognised AI

here's the status of bwipjs-api.metafloor.com:

`OK 695812 requests, 3995.91 secs, 2376.18M bytes

STARTED=2023-11-14 22:59:23
UPTIME=13:46:13
BWIPJS_VERSION=4.1.2 (2023-11-01)
BWIPP_VERSION=2023-04-03
INVALID=485
ERRORS=3
CAUGHT=12917
TIMEOUTS=128

api-bwipjs.metafloor.com=34910
bwip-js.herokuapp.com=2463
bwipjs-api.metafloor.com=658439

azteccode=264
code128=358557
code2of5=235
code39=241744
code39ext=910
code93=10
databarlimited=22
datamatrix=7072
ean13=18271
ean14=12
gs1-128=702
gs1_128=1890
gs1datamatrix=863
gs1qrcode=37
hibcpdf417=4
interleaved2of5=369
isbn=4
itf14=198
kix=3
pdf417=1603
qrcode=55068
rationalizedCodabar=4787
upca=2881
upce=306`

I've tested on 3 resources :
bwipjs-api.metafloor.com
api-bwipjs.metafloor.com
bwip-js.herokuapp.com

The error is the same.

I see that AIs are parsed separately, since the error is now occurring in the 92 AI and the approach which is working for the AI 21 is not applied to any other.

Could you please extend the parentheses parsing function to all AIs ?

[metafloor]

Looking at your example, I don't see where you are escaping extra parentheses with ^040 and ^041. URL escapes are decoded prior to the call into the BWIPP code.

[Senshi26]

yes, indeed, as you may see I was not escaping them when you did an update.
Therefore I thought, I don't need to escape parentheses using ^41 and ^42.
I will escape them and let you know the results.

[Senshi26]

Tested with escaping sequence ^42
got an error
Error: bwipp.GS1badCSET82character#2517: AI 21: Invalid CSET 82 character

Used both url encoded string and url encoded string - error remains the same.

https://bwipjs-api.metafloor.com/?bcid=gs1datamatrix&text=%2801%2902900059737460%2821%29BS7%27B%5E42e25P%2CC%22%2891%29EE07%2892%29frTu8qwL1HH2VyNmR%2BJA5NU7bTaNaThtkGBqFjrFUQk%3D%26parse%26parsefnc

and

https://bwipjs-api.metafloor.com/?bcid=gs1datamatrix&text=(01)02900059737460(21)BS7%27B)e25P,C%22(91)EE07(92)frTu8qwL1HH2VyNmR+JA5NU7bTaNaThtkGBqFjrFUQk=&parse&parsefnc

Please note that my previous example with parentheses in 21 AI works fine
here it is:
http://bwipjs-api.metafloor.com/?bcid=gs1datamatrix&text=(01)03386460007092(21)r1O%3DuezE%29V%22Vb(91)EE07(92)Gr%2B9UO8qBZ4K0N6MVV%2BgOmyMX%2BzHixtFaIkrPj0AiWk%3D&parse&parsefnc
I'm not escaping it and getting correct result without escaping to ^42, however the first example is still returning error , even though it's very similar. Please verify

[metafloor]

In the second example, the + needs to be %-escaped as it is special to URLs. This works:

https://bwipjs-api.metafloor.com/?bcid=gs1datamatrix&text=(01)02900059737460(21)BS7%27B)e25P,C%22(91)EE07(92)frTu8qwL1HH2VyNmR%2BJA5NU7bTaNaThtkGBqFjrFUQk=&parse&parsefnc

And you are correct about the parse option being incompatible with GS-1 encoded text. It is not possible encode something like (92)ABC(11)XYZ, where the (11) is part of the 92-field. Encoding the (11) as ^04011^041 raises bwipp.GS1badCSET82character#2517: AI 92: Invalid CSET 82 character due to the ^. And not parse encoding (11) raises bwipp.GS1valueTooShort#3178: AI 11: Too short as an 11-field is supposed to be a 6-digit date.

[terryburton]

And you are correct about the parse option being incompatible with GS-1 encoded text.

@metafloor Just to confirm, in case it wasn't clear, that this is not an issue with the BWIPP library, e.g. this works perfectly well:

Barcode: gs1datamatrix
Contents: (01)09521234543213(10)AAA^04011^041AAA
Options: parse

For regular encoders, the parse option applies to the entire string before symbology encoding.

For the GS1 application-specific helpers, there is an upfront parser that splits the bracketed element string into AI-value pairs and then validates the extracted data. Afterwards, the parse option is applied to each pair's value, immediately prior to constructing the final "FNC1 in first" message that is encoded within the symbology.

The ordinal syntax (^NNN) is only required so that the AI parser can disambiguate AIs vs values in bracketed AI input without needing to resort to some less intuitive AI indicator (e.g. [01]) as used by other libraries. (GS1 periodically introduces additional CSETs for AI values. In the future, these new sets may include characters that are not in the current 82-character alphabet ("CSET X"), such as [ and ], so it seems best to parse after extraction than attempt to guess which character substitutions will remain disambiguated over time.)

[metafloor]

Thank you for the extra clarification Terry. The bug is understood but complex to fix while maintaining some semblance of performance. I am testing one fix right now but not happy with its brittleness. Have another idea to try out tomorrow...

[Senshi26]

Happy New Year!
please pay attention to the datamatrix code, which is working - it has a different parentheses than the one provided in your example.
http://bwipjs-api.metafloor.com/?bcid=gs1datamatrix&text=(01)02900059737460(21)BS7%27B(e25P,C%22(91)EE07(92)frTu8qwL1HH2VyNmR%2BJA5NU7bTaNaThtkGBqFjrFUQk&parse&parsefnc
(I've intentionally left parentheses unencoded for better explanation)

Now mine example which throws Error: bwipp.GS1unknownAI#3205: AI e25P%C"(91: Unrecognised AI

http://bwipjs-api.metafloor.com/?bcid=gs1datamatrix&text=(01)02900059737460(21)BS7%27B(e25P,C%22(91)EE07(92)frTu8qwL1HH2VyNmR%2BJA5NU7bTaNaThtkGBqFjrFUQk&parse&parsefnc
Basically, your has ) and mine (
My example doesn't work, while yours totally ok. May be there's a way you can work this issue out. That would be great.

The initial datamatrix code (without any encoding is the follows - again example from Perfumes)
010290005973746021BS7'B(e25P,C"91EE0792frTu8qwL1HH2VyNmR+JA5NU7bTaNaThtkGBqFjrFUQk=

[metafloor]

Currently testing a fix for this issue. The problem is in the emulation of the postscript dictionary stack. The BWIPP upstream code handles these test cases correctly. Unfortunately, the fix has caused a considerable change in the emitted PS-to-JS code so testing requires extra care.

[metafloor]

Version 4.2 should resolve this issue (finally!). Please give it a try and let me know.

[metafloor]

Closing this out as resolved.