Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Suggestion] Add base64 built in #1960

Open
Soapy7261 opened this issue Aug 30, 2024 · 5 comments
Open

[Suggestion] Add base64 built in #1960

Soapy7261 opened this issue Aug 30, 2024 · 5 comments
Labels
area-CraftOS This affects CraftOS, or any other Lua portions of the mod. enhancement An extension of a feature or a new feature.

Comments

@Soapy7261
Copy link

Soapy7261 commented Aug 30, 2024

Useful information to include:

  • A function like textutils.base64.encode(stringofbinaryornonbinarydatahere), or like how python has it with the base64 package having b64encode/b64decode along with b32encode/b32decode under the same base64 package
  • Things like web socket sending require encoding binary data as Base64, there are LUA packages that do this, but they are quite slow for large amounts of binary data, it would be nice if it was included (using a java implementation ideally so it could be done faster), other bases (like base32) would be nice too.
  • I have considered using textutils.urlEncode(), and it does work for smaller strings of binary data, but it has a bug(?) where if the string is too big, it never yields so you get the dreaded 'Too long without yielding' error, even if that was fixed it would still be nice to have base64 built in.
@Soapy7261 Soapy7261 added the enhancement An extension of a feature or a new feature. label Aug 30, 2024
@SquidDev SquidDev added the area-CraftOS This affects CraftOS, or any other Lua portions of the mod. label Aug 30, 2024
@SquidDev
Copy link
Member

Things like web socket sending require encoding binary data as Base64

Not sure if you're aware but websockets should support sending binary messages.

I have considered using textutils.urlEncode(), but it has a bug(?) where if the string is too big, it never yields. [...] There are LUA packages that do this, but they are quite slow for large amounts of binary data, it would be nice if it was included (using a java implementation ideally so it could be done faster)

textutils.urlEncode is probably not the right tool here, but I am curious how much data you're trying to send here? Looking at the implementation, there's some obvious optimisations there which I'll make, but it should be fine for anything less than a megabyte.

I generally prefer putting this stuff in Lua where possible, so would like to understand what performance expectations you have.

@Soapy7261
Copy link
Author

Not sure if you're aware but websockets should support sending binary messages.

Well, I guess there goes 3 hours of my life I'll never get back.

textutils.urlEncode is probably not the right tool here, but I am curious how much data you're trying to send here? Looking at the implementation, there's some obvious optimisations there which I'll make, but it should be fine for anything less than a megabyte.

I'll admit I was trying to encode a 150MB file, but the limit before that happens is probably quite a bit lower, I'll do more testing when I have more time

I generally prefer putting this stuff in Lua where possible, so would like to understand what performance expectations you have.

That's understandable, even if it was in LUA, it would still be nice to have included

@Soapy7261
Copy link
Author

image
Also, just looking at the documentation 'Whether this message should be treated as a', then it cuts off, i know its probably 'Whether this message should be treated as a binary message' but still

@Soapy7261
Copy link
Author

Soapy7261 commented Aug 30, 2024

Also, kind of unrelated but textutils.serialiseJSON() does not seem to like binary data, the chunk_data is exactly 131036 big
image
Yet somehow it grows to 535823? Didn't do this when i gave it base64 data
image

@Soapy7261
Copy link
Author

Soapy7261 commented Aug 30, 2024

image
Seems to be converting the binary data to string, although that actually makes sense now that i think about it, i should probably not use JSON, although using base64 is probably easier (and faster) then re-designing my entire system

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-CraftOS This affects CraftOS, or any other Lua portions of the mod. enhancement An extension of a feature or a new feature.
Projects
None yet
Development

No branches or pull requests

2 participants