Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement "Astraweb-style" compression #14

Open
sgg opened this issue Jun 11, 2020 · 0 comments
Open

Implement "Astraweb-style" compression #14

sgg opened this issue Jun 11, 2020 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@sgg
Copy link
Owner

sgg commented Jun 11, 2020

Implement Astraweb-style compression. This could be provided as part of overviews in the NntpClient and as separate deserialization functions.

Astraweb-style header compression leverages two custom commands, XZVER and XZHDR.

The closes thing I've found to documentatoin is an old post from their helpdesk (pasted below).

============================================================

We are proposing two new commands as an extension to the NNTP protocol: XZVER and XZHDR as counterparts to XOVER and XHDR respectively.
These two new optional commands will appear in the list returned from the "HELP" command.


Request: XZVER [range]
Response:

224 compressed data follows
=ybegin line=128 size=-1

=yend crc=FFFFFFFF
.


Request: XZHDR header [range]
Response

221
compressed data follows
=ybegin line=128 size=-1

=yend crc=FFFFFFFF
.


Where [range] is in the standard article number range as per XOVER.
Notes:
- The multi-line responses conform to standard NNTP protocol with a single period "." on a line by itself.
- Line terminals are \r\n
- The actual XOVER / XHDR response is compressed with zlib, then yEnc encoded.
- The compression conforms with zlib's deflate() method and there is NO gzip done on it. ie. Just pure gzip'ed data with no extra padding info to make it gzip-file compliant.
- The yEnc spec requires that the "size" parameter in "=ybegin line=123 size=456" be present, but for efficiency and performance sake, the total size is not known ahead of time so it has been set to some arbitrary value that should be ignored. The yEnc CRC32 value should be sufficient to validate the payload. On top of that, zlib has builtin adler32 which is an alternate form of CRC32.


It is assumed that the implementation for XZVER be duplicated from XOVER, with two additional steps at the end, that being: yEnc decode the data, then zlib inflate the data.
============================================================
@sgg sgg added this to the v0.2 milestone Jun 11, 2020
@sgg sgg added the enhancement New feature or request label Jun 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant