Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Footer to make it compatible with RFC1952 #190

Merged
merged 1 commit into from
Dec 9, 2020

Conversation

ktock
Copy link
Member

@ktock ktock commented Dec 8, 2020

This commit reflects the recent change on eStargz's footer format for making it
compatible with RFC1952[0].

Since this commit the footer is the following 51 bytes:

10 bytes  gzip header
2  bytes  XLEN (length of Extra field) = 26 (4 bytes header + 16 hex digits + len("STARGZ"))
2  bytes  Extra: SI1 = 'S', SI2 = 'G'
2  bytes  Extra: LEN = 22 (16 hex digits + len("STARGZ"))
22 bytes  Extra: subfield = fmt.Sprintf("%016xSTARGZ", offsetOfTOC)
5  bytes  flate header
8  bytes  gzip footer
(End of the eStargz blob)

According to RFC1952, Extra field is defined as the following (quoted from [1]):

  If the FLG.FEXTRA bit is set, an "extra field" is present in
  the header, with total length XLEN bytes.  It consists of a
  series of subfields, each of the form:

     +---+---+---+---+==================================+
     |SI1|SI2|  LEN  |... LEN bytes of subfield data ...|
     +---+---+---+---+==================================+

  SI1 and SI2 provide a subfield ID, typically two ASCII letters
  with some mnemonic value.  Jean-Loup Gailly
  <gzip@prep.ai.mit.edu> is maintaining a registry of subfield
  IDs; please send him any subfield ID you wish to use.  Subfield
  IDs with SI2 = 0 are reserved for future use.

This commit adds SI1, SI2 and LEN fields to Extra field, which are lacked in the
stargz. As SI1 and SI2, we use (SI1, SI2) = ('S', 'G') for eStargz footer.
(P.S. ('A', 'P') is used by "Apollo file type information"[1] and ('R', 'A') is
used by dictzip[2])

Notes:

  • This commit forks stargz lib for applying this change.
  • This change makes eStargz incompatible with stargz (but still compatible with
    tar.gz). Images provided by this commit's ctr-remote cannot be lazily pulled
    by the stargz snapshotter before this commit. --stargz-only option is also
    deprecated since this commit. Users should use --no-optimize option instead.
  • As a fallback behaviour, the legacy footer is still supported by Stargz
    Snapshotter plugin (i.e. it can lazily pull eStargz images older than this
    commit).

Some minor changes for making linter happy are also included in this patch.

[0] https://tools.ietf.org/html/rfc1952
[1] https://tools.ietf.org/html/rfc1952#section-2.3.1.1
[2] https://linux.die.net/man/1/dictzip

Signed-off-by: Kohei Tokunaga ktokunaga.mail@gmail.com

This commit reflects the recent change on eStargz's footer format for making it
compatible with RFC1952[0].

Since this commit the footer is the following 51 bytes:

```
10 bytes  gzip header
2  bytes  XLEN (length of Extra field) = 26 (4 bytes header + 16 hex digits + len("STARGZ"))
2  bytes  Extra: SI1 = 'S', SI2 = 'G'
2  bytes  Extra: LEN = 22 (16 hex digits + len("STARGZ"))
22 bytes  Extra: subfield = fmt.Sprintf("%016xSTARGZ", offsetOfTOC)
5  bytes  flate header
8  bytes  gzip footer
(End of the eStargz blob)
```

According to RFC1952, Extra field is defined as the following (quoted from [1]):

```
  If the FLG.FEXTRA bit is set, an "extra field" is present in
  the header, with total length XLEN bytes.  It consists of a
  series of subfields, each of the form:

     +---+---+---+---+==================================+
     |SI1|SI2|  LEN  |... LEN bytes of subfield data ...|
     +---+---+---+---+==================================+

  SI1 and SI2 provide a subfield ID, typically two ASCII letters
  with some mnemonic value.  Jean-Loup Gailly
  <gzip@prep.ai.mit.edu> is maintaining a registry of subfield
  IDs; please send him any subfield ID you wish to use.  Subfield
  IDs with SI2 = 0 are reserved for future use.
```

This commit adds SI1, SI2 and LEN fields to Extra field, which are lacked in the
stargz. As SI1 and SI2, we use (SI1, SI2) = ('S', 'G') for eStargz footer.
(P.S. ('A', 'P') is used by "Apollo file type information"[1] and ('R', 'A') is
used by dictzip[2])

Notes:
- This commit forks stargz lib for applying this change.
- This change makes eStargz incompatible with stargz (but still compatible with
  tar.gz). Images provided by this commit's `ctr-remote` cannot be lazily pulled
  by the stargz snapshotter before this commit. `--stargz-only` option is also
  deprecated since this commit. Users should use `--no-optimize` option instead.
- As a fallback behaviour, the legacy footer is still supported by Stargz
  Snapshotter plugin (i.e. it can lazily pull eStargz images older than this
  commit).

Some minor changes for making linter happy are also included in this patch.

[0] https://tools.ietf.org/html/rfc1952
[1] https://tools.ietf.org/html/rfc1952#section-2.3.1.1
[2] https://linux.die.net/man/1/dictzip

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
@AkihiroSuda
Copy link
Member

please send him any subfield ID you wish to use

Did you manage to make a contact to this guy (with his email address in 1990s)? 😂

@ktock
Copy link
Member Author

ktock commented Dec 8, 2020

I sent an email today. Hoping the response.

Interestingly dictzip uses ('R', 'A') but doesn't seems to be registered yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants