diff --git a/pep-0458-1.png b/pep-0458-1.png index b08801b5841..52bea1b2948 100644 Binary files a/pep-0458-1.png and b/pep-0458-1.png differ diff --git a/pep-0458-2.png b/pep-0458-2.png deleted file mode 100644 index 52bea1b2948..00000000000 Binary files a/pep-0458-2.png and /dev/null differ diff --git a/pep-0458.txt b/pep-0458.txt index 18d00bd2a9a..99fce5be085 100644 --- a/pep-0458.txt +++ b/pep-0458.txt @@ -337,12 +337,55 @@ paths of available target files together with their cryptographic hashes. The file paths must be specified relative to a base URL. This allows the actual target files to be served from anywhere, as long as the base URL can be accessed by the client. Each top-level role will serve its -responsibilities without exception. Figure 1 provides a table of the +responsibilities without exception. Table 1 provides an overview of the roles used in TUF. -.. image:: pep-0458-1.png - -Figure 1: An overview of the TUF roles. ++-----------------------------------------------------------------------------+ +| Roles and Responsibilities | ++-----------------+-----------------------------------------------------------+ +| root | The root roles is the locus of trust for the entire | +| | repository. The root role signs the root.json metadata | +| | file. This file indicates which keys are authorized for | +| | each of the top-level roles, including for the root role | +| | itself. The roles "root", "snapshot", "timestamp" and | +| | "targets" must be specified and each has a list of public | +| | keys. | ++-----------------+-----------------------------------------------------------+ +| targets | The targets role is responsible for indicating which | +| | target files are available from the repository. More | +| | precisely, it shares the responsibility of providing | +| | information about the content of updates. The targets | +| | role signs targets.json metadata, and can delegate trust | +| | for repository files to other roles (delegated roles). | ++-----------------+-----------------------------------------------------------+ +| delegated roles | If the top-level targets role performs delegation, the | +| | resulting delegated roles can then provide their own | +| | metadata files. The format of the metadata files provided | +| | by delegated targets roles is the same as that of | +| | targets.json. As with targets.json, the latest version of | +| | metadata files belonging to delegated roles are described | +| | in the snapshot role's metadata. | ++-----------------+-----------------------------------------------------------+ +| snapshot | The snapshot role is responsible for ensuring that | +| | clients see a consistent repository state. It provides | +| | repository state information by indicating the latest | +| | versions of the top-level targets and delegated targets | +| | metadata files on the repository in snapshot.json. root | +| | and timestamp are not listed in snapshot.json, because | +| | timestamp signs for its freshness, after snapshot.json | +| | has been created, and root, which has all top-level keys, | +| | is required ahead of time to trust any of the top-level | +| | roles. | ++-----------------+-----------------------------------------------------------+ +| timestamp | The timestamp role is responsible for providing | +| | information about the timeliness of available updates. | +| | Timeliness information is made available by frequently | +| | signing a new timestamp.json file that has a short | +| | expiration time. This file indicates the latest version | +| | of snapshot.json. | ++-----------------+-----------------------------------------------------------+ + +Table 1: An overview of the TUF roles. Unless otherwise specified, this PEP RECOMMENDS that every metadata or target file be hashed using the SHA2-512 function of @@ -371,15 +414,15 @@ new snapshot of the repository metadata. The *snapshot* role signs for *root*, further delegates to the *bin-n* roles, which sign for all distribution files belonging to registered PyPI projects. -Figure 2 provides an overview of the roles available within PyPI, which +Figure 1 provides an overview of the roles available within PyPI, which includes the top-level roles and the roles delegated to by *targets*. The figure also indicates the types of keys used to sign each role, and which roles are trusted to sign for files available on PyPI. The next two sections cover the details of signing repository files and the types of keys used for each role. -.. image:: pep-0458-2.png +.. image:: pep-0458-1.png -Figure 2: An overview of the role metadata available on PyPI. +Figure 1: An overview of the role metadata available on PyPI. The roles that change most frequently are *timestamp*, *snapshot* and roles delegated to by *bins* (i.e., *bin-n*). The *timestamp* and *snapshot* @@ -506,15 +549,16 @@ install or upgrade a PyPI project via TUF. __ https://github.com/theupdateframework/tuf/blob/v0.11.1/docs/TUTORIAL.md#delegate-to-hashed-bins Based on our findings as of the time this document was updated for -implementation (Nov 7 2019), summarized in Tables 1-2, PyPI SHOULD +implementation (Nov 7 2019), summarized in Tables 2-3, PyPI SHOULD split all targets in the *bins* role by delegating them to 16,384 -*bin-n* roles (see C10 in Table 1). Each *bin-n* role would sign +*bin-n* roles (see C10 in Table 2). Each *bin-n* role would sign for the PyPI targets whose SHA2-512 hashes fall into that bin (see and Figure 2 and `Consistent Snapshots`_). It was found that this number of bins would result in a 6-10% metadata overhead (relative to the average size of downloaded distribution files; see V13 and -V15 in Table 2) for returning users, and a 70% overhead for new -users who are installing pip for the first time (see V17 in Table 2). +V15 in Table 3) for returning users, and a 70% overhead for new +users who are installing pip for the first time (see V17 in Table 3). + A few assumptions used in calculating these metadata overhead percentages: @@ -552,7 +596,7 @@ size of release files *downloaded* over the past 31 days (1,628,321 bytes), and the average size of releases files on disk (2,740,465 bytes). Ernest W. Durbin III helped to provide these numbers on November 7, 2019. -Table 1: A list of constants used to calculate metadata overhead. +Table 2: A list of constants used to calculate metadata overhead. +------+------------------------------------------------------------------------------------+------------------------------+-----------+ | Name | Description | Formula | Value | @@ -592,7 +636,7 @@ Table 1: A list of constants used to calculate metadata overhead. | V17 | Est. metadata overhead per distribution per new user | round((V16/C9)*100) | 69% | +------+------------------------------------------------------------------------------------+------------------------------+-----------+ -Table 2: Estimated metadata overheads for new and returning users. +Table 3: Estimated metadata overheads for new and returning users. The interested reader may find an interactive version of the metadata overhead calculator here__: @@ -620,7 +664,7 @@ It is possible to make TUF metadata more compact by representing it in a binary format, as opposed to the JSON text format. Nevertheless, a sufficiently large number of projects and distributions will introduce scalability challenges at some point, and therefore the *bins* role will still need delegations (as -outlined in figure 2) in order to address the problem. The JSON format is an +outlined in Figure 1) in order to address the problem. The JSON format is an open and well-known standard for data interchange, which is already supported by the TUF reference implementation, and therefore the recommended data format by this PEP. However, due to the large number of delegations, compressed @@ -961,7 +1005,7 @@ sign the metadata for each role. The remaining sections discuss how PyPI SHOULD audit repository metadata, and the methods PyPI can use to detect and recover from a PyPI compromise. -Table 3 summarizes a few of the attacks possible when a threshold number of +Table 4 summarizes a few of the attacks possible when a threshold number of private cryptographic keys (belonging to any of the PyPI roles) are compromised. The leftmost column lists the roles (or a combination of roles) that have been compromised, and the columns to its right show whether the @@ -988,7 +1032,7 @@ is included in PEP 480 [21]_. | root | YES | +-----------------+-------------------+----------------+--------------------------------+ -Table 3: Attacks possible by compromising certain combinations of role keys. +Table 4: Attacks possible by compromising certain combinations of role keys. In `September 2013`__, it was shown how the latest version (at the time) of pip was susceptible to these attacks and how TUF could protect users against them [14]_.