security: postmortem for CVE-2019-990[01]

security/postmortems/cve-2019-9900.md

@@ -0,0 +1,323 @@
+# Security postmortem for CVE-2019-9900, CVE-2019-9901
+
+## Incident date(s)
+
+2019-02-18 - 2019-04-05
+
+## Authors
+
+@htuch
+
+## Status
+
+Final
+
+## Summary
+
+Two independent vulnerabilities related to a mismatch between the information used for request
+matching and routing were discovered in February/March 2019, leading to the potential ability for an
+attacker to bypass access control checks and route table intent. Since these issues had similar
+attack vectors and were discovered within the same embargo window, the issues were grouped and
+resolved (mostly) privately by Envoy and Istio fix teams, resulting in the Envoy 1.9.1 security
+release issued on 2019-04-05.
+
+This was the first time in which the Envoy security release process was followed and provided a
+learning opportunity to refine the process, originally borrowed from the Kubernetes project, to
+Envoy's requirements. While the issues were upgraded from medium to high criticality during the fix
+process, they were limited in impact to a subset of users and specific configuration patterns. This
+postmortem captures the issues encountered during the fix process and provides actionable next
+steps.
+
+## CVE issue(s)
+
+* https://github.com/envoyproxy/envoy/issues/6434
+* https://github.com/envoyproxy/envoy/issues/6435
+
+## Root Causes
+
+CVE-2019-9900 resulted from Envoy assuming that its codec libraries (http-parser, nghttp2) followed
+RFC 7230 and would reject any header value with an embedded NUL character. Unfortunately,
+http-parser did not do this due to an optimization in header value processing
+(https://github.com/nodejs/http-parser/issues/468, https://github.com/nodejs/http-parser/pull/469).
+In addition, Envoy viewed header strings with a mixture of `c_str()` and `string_view`, allowing the
+possibility of inconsistent views between checks and resulting action. A combination of a buggy
+external dependency and problematic use of C string views led to this vulnerability.
+
+CVE-2019-9901 resulted from two distinct views of the role of a proxy in path handling. On the one
+hand, Envoy was considered a data forwarding engine for HTTP requests that did not need to perform
+path normalization, with this concern left to client and backend. However, at the same time, Envoy
+was being used in applications where it intermediated on requests for access control purposes (e.g.
+RBAC, `ext_authz`) and performed path matching against policy. Especially in the presence of a
+backend that itself normalizes, this access control role required that path normalization be applied
+in the proxy.
+
+## Resolution
+
+CVE-2019-9900 was reported by Envoy maintainer @htuch on 2019-03-10 to
+envoy-security@googlegroups.com. After some discussion, it was agreed that this warranted invoking
+the security release process. The issue was mitigated in the Envoy private security repository by
+@htuch and the Envoy security fix team. A single patch
+(https://github.com/envoyproxy/envoy/commit/b155af75fad7861e941b5939dc001abf581c9203) was required
+to workaround the http-parser behavior. In addition, both tests and fuzzers were
+created to validate the behavior when NULs were introduced anywhere in an HTTP/1 or HTTP/2 request
+(https://github.com/envoyproxy/envoy/commit/1e61a3f95f2c4d9ac1e54feae8693cee7906e2eb). Manual code
+inspection was also performed in nghttp2 to verify the absence of vulnerability. While doing so, a
+non-security related bug was discovered (https://github.com/nghttp2/nghttp2/issues/1331).
+
+Shortly after discovery of CVE-2019-9900, the http-parser issue was reported to the Node.js security
+working group at security@nodejs.org, since http-parser lives under the umbrella of the Node.js
+project. The full vulnerability was described and the Envoy security team proposed working with
+Node.js PST. As there was no reply, we proceeded independently. Unfortunately, it appears that the
+Node.js security WG never received the e-mail, due to the reliance of Node.js on HackerOne to gate
+incoming issues and a problematic e-mail forwarding chain
+(https://github.com/nodejs/security-wg/issues/454#issuecomment-481919759). We have since filed a
+HackerOne issue with the original report e-mail.
+
+CVE-2019-9901 was privately disclosed to the Istio security team by an external researcher on
+2019-02-18 and accidentally publicly disclosed in part in
+https://github.com/envoyproxy/envoy/issues/6008 on 2019-03-13. Once the severity of this was
+realized via offline discussion between the Envoy security team and the PR authors, we moved to a
+private fix process in conjunction with CVE-2019-9900, targeting the 1.9.1 release. The Google Istio
+security and networking teams led the efforts to fix this vulnerability in Envoy's private security
+repository. The workaround implementation of path normalization borrowed from Chromium's URL
+library, adapted and minified for the Envoy context. The follow patches were
+produced:
+* https://github.com/envoyproxy/envoy/commit/c22cfd2c483fc26534382a0b6835f45264bb137a
+* https://github.com/envoyproxy/envoy/commit/7ed6d2187df94c4cb96f7dccb8643bf764af2ccb
+
+In both cases, the Envoy security team considered the issues of medium criticality (CVSS 6.5)
+initially, since it was thought that the attack complexity was high, requiring special circumstances
+to apply. As we continued discussion with Istio and Google teams, it became apparent that the
+exploits were trivial to automate and we upgraded to high criticality (CVSS 8.3), due to the lower
+attack complexity.
+
+A 1.9.1 security release was initially targeted for 2019-04-02 and announced on 2019-03-22. An
+e-mail was sent to the Envoy private distributor list sharing CVE details.
+
+After private discussions with a distributor on 2019-03-28, who expressed concern over the very
+short (3 working day) distance between fix patch availability and release, the Envoy security team
+decided to delay the 1.9.1 release until 2019-04-05. This provided 1 week for distributors to
+prepare their software for the security release date.
+
+Fix patches were shared with the private distributor list late on 2019-03-28.
+
+During the fix process, two distributors reached out to us to request the ability to stage in
+publicly accessible locations binary images with the fixes applied. While technically this would
+violate embargo, we decided to allow this due to a lack of a clear alternative; Envoy's sidecar
+use cases and reliance on Docker for distribution, where images are generally staged on public hubs,
+did not lend itself to opaque rollout.
+
+## Detection
+
+The underlying issue behind CVE-2019-9900 was first noticed via fuzzers when an explicit `ASSERT`
+check for embedded NULs was added in #6170. The following issue was tripped by
+`h1_capture_fuzz_test`:
+https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=13613. Some experiments with `netcat`, `tcpdump`
+and an Envoy binary demonstrated that it was viable to at least bypass header suffix matches via
+this mechanism.
+
+CVE-2019-9901 was reported by an external researcher (Erlend Oftedal) to the Istio security team.
+
+## Action Items
+
+* https://github.com/envoyproxy/envoy/issues?utf8=%E2%9C%93&q=is%3Aissue+%22Action+item+for+CVE-2019-9900%22+
+* https://github.com/envoyproxy/envoy/issues?utf8=%E2%9C%93&q=is%3Aissue+%22Action+item+for+CVE-2019-9901%22+
+
+## Lessons Learned
+
+### What went well
+
+* Fix patches were available within 2 weeks of vulnerability disclosure. The
+  changes were localized and relatively clean.
+
+### What went wrong
+
+* The Envoy private distributor list was initially almost empty. We sent out an
+  e-mail to remind distributors to sign up on 2019-03-14 and the list is now O(10).
+
+* The security impact of https://github.com/envoyproxy/envoy/issues/6008
+  was not caught by Envoy until this was brought to our attention ~20 days after
+  the issue was first pushed. Ideally such issues should be routed to
+  envoy-security@googlegroups.com first in the future and Envoy
+  reviewers/maintainers should keep an eye out for inadvertent security
+  disclosures through public channels. In addition, an earlier issue
+  https://github.com/envoyproxy/envoy/issues/2956 was opened a year previous, but was not tagged as
+  being security sensitive.
+
+* Applicants for the private distributor list were turned down based on
+  membership criteria that was adopted from k8s. This is now being revisited in
+  https://github.com/envoyproxy/envoy/issues/6586.
+
+* Distributors were only provided 3 days from candidate fix patch availability
+  until public release at first. While this was extended to 1 week, even this
+  might be too little. This is now being codified in
+  https://github.com/envoyproxy/envoy/issues/6587.
+
+* The Chromium URL library was forked, minified and adapted to Envoy. This was
+  expedient but not a maintainable long term solution, see
+  https://github.com/envoyproxy/envoy/issues/6588.
+
+* Only coarse grained control over path normalization was provided, since this
+  was expedient and mitigated the vulnerability. We should provide finer grained
+  controls, see https://github.com/envoyproxy/envoy/issues/6589.
+
+* Our report to the Node.js security working group was lost due to
+  https://github.com/nodejs/security-wg/issues/454#issuecomment-481919759.
+  We should avoid this happening Envoy-side, see https://github.com/envoyproxy/envoy/issues/6590.
+  More generally, we should err on the side of reaching out over more channels
+  in the future, since it's unclear how effective any given disclosure channel
+  is.
+
+* The security release day (2019-04-05) was Friday PDT. We should pick a
+  globally friendly day-of-week, e.g. Tue-Thu, for security releases.
+
+* Nginx already had a CVE for path normalization
+  (https://www.rapid7.com/db/vulnerabilities/nginx-cve-2009-3898) similar to
+  CVE-2019-9901, but we did not know this until after the fact. We should audit
+  CVEs for similar class software, see
+  https://github.com/envoyproxy/envoy/issues/6592.
+
+* A distributor reached out to the security team for permission to perform
+  silent binary rollouts as discussed above. While in principle our relaxation
+  of the embargo policy applied to all distributors, an e-mail was not sent to
+  the list. This resulted in confusion when a second distributor observed this
+  rollout. We should ensure going forward that any policy relaxation during CVE
+  handling is clearly communicated across the board.
+
+* Public, albeit silent, staging of Docker images before the public security
+  release date was a necessary pragmatic tradeoff. We need to refine the
+  security release process to deal with this explicitly, see
+  https://github.com/envoyproxy/envoy/issues/6593.
+
+* The security release forced `envoy-dev:latest` back to the 1.9.1 release
+  branch. This should be fixed, see
+  https://github.com/envoyproxy/envoy/issues/6595.
+
+* There was a window of ~50 minutes between the release tagging of the Envoy
+  1.9.1 branch and availability of Docker images. Ideally we shrink this to
+  allow users to upgrade faster. See
+  https://github.com/envoyproxy/envoy/issues/6596.
+
+* The CVE-2019-9901 fix required either control plane or runtime changes. This
+  orchestration was not well suited to all deployment environments, so some
+  distributions, e.g. Istio, applied additional patches to enable at compile
+  time. Ideally we support control plane, runtime and CLI or compile-time fix
+  opt-in abilities.
+
+### Where we got lucky
+
+* The defects were not critical (by CVSS scoring and intuition) and (mostly)
+  privately disclosed. This provided an opportunity to exercise and refine the
+  Envoy security release process.
+
+* Huffman and HPACK in general frustrates HTTP/2 testing and fuzzing for security
+  properties. We had no effective fuzzing or testing for this previously as a
+  result, we were lucky that the scope of CVE-2019-9900 was limited to HTTP/1.1.
+
+* CVE-2019-9900 was only discovered as a result of additional `ASSERT`s added to
+  verify a property that Envoy developers were certain held. Fuzzing alone had
+  not previously caught this.
+
+* Distributors were able to execute their own security releases within the 1
+  week provided from patch availability. Anecdotally, this involved effort
+  beyond that which we should expect normally to manage an Envoy fix.
+
+* No known instances reported of pre-release embargo breakage due to silent
+  public staging of Docker images.
+
+## Timeline
+
+All times US/Pacific
+
+2019-02-18:
+* [CVE-2019-9901] Path normalization issue was reported to Istio security team at vulnerabilities@discuss.istio.io.
+
+2019-02-19:
+* [CVE-2019-9901] https://github.com/envoyproxy/envoy/issues/6008 was opened. This was not the first Envoy report of
+  missing path normalization (see https://github.com/envoyproxy/envoy/issues/2956). Neither issue
+  mentioned the security basis and Envoy reviewers speculated on the potential for path traversal
+  attacks.
+
+2019-03-08:
+* [CVE-2019-9900] oss-fuzz reports https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=13613 under embargo.
+
+2019-03-10:
+* [CVE-2019-9900] E-mail thread on envoy-security@googlegroups.com regarding the
+  potential effects of this bugs. While it was unclear whether there would be
+  impact beyond some narrow circumstances, agreement was reached to start the
+  security release process. Analysis began to determine the extent of impact on
+  HTTP/2 and Envoy's code base was audited.
+
+2019-03-11:
+* [CVE-2019-9901] https://github.com/envoyproxy/envoy/pull/6258 was opened to address
+  https://github.com/envoyproxy/envoy/issues/6008.
+
+2019-03-13:
+* [CVE-2019-9901] https://github.com/envoyproxy/envoy/pull/6258 was closed after offline discussions
+  between Envoy security team and the author, once the Envoy security team became aware of the
+  potential severity in the Istio setup (in particular with RBAC and Mixer in play).
+
+2019-03-14:
+* [CVE-2019-9900] Finding were presented to envoy-security@. A fix plan was
+  agreed upon and a candidate fix PR was shared with the team by e-mail. At this
+  point, no private fix repository existed.
+* [CVE-2019-9901] The Istio fix leads initiated private work on a fix patch.
+  Since it was likely that this would land within the 1.9.1 release
+  window for CVE-2019-9900, CVE-2019-9901 was also scheduled for the release.
+* [Announcement](https://groups.google.com/forum/#!topic/envoy-announce/dEOLqAiaSUI) sent to remind
+  distributors to join cncf-envoy-distributors-announce@lists.cncf.io.
+
+2019-03-20:
+* CVEs were requested from MITRE for both issues.
+* Draft fix PRs for CVE-2019-9900 and CVE-2019-9901 were shared on private Envoy
+  security repository. Reviews and further development occurred over the
+  following week.
+
+2019-03-22:
+* 11:20 1.9.1 security release for the two vulnerabilities was
+  [announced](https://groups.google.com/d/msg/envoy-announce/6fwGB2TxB74/dKeURAdfAgAJ).
+* 11:24 CVE summary details shared with cncf-envoy-distributors-announce@lists.cncf.io.
+
+2019-03-28:
+* Envoy security team met with a distributor to discuss their concerns over the lack of time between
+  patch availability and the release date. We agreed that three days was insufficient and agreed to
+  extend to a week.
+* 13:53 A delay of the 1.9.1 release until 2019-04-05 was
+  [announced](https://groups.google.com/d/msg/envoy-announce/6fwGB2TxB74/Pe3PPFbPBAAJ).
+* 20:07 Candidate fix patches for both CVE shared with
+  cncf-envoy-distributors-announce@lists.cncf.io.
+
+2019-03-29:
+* Envoy security team was contacted by a distributor regarding the permissibility of silently
+  staging binary images in public locations in advance of the security release due to a lack of
+  viable alternatives. The Envoy security team agreed that there was no better alternative and
+  provided an exemption.
+
+2019-04-02:
+* 08:15 The increase of severity from medium to high was
+  [announced](https://groups.google.com/d/msg/envoy-announce/6fwGB2TxB74/qiDEgclFBgAJ).
+  This followed several days of offline discussion between Istio and Envoy teams
+  on Istio's independent assessment of the issues as high severity, and a better
+  awareness of how to score. What was missing was the intuition that a
+  vulnerability can be high severity even if it only affects a rather limited
+  number of users.
+
+2019-04-04:
+* 15:41 The Envoy master branch was frozen to prepare for the security release. PRs were rebased
+  against master and prepared for the release push.
+* 18:33 Envoy security team was contacted by a distributor who had noticed public visibility of
+  binary images with the fix patch by other vendors. After discussion, we agreed on a general
+  exemption for these CVEs to the embargo policy for binary images with some constraints.
+* 19:18 cncf-envoy-distributors-announce@lists.cncf.io was e-mailed to clarify position on staging
+  of binary images on public sites prior to the release date. A narrow set of circumstances under
+  which this was permissible were outlined.
+
+2019-04-05:
+* 10:00 - 10:05 The [v1.9.1](https://github.com/envoyproxy/envoy/tree/v1.9.1) release branch was
+  pushed and the 1.9.1 releaes was tagged. This started the Docker build process for the release.
+  The same PRs were pushed to master.
+* 10:05 The Envoy 1.9.1 security release was
+  [announced](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/envoy-announce/VoHfnDqZiAM).
+* 10:57 The v1.9.1 image was available at https://hub.docker.com/r/envoyproxy/envoy/tags.
+
+## Supporting information

security/postmortems/cve-2019-9901.md

@@ -0,0 +1 @@
+See [cve-2019-9900.md](cve-2019-9900.md)