Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance external dependency documentation #12673

Closed
9 of 11 tasks
htuch opened this issue Aug 16, 2020 · 2 comments · Fixed by #13571
Closed
9 of 11 tasks

Enhance external dependency documentation #12673

htuch opened this issue Aug 16, 2020 · 2 comments · Fixed by #13571
Assignees
@htuch
Labels
area/build area/docs area/security enhancement Feature requests. Not bugs or questions. no stalebot Disables stalebot from closing an issue

Comments

@htuch
Copy link
Member

htuch commented Aug 16, 2020 •
edited
Loading

#12639 introduced a dashboard documenting our external dependencies and their version/CPE. There's a number of followup improvements that this issue tracks:

  • Add project description field that is human readable.
  • Improve handling of version strings that require mangling to different forms, see docs: external dependency dashboard. #12639 (comment).
  • Split out metadata and join with the labels produced by bazel query //external:all --output
  • Link GH dependencies directly to the GH tree at the correct tag/SHA.
  • Include api/bazel/repository_locations.py
  • Include bazel/dependency_imports.bzl, possibly UDPA deps.
  • Track "last updated" status.
  • Distinguish core vs. extension dependencies. Ideally this is done using bazel query smarts, so that we can get fine grained and live information, e.g. which extensions are using which deps over time.
  • Include version in CPE links
  • Update CPE metadata where missing.
  • Update use_category where incorrect, there are some missing dataplane and controlplane tags on some deps.
@htuch htuch added area/build area/docs area/security enhancement Feature requests. Not bugs or questions. labels Aug 16, 2020
@htuch htuch self-assigned this Aug 16, 2020
@htuch htuch mentioned this issue Aug 16, 2020
Merged
htuch added a commit that referenced this issue Aug 18, 2020
@htuch
docs: external dependency dashboard. (#12639)
b0e391f 
This patch introduces a set of automatically generated tables (based on repository_locations.bzl)
that enumerate the external dependencies that feature on Envoy's data/control planes, test, build, etc.

Version and CPE information is currently included. In the future, we will also have last updated,
distinguish core vs. extensions and populate with external dependency process maturity information.

Part of #10471. This is essentially providing a programmatic variant of #10471 (comment). Future enhancements are
tracked at #12673.

Signed-off-by: Harvey Tuch <htuch@google.com>
stedsome pushed a commit to stedsome/envoy that referenced this issue Aug 22, 2020
manually add the build flag
ce96eef 
Signed-off-by: Yifan Yang <needyyang@google.com>

the introduction of post processing script

Signed-off-by: Yifan Yang <needyyang@google.com>

add the macos dependency installation

Signed-off-by: Yifan Yang <needyyang@google.com>

install macos dependency

Signed-off-by: Yifan Yang <needyyang@google.com>

comment out the slack functionality

Signed-off-by: Yifan Yang <needyyang@google.com>

introduce flaky tests

Signed-off-by: Yifan Yang <needyyang@google.com>

formatting the flaky tests

Signed-off-by: Yifan Yang <needyyang@google.com>

commentout the format checks for debugging

Signed-off-by: Yifan Yang <needyyang@google.com>

configuration

Signed-off-by: Yifan Yang <needyyang@google.com>

configuration

Signed-off-by: Yifan Yang <needyyang@google.com>

configuration

Signed-off-by: Yifan Yang <needyyang@google.com>

configuration

Signed-off-by: Yifan Yang <needyyang@google.com>

more configuration

Signed-off-by: Yifan Yang <needyyang@google.com>

debugging

Signed-off-by: Yifan Yang <needyyang@google.com>

install slack package for real

Signed-off-by: Yifan Yang <needyyang@google.com>

add_condition

Signed-off-by: Yifan Yang <needyyang@google.com>

try to install slack

Signed-off-by: Yifan Yang <needyyang@google.com>

changed find path

Signed-off-by: Yifan Yang <needyyang@google.com>

trying to locate the test result folders

Signed-off-by: Yifan Yang <needyyang@google.com>

trying to find the test folders

Signed-off-by: Yifan Yang <needyyang@google.com>

trying to find the test folders

Signed-off-by: Yifan Yang <needyyang@google.com>

trying to find the test folders

Signed-off-by: Yifan Yang <needyyang@google.com>

trying to find the test folders

Signed-off-by: Yifan Yang <needyyang@google.com>

trying to find the test folders

Signed-off-by: Yifan Yang <needyyang@google.com>

finding test files

Signed-off-by: Yifan Yang <needyyang@google.com>

finding test files

Signed-off-by: Yifan Yang <needyyang@google.com>

finding test files

Signed-off-by: Yifan Yang <needyyang@google.com>

finding test files

Signed-off-by: Yifan Yang <needyyang@google.com>

finding test files

Signed-off-by: Yifan Yang <needyyang@google.com>

finding test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

find the test files

Signed-off-by: Yifan Yang <needyyang@google.com>

python dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

python dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

python dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

python dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

python dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

python dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

python dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

testing coverage builds

Signed-off-by: Yifan Yang <needyyang@google.com>

testing coverage builds

Signed-off-by: Yifan Yang <needyyang@google.com>

try to figure out coverage builds

Signed-off-by: Yifan Yang <needyyang@google.com>

comment out slack

Signed-off-by: Yifan Yang <needyyang@google.com>

testing coverage build

Signed-off-by: Yifan Yang <needyyang@google.com>

testing coverage build

Signed-off-by: Yifan Yang <needyyang@google.com>

trying slack features

Signed-off-by: Yifan Yang <needyyang@google.com>

more formatting

Signed-off-by: Yifan Yang <needyyang@google.com>

slack

Signed-off-by: Yifan Yang <needyyang@google.com>

adding CI_Target to output msg

Signed-off-by: Yifan Yang <needyyang@google.com>

force a run

Signed-off-by: Yifan Yang <needyyang@google.com>

fuzz: added fuzz test for listener filter tls_inspector (envoyproxy#12617)

Created tls_inspector_corpus and populated with testcases (valid and invalid client hellos)

Risk Level: Low
Testing: increased function coverage of tls_inspector.cc to 100.0% and line coverage to 87.3% after running fuzzer (covers all parse states except errors related to socket read failure).
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Arthur Yan <arthuryan@google.com>

scoped_rds_integration_test migrate from api v2 to api v3. (envoyproxy#12633)

Migrate the integration test of scoped rds from api v2 to api v3. Fix a bug in scoped_rds.cc: ScopedRdsConfigSubscription should use the resource version of srds, not the resource version of rds.

Risk Level: Low

Signed-off-by: chaoqinli <chaoqinli@google.com>

add 'explicit' restriction. (envoyproxy#12643)

Commit Message: The intent, when providing Stats::Utility::counterFromElements, is that dynamic segments should be easy to construct, but still searchable. We should be trying to avoid dynamic segments whenever possible, so having them implicitly created from string data is not idea.
Additional Description:
Risk Level: none for the repo, but possibly will require trivial edits outside the repo
Testing: //test/...
Docs Changes: n/a
Release Notes: n/a

Signed-off-by: Joshua Marantz <jmarantz@google.com>

[fuzz]added an input check in writefilter fuzzer and added test cases (envoyproxy#12628)

Added a handle for nullptr in HeaderPercentageProvider::percentage to avoid crash in mongo_proxy.
Added many unit test cases into corpus so that the coverage can be improved. All those filters' coverage was increased by 20%-40%.
Signed-off-by: jianwen <jianwendong@google.com>

test: fix http_timeout_integration_test flake (envoyproxy#12654)

Fixes envoyproxy#12653

Signed-off-by: Matt Klein <mklein@lyft.com>

logger: support log control in admin interface and command line option for Fancy Logger (envoyproxy#12369)

Add log control (list and modify log level) in admin interface for Fancy Logger, a new fine-grained logger for Envoy, and provide command line option --enable-fine-grain-logging for developers.

Additional Description: A doc of overview is provided here: source/docs/fancy_logger.md.
Risk Level: Medium
Testing: Unit tests.
Docs Changes: Added a new option --enable-fine-grain-logging and doc it.
Release Notes: Added to current.rst.

Signed-off-by: Jinhui Song <jinhuisong@google.com>

Decreased the flakiness of Watchdog tests running real time system. (envoyproxy#12659)

Signed-off-by: Kevin Baichoo <kbaichoo@google.com>

test: fix ext auth flake (envoyproxy#12660)

Fixes envoyproxy#12657

Signed-off-by: Matt Klein <mklein@lyft.com>

test: deflake timer test, not completely (envoyproxy#12642)

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

test: fix ProtocolIntegrationTest.LargeRequestMethod flake (envoyproxy#12661)

Fixes envoyproxy#12484

Signed-off-by: Matt Klein <mklein@lyft.com>

Fix proto_sync.py (envoyproxy#12434)

Fix path in proto_sync.py generated comments and regenerate.

Signed-off-by: wzshiming <wzshiming@foxmail.com>

udp: Add some log when session is deleted (envoyproxy#12669)

It is very helpful for debugging.

Signed-off-by: DongRyeol Cha <dr83.cha@samsung.com>

DNS filter: set default resolver timeout (envoyproxy#12293)

Fix an ASAN failure in certain env.

Risk Level: low
Testing: n/a
Docs Changes: n/a
Release Notes: n/a
Signed-off-by: Yuchen Dai <silentdai@gmail.com>

http: remove deprecated envoy.reloadable_features.connection_header_sanitization runtime guard (envoyproxy#12500)

http: Remove deprecated envoy.reloadable_features.connection_header_sanitization runtime guard

Signed-off-by: Alvin Baptiste alvinsb@gmail.com

Commit Message: http: remove deprecated runtime guard for connection header sanitization
Risk Level: Low
Testing: bazel test //test/...
Release Notes: Added
Fixes envoyproxy#11933
Removed: envoy.reloadable_features.connection_header_sanitization

Signed-off-by: Alvin Baptiste <alvinsb@gmail.com>

Fix broken codeb lock style (envoyproxy#12667)

Fix broken code block style in docs
Signed-off-by: Takao Shibata <chise.alter.pasta@gmail.com>

lua: Manage imported public keys in stream handle (envoyproxy#12664)

This patch manages the imported public keys in the stream-handle object instead of "exposing" it as pointer through lua_pushlightuserdata while preserving the current Lua APIs.

Signed-off-by: Dhi Aurrahman <dio@tetrate.io>

lua API: add base64Escape function to stream handle (envoyproxy#12552)

This makes it easy for Lua filters to base64 escape strings without
needing to provide their own base64 helper.

Signed-off-by: Michael Puncel <mpuncel@squareup.com>

[Windows] Fixes Udp listener tests (envoyproxy#12635)

Fixes UDP listener tests on Windows by modernizing iovecToWSABUF and msghdrToWSAMSG from pointer arithmetic to c++ and makes message.msg_controllen the proper length.

Signed-off-by: Sotiris Nanopoulos <sonanopo@microsoft.com>

api: deprecate the node.listening_addresses field (envoyproxy#12691)

This was added for gRPC server support, but we've decided to use resource names instead to explicitly request the listeners we want by name. This is more in-line with the new naming scheme described in the "xDS Transport Next Steps" design.

Signed-off-by: Mark D. Roth <roth@google.com>

Fix broken reST style (envoyproxy#12668)

Signed-off-by: Takao Shibata <chise.alter.pasta@gmail.com>

Windows: Fix filesystem subscription impl tests (envoyproxy#12597)

Windows: Fix filesystem subscription impl tests

Replace dispatcher and watcher with mocks and orchestrate the test
harness to capture filesystem subscription impl file event callback and
invoke it directly. Test no longer relies on watching real filesystem
events and avoids timing discrepancies causing flakiness.

Additional Description: N/A
Risk Level: Low
Testing: Modifies unit tests, tested locally on Windows
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Sunjay Bhatia <sunjayb@vmware.com>
Co-authored-by: William A Rowe Jr <wrowe@vmware.com>

Fixes stack overflow in http inspector test (envoyproxy#12577)

Fixes stack overflow exception in HttpInspectorTest.Http1WithLargeRequestLine and makes the test faster.

Additional Description:
While I was working on level vs edge based events I observed that the test is causing a stack overflow on Windows/MSVC.

The testlist http_inspector_test now passes on Windows but it should not because it relies on Event::FileTriggerType::Edge which are not supported. This is why I did not enable it for the CI.

Risk Level: N/A
Testing: N/A
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Sotiris Nanopoulos <sonanopo@microsoft.com>

docs: external dependency dashboard. (envoyproxy#12639)

This patch introduces a set of automatically generated tables (based on repository_locations.bzl)
that enumerate the external dependencies that feature on Envoy's data/control planes, test, build, etc.

Version and CPE information is currently included. In the future, we will also have last updated,
distinguish core vs. extensions and populate with external dependency process maturity information.

Part of envoyproxy#10471. This is essentially providing a programmatic variant of envoyproxy#10471 (comment). Future enhancements are
tracked at envoyproxy#12673.

Signed-off-by: Harvey Tuch <htuch@google.com>

[fuzz]expand readfilter_fuzzer to cover mongo_proxy and mysql_proxy (envoyproxy#12612)

Added coverage for mongo_proxy and mysql_proxy
Added test cases(corpus) for them.
Signed-off-by: jianwen <jianwendong@google.com>

xds: allow updating listener back to original state (envoyproxy#12645)

since addOrUpdateListenerInternal returns early in the case of a duplicate active/warming listener being added, it means you cannot update a listener back to its original state after updating it to a warming state

consider the following sequence of actions:
* add listener_0 referencing route_config_0, make it active
* update listener_0 referencing route_config_1, keep it warming (Envoy keeps the original listener_0 active until the new warms)
* update listener_0 back to route_config_0, which should remove the warming listener and return Envoy to its initial state, so that a future addition of route_config_1 won't cause the listener to change to that state, but right now it doesn't make a small change in listener_manager_impl.cc to allow this to happen

update xds_verifier.cc and add a unit test for this case

Risk Level: Medium
Testing: Passes existing fuzz corpora and a few minutes on libfuzzer, passes ads_integration_test and lds_api_test
Docs Changes: N/A
Release Notes: N/A
Fixes: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=24888

Signed-off-by: Sam Flattery <samflattery@google.com>

ci: refactor docker ci script and enable docker job in presubmit (envoyproxy#12662)

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

[test] using reserve and commit in watermark_buffer overflow high value test (envoyproxy#12494)

Improving the test by avoiding actual copying of bytes, but just reserving and committing slices.

Signed-off-by: Adi Suissa-Peleg <adip@google.com>

Greenlight and reclassify //test/... with the current progress for Windows (envoyproxy#12695)

Co-authored-by: William A Rowe Jr <wrowe@vmware.com>
Co-authored-by: Sunjay Bhatia <sunjayb@vmware.com>
Signed-off-by: William A Rowe Jr <wrowe@vmware.com>
Signed-off-by: Sunjay Bhatia <sunjayb@vmware.com>

hds: group endpoint health response by cluster and locality (envoyproxy#12452)

Currently, the health discovery service takes a specifier with information about which endpoints to perform health checks on, which already supports information about cluster structure and locality information. However, when when forming a response holding endpoint health information, all notion of cluster or locality is dropped and endpoint information is stored in a flat list. This is problematic when there are several endpoints with the same address or port but with a different locality, cluster, or path.

This PR uses the previous proto change in Issue envoyproxy#12153 which added support for grouping endpoint health information by their respective cluster and locality.

Risk Level: Low
Testing: Added a unit test in test/common/upstream/hds_test, which sends a specifier to HdsDelegate with several clusters, localties, and endpoints. It then verifies that the response holds the same structure. Existing integration tests were also changed to check for the new proto structure, specifically ones that already group several endpoints by differing clusters or localities.

Signed-off-by: Drew S. Ortega <drewortega@google.com>

network: socket and address build cleanup (envoyproxy#12710)

- split socket interface from socket
- add default socket interface library
- move io handle to default socket interface library from address

Signed-off-by: Florin Coras <fcoras@cisco.com>

thrift: envoy_cc_test -> envoy_extension_cc_test (envoyproxy#12697)

Signed-off-by: Roelof DuToit <roelof.dutoit@broadcom.com>

test: Add test socket interface that allows overriding IoHandle behavior (envoyproxy#12528)

Add test socket interface that allows overriding IoHandle behavior of accepted sockets.
Change flood tests to use exact frame counts needed for flooding.
Fix DATA frame flood test.

Signed-off-by: Yan Avlasov <yavlasov@google.com>

fix main branch merge issue (envoyproxy#12722)

Signed-off-by: Florin Coras <fcoras@cisco.com>

ci: fix VRP image push (envoyproxy#12715)

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

cosmetic changes

Signed-off-by: Yifan Yang <needyyang@google.com>

add a flaky test in macOS build

Signed-off-by: Yifan Yang <needyyang@google.com>

add a flaky test in macOS build

Signed-off-by: Yifan Yang <needyyang@google.com>

changed the flaky test

Signed-off-by: Yifan Yang <needyyang@google.com>

changed the flaky test

Signed-off-by: Yifan Yang <needyyang@google.com>

have a wrapper script

Signed-off-by: Yifan Yang <needyyang@google.com>

trying necessary dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

trying necessary dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

testing python dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

add a shell script wrapper and use python_venv

Signed-off-by: Yifan Yang <needyyang@google.com>

moved functionality into do_ci.sh and added the hyperlink

Signed-off-by: Yifan Yang <needyyang@google.com>

moved functionality into do_ci.sh and added the hyperlink

Signed-off-by: Yifan Yang <needyyang@google.com>

put it out of ci_do.sh

Signed-off-by: Yifan Yang <needyyang@google.com>

put it out of ci_do.sh

Signed-off-by: Yifan Yang <needyyang@google.com>

put it out of ci_do.sh

Signed-off-by: Yifan Yang <needyyang@google.com>

ci_do.sh

Signed-off-by: Yifan Yang <needyyang@google.com>

ci_do.sh

Signed-off-by: Yifan Yang <needyyang@google.com>

ci_do.sh

Signed-off-by: Yifan Yang <needyyang@google.com>

more setup

Signed-off-by: Yifan Yang <needyyang@google.com>

more setup

Signed-off-by: Yifan Yang <needyyang@google.com>

more setup

Signed-off-by: Yifan Yang <needyyang@google.com>

more setup

Signed-off-by: Yifan Yang <needyyang@google.com>

more setup

Signed-off-by: Yifan Yang <needyyang@google.com>

more setup

Signed-off-by: Yifan Yang <needyyang@google.com>

more setup

Signed-off-by: Yifan Yang <needyyang@google.com>

test hyperlink

Signed-off-by: Yifan Yang <needyyang@google.com>

installing with py3

Signed-off-by: Yifan Yang <needyyang@google.com>

making it work

Signed-off-by: Yifan Yang <needyyang@google.com>

pass the CI-target in

Signed-off-by: Yifan Yang <needyyang@google.com>

changing the search path

Signed-off-by: Yifan Yang <needyyang@google.com>

changing the search path

Signed-off-by: Yifan Yang <needyyang@google.com>

add a set to keep track of seen problem

Signed-off-by: Yifan Yang <needyyang@google.com>

fix of something stupid

Signed-off-by: Yifan Yang <needyyang@google.com>

add some comments and retry arm64

Signed-off-by: Yifan Yang <needyyang@google.com>

getting it ready for shipping

Signed-off-by: Yifan Yang <needyyang@google.com>

format

Signed-off-by: Yifan Yang <needyyang@google.com>

more format

Signed-off-by: Yifan Yang <needyyang@google.com>

format

Signed-off-by: Yifan Yang <needyyang@google.com>

try again with arm64

Signed-off-by: Yifan Yang <needyyang@google.com>

testing arm build

Signed-off-by: Yifan Yang <needyyang@google.com>

update requirement for arm

Signed-off-by: Yifan Yang <needyyang@google.com>

trying out arm dependencies

Signed-off-by: Yifan Yang <needyyang@google.com>

debugging

Signed-off-by: Yifan Yang <needyyang@google.com>

opting out of arm arch for now

Signed-off-by: Yifan Yang <needyyang@google.com>

cleanup

Signed-off-by: Yifan Yang <needyyang@google.com>

commentout formatting for faster builds

Signed-off-by: Yifan Yang <needyyang@google.com>

upgrading setuptools

Signed-off-by: Yifan Yang <needyyang@google.com>

dependency

Signed-off-by: Yifan Yang <needyyang@google.com>

dependency

Signed-off-by: Yifan Yang <needyyang@google.com>

dependency

Signed-off-by: Yifan Yang <needyyang@google.com>

dependency

Signed-off-by: Yifan Yang <needyyang@google.com>

dependency

Signed-off-by: Yifan Yang <needyyang@google.com>

dependency

Signed-off-by: Yifan Yang <needyyang@google.com>

dependency

Signed-off-by: Yifan Yang <needyyang@google.com>

dependency

Signed-off-by: Yifan Yang <needyyang@google.com>

sanity check

Signed-off-by: Yifan Yang <needyyang@google.com>
@stale
Copy link

stale bot commented Sep 20, 2020

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

@stale stale bot added the stale stalebot believes this issue/PR has not been touched recently label Sep 20, 2020
@dio dio added no stalebot Disables stalebot from closing an issue and removed stale stalebot believes this issue/PR has not been touched recently labels Sep 20, 2020
htuch added a commit to htuch/envoy that referenced this issue Sep 24, 2020
@htuch
dependencies: last_updated and project_desc metadata.
515bbe3 
* Added mandatory last_updated metadata to track when a dependency was
  last updated. Backfilled existing dependencies based primarily on
  when the dependency was released, and in some cases based on Envoy
  git history.

* Added project_desc metadata to provide a human understandable
  statement about projects.

* Updated CPEs using the latest CPE database. There were a number of
  dataplane components with missing CPEs.

* Corrected use_category for a number of components.

Risk level: Low
Testing: Build, dashboard inspection.

Part of envoyproxy#12673.

Signed-off-by: Harvey Tuch <htuch@google.com>
@htuch htuch mentioned this issue Sep 24, 2020
Merged
htuch added a commit that referenced this issue Sep 25, 2020
@htuch
dependencies: last_updated and project_desc metadata. (#13263)
47dae09 
* Added mandatory last_updated metadata to track when a dependency was
  last updated. Backfilled existing dependencies based primarily on
  when the dependency was released, and in some cases based on Envoy
  git history.

* Added project_desc metadata to provide a human understandable
  statement about projects.

* Updated CPEs using the latest CPE database. There were a number of
  dataplane components with missing CPEs.

* Corrected use_category for a number of components.

Risk level: Low
Testing: Build, dashboard inspection.

Part of #12673.

Signed-off-by: Harvey Tuch <htuch@google.com>
htuch added a commit to htuch/envoy that referenced this issue Sep 30, 2020
@htuch
dependencies: associate extensions with deps, validate use_category.
e281064 
This PR introduces a few related changes:

* use_category is restructured to distinguish core/extension deps. There's also an extension
  allowlist added for each dependency in the dataplane_ext and observability_ext category.

* tools/dependency/validate.py is introduced to validate a bunch of structural relationships
  implied by the bazel/repository_locations.bzl metadata. This includes that test-only deps
  aren't used in //source/.., that some obvious dataplane/controlplane packages taint the
  appropriate reachable deps and that the association between extensions/deps holds.

* The CI docs jobs now runs tools/dependency/validate.py.

* The dependency dashboard at
  https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/external_deps
  to include granular information on which dataplane/observability deps come from core/extensions.

* Some misc. cleanup of source code dep that came up while working on this.

Risk level: Low
Testing: Pyunit tests added for validate.py.

Part of envoyproxy#12673.

Signed-off-by: Harvey Tuch <htuch@google.com>
@htuch htuch mentioned this issue Sep 30, 2020
Merged
htuch added a commit that referenced this issue Oct 1, 2020
@htuch
dependencies: associate extensions with deps, validate use_category. (#…
2550806 
…13340)

This PR introduces a few related changes:

* use_category is restructured to distinguish core/extension deps. There's also an extension
  allowlist added for each dependency in the dataplane_ext and observability_ext category.

* tools/dependency/validate.py is introduced to validate a bunch of structural relationships
  implied by the bazel/repository_locations.bzl metadata. This includes that test-only deps
  aren't used in //source/.., that some obvious dataplane/controlplane packages taint the
  appropriate reachable deps and that the association between extensions/deps holds.

* The CI docs jobs now runs tools/dependency/validate.py.

* The dependency dashboard at
  https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/external_deps
  to include granular information on which dataplane/observability deps come from core/extensions.

* Some misc. cleanup of source code dep that came up while working on this.

Risk level: Low
Testing: Pyunit tests added for validate.py.

Part of #12673.

Signed-off-by: Harvey Tuch <htuch@google.com>
htuch added a commit to htuch/envoy that referenced this issue Oct 8, 2020
@htuch
dependencies: enhance CVE and version links.
9bcc59f 
* CPE link now performs NIST search on all CVEs, rather than linking to
  a collection of version CPEs and requiring the user to click on each
  for CVEs.

* Version links now point to GitHub release tag pages or the GitHub
  tree at a particular hash for SHA versions. Previously this was just
  the tarball download. For non-GitHub blobs, e.g. on GCS, we still just
  provide the tarball download link.

Risk level: Low
Testing: Manual clicking on links, seems to work for NIST CVEs and a
  wide number of sampled tagged release versions.

Part of envoyproxy#12673

Signed-off-by: Harvey Tuch <htuch@google.com>
@htuch htuch mentioned this issue Oct 8, 2020
Merged
htuch added a commit that referenced this issue Oct 8, 2020
@htuch
dependencies: enhance CVE and version links. (#13434)
c9aca3c 
* CPE link now performs NIST search on all CVEs, rather than linking to
  a collection of version CPEs and requiring the user to click on each
  for CVEs.

* Version links now point to GitHub release tag pages or the GitHub
  tree at a particular hash for SHA versions. Previously this was just
  the tarball download. For non-GitHub blobs, e.g. on GCS, we still just
  provide the tarball download link.

Risk level: Low
Testing: Manual clicking on links, seems to work for NIST CVEs and a
  wide number of sampled tagged release versions.

Part of #12673

Signed-off-by: Harvey Tuch <htuch@google.com>
@htuch htuch mentioned this issue Oct 8, 2020
Merged
htuch added a commit to htuch/envoy that referenced this issue Oct 8, 2020
@htuch
dependencies: refactor repository location schema utils, cleanups.
dc65abb 
* Refactor code responsible for processing repository location specs,
  i.e. checking for the presence of fields like last_updated and
  interpolation of version. The same code is now usable by both
  API repository_locations.bzl and bazel/repository_locations.bzl.

* Cleanup reference to repo locations in repository_locations.bzl,
  now using a consistent set of macros.

* Add API dependencies to dependency dashboard.

Risk level: Low
Testing: Docs build.

Part of envoyproxy#12673

Signed-off-by: Harvey Tuch <htuch@google.com>
@htuch
Copy link
Member Author

htuch commented Oct 8, 2020

Not planning on tackling the following:

  • Split out metadata and join with the labels produced by bazel query //external:all --output. The same effect is achieved via inplace metadata and tools/dependency/validate.py ensuring consistency.
  • Include bazel/dependency_imports.bzl, possibly UDPA deps.. The Go and pip3 dependencies aren't in the build. Transitive dependencies will be called out explicitly but a redirect added to the implying direct dependency.

htuch added a commit that referenced this issue Oct 13, 2020
@htuch
dependencies: refactor repository location schema utils, cleanups. (#…
9181790 
…13452)

- Refactor code responsible for processing repository location specs, i.e. checking for the presence of fields like last_updated and interpolation of version. The same code is now usable by both API repository_locations.bzl and bazel/repository_locations.bzl.

- Cleanup reference to repo locations in repository_locations.bzl, now using a consistent set of macros.

- Add API dependencies to dependency dashboard.

Risk level: Low
Testing: Docs build.

Part of #12673

Signed-off-by: Harvey Tuch <htuch@google.com>
@htuch htuch closed this as completed Oct 13, 2020
htuch added a commit to htuch/envoy that referenced this issue Oct 14, 2020
@htuch
dependencies: track untracked implied dependencies, wrapup dashboard.
5958383 
A few changes that wrapup envoyproxy#12673.

* Python/Go dependencies that aren't part of the Envoy binary build
  don't make sense to track in repository_locations.bzl, since they
  have their own language specific metadata (e.g. requirements.txt)
  or are in many cases transitively implied.

* Ensure that the full set of dependencies visible to bazel query
  is now validated. This requires that we explicitly call out
  transitive dependencies that are implied by direct dependencies
  in repository_locations.bzl. A new annotation `implied_untracked_deps`
  is used.

Fixes envoyproxy#12673

Risk level: Low
Testing: validate.py.

Signed-off-by: Harvey Tuch <htuch@google.com>
@htuch htuch mentioned this issue Oct 14, 2020
Merged
htuch added a commit that referenced this issue Oct 16, 2020
@htuch
dependencies: track untracked implied dependencies, wrapup dashboard. (…
499f46a 
…#13571)

A few changes that wrapup #12673.

* Python/Go dependencies that aren't part of the Envoy binary build
  don't make sense to track in repository_locations.bzl, since they
  have their own language specific metadata (e.g. requirements.txt)
  or are in many cases transitively implied.

* Ensure that the full set of dependencies visible to bazel query
  is now validated. This requires that we explicitly call out
  transitive dependencies that are implied by direct dependencies
  in repository_locations.bzl. A new annotation `implied_untracked_deps`
  is used.

Fixes #12673

Risk level: Low
Testing: validate.py.

Signed-off-by: Harvey Tuch <htuch@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@htuch htuch

Labels
area/build area/docs area/security enhancement Feature requests. Not bugs or questions. no stalebot Disables stalebot from closing an issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

dependencies: track untracked implied dependencies, wrapup dashboard. htuch/envoy
2 participants
@dio @htuch