Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stats: integrate real symbol table into stats system #4980

Merged
merged 186 commits into from
Jun 28, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
186 commits
Select commit Hold shift + click to select a range
ed31d09
checkpoint
jmarantz Oct 31, 2018
1cd7a7b
got heap_stat_data_test working.
jmarantz Nov 2, 2018
d62f533
all tests working; but still taking locks in hot-path.
jmarantz Nov 7, 2018
b3cad8a
format
jmarantz Nov 7, 2018
63138e8
Merge branch 'master' into integration-symtab
jmarantz Nov 7, 2018
524cf02
format
jmarantz Nov 7, 2018
bb84b90
fix botched merge.
jmarantz Nov 7, 2018
f9c0441
perf test and tweaks
jmarantz Nov 7, 2018
7c5ac2f
formatting, comments, and tests.
jmarantz Nov 7, 2018
5e1c713
Merge branch 'master' into integration-symtab
jmarantz Nov 7, 2018
bdbf340
use std::make_unique for clang-tidy
jmarantz Nov 8, 2018
d70cb18
Make a class for Http::CodeUtility::chargeResponseTiming et al, plumb…
jmarantz Nov 8, 2018
9c49dec
Use join() rather than fmt::format, and hold static strings in string…
jmarantz Nov 8, 2018
df7d7a1
fix router test (empty prefix) and use const string_view for member v…
jmarantz Nov 8, 2018
da648b3
add speed-test.
jmarantz Nov 8, 2018
ec66aaf
Remove libraries not needed by speed test.
jmarantz Nov 8, 2018
461471a
Merge branch 'master' into integration-symtab
jmarantz Nov 8, 2018
32ff476
Merge branch 'http-code-stats-as-object' into integration-symtab
jmarantz Nov 8, 2018
b711b1b
checkpoint
jmarantz Nov 8, 2018
7723309
checkpoint
jmarantz Nov 9, 2018
a892f70
partially symbolize http-response-code stats.
jmarantz Nov 9, 2018
5fadb96
Virtualize SymbolTable (but not symbols or StatName) for use in mocks…
jmarantz Nov 11, 2018
3dacb8e
Merge branch 'master' into integration-symtab
jmarantz Nov 11, 2018
aaca920
Pre-allocate symbols for 200s and 404s, alloc file-system stats per f…
jmarantz Nov 12, 2018
e25f7cd
Finishing plumbing up Filesystem::Instance as an object via API and g…
jmarantz Nov 13, 2018
ec78dca
Merge branch 'master' into integration-symtab
jmarantz Nov 13, 2018
ae0d345
format
jmarantz Nov 13, 2018
1727daa
Merge branch 'master' into integration-symtab
jmarantz Nov 14, 2018
739bf06
revert API change - make an IsolatedStoreImpl in the RawConnectionDri…
jmarantz Nov 14, 2018
8336650
force the SymbolTable backing MockIsolatedStore to be a singleton.
jmarantz Nov 16, 2018
c17cca9
format
jmarantz Nov 16, 2018
04cbc9c
comment out singleton symbol-table usage in MockIsolatedStatsStore.
jmarantz Nov 16, 2018
101e913
use globals
jmarantz Nov 21, 2018
09d1b6e
merge in globals
jmarantz Nov 21, 2018
cee54fe
move file placed in wrong dir
jmarantz Nov 21, 2018
6883d6d
add missing file.
jmarantz Nov 21, 2018
c353a34
Merge branch 'master' into integration-symtab
jmarantz Nov 29, 2018
1601f6f
Merge branch 'master' into integration-symtab
jmarantz Nov 29, 2018
ac03dd9
All tests working again.
jmarantz Nov 30, 2018
0a2fbe8
format
jmarantz Nov 30, 2018
2c92ce7
Merge branch 'master' into integration-symtab
jmarantz Dec 6, 2018
4bb7c0d
format
jmarantz Dec 6, 2018
b7acdd5
prune dead code.
jmarantz Dec 6, 2018
6b4cbb8
Merge branch 'master' into integration-symtab
jmarantz Dec 10, 2018
0cf3f92
Merge branch 'master' into integration-symtab
jmarantz Dec 13, 2018
0eb83f6
Merge branch 'master' into integration-symtab
jmarantz Dec 15, 2018
d087924
Merge branch 'master' into integration-symtab
jmarantz Dec 17, 2018
af1d8d0
Merge branch 'master' into integration-symtab
jmarantz Dec 17, 2018
4014dde
Remove the symbol-table interface; just call symbol table directly.
jmarantz Dec 18, 2018
346e9e5
move toString to SymbolTable.
jmarantz Dec 18, 2018
1343181
Merge branch 'master' into integration-symtab
jmarantz Dec 25, 2018
889f819
format
jmarantz Dec 25, 2018
cc87f92
fix tsan error (from a stale version of a test due to a botched git m…
jmarantz Dec 25, 2018
f81f794
Re-order member variables in the hope of improving test flakes due to…
jmarantz Dec 27, 2018
42bd6b8
Merge branch 'master' into integration-symtab
jmarantz Dec 27, 2018
c375cf6
Merge branch 'master' into integration-symtab
jmarantz Dec 28, 2018
5fe26f5
Move the aggregated stat-name storage mechanism out of MetricImpl and…
jmarantz Jan 27, 2019
3bdbdc2
Merge branch 'master' into integration-symtab
jmarantz Jan 27, 2019
0b39814
Got compile to work.
jmarantz Jan 27, 2019
665897f
Get tests to work.
jmarantz Jan 28, 2019
7c9471e
Merge branch 'master' into integration-symtab
jmarantz Jan 28, 2019
ef23a21
Merge branch 'master' into integration-symtab
jmarantz Feb 3, 2019
40046c7
Merge branch 'master' into integration-symtab
jmarantz Feb 3, 2019
1028dbe
format
jmarantz Feb 3, 2019
45ef917
all tests passing.
jmarantz Mar 3, 2019
fd8db0e
Use FakeSymbolTable in prod instances.
jmarantz Mar 3, 2019
c6ec598
format
jmarantz Mar 3, 2019
88f1f4d
Use FakeSymbolTableImpl in most tests too (except the SymbolTable uni…
jmarantz Mar 4, 2019
94c5205
Don't pass SymbolTable in to stat Mocks but use the global; factor ou…
jmarantz Mar 4, 2019
b8cbed3
Remove comments and dead code.
jmarantz Mar 4, 2019
d53d7f6
Merge branch 'master' into integration-symtab
jmarantz Mar 4, 2019
4a0d37d
Go back to using real symbol tables in this client, and fix a content…
jmarantz Mar 6, 2019
b2a1c9f
Merge branch 'master' into integration-symtab (hystrix_test fails)
jmarantz Mar 9, 2019
8bd401b
format
jmarantz Mar 9, 2019
494f366
All tests working.
jmarantz Mar 9, 2019
aa60638
spelling
jmarantz Mar 9, 2019
8626e51
Merge branch 'master' into integration-symtab
jmarantz Mar 11, 2019
9ffa66f
Merge branch 'master' into integration-symtab
jmarantz Mar 11, 2019
b312ecd
Backed out a few changes that are not needed.
jmarantz Mar 15, 2019
a0f0c9e
simplify the router changes.
jmarantz Mar 15, 2019
156434e
Clean up router changes slightly.
jmarantz Mar 15, 2019
aa6ecff
checkpoint merge; does not compile.
jmarantz Apr 4, 2019
54d9307
Merge branch 'master' into integration-symtab
jmarantz Apr 4, 2019
9e13376
everything compiles, all tests pass except StatsThreadLocalStoreTest.…
jmarantz Apr 6, 2019
59a36d7
Merge branch 'master' into integration-symtab
jmarantz Apr 7, 2019
33f36f3
format
jmarantz Apr 7, 2019
64d94fa
Fix tests and resync with #6161 and #6504
jmarantz Apr 7, 2019
b1a7db0
Merge branch 'master' into integration-symtab
jmarantz Apr 8, 2019
a57a859
Merge branch 'master' into integration-symtab
jmarantz Apr 10, 2019
5ad3312
Merge branch 'master' into integration-symtab
jmarantz Apr 22, 2019
4f48459
Merge branch 'master' into integration-symtab
jmarantz Apr 22, 2019
7082921
Merge branch 'master' into integration-symtab
jmarantz Apr 26, 2019
4c89a6f
fix functional issues from merge.
jmarantz Apr 26, 2019
788efb6
Merge branch 'master' into integration-symtab
jmarantz Apr 26, 2019
27b9c06
remove some detritus
jmarantz Apr 26, 2019
d27394c
Merge branch 'master' into integration-symtab
jmarantz May 3, 2019
214d3a0
Merge branch 'master' into integration-symtab
jmarantz May 3, 2019
60b82cf
get build to work again.
jmarantz May 3, 2019
9d2d673
fix merge conflicts.
jmarantz May 3, 2019
9e3986c
Merge branch 'master' into integration-symtab
jmarantz May 3, 2019
49b8078
Remove stale test file.
jmarantz May 6, 2019
3206f30
Merge branch 'master' into integration-symtab
jmarantz May 6, 2019
aa15d73
Merge branch 'master' into integration-symtab
jmarantz May 7, 2019
47a3219
Merge branch 'master' into integration-symtab
jmarantz May 7, 2019
81b9779
Merge branch 'master' into integration-symtab
jmarantz May 10, 2019
f242443
Merge branch 'master' into integration-symtab
jmarantz May 13, 2019
afbb94a
clean up commented-out code.
jmarantz May 13, 2019
dfbe16b
Merge branch 'master' into integration-symtab
jmarantz May 21, 2019
76ac2aa
Merge branch 'master' into integration-symtab
jmarantz May 21, 2019
95c82b2
Skip over empty StatNames.
jmarantz May 21, 2019
64f8099
Remove superfluous changes.
jmarantz May 22, 2019
801b1b6
Remove superfluous comments.
jmarantz May 22, 2019
843a35a
Merge branch 'master' into integration-symtab
jmarantz May 28, 2019
86aa06c
Merge branch 'master' into integration-symtab
jmarantz May 28, 2019
647fd6d
Merge branch 'master' into integration-symtab
jmarantz Jun 18, 2019
ce7417d
get CI mostly working; I think there's an unrelated problem in hotres…
jmarantz Jun 18, 2019
713d97d
format
jmarantz Jun 18, 2019
5980a10
updated memory usage
jmarantz Jun 18, 2019
199ef47
fix comment
jmarantz Jun 18, 2019
137315e
Merge branch 'master' into integration-symtab
jmarantz Jun 27, 2019
8028807
Merge branch 'master' into integration-symtab
jmarantz Jun 27, 2019
fe153be
Merge branch 'master' into integration-symtab
jmarantz Jul 1, 2019
15034e9
Merge branch 'master' into integration-symtab
jmarantz Jul 2, 2019
e0c4711
Merge branch 'master' into integration-symtab
jmarantz Jul 4, 2019
c831e37
cleanup tests.
jmarantz Jul 5, 2019
3cb2b91
fix expected value.
jmarantz Jul 6, 2019
3e931f6
Merge branch 'master' into integration-symtab
jmarantz Jul 16, 2019
1e69649
Merge branch 'master' into integration-symtab
jmarantz Jul 17, 2019
bd95076
Merge branch 'master' into integration-symtab
jmarantz Jul 24, 2019
1016ae6
shared the main symbol-table with the isolated stats used for cluster…
jmarantz Jul 24, 2019
e45e905
Update memory usage to account for isolated stats sharing symbol tables.
jmarantz Jul 25, 2019
e1787cb
pedantic spelling
jmarantz Jul 25, 2019
600977e
update PR #
jmarantz Jul 25, 2019
5f48598
Merge branch 'shared-cluster-isolated-stats' into integration-symtab
jmarantz Jul 25, 2019
44bd5a5
format
jmarantz Jul 25, 2019
9b075bb
Merge branch 'master' into integration-symtab
jmarantz Aug 8, 2019
d4744bd
Add option to switch between fake and real symbol-tables on the comma…
jmarantz Aug 8, 2019
313d591
Add command-line option to switch between real/fake symbol tables, an…
jmarantz Aug 9, 2019
bee7b67
Merge branch 'master' into symbol-table-option
jmarantz Aug 9, 2019
56ab602
use configured symbol-table impl in integration tests.
jmarantz Aug 9, 2019
a0e1d7c
clang-tidy error
jmarantz Aug 9, 2019
92442ef
Cover both real & fake symbol tables in symbol-table memory test.
jmarantz Aug 10, 2019
96f4e85
cleanups
jmarantz Aug 10, 2019
f9c4f82
Merge branch 'master' into symbol-table-option
jmarantz Aug 10, 2019
2a2f1a0
Merge branch 'master' into integration-symtab
jmarantz Aug 10, 2019
53ba649
Merge branch 'symbol-table-option' into integration-symtab
jmarantz Aug 10, 2019
08f444e
convert more tests to use the symbol-table creator rather than hard-c…
jmarantz Aug 10, 2019
ef57468
Merge branch 'symbol-table-option' into integration-symtab
jmarantz Aug 10, 2019
1c08f0a
format
jmarantz Aug 10, 2019
25367f6
use real symbol tables by default.
jmarantz Aug 10, 2019
6551dee
format
jmarantz Aug 10, 2019
8a6521c
swap default
jmarantz Aug 10, 2019
347cb3c
Merge branch 'master' into integration-symtab
jmarantz Aug 21, 2019
a17e17c
fix test
jmarantz Aug 21, 2019
bf6fcd1
back out some earlier attempts at patching in the options PR.
jmarantz Aug 22, 2019
dc690b2
update byte-size golds.
jmarantz Aug 22, 2019
71e0fcd
back out the sharing of symbol tables for upstream cluster isolated s…
jmarantz Aug 22, 2019
f5d1dbb
Merge branch 'master' into integration-symtab
jmarantz Aug 23, 2019
b9a9ccc
Merge branch 'master' into integration-symtab
jmarantz Aug 28, 2019
b9b26d4
Merge branch 'master' into integration-symtab
jmarantz Oct 17, 2019
bf022f1
Merge branch 'master' into integration-symtab
jmarantz Oct 17, 2019
8be749b
Merge branch 'master' into integration-symtab
jmarantz Nov 22, 2019
b855889
Merge branch 'master' into integration-symtab
jmarantz Dec 1, 2019
91f0f92
Merge branch 'master' into integration-symtab
jmarantz Jan 21, 2020
5ff494a
remove superfluous check.
jmarantz Jan 21, 2020
eb79c95
Merge branch 'master' into integration-symtab
jmarantz Mar 18, 2020
b8dfdc0
switch another default to use real symbol tables.
jmarantz Mar 19, 2020
141e711
Merge branch 'master' into integration-symtab
jmarantz Apr 20, 2020
f834e4e
move version history where it belongs.
jmarantz Apr 20, 2020
52eca10
Merge branch 'master' into integration-symtab
jmarantz Apr 26, 2020
7c3ce33
Merge branch 'master' into integration-symtab
jmarantz May 5, 2020
febd650
format
jmarantz May 5, 2020
9c79dc2
Merge branch 'master' into integration-symtab
jmarantz Jun 24, 2020
16077ab
Instantiate a TestSymboLTable so that mocks reference the same one.
jmarantz Jun 25, 2020
dcabf96
reformat with clang-10.
jmarantz Jun 25, 2020
c0e45c6
improve assert message for dynamic/symbolic mismatch, and don't check…
jmarantz Jun 25, 2020
f954e35
clean up the config_test so it has minimal documnted changes from the…
jmarantz Jun 25, 2020
c3bf36d
Merge branch 'master' into integration-symtab
jmarantz Jun 25, 2020
3193beb
Merge branch 'master' into integration-symtab
jmarantz Jun 26, 2020
55e2060
Merge branch 'master' into integration-symtab
jmarantz Jun 26, 2020
4d3043e
Merge branch 'master' into integration-symtab
jmarantz Jun 26, 2020
5695a17
try to avoid race by waiting for a timeout stat rather than just stro…
jmarantz Jun 27, 2020
9d33cce
Declare ads_integregion_test as enormous to try to avoid tsan timeouts.
jmarantz Jun 27, 2020
96e2c6c
remove superfluous line, that probably was from a merge.
jmarantz Jun 28, 2020
141c716
back out accidental commit
jmarantz Jun 28, 2020
fabc533
Merge branch 'master' into integration-symtab
jmarantz Jun 28, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/root/version_history/current.rst
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ New Features
* metrics service: added added :ref:`API version <envoy_v3_api_field_config.metrics.v3.MetricsServiceConfig.transport_api_version>` to explicitly set the version of gRPC service endpoint and message to be used.
* network filters: added a :ref:`postgres proxy filter <config_network_filters_postgres_proxy>`.
* network filters: added a :ref:`rocketmq proxy filter <config_network_filters_rocketmq_proxy>`.
* performance: stats symbol table implementation (enabled by default; to disable it, add
`--use-fake-symbol-table 1` to the command-line arguments when starting Envoy).
* ratelimit: add support for use of dynamic metadata :ref:`dynamic_metadata <envoy_v3_api_field_config.route.v3.RateLimit.Action.dynamic_metadata>` as a ratelimit action.
* ratelimit: added :ref:`API version <envoy_v3_api_field_config.ratelimit.v3.RateLimitServiceConfig.transport_api_version>` to explicitly set the version of gRPC service endpoint and message to be used.
* redis: added acl support :ref:`downstream_auth_username <envoy_v3_api_field_extensions.filters.network.redis_proxy.v3.RedisProxy.downstream_auth_username>` for downstream client ACL authentication, and :ref:`auth_username <envoy_v3_api_field_extensions.filters.network.redis_proxy.v3.RedisProtocolOptions.auth_username>` to configure authentication usernames for upstream Redis 6+ server clusters with ACL enabled.
Expand Down
2 changes: 1 addition & 1 deletion source/common/stats/symbol_table_creator.cc
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ namespace Envoy {
namespace Stats {

bool SymbolTableCreator::initialized_ = false;
bool SymbolTableCreator::use_fake_symbol_tables_ = true;
bool SymbolTableCreator::use_fake_symbol_tables_ = false;

SymbolTablePtr SymbolTableCreator::initAndMakeSymbolTable(bool use_fake) {
ASSERT(!initialized_ || (use_fake_symbol_tables_ == use_fake));
Expand Down
2 changes: 1 addition & 1 deletion source/server/options_impl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ OptionsImpl::OptionsImpl(std::vector<std::string> args,
"", "cpuset-threads", "Get the default # of worker threads from cpuset size", cmd, false);

TCLAP::ValueArg<bool> use_fake_symbol_table("", "use-fake-symbol-table",
"Use fake symbol table implementation", false, true,
"Use fake symbol table implementation", false, false,
"bool", cmd);

TCLAP::ValueArg<std::string> disable_extensions("", "disable-extensions",
Expand Down
9 changes: 6 additions & 3 deletions test/common/stats/stat_test_utility.cc
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,8 @@ Counter& TestStore::counterFromStatNameWithTags(const StatName& stat_name,
} else {
// Ensures StatNames with the same string representation are specified
// consistently using symbolic/dynamic components on every access.
ASSERT(counter_ref->statName() == stat_name);
ASSERT(counter_ref->statName() == stat_name, "Inconsistent dynamic vs symbolic "
"stat name specification");
}
return *counter_ref;
}
Expand All @@ -173,7 +174,8 @@ Gauge& TestStore::gaugeFromStatNameWithTags(const StatName& stat_name,
if (gauge_ref == nullptr) {
gauge_ref = &IsolatedStoreImpl::gaugeFromStatNameWithTags(stat_name, tags, mode);
} else {
ASSERT(gauge_ref->statName() == stat_name);
ASSERT(gauge_ref->statName() == stat_name, "Inconsistent dynamic vs symbolic "
"stat name specification");
}
return *gauge_ref;
}
Expand All @@ -194,7 +196,8 @@ Histogram& TestStore::histogramFromStatNameWithTags(const StatName& stat_name,
if (histogram_ref == nullptr) {
histogram_ref = &IsolatedStoreImpl::histogramFromStatNameWithTags(stat_name, tags, unit);
} else {
ASSERT(histogram_ref->statName() == stat_name);
ASSERT(histogram_ref->statName() == stat_name, "Inconsistent dynamic vs symbolic "
"stat name specification");
}
return *histogram_ref;
}
Expand Down
17 changes: 12 additions & 5 deletions test/extensions/filters/http/grpc_stats/config_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -358,11 +358,18 @@ TEST_F(GrpcStatsFilterConfigTest, MessageCounts) {
.counterFromString(
"grpc.lyft.users.BadCompanions.GetBadCompanions.request_message_count")
.value());
EXPECT_EQ(0U, decoder_callbacks_.clusterInfo()
->statsScope()
.counterFromString(
"grpc.lyft.users.BadCompanions.GetBadCompanions.response_message_count")
.value());

// Check that there is response_message_count stat yet. We use
// stats_store_.findCounterByString rather than looking on
// clusterInfo()->statsScope() because findCounterByString is not an API on
// Stats::Store, and there is no prefix so the names will match. We verify
// that by double-checking we can find the request_message_count using the
// same API.
EXPECT_FALSE(stats_store_.findCounterByString(
"grpc.lyft.users.BadCompanions.GetBadCompanions.response_message_count"));
EXPECT_TRUE(stats_store_.findCounterByString(
"grpc.lyft.users.BadCompanions.GetBadCompanions.request_message_count"));

const auto& data = stream_info_.filterState()->getDataReadOnly<GrpcStatsObject>(
HttpFilterNames::get().GrpcStats);
EXPECT_EQ(2U, data.request_message_count);
Expand Down
1 change: 1 addition & 0 deletions test/integration/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ envoy_cc_test_library(

envoy_cc_test(
name = "ads_integration_test",
size = "enormous",
srcs = ["ads_integration_test.cc"],
tags = ["fails_on_windows"],
deps = [
Expand Down
3 changes: 1 addition & 2 deletions test/integration/http2_integration_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1562,8 +1562,7 @@ void Http2FloodMitigationTest::floodServer(const Http2Frame& frame, const std::s

EXPECT_LE(total_bytes_sent, TransmitThreshold) << "Flood mitigation is broken.";
EXPECT_EQ(1, test_server_->counter(flood_stat)->value());
EXPECT_EQ(1,
test_server_->counter("http.config_test.downstream_cx_delayed_close_timeout")->value());
test_server_->waitForCounterGe("http.config_test.downstream_cx_delayed_close_timeout", 1);
}

// Verify that the server detects the flood using specified request parameters.
Expand Down
6 changes: 2 additions & 4 deletions test/server/admin/prometheus_stats_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,7 @@ class HistogramWrapper {

class PrometheusStatsFormatterTest : public testing::Test {
protected:
PrometheusStatsFormatterTest()
: symbol_table_(Stats::SymbolTableCreator::makeSymbolTable()), alloc_(*symbol_table_),
pool_(*symbol_table_) {}
PrometheusStatsFormatterTest() : alloc_(*symbol_table_), pool_(*symbol_table_) {}

~PrometheusStatsFormatterTest() override { clearStorage(); }

Expand Down Expand Up @@ -92,7 +90,7 @@ class PrometheusStatsFormatterTest : public testing::Test {
EXPECT_EQ(0, symbol_table_->numSymbols());
}

Stats::SymbolTablePtr symbol_table_;
Stats::TestSymbolTable symbol_table_;
Stats::AllocatorImpl alloc_;
Stats::StatNamePool pool_;
std::vector<Stats::CounterSharedPtr> counters_;
Expand Down