Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloud Telemetry #775

Merged
merged 48 commits into from
Nov 26, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
cb83e0f
Add metric for tracking feature usage and results
jnunemaker Oct 8, 2023
ae7bf70
Add metric storage for metrics
jnunemaker Oct 8, 2023
a4dc8b2
Add empty to metric storage
jnunemaker Oct 9, 2023
dda34fd
Avoid matrix locally so its easier to use these
jnunemaker Oct 12, 2023
d7c8f3c
Merge branch 'main' into rollup-instrumenter
jnunemaker Oct 12, 2023
27571cb
Some WIP on telemetry
jnunemaker Oct 12, 2023
64d16f2
Switch to symbol headers, schema version number and array of metrics
jnunemaker Oct 17, 2023
838f5a6
Slow down polling in threaded example and check multiple flags with d…
jnunemaker Oct 18, 2023
d8e85fa
Telemetry improvements
jnunemaker Oct 18, 2023
d324f47
Minor whitespace
jnunemaker Oct 19, 2023
f40352e
Add comment for start
jnunemaker Oct 19, 2023
b2ca64d
Add assertions for bar and baz
jnunemaker Oct 19, 2023
9dcd5ba
Merge branch 'main' into rollup-instrumenter
jnunemaker Oct 19, 2023
59c06f0
Add request id to payload
jnunemaker Oct 29, 2023
09bf4d6
Add note for getting true and false results
jnunemaker Oct 29, 2023
79da99f
fix failing specs when running some individually
jnunemaker Oct 29, 2023
6034d65
Tweaks to json generation
jnunemaker Oct 29, 2023
0d054c6
Pid is in log output already
jnunemaker Nov 12, 2023
95b6150
Compress body
jnunemaker Nov 12, 2023
f145ee2
Add serializers for sharing to and from json and compression
jnunemaker Nov 13, 2023
e2f7ab3
Add typecast stuff for json and gzip
jnunemaker Nov 13, 2023
041840d
Switch JSON.dump to Typecast.to_json
jnunemaker Nov 13, 2023
771c531
Change JSON.parse to Typecast.from_json
jnunemaker Nov 13, 2023
bdf5f84
Use shortcuts
jnunemaker Nov 18, 2023
4f4da36
Merge branch 'main' into rollup-instrumenter
jnunemaker Nov 18, 2023
27b1fc1
More flexible env vars
jnunemaker Nov 19, 2023
f15450a
Add backoff policy
jnunemaker Nov 21, 2023
c29c168
Just use vanilla gzip
jnunemaker Nov 21, 2023
0d21a12
Add initial specs for submission
jnunemaker Nov 21, 2023
c45e987
Add retry specs for 429 and 5xx
jnunemaker Nov 21, 2023
51d7228
More retry specs
jnunemaker Nov 21, 2023
f0b8852
A bit of language in example
jnunemaker Nov 21, 2023
fd600ba
Reduce queue size
jnunemaker Nov 21, 2023
877ef56
Adjust defaults for back off policy
jnunemaker Nov 22, 2023
268192d
Fix failing specs
jnunemaker Nov 22, 2023
1825be8
Pass response to error
jnunemaker Nov 22, 2023
d0bc6cd
Fix submitter error raise
jnunemaker Nov 22, 2023
15c4cc4
Create writer for telemetry interval
jnunemaker Nov 22, 2023
92454ef
Specs for telemetry_interval updating from server responses
jnunemaker Nov 22, 2023
cf3a005
Remove configure of cloud and use default config
jnunemaker Nov 25, 2023
de14d12
Remove comments
jnunemaker Nov 25, 2023
233e00e
Combine read and accessor
jnunemaker Nov 25, 2023
00f2595
Remove explicit sync interval in example
jnunemaker Nov 25, 2023
d8c042d
More generic name
jnunemaker Nov 25, 2023
00e5971
Delegate the right object
jnunemaker Nov 26, 2023
0adf54f
Remove unused constants
jnunemaker Nov 26, 2023
654eb41
Remove unnecessary return
jnunemaker Nov 26, 2023
de75893
Make reset private
jnunemaker Nov 26, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions examples/cloud/backoff_policy.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Just a simple example that shows how the backoff policy works.
require 'bundler/setup'
require 'flipper/cloud/telemetry/backoff_policy'

intervals = []
policy = Flipper::Cloud::Telemetry::BackoffPolicy.new

10.times do |n|
intervals << policy.next_interval
end

pp intervals.map { |i| i.round(2) }
puts "Total: #{intervals.sum.round(2)}ms (#{(intervals.sum/1_000.0).round(2)} sec)"
10 changes: 7 additions & 3 deletions examples/cloud/cloud_setup.rb
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,13 @@
exit
end

suffix_rails = ENV["RAILS_VERSION"].split(".").take(2).join
suffix_ruby = RUBY_VERSION.split(".").take(2).join
matrix_key = "FLIPPER_CLOUD_TOKEN_#{suffix_ruby}_#{suffix_rails}"
matrix_key = if ENV["CI"]
suffix_rails = ENV["RAILS_VERSION"].split(".").take(2).join
suffix_ruby = RUBY_VERSION.split(".").take(2).join
"FLIPPER_CLOUD_TOKEN_#{suffix_ruby}_#{suffix_rails}"
else
"FLIPPER_CLOUD_TOKEN"
end

if matrix_token = ENV[matrix_key]
puts "Using #{matrix_key} for FLIPPER_CLOUD_TOKEN"
Expand Down
9 changes: 7 additions & 2 deletions examples/cloud/forked.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,16 @@
require 'bundler/setup'
require 'flipper/cloud'

pids = 5.times.map do |n|
puts Process.pid

# Make a call in the parent process so we can detect forking.
Flipper.enabled?(:stats)

pids = 2.times.map do |n|
fork {
# Check every second to see if the feature is enabled
threads = []
5.times do
2.times do
threads << Thread.new do
loop do
sleep rand
Expand Down
33 changes: 15 additions & 18 deletions examples/cloud/threaded.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,33 +4,30 @@
require_relative "./cloud_setup"
require 'bundler/setup'
require 'flipper/cloud'
require "active_support/notifications"
require "active_support/isolated_execution_state"

ActiveSupport::Notifications.subscribe(/poller\.flipper/) do |*args|
p args: args
end
puts Process.pid

Flipper.configure do |config|
config.default {
Flipper::Cloud.new(local_adapter: config.adapter, instrumenter: ActiveSupport::Notifications)
Flipper::Cloud.new(
local_adapter: config.adapter,
debug_output: STDOUT,
)
}
end

# You might want to do this at some point to see different results:
# Flipper.enable(:search)
# Flipper.disable(:stats)

# Check every second to see if the feature is enabled
threads = []
10.times do
threads << Thread.new do
5.times.map { |i|
Thread.new {
loop do
sleep rand

if Flipper[:stats].enabled?
puts "#{Time.now.to_i} Enabled!"
else
puts "#{Time.now.to_i} Disabled!"
end
Flipper.enabled?(:stats)
Flipper.enabled?(:search)
end
end
end

threads.map(&:join)
}
}.each(&:join)
1 change: 0 additions & 1 deletion flipper.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -35,5 +35,4 @@ Gem::Specification.new do |gem|
gem.metadata = Flipper::METADATA

gem.add_dependency 'concurrent-ruby', '< 2'
gem.add_dependency 'brow', '~> 0.4.1'
end
4 changes: 2 additions & 2 deletions lib/flipper/adapters/active_record.rb
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ def set(feature, gate, thing, options = {})
@gate_class.create! do |g|
g.feature_key = feature.key
g.key = gate.key
g.value = json_feature ? JSON.dump(thing.value) : thing.value.to_s
g.value = json_feature ? Typecast.to_json(thing.value) : thing.value.to_s
end
rescue ::ActiveRecord::RecordNotUnique
# assume this happened concurrently with the same thing and its fine
Expand Down Expand Up @@ -263,7 +263,7 @@ def result_for_gates(feature, gates)
end
when :json
if row = gates.detect { |key, value| !key.nil? && key.to_sym == gate.key }
JSON.parse(row.last)
Typecast.from_json(row.last)
end
when :set
gates.select { |key, value| !key.nil? && key.to_sym == gate.key }.map(&:last).to_set
Expand Down
8 changes: 4 additions & 4 deletions lib/flipper/adapters/http.rb
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def initialize(options = {})
def get(feature)
response = @client.get("/features/#{feature.key}")
if response.is_a?(Net::HTTPOK)
parsed_response = JSON.parse(response.body)
parsed_response = Typecast.from_json(response.body)
result_for_feature(feature, parsed_response.fetch('gates'))
elsif response.is_a?(Net::HTTPNotFound)
default_config
Expand All @@ -41,7 +41,7 @@ def get_multi(features)
response = @client.get("/features?keys=#{csv_keys}&exclude_gate_names=true")
raise Error, response unless response.is_a?(Net::HTTPOK)

parsed_response = JSON.parse(response.body)
parsed_response = Typecast.from_json(response.body)
parsed_features = parsed_response.fetch('features')
gates_by_key = parsed_features.each_with_object({}) do |parsed_feature, hash|
hash[parsed_feature['key']] = parsed_feature['gates']
Expand All @@ -59,7 +59,7 @@ def get_all
response = @client.get("/features?exclude_gate_names=true")
raise Error, response unless response.is_a?(Net::HTTPOK)

parsed_response = JSON.parse(response.body)
parsed_response = Typecast.from_json(response.body)
parsed_features = parsed_response.fetch('features')
gates_by_key = parsed_features.each_with_object({}) do |parsed_feature, hash|
hash[parsed_feature['key']] = parsed_feature['gates']
Expand All @@ -78,7 +78,7 @@ def features
response = @client.get('/features?exclude_gate_names=true')
raise Error, response unless response.is_a?(Net::HTTPOK)

parsed_response = JSON.parse(response.body)
parsed_response = Typecast.from_json(response.body)
parsed_response['features'].map { |feature| feature['key'] }.to_set
end

Expand Down
18 changes: 11 additions & 7 deletions lib/flipper/adapters/http/client.rb
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ def initialize(options = {})
@debug_output = options[:debug_output]
end

def add_header(key, value)
@headers[key] = value
end

def get(path)
perform Net::HTTP::Get, path, @headers
end
Expand Down Expand Up @@ -77,13 +81,13 @@ def build_http(uri)

def build_request(http_method, uri, headers, options)
request_headers = {
"Client-Language" => "ruby",
"Client-Language-Version" => "#{RUBY_VERSION} p#{RUBY_PATCHLEVEL} (#{RUBY_RELEASE_DATE})",
"Client-Platform" => RUBY_PLATFORM,
"Client-Engine" => defined?(RUBY_ENGINE) ? RUBY_ENGINE : "",
"Client-Pid" => Process.pid.to_s,
"Client-Thread" => Thread.current.object_id.to_s,
"Client-Hostname" => Socket.gethostname,
client_language: "ruby",
client_language_version: "#{RUBY_VERSION} p#{RUBY_PATCHLEVEL} (#{RUBY_RELEASE_DATE})",
client_platform: RUBY_PLATFORM,
client_engine: defined?(RUBY_ENGINE) ? RUBY_ENGINE : "",
client_pid: Process.pid.to_s,
client_thread: Thread.current.object_id.to_s,
client_hostname: Socket.gethostname,
}.merge(headers)

body = options[:body]
Expand Down
2 changes: 1 addition & 1 deletion lib/flipper/adapters/http/error.rb
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ def initialize(response)
message = "Failed with status: #{response.code}"

begin
data = JSON.parse(response.body)
data = Typecast.from_json(response.body)

if error_message = data["message"]
message << "\n\n#{data["message"]}"
Expand Down
4 changes: 2 additions & 2 deletions lib/flipper/adapters/mongo.rb
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ def enable(feature, gate, thing)
}
when :json
update feature.key, '$set' => {
gate.key.to_s => JSON.dump(thing.value),
gate.key.to_s => Typecast.to_json(thing.value),
}
else
unsupported_data_type gate.data_type
Expand Down Expand Up @@ -175,7 +175,7 @@ def result_for_feature(feature, doc)
doc.fetch(gate.key.to_s) { Set.new }.to_set
when :json
value = doc[gate.key.to_s]
JSON.parse(value) if value
Typecast.from_json(value)
else
unsupported_data_type gate.data_type
end
Expand Down
4 changes: 2 additions & 2 deletions lib/flipper/adapters/pstore.rb
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ def enable(feature, gate, thing)
when :set
set_add key(feature, gate), thing.value.to_s
when :json
write key(feature, gate), JSON.dump(thing.value)
write key(feature, gate), Typecast.to_json(thing.value)
else
raise "#{gate} is not supported by this adapter yet"
end
Expand Down Expand Up @@ -161,7 +161,7 @@ def result_for_feature(feature)
set_members key
when :json
value = read(key)
JSON.parse(value) if value
Typecast.from_json(value)
else
raise "#{gate} is not supported by this adapter yet"
end
Expand Down
4 changes: 2 additions & 2 deletions lib/flipper/adapters/redis.rb
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ def enable(feature, gate, thing)
when :set
@client.hset feature_key, to_field(gate, thing), 1
when :json
@client.hset feature_key, gate.key, JSON.dump(thing.value)
@client.hset feature_key, gate.key, Typecast.to_json(thing.value)
else
unsupported_data_type gate.data_type
end
Expand Down Expand Up @@ -174,7 +174,7 @@ def result_for_feature(feature, doc)
fields_to_gate_value fields, gate
when :json
value = doc[gate.key.to_s]
JSON.parse(value) if value
Typecast.from_json(value)
else
unsupported_data_type gate.data_type
end
Expand Down
4 changes: 2 additions & 2 deletions lib/flipper/adapters/sequel.rb
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ def gate_attrs(feature, gate, thing, json: false)
{
feature_key: feature.key.to_s,
key: gate.key.to_s,
value: json ? JSON.dump(thing.value) : thing.value.to_s,
value: json ? Typecast.to_json(thing.value) : thing.value.to_s,
}
end

Expand All @@ -227,7 +227,7 @@ def result_for_feature(feature, db_gates)
db_gates.select { |db_gate| db_gate.key == gate.key.to_s }.map(&:value).to_set
when :json
if detected_db_gate = db_gates.detect { |db_gate| db_gate.key == gate.key.to_s }
JSON.parse(detected_db_gate.value)
Typecast.from_json(detected_db_gate.value)
end
else
unsupported_data_type gate.data_type
Expand Down
2 changes: 1 addition & 1 deletion lib/flipper/api/action.rb
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ def halt(response)
def json_response(object, status = 200)
header 'content-type', Api::CONTENT_TYPE
status(status)
body = JSON.dump(object)
body = Typecast.to_json(object)
halt [@code, @headers, [body]]
end

Expand Down
2 changes: 1 addition & 1 deletion lib/flipper/api/json_params.rb
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ def call(env)
# This method accomplishes similar functionality
def update_params(env, data)
return if data.empty?
parsed_request_body = JSON.parse(data)
parsed_request_body = Typecast.from_json(data)
env["parsed_request_body".freeze] = parsed_request_body
parsed_query_string = parse_query(env[QUERY_STRING])
parsed_query_string.merge!(parsed_request_body)
Expand Down
Loading