Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add UDP telemetry support #7

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

add UDP telemetry support #7

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c3c3d99
add UDP telemetry support
weijiage Apr 20, 2023
6c4a483
remove udp CLI option
weijiage Apr 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
10 changes: 10 additions & 0 deletions config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ type Config struct {
Meta bool `json:"meta"`
EOS bool `json:"eos"`
GRPC GRPCConfig `json:"grpc"`
UDP UDPConfig `json:"udp"`
TLS TLSConfig `json:"tls"`
Influx InfluxConfig `json:"influx"`
Kafka *KafkaConfig `json:"kafka"`
Expand Down Expand Up @@ -75,6 +76,12 @@ type GRPCConfig struct {
WS int32 `json:"ws"`
}

type UDPConfig struct {
Port int `json:"port"`
Host string `json:"host"`
Protos []string `json:"protos"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makefile target to compile protos are missing ?

Copy link
Author

@weijiage weijiage Apr 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that the user may not provide specific proto files in the JTIMON config, and it may be time consuming to compile all the protos, therefore I just uploaded all the .pb.go files along with the proto files under /schema.
I'd love to add make proto in the Makefile as needed.

Just to clarify, the Makefile target is expected to compile all the protos?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes,

}

// TLSConfig is to specify TLS params
type TLSConfig struct {
ClientCrt string `json:"clientcrt"`
Expand Down Expand Up @@ -277,6 +284,9 @@ func (jctx *JCtx) isConfigChanged(new Config) bool {
if !reflect.DeepEqual(old.Influx, new.Influx) {
return true
}
if !reflect.DeepEqual(old.UDP, new.UDP) {
return true
}

return false
}
Expand Down
2 changes: 2 additions & 0 deletions defaults.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,6 @@ const (
MatchExpressionXpath = "\\/([^\\/]*)\\[(.*?)+?(?:\\])"
// MatchExpressionKey is for pattern matching the single and multiple key value pairs
MatchExpressionKey = "([A-Za-z0-9-/]*)=(.*?)?(?: and |$)+"
// UDPMatchExpressionKey is for pattern matching the single and multiple key value pairs for UDP data
UDPMatchExpressionKey = "([A-Za-z0-9-_]*)=(?:'(.*?)'|\"(.*?)\")?(?: and |$)+"
)
50 changes: 39 additions & 11 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,50 @@
module github.com/Juniper/jtimon

go 1.13
go 1.17

require (
github.com/Juniper/jtimon/schema v0.0.0
github.com/Shopify/sarama v1.26.1
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 // indirect
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
github.com/golang/protobuf v1.0.0
github.com/beorn7/perks v1.0.1 // indirect
github.com/gogo/protobuf v1.3.2
github.com/golang/protobuf v1.5.2
github.com/gorilla/mux v1.8.0
github.com/influxdata/influxdb v1.5.2
github.com/matttproud/golang_protobuf_extensions v1.0.0 // indirect
github.com/jhump/protoreflect v1.15.1
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/prometheus/client_golang v0.8.0
github.com/prometheus/client_model v0.0.0-20171117100541-99fa1f4be8e5 // indirect
github.com/prometheus/client_model v0.2.0 // indirect
github.com/spf13/pflag v1.0.1
golang.org/x/net v0.7.0
google.golang.org/grpc v1.53.0
google.golang.org/protobuf v1.28.2-0.20230222093303-bc1253ad3743
)

require (
github.com/bufbuild/protocompile v0.4.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/eapache/go-resiliency v1.2.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/golang/snappy v0.0.3 // indirect
github.com/hashicorp/go-uuid v1.0.2 // indirect
github.com/jcmturner/gofork v1.0.0 // indirect
github.com/klauspost/compress v1.9.8 // indirect
github.com/pierrec/lz4 v2.4.1+incompatible // indirect
github.com/prometheus/common v0.0.0-20180413074202-d0f7cd64bda4 // indirect
github.com/prometheus/procfs v0.0.0-20180408092902-8b1c2da0d56d // indirect
github.com/spf13/pflag v1.0.1
golang.org/x/net v0.0.0-20210917221730-978cfadd31cf
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a // indirect
google.golang.org/genproto v0.0.0-20180413175816-7fd901a49ba6 // indirect
google.golang.org/grpc v1.11.3
github.com/rcrowley/go-metrics v0.0.0-20190826022208-cac0b30c2563 // indirect
golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa // indirect
golang.org/x/sync v0.1.0 // indirect
golang.org/x/sys v0.5.0 // indirect
golang.org/x/text v0.7.0 // indirect
google.golang.org/genproto v0.0.0-20230110181048-76db0878b65f // indirect
gopkg.in/jcmturner/aescts.v1 v1.0.1 // indirect
gopkg.in/jcmturner/dnsutils.v1 v1.0.1 // indirect
gopkg.in/jcmturner/gokrb5.v7 v7.5.0 // indirect
gopkg.in/jcmturner/rpc.v1 v1.1.0 // indirect
)

replace github.com/Juniper/jtimon/schema => ./schema

replace github.com/Juniper/jtimon/schema/telemetry_top => ./schema/telemetry_top
1,137 changes: 1,113 additions & 24 deletions go.sum
View file
Open in desktop

Large diffs are not rendered by default.

11 changes: 8 additions & 3 deletions main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ var (
prom = flag.Bool("prometheus", false, "Stats for prometheus monitoring system")
promHost = flag.String("prometheus-host", "127.0.0.1", "IP to bind Prometheus service to")
promPort = flag.Int32("prometheus-port", 8090, "Prometheus port")
udp = flag.Bool("udp", false, "Stats for udp monitoring system")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why CLI option given that there is already udp in .json file ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for advice. Please find the changes in commit remove udp CLI option.

prefixCheck = flag.Bool("prefix-check", false, "Report missing __prefix__ in telemetry packet")
pProf = flag.Bool("pprof", false, "Profile JTIMON")
pProfPort = flag.Int32("pprof-port", 6060, "Profile port")
Expand Down Expand Up @@ -73,9 +74,13 @@ func main() {
return
}

workers := NewJWorkers(*configFiles, *configFileList, *maxRun)
workers.StartWorkers()
workers.Wait()
if *udp {
udpWork(configFiles)
} else {
workers := NewJWorkers(*configFiles, *configFileList, *maxRun)
workers.StartWorkers()
workers.Wait()
}

log.Printf("all done ... exiting!")
}
Loading