Dynamic Service Creation and Registration at Runtime.

[shaj13]

I’d like to check whether the following approach is considered best practice and if future releases are unlikely to break it. Additionally, any suggestions or recommendations would be greatly appreciated.

We are designing a system to scrape metrics from multiple replicas (X). The server itself can expose hundreds of metric types, grouping them accordingly, and all metrics will ultimately follow the same format. Our goal is to allow scraping at different intervals for different metrics. In some cases, we may also need to query different servers for specific metric groups and types.

We need a generic or common service interface that all metric services implement.

One possible solution is to generate the gRPC definitions and implement dynamic service registration at runtime based on the available metric services, allowing the client to invoke them accordingly.

syntax = "proto3";

option go_package = "./pb";

import "google/protobuf/empty.proto";

service metrics {
    rpc Scrape(google.protobuf.Empty) returns Resources;
}

message Resources {
    repeated bytes data = 1;
}

import grpc "google.golang.org/grpc"

func RegisterMetricsXServer(s grpc.ServiceRegistrar, srv MetricsServer, ServiceName string) {
	d := grpc.ServiceDesc{
		ServiceName: ServiceName,
		HandlerType: (*MetricsServer)(nil),
		Methods: []grpc.MethodDesc{
			{
				MethodName: "Scrape",
				Handler:    _Metrics_Scrape_Handler,
			},
		},
		Streams:  []grpc.StreamDesc{},
		Metadata: "scrape.proto",
	}

	s.RegisterService(&d, srv)
}

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"net"

	"github.com/golang/protobuf/ptypes/empty"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/protobuf/types/known/emptypb"
)

func main() {
	s := flag.Bool("s", false, "run server")
	flag.Parse()
	if *s {
		if err := server(); err != nil {
			log.Fatal(err)
		}
	}

	for _, m := range []string{"cpu", "memory"} {
		cli(m)
	}

}
func cli(service string) error {
	cc, err := grpc.NewClient(":8080", grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		return err
	}

	in := new(emptypb.Empty)
	out := new(Resources)
	return cc.Invoke(context.Background(), "/"+service+"/Scrape", in, out)
}

func server() error {
	gs := grpc.NewServer(
		grpc.Creds(insecure.NewCredentials()),
	)

	for _, s := range []string{"cpu", "memory"} {
		RegisterMetricsXServer(gs, &service{kind: s}, s)
	}

	l, err := net.Listen("tcp", ":8080")
	if err != nil {
		panic(err)
	}

	fmt.Println("server listen and serve")
	return gs.Serve(l)
}

type service struct {
	kind string
}

func (s *service) Scrape(context.Context, *empty.Empty) (*empty.Empty, error) {
	fmt.Println(s.kind, "called")
	return &emptypb.Empty{}, nil
}

Another approach we are considering is maintaining a single service registration while wrapping the service to route requests to the appropriate implementation based on metadata. Alternatively, we could use an interceptor to direct requests to the relevant handler.

To be honest, I'm inclined toward dynamic service registration and letting go-gRPC handle the logic, as some gRPC gateways and ingress controllers use similar mechanisms, leveraging reflection.

I've encountered #1825, but I couldn't find any reference or example related to it.

[purnesh42H]

@shaj13 thanks for the question. Could you confirm if you are planning to have a batch service which scrape metrics in regular intervals or you are planning to invoke scrapers for each metric through a client and invocations happen at regular intervals?

If its the latter case (this is what I understand from your example), I think a single handler that takes in the type of metric to scrape as input which invoke the required scrape method makes more sense to me compared to managing large amount of services since you mentioned there can be hundreds of metrics possible.

Let me know if I understood your use case correctly or could you provide more details on how you are planning to invoke the services that having one service per metric is more feasible to you.