Skip to content

A one stop shop for your Signoz-ECS needs. This repository contains crafty scripts to supercharge your application observability using SigNoz and a single command

Notifications You must be signed in to change notification settings

wednesday-solutions/ecs-signoz

Repository files navigation

This template allows you to deploy SigNoz on aws ecs fargate. (what is signoz?)

latency of api calls of a sample application

System Architecture of everything we are going to deploy.

This is going to deploy a clickhouse cluster in our vpc and and all the signoz services in our ecs fargate cluster.

Table of contents

About SigNoz:

SigNoz provides comprehensive monitoring for your application. It tracks and monitors all the important metrics and logs related to your application, infrastructure, and network, and provides real-time alerts for any issues.

You can get traces,metrics and logs of your application.

latency of api calls of a sample application Latency of api calls to a sample application in signoz.

traces of api calls of a sample application Traces of our sample application in signoz.

latency of api calls of a sample application Logs of our sample application in signoz

latency of api calls of a sample application CPU metrics of our sample application in signoz

Self hosting SigNoz on aws ecs fargate is also significantly cheaper than using aws native services like xray and cloudwatch to collect application metrics,traces and logs.SigNoz consists of a clikhouse cluster and multiple services, hosting them is tedious and inconvenient process but this template allows you to do so using a single command.


SigNoz architecture:

signoz architecture

We will be hosting our own clickhouse cluster using aws cloudformation and host all the services on aws ecs faragate.


Pre-Requisites:

  1. please have bash utility jq installed to process json.
  2. please have bash utility yq installed to process yml.
  3. please install the aws-cli and docker.
  4. Please have aws cli To configured with access key,secret and region.
  5. please have aws copilot version of the current develop branch on github. To install - use make to install a standalone binary
  6. please configure signoz-ecs-config.yml file with appropriate values.

Config File:

The signoz-ecs-config.yml files containes all our configuration :

Configuration

  • application-name

    name of your aws copilot application, if you want to use an existing app please use the same app name
  • environment-name

    environment name of your aws copilot application
  • clickhouseConf:

    Config for the clickhouse cluster
    • stackName

      Name of the cloudformation stack which will create the clickhouse cluster
    • clickhouseDiskSize

      Disk size of the ec2 instance of the clickhouse in your clickhouse cluster
    • zookeeperDiskSize

      Disk size of the EC2 instance of the zookeeper in your clickhouse cluster
    • zookeeperInstanceType

      EC2 instance type to provision for Zookeeper
    • instanceType

      EC2 instance type to provision for the clickhosue shard
    • hostName

      Host of the clickhouse intance, please keep it black if you want to create your own clickhouse cluster, if your want to use an existing one please replace it with value of the host of one of your clickhouse shards
  • existingVpc:

    In case you already havee an existing VPC and would like to deploy your application in it, please add in the required details below
    • vpcId

      Please fill the value if you want to use an existing vpc, if your want to create a new one please keep it empty
    • publicSubnetAId

      Subnet of the pre-existing public subnet, otherwise please leave it blank
    • publicSubnetBId

      Subnet of the pre-existing public subnet, otherwise please leave it blank
    • privateSubnetAId

      Subnet of the pre-existing private subnet, otherwise please leave it blank
    • privateSubnetBId

      Subnet of the pre-existing private subnet, otherwise please leave it blank
  • fluentbitConf:

    The config related to the custom fluentbit image we will be uploading to aws ecr
    • repoName

      name of the repo in aws ecr for the fluentbit image
    • localImageName

      name of the local fluentbit image we will be creating
  • otelSidecarConf:

    The config related to the custom otel collector image we will be uploading to aws ecr
    • repoName

      name of the repo in aws ecr for the sidecar-otel image
    • localImageName

      name of the local sidecar-otel image we will be creating
  • serviceNames

    name of the variouse SigNoz service we will be deploying to aws ecs fargate
    • otel

      name of the SigNoz otel collector service
    • query

      name of the SigNoz query service
    • alert

      name of the SigNoz alert service
    • frontend

      name of the SigNoz frontend service

To instrument your applications and send data to SigNoz please refer- https://signoz.io/docs/instrumentation/


Configuring values in signoz-ecs-config.yml

  1. If your want to use an existing vpc please add the existing vpc block in the signoz-ecs-config.yml file.
  2. If you want to use your own clickhouse cluster please mention the clickhouse host in the signoz-ecs-config.yml file(by default a new clickhouse cluster would be created).
  3. If you want to host SigNoz in an existing copilot application, just keep your application name and the enviroment same as the one you want to host SigNoz in.
  4. You can change the name of cloudformation stack of your clickhouse cluster.
  5. can change the instance type of your clikhouse or zookeeper hosts.
  6. You can change the name of various SigNoz services in the config file.

Configuring a config file with custom prefix

You can also mention the name of a custom config file of the form signoz-ecs-config-suffix, you can mention the suffix by defining an environment variable "$environmentName" and then the scripts will use the custom config file. you revise a wordy sentence in an effortless manner.


To deploy Signoz:

If you want to deploy from scratch - to deploy your own vpc,clickhouse cluster and fargate cluster

clone this repository locally then, configure signoz-ecs-config.yml with appropriate values:

      signoz-app:
      application-name: "tp-signoz"
      environment-name: "dev"
      clickhouseConf:
        stackName: "tp-signoz"
        clickhouseDiskSize: 30
        zookeeperDiskSize: 30
        zookeeperInstanceType: "t2.small"
        instanceType: "t2.small"
        hostName: 

      fluentbitConf:
        repoName: "fbit-repo"
        localImageName: "fbit"
      otelSidecarConf:
        repoName: "sidecar-otel"
        localImageName: "sotel"  
      serviceNames:
        otel: "otel"
        query: "query"
        alert: "alert"
        frontend: "frontend"

your signoz-ecs-config.yml file

Then execute the script with

make deploy

If you want to deploy clickhouse cluster and services in your own vpc:

clone this repository locally then, configure vpc id and subnet id in signoz-ecs-config.yml add the vpc id and all the subnets id

signoz-app:
  application-name: "tp-signoz"
  environment-name: "dev"
  clickhouseConf:
    stackName: "tp-signoz"
    clickhouseDiskSize: 30
    zookeeperDiskSize: 30
    zookeeperInstanceType: "t2.small"
    instanceType: "t2.small"
    hostName: 

  fluentbitConf:
    repoName: "fbit-repo"
    localImageName: "fbit"
  otelSidecarConf:
    repoName: "sidecar-otel"
    localImageName: "sotel"  
  serviceNames:
    otel: "otel"
    query: "query"
    alert: "alert"
    frontend: "frontend"
  existingVpc:
    vpcId: vpc-09cc39b8b0fa6b10f
    publicSubnetAId: subnet-0787b5ad8b2d8aa41
    publicSubnetBId: subnet-07d3d63b9c87501b0
    privateSubnetAId: subnet-05870291afed3115e
    privateSubnetBId: subnet-045544de09bff69be

your signoz-ecs-config.yml file will look like this

make deploy

If you have already deployed clickhouse cluster and want to deploy all services in a new vpc and fargate cluster:

clone this repository locally then, configure the clickhouse host in signoz-ecs-config.yml

signoz-app:
  application-name: "tp-signoz"
  environment-name: "dev"
  clickhouseConf:
    stackName: "tp-signoz"
    clickhouseDiskSize: 30
    zookeeperDiskSize: 30
    zookeeperInstanceType: "t2.small"
    instanceType: "t2.small"
    hostName: my-clickhouse-host

  fluentbitConf:
    repoName: "fbit-repo"
    localImageName: "fbit"
  otelSidecarConf:
    repoName: "sidecar-otel"
    localImageName: "sotel"  
  serviceNames:
    otel: "otel"
    query: "query"
    alert: "alert"
    frontend: "frontend"

your signoz-ecs-config.yml file will look like this

make deploy

If you have already deployed clickhouse cluster and want to deploy all services in an existing vpc:

clone this repository locally then, configure the clickhouse host in signoz-ecs-config.yml configure vpc id and subnet id in signoz-ecs-config.yml

signoz-app:
  application-name: "tp-signoz"
  environment-name: "dev"
  clickhouseConf:
    stackName: "tp-signoz"
    clickhouseDiskSize: 30
    zookeeperDiskSize: 30
    zookeeperInstanceType: "t2.small"
    instanceType: "t2.small"
    hostName: my-clickhouse-host

  fluentbitConf:
    repoName: "fbit-repo"
    localImageName: "fbit"
  otelSidecarConf:
    repoName: "sidecar-otel"
    localImageName: "sotel"  
  serviceNames:
    otel: "otel"
    query: "query"
    alert: "alert"
    frontend: "frontend"
  existingVpc:
    vpcId: vpc-09cc39b8b0fa6b10f
    publicSubnetAId: subnet-0787b5ad8b2d8aa41
    publicSubnetBId: subnet-07d3d63b9c87501b0
    privateSubnetAId: subnet-05870291afed3115e
    privateSubnetBId: subnet-045544de09bff69be

your signoz-ecs-config.yml file will look like this

make deploy

If you have already configured copilot with an app name and environment(the subnets should be the same as the one where clickhouse cluster is present):

clone this repository locally then, configure the clickhouse host in signoz-ecs-config.yml configure vpc id and subnet id in signoz-ecs-config.yml

make deploy-existing-copilot-app

In this template we are using cloudformation to host our clickhouse cluster and aws copilot to host our services on ecs fargate.


Hosting a clickhouse cluster using aws cloudformation:

The template can be configured if you want to create a new vpc for our clickhouse-cluster or create the cluster in an existing vpc. To create the cluster in an existing vpc just mention the value of existing vpc and subnets in the config file. existingVpc: vpcId: vpc-09cc39b8b0fa6b10f publicSubnetAId: subnet-0787b5ad8b2d8aa41 publicSubnetBId: subnet-07d3d63b9c87501b0 privateSubnetAId: subnet-05870291afed3115e privateSubnetBId: subnet-045544de09bff69be

When existing-vpc value is present it will create the following resources:

clickhouse.yaml resources diagram

This will create a new vpc with two private subnets and two public subnets. We will host 3 zookeeper instances and 3 shards of clickhouse instacnce in our private subnets. This will also host a bastion instance in our public instance if we ever want to ssh into our instaces to debug some issues.

When existing-vpc option is toggled true:

You will also have to configure vpc id,public and private subnets option in signoz-ecs-config.yml file

                              existingVpc:
                                vpcId: vpc-09cc39b8b0fa6b10f
                                publicSubnetAId: subnet-0787b5ad8b2d8aa41
                                publicSubnetBId: subnet-07d3d63b9c87501b0
                                privateSubnetAId: subnet-05870291afed3115e
                                privateSubnetBId: subnet-045544de09bff69be

clickhouse-custom-vpc.yaml resources diagram

This creates three zookeeper instances and three clickhouse shards in private subnets of your vpc.

If you are using a managed clickhouse-service, you can mention the host of one of the shards, then will not create it's own clickhouse cluster(on port 9000): signoz-app: clickhouseConf: hostName: my-clickhouse-host

You can use custom ami's for zookeeper or clickhouse instances if your organization has special security needs or want to add more functionality (we have made our own user-data scripts, so each clichouse shard and zookeeper can know about each other at start time). If you do not want to copy your ami's outside of a single region just replace the ImageId field in clickhouse.yaml or clickhous-custom-vpc.yaml cloudformation template. If you want to copy ami's to all region then use ./scripts/copy-ami.sh script and ./scripts/amimap.sh to generate the mappings,then replace the existing mappings with the output.


Why are we using aws copilot?

AWS Copilot CLI simplifies the deployment of your applications to AWS. It automates the process of creating AWS resources, configuring them, and deploying your application. This can save you time and effort compared to manual deployment.It simplifies the deployment of your applications to AWS. It automates the process of creating AWS resources, configuring them, and deploying your application. This can save you time and effort compared to manual deployment. Aws copilot will allow us to deploy all our SigNoz services with minimum configuration

You are free to use your own custon vpc and subnets, though they should be the same as your clickhouse cluster(if hosting in a private subnet). You will also have to configure vpc id,public and private subnets option in signoz-ecs-config.yml file

                                existingVpc:
                                  vpcId: vpc-09cc39b8b0fa6b10f
                                  publicSubnetAId: subnet-0787b5ad8b2d8aa41
                                  publicSubnetBId: subnet-07d3d63b9c87501b0
                                  privateSubnetAId: subnet-05870291afed3115e
                                  privateSubnetBId: subnet-045544de09bff69be


Guides:

How to send instrumentation data from your existing fargate service to signoz?

If you have an existing service that you want to instrument, you will have to configure an otel sidecar container to a service which will forward all data to the signoz service.

Please look at the following documentation to add instrumentation to your application. After you have added the code to your application, we will be abel to generate traces and metrics, and we will have to send this data to the SigNoz otel collector. In our setup we will first configure an otel collector sidecar, which will then forward our data to the SigNoz otel collector, to do so we will add the following to our manifest file: (please ensure you have uploaded the appropriate docker image for otel sidecar to aws ecr, you can get the image uri in output.yml file)

                                sidecars:
                                    otel:
                                        port: 4317
                                        image: <AWS_ACCOUNT_ID>.dkr.ecr.ap-southeast-1.amazonaws.com/sidecar-otel:latest # uri of your sidecar otel image

We have configured an otel sidecar to forward our metrics and traces instead of directly sending them to SigNoz otel collector because, this will reduce the latency of our service whenever we are trying to connect to another database or service as we now have to talk to localhost instead of another ecs service.

To manuall upload the sidecar otel image use command:

    make otel-sidecar-upload

How to send logs of your existing ecs fargate service to signoz?

If you have an existing service whose logs you want to send to signoz, you will have to configure an aws firelens service which will forward all the logs using fluentforward protocol.

To send logs of your application to SigNoz we are going to use aws firelens.FireLens works with Fluentd and Fluent Bit. We provide the AWS for Fluent Bit image or you can use your own Fluentd or Fluent Bit image. We will create our own custom image where will configure rules which will forward logs from our application to the SigNoz collector using the fluentforward protocol. When you deploy the template it will automatically deploy our custom fluentbit image to aws ecr and we have configured our SigNoz otel collector to accept logs via firelens.Using the command make scaffold svcName we can create a sample manifest file for you with firelens preconfigured. Configuring firelens using aws copilot is extremely easy, just add follwing to the mainfest file

                                    logging:
                                        image: public.ecr.aws/k8o0c2l3/fbit:latest
                                        configFilePath: /logDestinations.conf

To manuall upload the fluenbit image use command:

    make fluentbit-upload

To scaffold a service with a sample file use command:

make scaffold $service-name

To use your own custom ami's for clickhouse and zookeeper :

you can copy the ami's in all region using the script :

./scripts/copy-ami.sh

then you can use the script to get a mapping of all ami's using:

./scripts/amimap.sh

Then replace the mappings in the clickhouse.yml cloudformation template


For deleting all the services and cleaning up all the resources(is an irreversible action)

This will stop all the ecs services and your clickhouse cloudformation stack.

make delete-all

To change the backup frequency of the clickhouse cluster:

Currently the clickhouse servers are configured to backup their data to aws s3 every 24 hours, if your want to change the backup frequency you will have to create your own custom clickhouse ami's. To make your own ami's please use our amis (present in clickhouse.yml clouformation file) as the base and then configure the cron job to run at whateverfrequency you desire.

script for backing up the clickhouse databse(configure you cron job to run this script):

  #!/bin/bash
  BACKUP_NAME=my_backup_$(date -u +%Y-%m-%dT%H-%M-%S)
  clickhouse-backup create --config /etc/clickhouse-backup/config.yml $BACKUP_NAME >> /var/log/clickhouse-backup.log
  if [[ $? != 0 ]]; then
    echo "clickhouse-backup create $BACKUP_NAME FAILED and return $? exit code"
  fi

  clickhouse-backup upload --config /etc/clickhouse-backup/config.yml $BACKUP_NAME >> /var/log/clickhouse-backup.log
  if [[ $? != 0 ]]; then
    echo "clickhouse-backup upload $BACKUP_NAME FAILED and return $? exit code"
  fi

About

A one stop shop for your Signoz-ECS needs. This repository contains crafty scripts to supercharge your application observability using SigNoz and a single command

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published