Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Init ignore_firelens_config option #798

Merged
merged 5 commits into from
Mar 25, 2024

Conversation

PettitWesley
Copy link
Contributor

@PettitWesley PettitWesley commented Mar 20, 2024

Background

Issue #, if available:

Use case and design discussion here: aws/containers-roadmap#964 (comment)

Description of changes:

See use_cases section update for a description of the feature.

Testing

Task Def

{
    "family": "init-ignore-firelens",
    "taskRoleArn": "arn:aws:iam::XXXXXXXXXX:role/ecs_task_role",
    "executionRoleArn": "arn:aws:iam::XXXXXXXXXX:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "memory": "1024",
    "cpu": "512",
    "containerDefinitions": [
        {
            "essential": true,
            "image": "XXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:init-ignore-firelens-2",
            "name": "log_router",
            "firelensConfiguration": {
                "type": "fluentbit"
            },
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "firelens-container",
                    "awslogs-region": "us-west-2",
                    "awslogs-create-group": "true",
                    "awslogs-stream-prefix": "init-ignore-firelens"
                }
            },
            "environment": [
                {
                    "name": "aws_fluent_bit_init_s3_1",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/dummy-filter.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_2",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/dummy-input.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_3",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/dummy-parser.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_4",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/stdout.conf"
                },
                {
                    "name": "aws_fluent_bit_init_ignore_firelens_config",
                    "value": "true"
                },
                {
                    "name": "FLB_LOG_LEVEL",
                    "value": "debug"
                }
            ],
            "memoryReservation": 50
        },
        {
            "essential": true,
            "image": "httpd",
            "name": "app",
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Name": "cloudwatch_logs",
                    "region": "${AWS_REGION}",
                    "log_group_name": "${ECS_CLUSTER}/application",
                    "auto_create_group": "true",
                    "log_stream_name": "${ECS_TASK_ID}",
                    "retry_limit": "2"
                }
            },
            "memoryReservation": 100
        }
    ]
}

Case 1: ignore set to true

Result from exec'ing into the task and checking the config:

sh-4.2# ls
dummy-filter.conf  dummy-input.conf  dummy-parser.conf	stdout.conf
sh-4.2# ls
dummy-filter.conf  dummy-input.conf  dummy-parser.conf	stdout.conf
sh-4.2# cd ..
sh-4.2# ls
fluent-bit-init-s3-files  fluent-bit-init.conf	fluent_bit_init_entrypoint.sh  fluent_bit_init_process	invoke_fluent_bit.sh
sh-4.2# cat fluent-bit-init.conf
@INCLUDE /init/fluent-bit-init-s3-files/stdout.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-input.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-filter.conf
sh-4.2# cat fluent_bit_init_entrypoint.sh
./init/fluent_bit_init_process
source /init/invoke_fluent_bit.sh
sh-4.2# cat invoke_fluent_bit.sh
export FLB_AWS_USER_AGENT=ecs-init
export AWS_REGION=us-west-2
export AWS_AVAILABILITY_ZONE=us-west-2a
export ECS_CLUSTER=firelens-private-subnets
export ECS_TASK_ARN=arn:aws:ecs:us-west-2:144718711470:task/firelens-private-subnets/002dd1eaa1544ac79cde672daedb4a83
export ECS_TASK_ID=002dd1eaa1544ac79cde672daedb4a83
export ECS_FAMILY=init-ignore-firelens
export ECS_LAUNCH_TYPE=FARGATE
export ECS_REVISION=3
export ECS_TASK_DEFINITION=init-ignore-firelens:3
exec /fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -c /init/fluent-bit-init.conf -R /init/fluent-bit-init-s3-files/dummy-parser.conf

Note there is no include for the default FireLens Fluent Bit config path.

Case 2: Ignore set to false

Starting session with SessionId: ecs-execute-command-02fe7250f544ea00e
sh-4.2# ls
AWS_FOR_FLUENT_BIT_VERSION  bin   dev  entrypoint.sh  fluent-bit  init	lib64  managed-agents  mnt  proc  run	srv  tmp  var
Dockerfile.1.main-release   boot  ecs  etc	      home	  lib	local  media	       opt  root  sbin	sys  usr
sh-4.2# cd init
sh-4.2# ls
fluent-bit-init-s3-files  fluent-bit-init.conf	fluent_bit_init_entrypoint.sh  fluent_bit_init_process	invoke_fluent_bit.sh
sh-4.2# cat fluent-bit-init.conf
@INCLUDE /fluent-bit/etc/fluent-bit.conf
@INCLUDE /init/fluent-bit-init-s3-files/stdout.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-input.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-filter.conf
sh-4.2# cat fluent_bit_init_entrypoint.sh
./init/fluent_bit_init_process
source /init/invoke_fluent_bit.sh
sh-4.2#

Note the @INCLUDE /fluent-bit/etc/fluent-bit.conf.

Case 3: Ignore not defined (regression check)

sh-4.2# ls
dummy-filter.conf  dummy-input.conf  dummy-parser.conf	stdout.conf
sh-4.2# cd ..
sh-4.2# ls
fluent-bit-init-s3-files  fluent-bit-init.conf	fluent_bit_init_entrypoint.sh  fluent_bit_init_process	invoke_fluent_bit.sh
sh-4.2# cat fluent-bit-init.conf
@INCLUDE /fluent-bit/etc/fluent-bit.conf
@INCLUDE /init/fluent-bit-init-s3-files/stdout.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-input.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-filter.conf

Normal behavior.

Unit tests

go test -v passes in the init directory.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
… config

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
@PettitWesley PettitWesley requested a review from a team as a code owner March 20, 2024 18:08
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
@PettitWesley
Copy link
Contributor Author

FireLens example: aws-samples/amazon-ecs-firelens-examples#127

Comment on lines +162 to +163
Mem_Buf_Limit 100MB
```
Copy link
Contributor

@matthewfala matthewfala Mar 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be helpful to add an example with the appropriate match statement for docker logs.

    Match               Application-firelens-*

Copy link
Contributor

@matthewfala matthewfala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! One comment.

@PettitWesley PettitWesley merged commit 2d2eb2f into aws:develop Mar 25, 2024
@PettitWesley PettitWesley deleted the init-ignore-firelens branch March 25, 2024 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants