Kube Logs are stored and retrieved from Cloud Storage. #4053

davinchia · 2021-06-11T11:23:47Z

Remaining work
[X] Modify API to read from Cloud. In general, we should defer to local if the log file exists, and Cloud Storage if it does not.
[x] Figure out configuration to also support GCS.
[X] Modify Kube deployment to support Cloud.
[x] Add better comments and log test to the Docker and Kube acceptance tests.

Future work (will create tickets for these):

Implement for GCS and Azure. Implement Kube Logging for other Cloud Providers. #4200
Make list operation more efficient. airbytehq/airbyte-internal-issues#143
Add a validator in the server that checks at least one Cloud Storage logger has been correctly configured. This should error out if we are in the Kubernetes environment. Airbyte Kubernetes should error if Cloud Storage logs aren't configured. #4202

… preserving local logs.

… substitution in configuration.

…e aws s3 v2 sdk. Also rename aws credentials env var to match what is expected by aws.

… iteration.

davinchia · 2021-06-16T16:02:23Z

airbyte-commons/src/main/resources/log4j2.xml

                        <PatternLayout pattern="${default-worker-file-pattern}"/>
                    </File>
                </Route>
            </Routes>
            <IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
        </Routing>
+        <Routing name="LogSplitCloud">


will comment why we had to configure in this way - essentially only accepts one appender.

davinchia · 2021-06-16T16:03:00Z

airbyte-config/models/src/main/java/io/airbyte/config/helpers/LogHelpers.java


 public class LogHelpers {

-  // if you update these values, you must also update log4j2.xml
+  private static final Logger LOGGER = LoggerFactory.getLogger(LogHelpers.class);


should all the logging related stuff be moved to it's own module? or can it stay in models for now?

Can stay for now but should be split eventually.

airbyte-config/models/build.gradle

airbyte-server/src/main/java/io/airbyte/server/converters/JobConverter.java

…rking on Kube.

This reverts commit 19db827.

…g this ID. Consolidate all MDC operations to setMDC function in WorkerUtils. Remove unused functions.

…orking. Move logging test to be part of Kube integration tests.

.env

jrhizor · 2021-06-18T15:24:09Z

airbyte-commons/src/main/java/io/airbyte/commons/json/Jsons.java

@@ -50,7 +50,7 @@
  private static final ObjectWriter OBJECT_WRITER = OBJECT_MAPPER.writer(new JsonPrettyPrinter());

  private static ObjectMapper initMapper() {
-    final ObjectMapper result = new ObjectMapper();
+    final ObjectMapper result = new ObjectMapper().findAndRegisterModules();


jrhizor · 2021-06-18T15:24:12Z

airbyte-commons/build.gradle

@@ -3,6 +3,5 @@ plugins {
 }

 dependencies {
-    implementation 'org.apache.commons:commons-compress:1.20'
-    implementation 'org.apache.commons:commons-lang3:3.11'
+    // Dependencies for this module should be specified in the top-level build.gradle.


Because of https://github.com/airbytehq/airbyte/blob/master/build.gradle#L181, if everything is depending on airbyte-commons, it seems cleaner to have all of common's dependencies in the build.gradle as well.

jrhizor · 2021-06-18T15:24:24Z

airbyte-commons/src/main/java/io/airbyte/commons/yaml/Yamls.java

@@ -45,7 +45,7 @@

  public static final YAMLFactory YAML_FACTORY = new YAMLFactory();
  // Object Mapper is thread-safe
-  private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(YAML_FACTORY);
+  private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(YAML_FACTORY).findAndRegisterModules();


airbyte-config/models/src/main/java/io/airbyte/config/EnvConfigs.java

jrhizor · 2021-06-18T15:26:16Z

airbyte-config/models/src/main/java/io/airbyte/config/helpers/LogHelpers.java


 public class LogHelpers {

-  // if you update these values, you must also update log4j2.xml
+  private static final Logger LOGGER = LoggerFactory.getLogger(LogHelpers.class);


Can stay for now but should be split eventually.

airbyte-config/models/src/main/java/io/airbyte/config/helpers/LogHelpers.java

airbyte-scheduler/app/src/main/java/io/airbyte/scheduler/app/JobSubmitter.java

davinchia · 2021-06-19T02:51:10Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/951654970
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/951654970

davinchia · 2021-06-19T02:51:23Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/951655271
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/951655271

davinchia · 2021-06-19T02:51:27Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/951655452
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/951655452

Set Log4j2 debug mode. Successfull route job and app logs to S3 while…

a0827fa

… preserving local logs.

davinchia changed the title ~~Kube Logging POC.~~ WIP: Kube Logging POC. Jun 11, 2021

Turn everything into environment variables. Successfully use property…

814449f

… substitution in configuration.

davinchia changed the title ~~WIP: Kube Logging POC.~~ WIP: Kube Logging. Jun 14, 2021

davinchia added 2 commits June 14, 2021 21:10

Checkpoint: Get GCS working.

aafa5bd

Checkpoint: Get all objects out of a directory and write it to a file.

d4e611f

github-actions bot added the area/platform issues related to the platform label Jun 15, 2021

davinchia added 14 commits June 16, 2021 15:19

Checkpoint: Get scheduler and server logs from Cloud working using th…

ff959a3

…e aws s3 v2 sdk. Also rename aws credentials env var to match what is expected by aws.

Setup to test S3LoggingClient.

09c6d1d

One initial test in S3Logs class.

98bca11

Fill in normal test.

329bfd1

Look at a specific folder.

7d97575

Correct error.

e9bd71d

Get logs tail working.

35d1abf

Format.

6415d55

Pass test.

3ff34b7

Make logging test script executable. Disable most of build for faster…

c9e89cb

… iteration.

Mount S3 secret for logging test.

0299f44

Strip quotes from jq output.

746a593

Uncomment rest of build.

b3320c3

Add comments.

23c8c93

davinchia commented Jun 16, 2021

View reviewed changes

airbyte-config/models/build.gradle Outdated Show resolved Hide resolved

davinchia commented Jun 16, 2021

View reviewed changes

airbyte-server/src/main/java/io/airbyte/server/converters/JobConverter.java Show resolved Hide resolved

davinchia added 4 commits June 17, 2021 16:22

Checkpoint: Kube modifications. Scheduler, Server and Job logs are wo…

afbe9d2

…rking on Kube.

Remove affinity.

19db827

Revert "Remove affinity."

858090b

This reverts commit 19db827.

Checkpoint: Stop emitting job id with MDC as we are not actually usin…

0dcb92b

…g this ID. Consolidate all MDC operations to setMDC function in WorkerUtils. Remove unused functions.

github-actions bot added the area/worker Related to worker label Jun 17, 2021

davinchia added 4 commits June 18, 2021 16:00

Remove unused plugin.

e259fa3

Remove GCP configuration for now.

9801789

Remove logic added for GKE bug.

e062f71

Checkpoint: Job and app logs work. Get S3LogClient integration test w…

af615e1

…orking. Move logging test to be part of Kube integration tests.

davinchia changed the title ~~WIP: Kube Logging.~~ Kube Logs are stored and retrieved from Cloud Storage. Jun 18, 2021

davinchia added 9 commits June 18, 2021 17:38

Fix worker log test.

0d8ccbc

Fix job scheduler test.

35e998a

Correct cloud storage test path. Fix format.

ede5e42

Stash to check if master acceptance tests work.

2cb5730

Does registering all modules fix test?

b49c7c9

Add S3 credentials information to Kube acceptance tests.

8be0a6e

Add comments to explain Log4j config.

786d594

Remove unneeded package.

9858624

Remove unneeded comments.

d0043b3

jrhizor approved these changes Jun 18, 2021

View reviewed changes

davinchia added 4 commits June 19, 2021 09:28

Respond to PR feedback.

31dd280

Explicitly require Jackson date time module.

7014ea3

Format.

d6b949d

Migrate all mappers to use injected function.

22eb741

github-actions bot added the area/connectors Connector related issues label Jun 19, 2021

davinchia marked this pull request as ready for review June 19, 2021 02:50

More format.

2c4bd4c

davinchia merged commit e60f536 into master Jun 19, 2021

davinchia deleted the davinchia/s3-logging-experiment branch June 19, 2021 04:15

davinchia mentioned this pull request Jun 21, 2021

allow writing logs to cloud storage in kubernetes #3639

Closed

karinakuz added connectors/destinations-warehouse connectors/destination/snowflake labels Jan 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kube Logs are stored and retrieved from Cloud Storage. #4053

Kube Logs are stored and retrieved from Cloud Storage. #4053

davinchia commented Jun 11, 2021 •

edited

Loading

davinchia Jun 16, 2021

davinchia Jun 16, 2021

jrhizor Jun 18, 2021

jrhizor Jun 18, 2021

jrhizor Jun 18, 2021

davinchia Jun 19, 2021

jrhizor Jun 18, 2021

jrhizor Jun 18, 2021

davinchia commented Jun 19, 2021 •

edited by github-actions bot

Loading

davinchia commented Jun 19, 2021 •

edited by github-actions bot

Loading

davinchia commented Jun 19, 2021 •

edited by github-actions bot

Loading

Kube Logs are stored and retrieved from Cloud Storage. #4053

Kube Logs are stored and retrieved from Cloud Storage. #4053

Conversation

davinchia commented Jun 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davinchia commented Jun 19, 2021 • edited by github-actions bot Loading

davinchia commented Jun 19, 2021 • edited by github-actions bot Loading

davinchia commented Jun 19, 2021 • edited by github-actions bot Loading

davinchia commented Jun 11, 2021 •

edited

Loading

davinchia commented Jun 19, 2021 •

edited by github-actions bot

Loading

davinchia commented Jun 19, 2021 •

edited by github-actions bot

Loading

davinchia commented Jun 19, 2021 •

edited by github-actions bot

Loading