Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Improve Airbyte CI logging #20517

Merged
merged 26 commits into from
Dec 27, 2022
Merged

fix: Improve Airbyte CI logging #20517

merged 26 commits into from
Dec 27, 2022

Conversation

xpuska513
Copy link
Contributor

What

  • Improve Airbyte CI job logging by introducing fluent-bit agent which collect all the logs and sends it to the s3 bucket for future usage
  • fix: improve debug_mode parameter by setting its type to choice

@xpuska513
add fluent-bit logging to k3s setup
f5b2390
@xpuska513
Fix --set for helm
0a89af7
@xpuska513
add --generate-name
74f038d
@xpuska513
fix slash in multiline bash
264342b
@xpuska513
Update --set for helm
b08d2f2
@xpuska513
Use generate-name instead
45d0d9f
@xpuska513
Add debug
e2ac390
@xpuska513
Remove deubg for helm
0d468ce
@xpuska513
Fix env again
9763bb2
@xpuska513
Disable build
1fbfd9f
@xpuska513
Remove everything
e7efe75
@xpuska513
Disable everything again
13eac9e
@xpuska513
Snap back to reallity
7748328
@xpuska513
Update fluent_values.yaml
df99100
@xpuska513
Use role instead of AWS creds
7d12f9b
@xpuska513
Use new bucket
ded553e
@xpuska513
Merge branch 'kskobylko/improve-airbyte-ci-logging' of github.com:air…
a9ba314 
…bytehq/airbyte into kskobylko/improve-airbyte-ci-logging
@xpuska513
Revert changes
df7b55b
@xpuska513
Update bucket name
ead967d
@xpuska513
Play around with fluent-bit config
3e0d0a6
@xpuska513
Escape string in helm --set arg
20a8585
@xpuska513
Update fluent-bit config + update --set argument
8c7536e
@xpuska513
Update fluentbit config so it"ll create s3 path correctly
2dfd1a4
@xpuska513
Add logging to kustomize
acb17a6
@xpuska513
Revert debug mode
e92b9a8
@xpuska513 xpuska513 requested a review from a team December 15, 2022 14:19
@xpuska513 xpuska513 temporarily deployed to more-secrets December 15, 2022 14:20 — with GitHub Actions Inactive
@xpuska513 xpuska513 temporarily deployed to more-secrets December 15, 2022 14:21 — with GitHub Actions Inactive
@xpuska513 xpuska513 changed the title fix: Improve Airbyte CI logging for fix: Improve Airbyte CI logging Dec 15, 2022
@supertopher supertopher requested a review from a team December 15, 2022 14:35
c-p-b
c-p-b suggested changes Dec 15, 2022
View reviewed changes
Copy link
Contributor

@c-p-b c-p-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs some internal docs on how to access the logs for end users. Otherwise LGTM

@xpuska513 xpuska513 requested a review from c-p-b December 27, 2022 17:28
@c-p-b
Copy link
Contributor

c-p-b commented Dec 27, 2022

Verified that logs docs were deployed on our internal site. Can be accessed by searching for "Accessing Airbyte CI Job Logs"

@xpuska513 xpuska513 temporarily deployed to more-secrets December 27, 2022 17:34 — with GitHub Actions Inactive
@xpuska513 xpuska513 temporarily deployed to more-secrets December 27, 2022 17:35 — with GitHub Actions Inactive
@xpuska513 xpuska513 merged commit 9da0de1 into master Dec 27, 2022
@xpuska513 xpuska513 deleted the kskobylko/improve-airbyte-ci-logging branch December 27, 2022 17:53
@github-actions github-actions bot mentioned this pull request Dec 29, 2022
Merged
This was referenced Jan 6, 2023
Closed
Closed
This was referenced Jan 6, 2023
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@c-p-b c-p-b c-p-b approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@xpuska513 @c-p-b