create RP branch to use as local master per PR comment #2
Conversation
set local default to 5 days of log retention
.gitignore
Outdated
| # PyBuilder | ||
| target/ | ||
| ## Directory-based project format: | ||
| .idea/ |
There was a problem hiding this comment.
I believe this should cover all of the contents of .idea/ below
There was a problem hiding this comment.
I'd rather keep this as close as possible to the original. I think most of this stuff does not apply to this project. You can move it to a .gitignore_global if you need it for your own dev environment.
| @@ -20,7 +20,7 @@ | |||
| SCHEDULE_INTERVAL = "@daily" # How often to Run. @daily - Once a day at Midnight | |||
| DAG_OWNER_NAME = "operations" # Who is listed as the owner of this DAG in the Airflow Web Server | |||
| ALERT_EMAIL_ADDRESSES = [] # List of email address to send email alerts to if this job fails | |||
| DEFAULT_MAX_LOG_AGE_IN_DAYS = 30 # Length to retain the log files if not already provided in the conf. If this is set to 30, the job will remove those files that are 30 days old or odler | |||
| DEFAULT_MAX_LOG_AGE_IN_DAYS = 5 # Length to retain the log files if not already provided in the conf. If this is set to 30, the job will remove those files that are 30 days old or odler | |||
There was a problem hiding this comment.
Is it possible to use the maxLogAgeInDays config option instead of changing this?
There was a problem hiding this comment.
perhaps, but I don't know how to do that. The way this is written requires passing a command line arg into airflow itself. The comments show a manual invocation:
airflow trigger_dag --conf '{"maxLogAgeInDays":30}' airflow-log-cleanup
would that perhaps be done by putting a value into the "default_args" object?
There was a problem hiding this comment.
The above airflow cli command should work but it means having another service run on a timer (or yet another DAG to invoke a DAG).
There was a problem hiding this comment.
What do you think about the "another DAG to invoke a DAG" approach?
There was a problem hiding this comment.
I'd rather convert this to a systemd job than create more DAGs to manage DAGs to manage log files. The way the airflow vars are defined and passed seems to change frequently. The --conf mechanism appears to have be introduced as a partial response to this exact issue.
There was a problem hiding this comment.
we could rewrite this DAG to pass in a more standard "params" object I think.
|
I updated the default repo branch |
set local default to 5 days of log retention
I do not seem to have privilege to set 'rubiconproject' branch as the git default.