First steps to nginx config template #4397

acozine · 2023-10-11T19:34:12Z

Partial fix for #4349.

This PR:

creates a template for nginx config files
adds four sites to the vars consumed by the template (plus more, commented out)
adds tasks to create and upload site configs for those four sites using the template, to the /tmp directory so we can test/diff
removes tasks for Stream config, which we are not using
uses | instead of : in task names in the upload-config.yml file so we don't have to quote them
uses FQCN for tasks in the upload-config.yml file

bess

😍

maxkadel

Looks good! Left a couple questions, but I really like where this is going!

maxkadel · 2023-10-11T19:46:52Z

roles/nginxplus/tasks/conf/upload-config.yml

+  ansible.builtin.template:
+    src: http/library.conf.j2
+    dest: /tmp/{{ item.name }}.conf
+    # dest: /etc/nginx/conf.d/{{ item.name }}.conf


Should we put in a comment explaining that this is where they will eventually go, once testing is complete, or is that self-explanatory?

maxkadel · 2023-10-11T19:54:40Z

roles/nginxplus/templates/http/library.conf.j2

+        proxy_intercept_errors on;
+        # TODO don't do a health check if there's only one server
+      {% if item.health_check_URI is defined %}
+        health_check interval=10 fails=3 passes=2 uri={{ item.health_check_URI }};


Could health_check_URI have a default of / (which I'm guessing is the nginx default) and thus avoid a conditional? I think it's fine either way, just trying to think of ways to simplify.

I'm kinda cheating here. We don't want to do health checks on all sites, because for sites with only one server a health check is all overhead and no gain. This way we can use "health_check_URI" tell which sites should have one at all. But yeah, your solution is a good one longer-term.

maxkadel · 2023-10-11T19:57:37Z

roles/nginxplus/vars/main.yml

-# vars file for roles/nginxplus
+# the 'sites' dictionary is used by the library.conf.j2 template
+sites:
+  # - name: example-prod


Really appreciate this in-file documentation

maxkadel · 2023-10-11T19:59:34Z

roles/nginxplus/vars/main.yml

+    location: "/"
+    visibility: private
+    app_protect: enabled
+  - name: allsearch-api-prod


This might not have been in when you started this, but I think you could do allsearch and allsearch-staging as well (in addition to the api-only boxes)

Cool! I'll add them in.

acozine · 2023-10-17T16:51:21Z

Try tigerdata staging and prod, orcid staging and prod.

acozine · 2023-10-19T19:54:11Z

For bibdata, we need to have three additional proxy settings:

proxy_connect_timeout
proxy_send_timeout
proxy_read_timeout

All are set to 2h - is this a reasonable default for all our sites?

acozine · 2023-11-03T22:30:36Z

Known remaining work to a first-round merge:

add a directory for templated config
include the templated config directory in the main nginx config file, so both static and templated config files are loaded by the load balancer
update the task to send the templated files to that new directory
remove the template_test tag
find a way to remove unneeded/old files from the /etc/nginx/conf.d/dynamic directory
make sure the SSL directories on the load balancer match the directory names in the /etc/nginx/conf.d/dynamic/*.conf files. Use roles/nginxplus/vars/main.yml to create the SSL certificates in the appropriate directory structure.
Update /etc/nginx/nginx.conf to include the dynamic directory configs, either manually or by running playbooks/nginxplus_production_rebuild.yml.
test playbook again

acozine · 2023-12-09T00:00:12Z

Gist for current blocker: https://gist.github.com/acozine/889141cde3f713d8c4b26b83fe93d334

acozine · 2023-12-22T21:15:56Z

@sandbergja and I tested the tasks for identifying and removing obsolete dynamic config files today. I added tasks for doing the same for obsolete static config files. I think after the break we can start moving the easy configs over. Procedure:

confirm that the SSL certs for the first sites to migrate have the correct dir names
update the main config to include the new dynamic config directory
remove the static config files for the first sites to migrate
run the playbook and test

If that works as expected, then we can merge this PR and migrate one or two configs per PR for the next few months.

Co-authored-by: Jane Sandberg <sandbergja@users.noreply.github.com>

Co-authored-by: Christina Chortaria <christinach@users.noreply.github.com>

Co-authored-by: Jane Sandberg <sandbergja@users.noreply.github.com>

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>

Co-authored-by: Jane Sandberg <sandbergja@users.noreply.github.com>

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>

Co-authored-by: Christina Chortaria <christinach@users.noreply.github.com> Co-authored-by: Francis Kayiwa <kayiwa@users.noreply.github.com> Co-authored-by: Jane Sandberg <sandbergja@users.noreply.github.com> Co-authored-by: Ryan Laddusaw <rladdusaw@users.noreply.github.com> Co-authored-by: Vickie Karasic <vickiekarasic@users.noreply.github.com>

Co-authored-by: Francis Kayiwa <kayiwa@users.noreply.github.com> Co-authored-by: Vickie Karasic <vickiekarasic@users.noreply.github.com>

Co-authored-by: Jane Sandberg <sandbergja@users.noreply.github.com>

…lete the entire directory when removing a single site

acozine · 2023-12-22T21:17:18Z

Rebased ahead of the holidays.

acozine · 2024-01-30T17:49:07Z

Next (final?) steps for current PR:

backup config directories for easy rollback
run nginx -t after main config tasks are done
~~if nginx -t fails, restore prior config from the backed up directories~~

Future work before we move our most complicated sites to the config template:

engineer this so we can, from a branch, test our configs without going live - so for a test run we could just put config in non-active directories for comparison and debugging
Workflow for migrating a site from static to templatized config would then be:

open a branch, add vars to make the templatized config
run playbook from branch with the flag or other solution to output all configs to a non-working directory
compare static and templatized configs and do any needed debugging
delete static config on branch once things look good
run the playbook from branch "for realz" to affect actual working config

acozine · 2024-02-06T17:53:18Z

More progress today. When the nginx -t verification fails, instead of doing an automated recovery from the backed-up config directory, we decided it's better to post a message with the failure message - something like:
Nginx config update failed. Time to get a human involved, please restore from <dir>.

acozine · 2024-03-06T17:11:19Z

Closing in favor of #4654

bess approved these changes Oct 11, 2023

View reviewed changes

acozine mentioned this pull request Oct 11, 2023

Use a template for our simplest nginx configs #4231

Closed

maxkadel approved these changes Oct 11, 2023

View reviewed changes

acozine force-pushed the nginx_template_intro branch from 462a385 to 80b4544 Compare October 17, 2023 19:56

sandbergja force-pushed the nginx_template_intro branch from 855266a to 4ca996c Compare November 3, 2023 22:20

acozine and others added 20 commits December 22, 2023 15:16

moderately mature template

db653d2

add first template-able sites to vars/main.yml

33021fb

tasks for site config template and vars

52fb90b

we are not using stream config

c92c0f8

testing showed us we needed these fixes

b976602

Co-authored-by: Jane Sandberg <sandbergja@users.noreply.github.com>

fix typo

204623a

fix template to work for bibdata-staging config

2a6111e

Co-authored-by: Christina Chortaria <christinach@users.noreply.github.com>

adds fields for proxy timeouts, configures for bibdata and default

f5fff9c

Co-authored-by: Jane Sandberg <sandbergja@users.noreply.github.com>

Set proxy_*_timeout defaults in the template, rather than variables

18bc3d6

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>

add dir and task for dynamic (template-driven) config files for nginx

987bc1b

Co-authored-by: Jane Sandberg <sandbergja@users.noreply.github.com>

Dynamic config upload should have the update-conf tag

4f85e95

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>

Update dynamic nginx config to include new abid boxes

a391b6e

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>

Use dynamic configs for some servers, but comment out abid-prod

20633f0

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>

can we get specific data?

8335250

Co-authored-by: Francis Kayiwa <kayiwa@users.noreply.github.com> Co-authored-by: Vickie Karasic <vickiekarasic@users.noreply.github.com>

like this?

1f0b9a4

Co-authored-by: Francis Kayiwa <kayiwa@users.noreply.github.com> Co-authored-by: Vickie Karasic <vickiekarasic@users.noreply.github.com>

simpler yet

3e1d006

Co-authored-by: Francis Kayiwa <kayiwa@users.noreply.github.com> Co-authored-by: Vickie Karasic <vickiekarasic@users.noreply.github.com>

loop!

0130d7c

Co-authored-by: Francis Kayiwa <kayiwa@users.noreply.github.com> Co-authored-by: Vickie Karasic <vickiekarasic@users.noreply.github.com>

acozine and others added 9 commits December 22, 2023 15:16

restore view

b0a01ea

Co-authored-by: Francis Kayiwa <kayiwa@users.noreply.github.com> Co-authored-by: Vickie Karasic <vickiekarasic@users.noreply.github.com>

needs curly braces?

0f10617

and not here?

86f3f54

is this better?

f973a9a

yeehaw this works!

fd1cf7f

Co-authored-by: Jane Sandberg <sandbergja@users.noreply.github.com>

tidy var names

e975a0f

oops, tidy that last task too

3527d2e

find and delete obsolete static config files, so we do not have to de…

df64ab6

…lete the entire directory when removing a single site

do not make backups of nginx configs, unknown files will bork the server

caaf0e3

acozine force-pushed the nginx_template_intro branch from 3e9b7c4 to caaf0e3 Compare December 22, 2023 21:16

sandbergja mentioned this pull request Jan 12, 2024

Adjust Error Handling for the All Search API #4396

Closed

adds tag for testing templated configs, so we get the registered var

61236ce

acozine mentioned this pull request Feb 9, 2024

Offer a template for nginx config files #4654

Draft

acozine closed this Mar 6, 2024

acozine deleted the nginx_template_intro branch May 24, 2024 16:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First steps to nginx config template #4397

First steps to nginx config template #4397

acozine commented Oct 11, 2023

bess left a comment

maxkadel left a comment

maxkadel Oct 11, 2023

maxkadel Oct 11, 2023

acozine Oct 11, 2023

maxkadel Oct 11, 2023

maxkadel Oct 11, 2023

acozine Oct 11, 2023

acozine commented Oct 17, 2023

acozine commented Oct 19, 2023

acozine commented Nov 3, 2023 •

edited

Loading

acozine commented Dec 9, 2023

acozine commented Dec 22, 2023

acozine commented Dec 22, 2023

acozine commented Jan 30, 2024 •

edited

Loading

acozine commented Feb 6, 2024 •

edited

Loading

acozine commented Mar 6, 2024

First steps to nginx config template #4397

First steps to nginx config template #4397

Conversation

acozine commented Oct 11, 2023

bess left a comment

Choose a reason for hiding this comment

maxkadel left a comment

Choose a reason for hiding this comment

maxkadel Oct 11, 2023

Choose a reason for hiding this comment

maxkadel Oct 11, 2023

Choose a reason for hiding this comment

acozine Oct 11, 2023

Choose a reason for hiding this comment

maxkadel Oct 11, 2023

Choose a reason for hiding this comment

maxkadel Oct 11, 2023

Choose a reason for hiding this comment

acozine Oct 11, 2023

Choose a reason for hiding this comment

acozine commented Oct 17, 2023

acozine commented Oct 19, 2023

acozine commented Nov 3, 2023 • edited Loading

acozine commented Dec 9, 2023

acozine commented Dec 22, 2023

acozine commented Dec 22, 2023

acozine commented Jan 30, 2024 • edited Loading

acozine commented Feb 6, 2024 • edited Loading

acozine commented Mar 6, 2024

acozine commented Nov 3, 2023 •

edited

Loading

acozine commented Jan 30, 2024 •

edited

Loading

acozine commented Feb 6, 2024 •

edited

Loading