[WIP] Add nginx-level API endpoint caching #943
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aronasorman
commented
Sep 1, 2018
| # ignore any get params, cache by our cookie sessionid value | ||
| proxy_cache_key $scheme$proxy_host$uri$cookie_kolibri_studio_sessionid; | ||
| # next two directives make nginx serve the cached value even when we're refreshing it | ||
| proxy_cache_use_stale updating error; |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
kollivier
approved these changes
Sep 1, 2018
micahscopes
added a commit
that referenced
this pull request
Sep 8, 2018
…ass (#940) * Add chef method unit tests and ensure that Django views return a HTTP response when they raise errors. * Add more file upload and add node tests. * Add finish_commit tests and return a HTTP 500 code instead of raising if an error occurs. * Get Sentry reports for handled internal server errors in Internal API calls. * Add yarn run devsetup command (#925) * add new yarn command to set up dev environment * change commands to remove redundant pieces to prioritize yarn run devsetup Streamlines the new dev experience * Updated translations with epub string * Fixed default preview styling * Add caching to the get_user_public_channels endpoint (#939) * Move view get_public_channels logic to Channel model * add the Channel.make_public function * use StudioTestCase to create buckets * add ChannelCacher class to implement channel caching * add channel token related convenience functions * make exporchannel use Channel.make_token() * add channel specific cache for tokens Cache that, one less query to make. * add caching to channel token serializer endpoint * move serializers.get_resource_count implementation to the Channel model * use channel.get_thumbnail() function on serializers.py * remove redefinition of generate_thumbnail_url * add ChannelCacher.get_resource_count() cache function Used to cache the channel's get_resource_count * add caching to get_resource_count API attribute * add channel.get_date_modified function * make channelfieldmixin use channel.get_date_modified * add channel.get_date_modified cache * Use the channel cache for get_date_modified * remove redundant generate_thumbnail_url Already defined in ChannelFieldMixin * pass cache get_public_channels args to real function * cache the entire get_user_public_channels view * fix tests for ChannelCacher.get_public_channels() by comparing actual channel ids rather than objects * Create SecretToken.exists() convenience method * refactor make_token to definitely end after 100 attempts And use for-else loop construct * add extra non-public assert on the make_public test * add test for SecretToken.exists() * [WIP] Add nginx-level API endpoint caching (#943) * add long running caching to the get_user_public_channels endpoint * add caching to the get_user_edit_channels endpoint * add 4 hour browser caching on all static files (#945) * add caching to the get_user_edit_channels endpoint (#947) * Unlock le-utils' version * have pipenv update all updateable packages * Added comment * Made resource count required on storage requests * Merge in develop * Return 404s for any files not found in the zip, and have BadZipfile errors report the name, size, mode and download state to help us narrow down problems with opening. Also, tests. (#942) * Remove the hardcoded server IP in tests so that configuration changes don't break them. * Apply Micah's changes and also fix some new issues uncovered by tests to get the code to the same state as it was before revert. * Fix login template name. * Actually commit the updated dependency versions to requirements.txt... * use `==` instead of `=` in template `if` statement * Re-delete requirements.txt as a result of rebasing issues. * Update Django to 1.11 in pipfile after rebase. * Actually update the dependencies after accidentally using pip... * objects.create automatically calls save, so no need to call it again afterwards. * objects.create automatically calls save, so no need to call it again afterwards. * Revert removed calls to save as we are creating models directly rather than using objects.create. * Revert removed calls to save as we are creating models directly rather than using objects.create. * update django to 1.11.15! * use the recommended way of declaring our AppConfig Through our INSTALLED_APPS! See https://docs.djangoproject.com/en/1.11/ref/applications/#configuring-applications * use FormatPresetSerializer as a PrimaryKeyRelatedField serializer * disable breakpoints * attempted fix for file uploading issues * file uploading works... yay! * signup form multiselection area is working now * enable offline_helper * remove extra debugging leftovers from registration info template * fixes for several tests... remember: don't manually hardcode constants from le_utils. they may change some day! * fixed authentication tests * add a migration from django 1.11
jayoshih
pushed a commit
to jayoshih/content-curation
that referenced
this pull request
Sep 13, 2018
* add long running caching to the get_user_public_channels endpoint * add caching to the get_user_edit_channels endpoint
micahscopes
added a commit
that referenced
this pull request
Sep 17, 2018
…ass (#940) * Add chef method unit tests and ensure that Django views return a HTTP response when they raise errors. * Add more file upload and add node tests. * Add finish_commit tests and return a HTTP 500 code instead of raising if an error occurs. * Get Sentry reports for handled internal server errors in Internal API calls. * Add yarn run devsetup command (#925) * add new yarn command to set up dev environment * change commands to remove redundant pieces to prioritize yarn run devsetup Streamlines the new dev experience * Updated translations with epub string * Fixed default preview styling * Add caching to the get_user_public_channels endpoint (#939) * Move view get_public_channels logic to Channel model * add the Channel.make_public function * use StudioTestCase to create buckets * add ChannelCacher class to implement channel caching * add channel token related convenience functions * make exporchannel use Channel.make_token() * add channel specific cache for tokens Cache that, one less query to make. * add caching to channel token serializer endpoint * move serializers.get_resource_count implementation to the Channel model * use channel.get_thumbnail() function on serializers.py * remove redefinition of generate_thumbnail_url * add ChannelCacher.get_resource_count() cache function Used to cache the channel's get_resource_count * add caching to get_resource_count API attribute * add channel.get_date_modified function * make channelfieldmixin use channel.get_date_modified * add channel.get_date_modified cache * Use the channel cache for get_date_modified * remove redundant generate_thumbnail_url Already defined in ChannelFieldMixin * pass cache get_public_channels args to real function * cache the entire get_user_public_channels view * fix tests for ChannelCacher.get_public_channels() by comparing actual channel ids rather than objects * Create SecretToken.exists() convenience method * refactor make_token to definitely end after 100 attempts And use for-else loop construct * add extra non-public assert on the make_public test * add test for SecretToken.exists() * [WIP] Add nginx-level API endpoint caching (#943) * add long running caching to the get_user_public_channels endpoint * add caching to the get_user_edit_channels endpoint * add 4 hour browser caching on all static files (#945) * add caching to the get_user_edit_channels endpoint (#947) * Unlock le-utils' version * have pipenv update all updateable packages * Added comment * Made resource count required on storage requests * Merge in develop * Return 404s for any files not found in the zip, and have BadZipfile errors report the name, size, mode and download state to help us narrow down problems with opening. Also, tests. (#942) * Remove the hardcoded server IP in tests so that configuration changes don't break them. * Apply Micah's changes and also fix some new issues uncovered by tests to get the code to the same state as it was before revert. * Fix login template name. * Actually commit the updated dependency versions to requirements.txt... * use `==` instead of `=` in template `if` statement * Re-delete requirements.txt as a result of rebasing issues. * Update Django to 1.11 in pipfile after rebase. * Actually update the dependencies after accidentally using pip... * objects.create automatically calls save, so no need to call it again afterwards. * objects.create automatically calls save, so no need to call it again afterwards. * Revert removed calls to save as we are creating models directly rather than using objects.create. * Revert removed calls to save as we are creating models directly rather than using objects.create. * update django to 1.11.15! * use the recommended way of declaring our AppConfig Through our INSTALLED_APPS! See https://docs.djangoproject.com/en/1.11/ref/applications/#configuring-applications * use FormatPresetSerializer as a PrimaryKeyRelatedField serializer * disable breakpoints * attempted fix for file uploading issues * file uploading works... yay! * signup form multiselection area is working now * enable offline_helper * remove extra debugging leftovers from registration info template * fixes for several tests... remember: don't manually hardcode constants from le_utils. they may change some day! * fixed authentication tests * add a migration from django 1.11
kollivier
pushed a commit
that referenced
this pull request
Oct 1, 2018
* objects.create automatically calls save, so no need to call it again afterwards. * objects.create automatically calls save, so no need to call it again afterwards. * Revert removed calls to save as we are creating models directly rather than using objects.create. * Revert removed calls to save as we are creating models directly rather than using objects.create. * Apply Micah's changes and also fix some new issues uncovered by tests to get the code to the same state as it was before revert. * Fix login template name. * Actually commit the updated dependency versions to requirements.txt... * use `==` instead of `=` in template `if` statement * Re-delete requirements.txt as a result of rebasing issues. * Update Django to 1.11 in pipfile after rebase. * Actually update the dependencies after accidentally using pip... * update django to 1.11.15! * use the recommended way of declaring our AppConfig Through our INSTALLED_APPS! See https://docs.djangoproject.com/en/1.11/ref/applications/#configuring-applications * use FormatPresetSerializer as a PrimaryKeyRelatedField serializer * disable breakpoints * attempted fix for file uploading issues * [django1.11] fix file uploading, new user registration + make tests pass (#940) * Add chef method unit tests and ensure that Django views return a HTTP response when they raise errors. * Add more file upload and add node tests. * Add finish_commit tests and return a HTTP 500 code instead of raising if an error occurs. * Get Sentry reports for handled internal server errors in Internal API calls. * Add yarn run devsetup command (#925) * add new yarn command to set up dev environment * change commands to remove redundant pieces to prioritize yarn run devsetup Streamlines the new dev experience * Updated translations with epub string * Fixed default preview styling * Add caching to the get_user_public_channels endpoint (#939) * Move view get_public_channels logic to Channel model * add the Channel.make_public function * use StudioTestCase to create buckets * add ChannelCacher class to implement channel caching * add channel token related convenience functions * make exporchannel use Channel.make_token() * add channel specific cache for tokens Cache that, one less query to make. * add caching to channel token serializer endpoint * move serializers.get_resource_count implementation to the Channel model * use channel.get_thumbnail() function on serializers.py * remove redefinition of generate_thumbnail_url * add ChannelCacher.get_resource_count() cache function Used to cache the channel's get_resource_count * add caching to get_resource_count API attribute * add channel.get_date_modified function * make channelfieldmixin use channel.get_date_modified * add channel.get_date_modified cache * Use the channel cache for get_date_modified * remove redundant generate_thumbnail_url Already defined in ChannelFieldMixin * pass cache get_public_channels args to real function * cache the entire get_user_public_channels view * fix tests for ChannelCacher.get_public_channels() by comparing actual channel ids rather than objects * Create SecretToken.exists() convenience method * refactor make_token to definitely end after 100 attempts And use for-else loop construct * add extra non-public assert on the make_public test * add test for SecretToken.exists() * [WIP] Add nginx-level API endpoint caching (#943) * add long running caching to the get_user_public_channels endpoint * add caching to the get_user_edit_channels endpoint * add 4 hour browser caching on all static files (#945) * add caching to the get_user_edit_channels endpoint (#947) * Unlock le-utils' version * have pipenv update all updateable packages * Added comment * Made resource count required on storage requests * Merge in develop * Return 404s for any files not found in the zip, and have BadZipfile errors report the name, size, mode and download state to help us narrow down problems with opening. Also, tests. (#942) * Remove the hardcoded server IP in tests so that configuration changes don't break them. * Apply Micah's changes and also fix some new issues uncovered by tests to get the code to the same state as it was before revert. * Fix login template name. * Actually commit the updated dependency versions to requirements.txt... * use `==` instead of `=` in template `if` statement * Re-delete requirements.txt as a result of rebasing issues. * Update Django to 1.11 in pipfile after rebase. * Actually update the dependencies after accidentally using pip... * objects.create automatically calls save, so no need to call it again afterwards. * objects.create automatically calls save, so no need to call it again afterwards. * Revert removed calls to save as we are creating models directly rather than using objects.create. * Revert removed calls to save as we are creating models directly rather than using objects.create. * update django to 1.11.15! * use the recommended way of declaring our AppConfig Through our INSTALLED_APPS! See https://docs.djangoproject.com/en/1.11/ref/applications/#configuring-applications * use FormatPresetSerializer as a PrimaryKeyRelatedField serializer * disable breakpoints * attempted fix for file uploading issues * file uploading works... yay! * signup form multiselection area is working now * enable offline_helper * remove extra debugging leftovers from registration info template * fixes for several tests... remember: don't manually hardcode constants from le_utils. they may change some day! * fixed authentication tests * add a migration from django 1.11 * cleanup some debugging remnants * cleanup one last debugging remnant -_- * get file format from extension * Don't query the database to guess file formats. Instead, use le_utils.constants.file_formats. * use in test * Pipfile.lock * align auth_login/login URL route name to 'login' * updated cloudbuild.yaml, courtesy of Aron * Create simplified implementation for gcs_storage.generate_filename Just return the filename passed in, since the logic is implemented through generate_object_storage_name. * Remove assert about the filename and a string * Partial fix for #975. Restore the Other option textbox in the reg info form.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After adding django-level caching, unfortunately we still don't prevent the
thundering herd problem for our cache. Luckily nginx has some nice directives
for helping with that.