Store Pageviews in DB #6121

dojutsu-user · 2019-08-29T18:24:14Z

Related: #5968

ericholscher

This is a good first approach. It needs to be cleaned up a bit though with the modeling.

readthedocs/api/v2/views/footer_views.py

readthedocs/search/models.py

readthedocs/search/tasks.py

ericholscher

This looks quite close to being ready. Just a few places where it can be cleaned up or simplified.

ericholscher · 2019-11-01T14:41:02Z

readthedocs/api/v2/views/footer_views.py

@@ -210,6 +211,14 @@ def get(self, request, format=None):
            'version_supported': version.supported,
        }

+        # increase the page view count
+        page_slug = request.GET.get('page', 'index')


This should likely not default to index, no? If we don't have a page, we probably shouldn't be storing anything, or some other string like rtd-unknown-page, not index.

I haven't seen any real examples where the page parameter is blank. It is index for the root and even with the dirhtml builder it uses the subdirectory name. I'll add a simple if page_slug check.

readthedocs/projects/views/private.py

readthedocs/rtd_tests/tests/test_footer.py

readthedocs/search/models.py

readthedocs/search/utils.py

stale · 2019-12-16T16:11:43Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ericholscher · 2019-12-17T10:31:29Z

This is still useful, but I want to wait until our DB load calms down a bit before adding additional load to it.

readthedocs/search/tasks.py

readthedocs/search/models.py

- Added a periodic task to remove old pageviews - Show aggregated traffic on the admin view

davidfischer · 2020-05-08T21:20:03Z

I refactored this and brought it to a merge-ready state.

There's a periodic task to remove old page views
The screen (see screenshot) now shows aggregated pageviews by day. That's a total traffic graph.

I am interested to see what this does to our database as it will add a (simple) write query in a celery task on every Footer API call.

Screenshot

davidfischer · 2020-05-08T21:25:13Z

readthedocs/projects/views/private.py

@@ -977,15 +975,6 @@ class RegexAutomationRuleUpdate(RegexAutomationRuleMixin, UpdateView):
    pass


-@login_required
-def search_analytics_view(request, project_slug):


I have no idea what this view is. It doesn't appear to be used or valid.

ericholscher

This looks good to me. I'm curious if we should be keeping the pageview data in the search app, but I think it's fine for now?

I'm also super curious what this will do to our celery queues and DB, but I think it shouldn't be too much. A simple update isn't a ton of work, but we'll see :)

The nice thing is we can always limit the celery workers or turn them off in some fashion, so we have a reasonable escape hatch.

ericholscher

A few more tidbits.

ericholscher · 2020-05-11T15:13:33Z

readthedocs/search/tasks.py

+
+    This is intended to run from a periodic task daily.
+    """
+    thirty_days_ago = timezone.now().date() - timezone.timedelta(days=30)


This should probably log something, so we know it's running properly.

ericholscher · 2020-05-11T15:14:41Z

readthedocs/search/tasks.py

+
+
+@app.task(queue='web')
+def increase_page_view_count(project_slug, version_slug, path):


I wonder if running this as a task is actually a good deal slower then running it in the footer itself? We already have the Project & Version object in the footer, and not we're adding additional queries for them here. They should be fast queries, but definitely additional load.

I guess the proper solution is to cache these lookups, which we can do later if it's an issue.

davidfischer · 2020-05-11T16:01:54Z

This looks good to me. I'm curious if we should be keeping the pageview data in the search app, but I think it's fine for now?

I had this thought myself. I don't have a strong opinion either way. I'd be fine moving it into the Project app or a case could even be made for the Build app as it is Version specific. The case for the search app is that that is where the data is most useful: for ordering search results. However, as we have a page in the project admin, maybe that isn't the right place.

I'd rather get it into the right place the first time it's deployed. Moving it could be a bit of a pain. If you have an opinion, let's get that in.

ericholscher · 2020-05-11T16:31:35Z

Well, arguably the Versions should be in the Project app instead of Builds, so we're already in a confusing place to start :)

I think we put them in Search originally because we wanted to use them to weight search results by pageview count. I think it probably makes sense for them to be in their own app, something like analytics or pageviews, but again, I don't have super clear idea of whats the best.

ericholscher · 2020-05-11T16:32:46Z

Ah, we already have an analytics app, so that probably makes sense :)

ericholscher · 2020-05-18T20:59:12Z

@davidfischer it would be good to get this patched up and shipped this week if possible.

madwaxamillion · 2020-05-19T04:19:14Z

I'll take a look at it But I can't make any promises I might need Some help

…

On Mon, May 18, 2020, 4:59 PM Eric Holscher ***@***.***> wrote: @davidfischer <https://github.com/davidfischer> it would be good to get this patched up and shipped this week if possible. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6121 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO3EPOCEY7GAQKLF2L5C67LRSGOTXANCNFSM4ISFNKXA> .

davidfischer · 2020-05-19T04:49:48Z

I moved the pageview modeling from the search app to the analytics app. This is ready for a full review.

ericholscher · 2020-05-19T14:06:54Z

Looks good 🎉

dojutsu-user added 3 commits August 28, 2019 23:13

initial work

0998e8a

fix arguments

3e8a99a

update migration file

f5c82cf

dojutsu-user added the PR: work in progress Pull request is not ready for full review label Aug 29, 2019

dojutsu-user added 4 commits August 30, 2019 14:05

Merge branch 'master' into search-pageviews-sorting

e1e954c

show top 10 viewed page to the users.

9b03012

Merge branch 'master' into search-pageviews-sorting

152aff0

initial work for showing graphs to the user

f389476

ericholscher reviewed Sep 9, 2019

View reviewed changes

readthedocs/api/v2/views/footer_views.py Outdated Show resolved Hide resolved

readthedocs/search/models.py Outdated Show resolved Hide resolved

readthedocs/search/models.py Outdated Show resolved Hide resolved

dojutsu-user added 2 commits September 18, 2019 02:36

Merge branch 'master' into search-pageviews-sorting

1717cfc

show pageviews for a specific page

a19823f

ericholscher changed the title ~~Search results sorted by pageviews~~ Store Pageview in DB Sep 30, 2019

ericholscher changed the title ~~Store Pageview in DB~~ Store Pageviews in DB Sep 30, 2019

dojutsu-user added 11 commits October 2, 2019 15:07

Merge branch 'master' into search-pageviews-sorting

af056c8

change view to class based view

a1d0a9b

Merge branch 'master' into search-pageviews-sorting

82b3182

fix lint

415a3b3

fix more lint

2073a82

Merge branch 'master' into search-pageviews-sorting

b263c1b

store page_slug instead of page_path

3ec59dc

little refactor

108fd5f

update test

ddc96d9

fix tests

8da0f93

add test for search tasks

d0e1317

dojutsu-user mentioned this pull request Oct 16, 2019

Cache in footer view #6126

Closed

davidfischer reviewed Oct 16, 2019

View reviewed changes

readthedocs/search/tasks.py Outdated Show resolved Hide resolved

dojutsu-user added 4 commits October 17, 2019 19:04

use F expression

3946398

Merge branch 'master' into search-pageviews-sorting

b06d599

fix tests

11eb6ac

Merge branch 'master' into search-pageviews-sorting

95e7ee7

Merge branch 'master' into search-pageviews-sorting

67d4885

ericholscher requested a review from a team October 31, 2019 15:38

ericholscher reviewed Nov 1, 2019

View reviewed changes

stale bot added the Status: stale Issue will be considered inactive soon label Dec 16, 2019

ericholscher added the Accepted Accepted issue on our roadmap label Dec 17, 2019

stale bot removed the Status: stale Issue will be considered inactive soon label Dec 17, 2019

davidfischer reviewed Jan 23, 2020

View reviewed changes

readthedocs/search/tasks.py Outdated Show resolved Hide resolved

readthedocs/search/models.py Outdated Show resolved Hide resolved

readthedocs/search/models.py Outdated Show resolved Hide resolved

davidfischer added 3 commits May 7, 2020 16:26

Merge branch 'master' into search-pageviews-sorting

9cb2310

Index and refactor page view counting

3ce38c8

Feedback updates and renames for clarity

46747b0

- Added a periodic task to remove old pageviews - Show aggregated traffic on the admin view

davidfischer removed the PR: work in progress Pull request is not ready for full review label May 8, 2020

davidfischer reviewed May 8, 2020

View reviewed changes

ericholscher approved these changes May 11, 2020

View reviewed changes

ericholscher reviewed May 11, 2020

View reviewed changes

Merge branch 'master' into search-pageviews-sorting

7e5a1f0

Move pageview models to the analytics app

b530bbc

ericholscher merged commit 4fa419b into readthedocs:master May 19, 2020

stsewd mentioned this pull request Jun 1, 2020

expose server side analytics #5141

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store Pageviews in DB #6121

Store Pageviews in DB #6121

dojutsu-user commented Aug 29, 2019 •

edited

Loading

ericholscher left a comment

ericholscher left a comment

ericholscher Nov 1, 2019

davidfischer May 8, 2020

stale bot commented Dec 16, 2019

ericholscher commented Dec 17, 2019

davidfischer commented May 8, 2020

davidfischer May 8, 2020

ericholscher left a comment •

edited

Loading

ericholscher left a comment

ericholscher May 11, 2020

ericholscher May 11, 2020

davidfischer commented May 11, 2020

ericholscher commented May 11, 2020

ericholscher commented May 11, 2020

ericholscher commented May 18, 2020

madwaxamillion commented May 19, 2020 via email

davidfischer commented May 19, 2020

ericholscher commented May 19, 2020



		@app.task(queue='web')
		def increase_page_view_count(project_slug, version_slug, path):

Store Pageviews in DB #6121

Store Pageviews in DB #6121

Conversation

dojutsu-user commented Aug 29, 2019 • edited Loading

ericholscher left a comment

Choose a reason for hiding this comment

ericholscher left a comment

Choose a reason for hiding this comment

ericholscher Nov 1, 2019

Choose a reason for hiding this comment

davidfischer May 8, 2020

Choose a reason for hiding this comment

stale bot commented Dec 16, 2019

ericholscher commented Dec 17, 2019

davidfischer commented May 8, 2020

Screenshot

davidfischer May 8, 2020

Choose a reason for hiding this comment

ericholscher left a comment • edited Loading

Choose a reason for hiding this comment

ericholscher left a comment

Choose a reason for hiding this comment

ericholscher May 11, 2020

Choose a reason for hiding this comment

ericholscher May 11, 2020

Choose a reason for hiding this comment

davidfischer commented May 11, 2020

ericholscher commented May 11, 2020

ericholscher commented May 11, 2020

ericholscher commented May 18, 2020

madwaxamillion commented May 19, 2020 via email

davidfischer commented May 19, 2020

ericholscher commented May 19, 2020

dojutsu-user commented Aug 29, 2019 •

edited

Loading

ericholscher left a comment •

edited

Loading