Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Summarize what changed inbetween two runs #13

Merged
merged 4 commits into from
Jun 26, 2018

Conversation

ghostwords
Copy link
Member

@ghostwords ghostwords commented Jun 26, 2018

This compares blocked and cookieblocked base domains. Blocking different subdomains of the same base domain is not reported.

Example output from a 500 site run comparing master and the navigator.sendBeacon branch:
screenshot from 2018-06-25 21 32 19

Closes EFForg/privacybadger#1019. We can open followup issues/PRs as needed.

This compares blocked and cookieblocked base domains.

Blocking different subdomains of the same base domain is not reported.
@ghostwords ghostwords requested a review from bcyphers June 26, 2018 01:54
@ghostwords ghostwords force-pushed the add-domain-change-summary branch from 09c685c to c5ef35a Compare June 26, 2018 12:25
@ghostwords ghostwords changed the title Add summary of changes to blocked domains Add change summary Jun 26, 2018
@ghostwords
Copy link
Member Author

ghostwords commented Jun 26, 2018

Would be nice to get these emailed from scheduled runs eventually.

@ghostwords ghostwords changed the title Add change summary Summarize what changed inbetween two runs Jun 26, 2018
@@ -1,2 +1,4 @@
colorama==0.3.9
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These requirements are installed inside the docker image, and validate.py is run outside of it. We can either add a separate requirements file to install dependencies for the outer script, or we could run validate.py inside Docker.

Copy link
Member Author

@ghostwords ghostwords Jun 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any downsides to running it inside Docker? We'd still be able to run it by itself manually on arbitrary result files I presume.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inside Docker is fine, I think, though it will split some of the logic out of runscan.sh. Update Dockerfile to copy validate.py and results.json into the docker container, then have docker-entry.sh call this script after it runs the crawler.

Run validate.py from docker-entry.sh. Update Dockerfile to copy validate.py and
results.json into docker container.
@bcyphers bcyphers merged commit 0b2efec into master Jun 26, 2018
@ghostwords ghostwords deleted the add-domain-change-summary branch June 28, 2018 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants