Help Us Develop the New ToSBack
We could use your help building the new Rails version of ToSBack! The ToSBack3 readme has a good overview of basic site functionality if you aren't familiar, and this page should help you see how the components come together. Open your favorite text editor and start contributing!
Here's a basic description of how it should come together:
- Policy information in the policies table that isn't marked as "needs_revision" will be scheduled to scrape policy data from the original site and added to the "crawls" table.
- Admins will approve the "pending_changes" and "crawls" (scrape data) before it is visible to most users.
- Approving crawls will replace the current data in the policies table and add a row to the versions table.
- Users will be able to view which policies have changed and view a diff of what has changed.
Eventually, we may have a way for users to easily help us add policies and subscribe to the changes that they're interested in.
This is an early diagram of the relationships between tables, but it's still pretty accurate:
Take a look at the current DB schema here.
With TOSBack2, the focus was just to get the scraper working with a good set of rules to use as a foundation. The HTML was being stripped from the crawl data with the 'sanitize' gem.
In the new version of TOSBack, we will keep the full archive in the database (full_page column) and also store the policy filtered by xpath. (We still only plan to store a new version the full archive if something within the CSS selector changed or we would be saving every new useless change, e.g. new comments, related articles, etc..) The final implementation will try to balance full archive flexibility with security and performance.
Would you like to contribute? Awesome! Here are some general guidelines to remember:
- Write a failing test before implementing the new feature (Red-Green-Refactor).
- Keep the controllers and views skinny.
- ToSBack is using factories and RSpec instead of fixtures and Test::Unit.
- Add the feature you're working on to the issue tracker if it's not there already!
Now, we have a separate DB table called "notifications" that's being modified by ToSBack2, but one of the next steps is importing the good data that we have in the ToSBack2 repo and crawling sites directly into the ToSBack3 DB. Work is already being done on getting a working import/export for the XML rules and the versions that we have stored in text files (see the rake tasks), but we will have to begin to work on how to display those things. Here are some tasks I see in ToSBack's future:
- Use the crawler code from ToSBack2 as a foundation, but update it to work with our ActiveRecord models.
- Strip out the authentication code for now so we can focus on getting a working diff system out there.
- (use devise later)
- Maybe use partials to build some extra content for some of the pages that seem empty.
- Modify the index page to use updated versions from our policies table as the notifications instead of having a separate table.
- Secure any location that may be vulnerable to XSS.
Join the #tosback channel on irc.oftc.net if you have any questions and we'll be glad to help!