Hits?

[nelsonic]

"Make Something People Want"

In August 2015, to "scratch my own itch," I created https://github.com/dwyl/hits
to track the "Page Views" on all of our learn-... tutorials and Open Source projects. 📈
It does (exactly) one thing and nothing else.

Rather than keeping it "private" (only for dwyl repos) I made it public so anyone could use it.
It's an incredibly simple (single-purpose) Web Service that only has one endpoint.

I did not promote it or try to ask anyone in my network to use it.
I did not do any "press" or "Show HN" ...
people have found the service 100% "organically", they want it

Last year I added a (basic) front-end to help people create their own badge: https://hits.dwyl.com

I linked to it on https://github.com/dwyl/repo-badges and waited to see if:

A) Anyone would Find it!

B) People would Use it without any prompting or promotion!

C) Running a basic (single metric) analytics service could be done on a low ($5/month) budget.

Fast forward a year and there are thousands of GitHub projects using the "Hits Badge"! 😮

It's been collecting Millions of "Hits" (Page Views) for some of the top projects on GitHub:

"XBMC" (originally "XBox Media Center" now "KODI" because it runs on more hardware)
https://github.com/xbmc/xbmc is a super popular GitHub repo
and the project maintainer added the Hits Badge ...

There's even a project from the Apache Software Foundation:
https://github.com/apache/incubator-servicecomb-service-center using a Hits Badge:

There is a lot of "noise" from random "pet projects" using the Hits "Badge" ...
But that's a good thing because it simulates usage (the Page View counts are all real!)
and has generated a lot of data that I can now analyse.

Product-Market Fit?

"Product/market fit is the degree to which a product satisfies a strong market demand."
"a first step to building a successful venture in which the company meets early adopters, gathers feedback and gauges interest in its product(s)."

• https://en.wikipedia.org/wiki/Product/market_fit
• https://a16z.com/2017/02/18/12-things-about-product-market-fit/ (highly recommend reading!)

Hits is a single function product/service - it does only one thing -
that Users have organically discovered and are using completely "under the radar"!

Note: I am aware that many people consider "Page Views" to be a "vanity metric",
for the most part I agree. However, there are also smart people who like the simplicity of the counter as a reminder of demand.

Cost?

initially, I ran the service on a Free Heroku Dyno: https://hitt.herokuapp.com
At the time the hits (Node.js) app was only incrementing a counter
for each page/project that was being tracked and I thought that was useless data for analysis.
So rather than paying $9/month for Heroku's PostgreSQL,
I decided to run the app on a $5/month Linode VM. see: learn-devops/linode-setup.md

https://manager.linode.com/account/billing_history

Resource Usage

CPU

The CPU usage never goes above 3%:

And the only reason for the "spikes" is that I'm using a (filesystem based) append-only log,
so each time the count needs to be displayed I have to stream the entire file which is quite "slow".
But still, CPU is definitely not a " performance bottleneck" in the App.

Traffic (Bandwidth in KiloBytes Per Second)

Bandwidth is negligible:

Because the response is almost always an SVG which is 511 Bytes:

The Linode Instance has 1000GB (1TB!) of Monthly Network Transfer and we are using less than 1%:

Storage

The App is using 20Gb of SSD Storage for the page view

Well, there's your problem ...

Data cost will only ever increase over time. 💸

Benefit?

Until now, the only benefit has been learning for me
and a counter for the people displaying the "badge".

Could we get a bit more out of this project ...?
(and could it be part of the "product roadmap"...? hence opening this issue/discussion here...)

Two <option>'s For ...

There are two options for what we can do with this product that clearly has traction:

1. Do Nothing. Leave it the way it is and gradually pay more for it over time as more people discover it and storage costs increase but do no more "dev" work on the idea. (lowest time/cash investment by far!)

2. Build Out an idea that has "Product-Market-Fit" (PMF) See How far it Goes!
   Build out the UI to include:

• Dashboard with graph of usage over time including (aggregate) User Agent and Location Stats
• Chart showing the progression of GitHub Stars over Time (using the GitHub API)

Note the "core" page view counting service would not change, it will still do exactly one thing.
all we would be doing is extending the UI such that repository owner can see a breakdown of the data that we have already been collecting.

Along the way:

• Learn a bunch about rendering dashboards for data and re-use my knowledge for other products!
• Write a step-by-step Real-time Graphing & Data Visualisation tutorial in Elm
• Share progress with the community on how to bootstrap an app/product
  with the absolute minimum cash. (staying within the AWS Free Tier Limits)
• Follow (Strict) "Performance Driven Development" and use the project as an Example
  • Migrate to using PostgreSQL append only log:
    https://github.com/dwyl/phoenix-ecto-append-only-log-example
  • Use ETS to cache count data for faster response times
    https://elixirschool.com/en/lessons/specifics/ets

[nelsonic]

The Linode server has run out of disk space ... 😞 dwyl/hits#62

[nelsonic]

Given that the "Hits" application (in it's current form) is "down" due to exhausted disk space on the server, I'm going to increase the disk space on Linode one last time. > dwyl/learn-devops#52

But we need to have a clear plan for migrating this from using a FileSystem based append-only log to using a Database ASAP (as discussed above).

As much as I like the simplicity of the Linode interface, see:
and their transparent pricing https://www.linode.com/pricing

I feel that their "inflexibility" on allowing scaling of just the SSD is a deal-breaker going forward.
I briefly investigated using patroni to create a PostgreSQL Cluster on Linode: https://www.linode.com/docs/databases/postgresql/create-a-highly-available-postgresql-cluster-using-patroni-and-haproxy and I would have considered the option.
But having two 80GB instances on Linode (one Master and the other a "hot replica") would cost $40/month whereas using AWS RDS will cost us (me) $26.21/month for 100GB of storage.

RDS supports scaling just the Storage (SSD) which means we can optimise for storage costs.
https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-rds-for-sql-server-supports-scaling-storage-size
With running our own "self-managed" Postgres cluster, the price doubles for each tier.
i.e. if we use 81GB of storage we need to upgrade to the instances with 160GB of space so it's 2x$40 or $80/month ... that's silly money for a "side project" with zero revenue.

By contrast even at the 200Gb point it's "only" $40/month on RDS:

According to the PostgreSQL DBaaS Calculator dwyl/learn-postgresql#59 $16.55/month

But this is only part of the story; the data part. (which is the biggest part, but not the only part)

Sadly, the (my) monthly cost for running the "Hits" Service is going to quadruple from $5/month
to at least $22.57/month

• DB pricing: https://aws.amazon.com/rds/postgresql/pricing
  • db.t3.micro instance ($0.02/h) => $14.60/month
    • SSD data storage 30gb ($0.127 per GB-month) => $3.81/month
• App server (EC2) pricing: https://aws.amazon.com/ec2/pricing/on-demand
  • t3.nano ($0.0057/h) => $4.16/month

Update

I've increased the Linode plan from $5/month to $10/month
and added full-disk backups for $2.50/mo.
$12.50/month is fine for now while we work on migrating this to Alog v2, PostgreSQL and AWS RDS.

[nelsonic]

Next: dwyl/hits#66

[nelsonic]

Still totally want to do some visualisation with Hits hits.dwyl.com ... 💭