Parse RSS feeds very quickly
Intro | Compatibility | Installation | Usage | Benchmarks | Deploying | License
Parse RSS feeds very quickly
Speed
Currently this is already much faster than most of the pure elixir/erlang packages out there. In benchmarks there are speed improvements anywhere between 6.12x - 50.09x over the next fastest package (feeder_ex) that was tested.
Compared to the slowest elixir options tested (feed_raptor, elixir_feed_parser), FastRSS was sometimes 259.91x faster and used 5,412,308.17x less memory (0.00156 MB vs 8423.70 MB).
See full benchmarks below:
FastRSS requires a minimum combination of Elixir 1.6.0 and Erlang/OTP 20.0, and is tested with a maximum combination of Elixir 1.11.1 and Erlang/OTP 22.0.
This package is available on hex.
It can be installed by adding fast_rss
to your list of dependencies in mix.exs
:
def deps do
[
{:fast_rss, "~> 0.5.0"}
]
end
You also need the rust compiler installed: https://www.rust-lang.org/tools/install
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
There is only two functions, one for parsing rss parse_rss/1
and one for parsing atom feeds parse_atom/1
they takes a string and outputs an {:ok, map()}
with string keys.
iex(1)> {:ok, map_of_rss} = FastRSS.parse_rss("...rss_feed_string...")
iex(2)> Map.keys(map_of_rss)
["categories", "cloud", "copyright", "description", "docs", "dublin_core_ext",
"extensions", "generator", "image", "items", "itunes_ext", "language",
"last_build_date", "link", "managing_editor", "namespaces", "pub_date",
"rating", "skip_days", "skip_hours", "syndication_ext", "text_input", "title",
"ttl", "webmaster"]
The docs can be found at https://hexdocs.pm/fast_rss.
Reading from the following RSS versions is supported:
- RSS 0.90
- RSS 0.91
- RSS 0.92
- RSS 1.0
- RSS 2.0
- iTunes
- Dublin Core
- Atom
HTML: https://avencera.github.io/fast_rss/
Benchmark run from 2020-02-22 05:23:47.524699Z UTC
Benchmark suite executing on the following system:
Operating System | macOS |
---|---|
CPU Information | Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz |
Number of Available Cores | 16 |
Available Memory | 32 GB |
Elixir Version | 1.10.1 |
Erlang Version | 22.2.6 |
Benchmark suite executing with the following configuration:
:time | 30 s |
---|---|
:parallel | 1 |
:warmup | 5 s |
Input: anxiety
Run Time
Name | IPS | Average | Devitation | Median | 99th % |
---|---|---|---|---|---|
fast_rss | 188.57 | 5.30 ms | ±8.26% | 5.45 ms | 6.43 ms |
feeder_ex | 3.70 | 269.92 ms | ±5.34% | 268.12 ms | 316.12 ms |
feed_raptor | 2.99 | 334.01 ms | ±2.44% | 331.03 ms | 371.28 ms |
elixir_feed_parser | 1.94 | 515.72 ms | ±1.94% | 516.10 ms | 536.05 ms |
Name | IPS | Slower |
---|---|---|
fast_rss | 188.57 | |
feeder_ex | 3.70 | 50.9x |
feed_raptor | 2.99 | 62.99x |
elixir_feed_parser | 1.94 | 97.25x |
Name | Memory | Factor |
---|---|---|
fast_rss | 0.00156 MB | |
feeder_ex | 17.21 MB | 11004.73x |
feed_raptor | 268.53 MB | 171693.91x |
elixir_feed_parser | 313.30 MB | 200316.09x |
Input: ben
Run Time
Name | IPS | Average | Devitation | Median | 99th % |
---|---|---|---|---|---|
fast_rss | 83.95 | 11.91 ms | ±10.29% | 12.23 ms | 16.17 ms |
feeder_ex | 13.33 | 75.04 ms | ±4.38% | 74.21 ms | 89.72 ms |
elixir_feed_parser | 3.52 | 284.18 ms | ±3.89% | 283.83 ms | 324.08 ms |
feed_raptor | 0.48 | 2078.76 ms | ±0.52% | 2076.27 ms | 2097.44 ms |
Name | IPS | Slower |
---|---|---|
fast_rss | 83.95 | |
feeder_ex | 13.33 | 6.3x |
elixir_feed_parser | 3.52 | 23.86x |
feed_raptor | 0.48 | 174.51x |
Name | Memory | Factor |
---|---|---|
fast_rss | 0.00155 MB | |
feeder_ex | 27.86 MB | 17990.96x |
elixir_feed_parser | 163.88 MB | 105811.88x |
feed_raptor | 1577.41 MB | 1018492.36x |
Input: daily
Run Time
Name | IPS | Average | Devitation | Median | 99th % |
---|---|---|---|---|---|
fast_rss | 32.98 | 0.0303 s | ±7.62% | 0.0313 s | 0.0339 s |
feeder_ex | 4.94 | 0.20 s | ±4.61% | 0.199 s | 0.24 s |
elixir_feed_parser | 0.64 | 1.57 s | ±1.50% | 1.57 s | 1.63 s |
feed_raptor | 0.127 | 7.88 s | ±0.23% | 7.88 s | 7.90 s |
Name | IPS | Slower |
---|---|---|
fast_rss | 32.98 | |
feeder_ex | 4.94 | 6.68x |
elixir_feed_parser | 0.64 | 51.86x |
feed_raptor | 0.127 | 259.91x |
Name | Memory | Factor |
---|---|---|
fast_rss | 0.00153 MB | |
feeder_ex | 109.73 MB | 71555.78x |
elixir_feed_parser | 880.51 MB | 574178.95x |
feed_raptor | 6386.12 MB | 4164382.64x |
Input: dave
Run Time
Name | IPS | Average | Devitation | Median | 99th % |
---|---|---|---|---|---|
fast_rss | 407.08 | 2.46 ms | ±9.83% | 2.41 ms | 3.16 ms |
feeder_ex | 56.52 | 17.69 ms | ±6.14% | 17.37 ms | 22.51 ms |
elixir_feed_parser | 8.90 | 112.31 ms | ±4.12% | 111.93 ms | 127.60 ms |
feed_raptor | 1.59 | 628.45 ms | ±1.60% | 626.71 ms | 656.74 ms |
Name | IPS | Slower |
---|---|---|
fast_rss | 407.08 | |
feeder_ex | 56.52 | 7.2x |
elixir_feed_parser | 8.90 | 45.72x |
feed_raptor | 1.59 | 255.83x |
Name | Memory | Factor |
---|---|---|
fast_rss | 0.00157 MB | |
feeder_ex | 9.25 MB | 5886.17x |
elixir_feed_parser | 80.42 MB | 51170.23x |
feed_raptor | 571.18 MB | 363425.45x |
Input: sleepy
Run Time
Name | IPS | Average | Devitation | Median | 99th % |
---|---|---|---|---|---|
fast_rss | 760.30 | 1.32 ms | ±16.62% | 1.21 ms | 2.03 ms |
feeder_ex | 124.28 | 8.05 ms | ±6.94% | 8.03 ms | 10.32 ms |
elixir_feed_parser | 26.26 | 38.09 ms | ±5.08% | 37.81 ms | 44.42 ms |
feed_raptor | 3.21 | 311.16 ms | ±2.85% | 307.86 ms | 345.09 ms |
Name | IPS | Slower |
---|---|---|
fast_rss | 760.30 | |
feeder_ex | 124.28 | 6.12x |
elixir_feed_parser | 26.26 | 28.96x |
feed_raptor | 3.21 | 236.57x |
Name | Memory | Factor |
---|---|---|
fast_rss | 0.00157 MB | |
feeder_ex | 4.28 MB | 2726.19x |
elixir_feed_parser | 35.88 MB | 22829.92x |
feed_raptor | 274.98 MB | 174963.99x |
Input: stuff
Run Time
Name | IPS | Average | Devitation | Median | 99th % |
---|---|---|---|---|---|
fast_rss | 19.19 | 0.0521 s | ±9.19% | 0.0546 s | 0.0635 s |
feeder_ex | 0.93 | 1.07 s | ±2.49% | 1.07 s | 1.15 s |
elixir_feed_parser | 0.53 | 1.88 s | ±1.22% | 1.89 s | 1.92 s |
feed_raptor | 0.0797 | 12.54 s | ±1.61% | 12.44 s | 12.77 s |
Name | IPS | Slower |
---|---|---|
fast_rss | 19.19 | |
feeder_ex | 0.93 | 20.59x |
elixir_feed_parser | 0.53 | 36.11x |
feed_raptor | 0.0797 | 240.68x |
Name | Memory | Factor |
---|---|---|
fast_rss | 0.00154 MB | |
feeder_ex | 140.58 MB | 91220.55x |
elixir_feed_parser | 1018.78 MB | 661058.28x |
feed_raptor | 8424.44 MB | 5466379.81x |
Deploying rust NIFs can be a little bit annoying as you have to install the rust compiler. We try to alleviate this with rustler_precopmiled
, which will create precompiled assets for a number of targets (see release.yml
for the full list), but does not cover all environments. If you are having trouble deploying this package make an issue and I will try and help you out.
I will then add it to the FAQ below.
A. I recommend using a multi-stage Dockerfile, and doing the following
-
On the stages where you build all your deps, and build your release make sure to install
build-base
andlibgcc
:# This step installs all the build tools we'll need RUN apk update && \ apk upgrade --no-cache && \ apk add --no-cache \ git \ curl \ build-base \ libgcc && \ mix local.rebar --force && \ mix local.hex --force
-
Install the rust compiler and allow dynamic linking to the C library by setting the rust flag
# install rustup RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y ENV RUSTUP_HOME=/root/.rustup \ RUSTFLAGS="-C target-feature=-crt-static" \ CARGO_HOME=/root/.cargo \ PATH="/root/.cargo/bin:$PATH"
-
On the stage where you actually run your elixir release install
libgcc
:################################################################################ ## STEP 4 - FINAL FROM alpine:3.11 ENV MIX_ENV=prod RUN apk update && \ apk add --no-cache \ bash \ libgcc \ openssl-dev COPY --from=release-builder /opt/built /app WORKDIR /app CMD ["/app/my_app/bin/my_app", "start"]
FastRSS is released under the Apache License 2.0 - see the LICENSE file.