-
Notifications
You must be signed in to change notification settings - Fork 6
/
what-is-the-air-speed-velocity-of-pythran.html
197 lines (193 loc) · 11.4 KB
/
what-is-the-air-speed-velocity-of-pythran.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>Pythran stories - What is the air speed velocity of Pythran?</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/normalize/8.0.1/normalize.min.css"/>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.2/css/all.min.css"/>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto+Slab|Ruda"/>
<link rel="stylesheet" type="text/css" href="./theme/css/main.css"/>
<link href="http://serge-sans-paille.github.io/pythran-stories/
feeds/all.atom.xml"
type="application/atom+xml" rel="alternate" title="Pythran stories Atom Feed"/>
</head>
<body>
<style>.github-corner:hover .octo-arm {
animation: octocat-wave 560ms ease-in-out
}
@keyframes octocat-wave {
0%, 100% {
transform: rotate(0)
}
20%, 60% {
transform: rotate(-25deg)
}
40%, 80% {
transform: rotate(10deg)
}
}
@media (max-width: 500px) {
.github-corner:hover .octo-arm {
animation: none
}
.github-corner .octo-arm {
animation: octocat-wave 560ms ease-in-out
}
}</style><div id="container">
<header>
<h1><a href="./">Pythran stories</a></h1>
<ul class="social-media">
<li><a href="https://github.com/serge-sans-paille/pythran"><i class="fab fa-github fa-lg" aria-hidden="true"></i></a></li>
<li><a href="http://serge-sans-paille.github.io/pythran-stories/
feeds/all.atom.xml"
type="application/atom+xml" rel="alternate"><i class="fa fa-rss fa-lg"
aria-hidden="true"></i></a></li>
</ul>
<p><em></em></p>
</header>
<nav>
<ul>
<li><a class="active" href="./category/benchmark.html"> benchmark </a></li>
<li><a href="./category/compilation.html"> compilation </a></li>
<li><a href="./category/engineering.html"> engineering </a></li>
<li><a href="./category/examples.html"> examples </a></li>
<li><a href="./category/mozilla.html"> mozilla </a></li>
<li><a href="./category/optimisation.html"> optimisation </a></li>
<li><a href="./category/release.html"> release </a></li>
</ul>
</nav>
<main>
<article>
<h1>What is the air speed velocity of Pythran?</h1>
<aside>
<ul>
<li>
<time datetime="2018-08-05 00:00:00+02:00">Aug 05, 2018</time>
</li>
<li>
Categories:
<a href="./category/benchmark.html"><em>benchmark</em></a>
</li>
</li>
</ul>
</aside>
<p>As many projects, <a class="reference external" href="https://github.com/serge-sans-paille/pythran">Pythran</a> has
its own test suite that runs on <a class="reference external" href="http://travis-ci.org/">travis-ci</a> and
<a class="reference external" href="http://appveyor.com/">AppVeyor</a>. It's a nice test suite: it tests different
compilers—namely Clang, GCC and MVSCC—with different set of flags—with and
without OpenMP and vectorization— and for the 2.7 and 3.5 Python versions.
Nice. Brest.</p>
<p>A while ago, I started to collect various high-level Numpy kernels in the
<a class="reference external" href="https://github.com/serge-sans-paille/numpy-benchmarks/">numpy-benchmark</a>
repo. The goal was to check how well Pythran behaves on these kernels, and to
compare it with other compilers. Over the time I've been adding new kernels
that showcases different optimization challenges, which leads us to more than 35
different kernels. I was occasionally using that set of kernels to check for
performance regression, but that was only semi-automated and rather tedious.</p>
<p>Then comes <a class="reference external" href="https://github.com/airspeed-velocity/asv/">airspeed velocity</a>, a
tool that exactly brings the remaining bits: automated benchmarking over
software revision, regression tracking and nice plots. So it was just a matter
of stirring everything together to give birth to
<a class="reference external" href="https://github.com/serge-sans-paille/pythran-asv">https://github.com/serge-sans-paille/pythran-asv</a>. In order to not break the
link with <em>numpy-benchmarks</em>, the original repo is imported as a <a class="reference external" href="https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt">git subtree</a>, and
the tests are automatically generated based on the kernel annotations :-)</p>
<div class="section" id="configuration">
<h2>Configuration</h2>
<p>I don't have a dedicated machine to run the performance tracking session, so I just ran the following:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>asv<span class="w"> </span><span class="m">0</span>.8.6..master
</pre></div>
<p>Basically, this commands runs <tt class="docutils literal">timeit</tt> on all kernels from <em>numpy-benchmarks</em>
according to the kernel annotations, but only once these kernels have been
compiled by the <em>Pythran</em> compiler extracted from the revisions ranging from
latest release, <em>0.8.6</em> to current <em>master</em>. That's a good way to check if the recent
commits brought any performance improvement or regressions.</p>
<p>In order to reduce system glitter, the following one-liner helps a lot:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>python<span class="w"> </span>-m<span class="w"> </span>perf<span class="w"> </span>system<span class="w"> </span>tune
</pre></div>
<p>These benchmarks were run during the night, with all apps closed, using a
blank <tt class="docutils literal"><span class="pre">~/.pythranrc</span></tt> and GCC 7.3 and on good ol' Python 2.7. For
reference, I have an <tt class="docutils literal">Intel(R) Core(TM) <span class="pre">i7-6600U</span> CPU @ 2.60GHz</tt>, but as
the goal is merely to track regression, the details are not as relevant as
they could be for a traditional benchmark setup.</p>
</div>
<div class="section" id="subtilities">
<h2>Subtilities</h2>
<p>Although <em>asv</em> works like a charm, some aspects require care.</p>
<ol class="arabic simple">
<li><em>Pythran</em> is an ahead of time compiler, so it does not make sense to track
compilation time as part of the actual benchmark. <em>asv</em> does provide a
<tt class="docutils literal">setup</tt> mechanism, but it proved to be too costly (the compilation is
called multiple times, and it costs too much time when ranging over hundreds
of commits). Fortunately, there is a <tt class="docutils literal">setup_cache</tt> mechanism that fulfills
our need perfectly!</li>
<li><em>numpy-benchmarks</em> tries hard to use randomized input, in order to avoid
over-specialization of a benchmark for a given input set. In some cases this
led to unstable behavior, so we occasionally forced the seed. Changing the
kernel body is not an option; since one of the interests behind using
<em>numpy-benchmarks</em> is that it come from third-party source and
are not tailored for a given compiler.</li>
<li>On very rare occasions, the <em>Pythran</em> annotation language evolves, and
kernels that use new syntax cannot be compiled by older versions of
<em>Pythran</em>. This actually happens only once, now that it is possible to
specify the value dimension of a given dimension. As a workaround, if a
kernel fails to compile, a rewrite rule automatically changes the annotation
back to the traditional, less specialized one.</li>
</ol>
</div>
<div class="section" id="results">
<h2>Results</h2>
<p>As of this post, the result of the regression test has been snapshot on
<a class="reference external" href="https://serge-sans-paille.github.io/pythran-asv-snapshot/">https://serge-sans-paille.github.io/pythran-asv-snapshot/</a>. It provides a good
illustration of the recent commits :-)</p>
<ul class="simple">
<li><a class="reference external" href="https://serge-sans-paille.github.io/pythran-asv-snapshot#benchmarks.TimeSuite.time_laplacien">laplacien</a>
greatly benefits from the partially static shape specialization introduced in
<a class="reference external" href="https://github.com/serge-sans-paille/pythran/tree/2e9dc6d694feae2be378fa5351e2cf5ad0c19f1">2e9dc6d69</a>.
This commit also has a positive impact on <a class="reference external" href="https://serge-sans-paille.github.io/pythran-asv-snapshot#benchmarks.TimeSuite.time_grayscott">grayscott</a>,
but a slightly negative impact on <a class="reference external" href="https://serge-sans-paille.github.io/pythran-asv-snapshot#benchmarks.TimeSuite.time_wdist">wdist</a>.</li>
<li><a class="reference external" href="https://serge-sans-paille.github.io/pythran-asv-snapshot#benchmarks.TimeSuite.time_make_decision">make_decision</a> exhibits a pattern captured by
<a class="reference external" href="https://github.com/serge-sans-paille/pythran/tree/a52ee30084549125ef34448f5ccf3013874331a9">a52ee300</a>
: the square of the norm of a complex number.</li>
<li><a class="reference external" href="https://serge-sans-paille.github.io/pythran-asv-snapshot#benchmarks.TimeSuite.time_reverse_cumsum">reverse_cumsum</a> is returning a complex Numpy view, something Pythran can
efficiently do since <a class="reference external" href="https://github.com/serge-sans-paille/pythran/tree/8e8711365899009634653cc7e11a8cd36001c0c7">8e871136</a>.</li>
</ul>
</div>
<div class="section" id="thanks-and-future-works">
<h2>Thanks and Future Works</h2>
<p>The original idea of using <em>asv</em> to track <em>Pythran</em> performance comes from a
discussion with the <a class="reference external" href="http://quantstack.net/">QuantStack</a> guys. They really
are amazing folks, all the interactions I have with them really keeps the
motivation up. They even funded all this regression tracking stuff for <em>Pythran</em>,
did I mention they're cool?</p>
<p>Next step is probably to run the regression suite on a larger commit range,
this may spot some more regressions or give hint for further improvements. And
I'm pretty confident <a class="reference external" href="https://twitter.com/wuoulf">@wolfv</a> will provide an
adaptation of <em>pythran-asv</em> for <a class="reference external" href="https://github.com/QuantStack/xtensor">xtensor</a> as he <a class="reference external" href="https://twitter.com/wuoulf/status/1016710926047825920">already did</a> for
<em>numpy-benchmarks</em>.</p>
<p>And thanks a lot to <a class="reference external" href="https://ashwinvis.github.io/">ashwinvis</a> for his review!</p>
</div>
</article>
<section class="post-nav">
<div id="left-page">
<div id="left-link">
</div>
</div>
<div id="right-page">
<div id="right-link">
</div>
</div>
</section>
<div>
</div>
</main>
<footer>
<h6>
Rendered by <a href="http://getpelican.com/">Pelican</a> • Theme by <a
href="https://github.com/aleylara/Peli-Kiera">Peli-Kiera</a> • Copyright
© ‑ serge-sans-paille and other pythraners </h6>
</footer>
</div>
</body>
</html>