Skip to content

Commit

Permalink
Deploying to gh-pages from @ 9e6d2e7 🚀
Browse files Browse the repository at this point in the history
maleadt committed May 28, 2024
1 parent dbb8508 commit c5e96be
Showing 149 changed files with 214 additions and 12,272 deletions.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -4,12 +4,12 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no">
<meta content="index, follow" name=robots>
<link rel="icon" href="/previews/PR44/assets/favicon.ico">
<link rel="alternate" type="application/rss+xml" href="/previews/PR44/post/index.xml" title="RSS Feed for JuliaGPU">
<link rel="icon" href="/assets/favicon.ico">
<link rel="alternate" type="application/rss+xml" href="/post/index.xml" title="RSS Feed for JuliaGPU">

<link rel="stylesheet" href="/previews/PR44/css/bootstrap.min.css">
<link rel="stylesheet" href="/css/bootstrap.min.css">

<link rel="stylesheet" href="/previews/PR44/libs/highlight/github.min.css">
<link rel="stylesheet" href="/libs/highlight/github.min.css">


<style>
@@ -109,21 +109,21 @@
<div id=nav-border class=container>
<nav class="navbar navbar-expand-lg navbar-light justify-content-center">
<ul class=navbar-nav>
<li class="nav-item "><a class=nav-link href="/previews/PR44/"><i data-feather=home></i>Home</a>
<li class="nav-item "><a class=nav-link href="/"><i data-feather=home></i>Home</a>
</li>
<li class="nav-item active"><a class=nav-link href="/previews/PR44/post/"><i data-feather=file-text></i>Blog</a>
<li class="nav-item active"><a class=nav-link href="/post/"><i data-feather=file-text></i>Blog</a>
</li>
<li class="nav-item "><a class=nav-link href="/previews/PR44/learn/"><i data-feather=book-open></i>Learn</a>
<li class="nav-item "><a class=nav-link href="/learn/"><i data-feather=book-open></i>Learn</a>
</li>
<li class="nav-item "><a class=nav-link href="/previews/PR44/cuda/">CUDA</a>
<li class="nav-item "><a class=nav-link href="/cuda/">CUDA</a>
</li>
<li class="nav-item "><a class=nav-link href="/previews/PR44/rocm/">ROCm</a>
<li class="nav-item "><a class=nav-link href="/rocm/">ROCm</a>
</li>
<li class="nav-item "><a class=nav-link href="/previews/PR44/oneapi/">oneAPI</a>
<li class="nav-item "><a class=nav-link href="/oneapi/">oneAPI</a>
</li>
<li class="nav-item "><a class=nav-link href="/previews/PR44/metal/">Metal</a>
<li class="nav-item "><a class=nav-link href="/metal/">Metal</a>
</li>
<li class="nav-item "><a class=nav-link href="/previews/PR44/other/">Other</a>
<li class="nav-item "><a class=nav-link href="/other/">Other</a>
</li>
</ul>
</nav>
@@ -344,7 +344,7 @@ <h2 id="other_changes"><a href="#other_changes" class="header-anchor">Other chan



<script src="/previews/PR44/libs/highlight/highlight.min.js"></script>
<script src="/libs/highlight/highlight.min.js"></script>
<script>hljs.initHighlightingOnLoad();hljs.configure({tabReplace: ' '});</script>


@@ -354,7 +354,7 @@ <h2 id="other_changes"><a href="#other_changes" class="header-anchor">Other chan
</footer>

<!-- FEATHER -->
<script src="/previews/PR44/libs/feather/feather.min.js"></script>
<script src="/libs/feather/feather.min.js"></script>
<script>feather.replace()</script>

<!-- GOOGLE ANALYTICS -->
8 changes: 8 additions & 0 deletions post/index.html
Original file line number Diff line number Diff line change
@@ -141,6 +141,14 @@ <h1>
<a href=/post/index.xml><i data-feather=rss></i></a>
</h1>

<p>
<a class=font-125 href="/post/2024-05-28-cuda_5.4/">
CUDA.jl 5.4: Memory management mayhem
</a><span>&nbsp;&#8599;</span>
<br>
<i data-feather=calendar></i>
<time datetime=2024-5-28>May 28, 2024</time><br>
</p>
<p>
<a class=font-125 href="/post/2024-05-24-oneapi_1.5/">
oneAPI.jl 1.5: Ponte Vecchio support and oneMKL improvements
183 changes: 183 additions & 0 deletions post/index.xml
Original file line number Diff line number Diff line change
@@ -18,6 +18,189 @@
type="application/rss+xml" />


<item>
<title><![CDATA[CUDA.jl 5.4: Memory management mayhem]]></title>
<link>https://juliagpu.org/post/2024-05-28-cuda_5.4/index.html</link>
<guid>https://juliagpu.org/2024-05-28-cuda_5.4/</guid>
<description><![CDATA[CUDA.jl 5.4 comes with many memory-management related changes that should improve performance of memory-heavy applications, and make it easier to work with heterogeneous set-ups involving multiple GPUs or using both the CPU and GPU.]]></description>

<content:encoded><![CDATA[
<p>CUDA.jl 5.4 comes with many memory-management related changes that should improve performance of memory-heavy applications, and make it easier to work with heterogeneous set-ups involving multiple GPUs or using both the CPU and GPU.</p>
<p>Before anything else, let&#39;s get the breaking changes out of the way. CUDA.jl v5.4 only bumps the minor version, so it should be compatible with existing codebases. However, there are a couple of API changes that, although covered by appropriate deprecation warnings, applications should be updated to:</p>
<ul>
<li><p>The <code>CUDA.Mem</code> submodule has been removed. All identifiers have been moved to the parent <code>CUDA</code> submodule, with a couple being renamed in the process:</p>
<ul>
<li><p><code>Mem.Device</code> and <code>Mem.DeviceBuffer</code> have been renamed to <code>CUDA.DeviceMemory</code> &#40;the same applies to <code>Mem.Host</code> and <code>Mem.Unified</code>&#41;;</p>
</li>
<li><p>enums from the <code>Mem</code> submodule have gained a <code>MEM</code> suffix, e.g., <code>Mem.ATTACH_GLOBAL</code> has been renamed to <code>CUDA.MEM_ATTACH_GLOBAL</code>;</p>
</li>
<li><p><code>Mem.set&#33;</code> has been renamed to <code>CUDA.memset</code>;</p>
</li>
<li><p><code>Mem.info&#40;&#41;</code> has been renamed to <code>CUDA.memory_info&#40;&#41;</code>;</p>
</li>
</ul>
</li>
<li><p><code>CUDA.memory_status&#40;&#41;</code> has been renamed to <code>CUDA.pool_status&#40;&#41;</code>;</p>
</li>
<li><p><code>CUDA.available_memory&#40;&#41;</code> has been renamed to <code>CUDA.free_memory&#40;&#41;</code>.</p>
</li>
</ul>
<p>The meat of this release is in the memory management improvements detailed below. These changes can have a significant impact of the performance of your application, so it&#39;s recommended to thoroughly test your application after upgrading&#33;</p>
<h2 id="eager_garbage_collection">Eager garbage collection</h2>
<p>Julia is a garbage collected language, which means that &#40;GPU&#41; allocations can fail because garbage has piled up, necessitating a collection cycle. Previous versions of CUDA.jl handled this at the allocation site, detecting out-of-memory errors and triggering the GC. This was not ideal, as it could lead to significant pauses and a bloated memory usage.</p>
<p>To improve this, <strong>CUDA.jl v5.4 more accurately keeps track of memory usage, and uses that information to trigger the GC early at appropriate times</strong>, e.g., when waiting for a kernel to finish. This should lead to more predictable performance, both by distributing the cost of garbage collection over time and by potentially masking it behind other operations.</p>
<p>For example, the following toy model implemented with Flux.jl allocates a ton of memory:</p>
<pre><code class="language-julia">using CUDA, Flux
using MLUtils: DataLoadern_obs &#61; 300_000
n_feature &#61; 1000
X &#61; rand&#40;n_feature, n_obs&#41;
y &#61; rand&#40;1, n_obs&#41;
train_data &#61; DataLoader&#40;&#40;X, y&#41; |&lt; gpu; batchsize &#61; 2048, shuffle&#61;false&#41;model &#61; Dense&#40;n_feature, &gt;&#41; |&lt; gpu
loss&#40;m, _x, _y&#41; &#61; Flux.Losses.mse&#40;m&#40;_x&#41;, _&gt;&#41;
opt_state &#61; Flux.setup&#40;Flux.Adam&#40;&#41;, model&#41;
Flux.train&#33;&#40;loss, model, train_data, opt_state&#41;
for epoch in 1:100
Flux.train&#33;&#40;loss, model, train_data, opt_state&#41;
end</code></pre>
<p>Without eager garbage collection, this leads to expensive pauses while freeing a large amount of memory at every epoch. We can simulate this by artificially limiting the memory available to the GPU, while also disabling the new eager garbage collection feature by setting the <code>JULIA_CUDA_GC_EARLY</code> environment variable to <code>false</code> &#40;this is a temporary knob that will be removed in the future, but may be useful now for evaluating the new feature&#41;:</p>
<pre><code class="language-text">❯ JULIA_CUDA_GC_EARLY&#61;false JULIA_CUDA_HARD_MEMORY_LIMIT&#61;4GiB \
julia --project train.jl
...
&#91; Info: Epoch 90 train time 0.031s
retry_reclaim: freed 2.865 GiB
&#91; Info: Epoch 91 train time 0.031s
&#91; Info: Epoch 92 train time 0.027s
retry_reclaim: freed 2.865 GiB
&#91; Info: Epoch 93 train time 0.03s
retry_reclaim: freed 2.873 GiB
&#91; Info: Epoch 94 train time 0.031s
retry_reclaim: freed 2.873 GiB
&#91; Info: Epoch 95 train time 0.03s
retry_reclaim: freed 2.873 GiB
&#91; Info: Epoch 96 train time 0.031s
&#91; Info: Epoch 97 train time 0.027s
retry_reclaim: freed 2.873 GiB
&#91; Info: Epoch 98 train time 0.031s
retry_reclaim: freed 2.865 GiB
&#91; Info: Epoch 99 train time 0.031s
retry_reclaim: freed 2.865 GiB
&#91; Info: Epoch 100 train time 0.031s
&#91; Info: Total time 4.307s</code></pre>
<p>With eager garbage collection enabled, more frequent but less costly pauses result in significantly improved performance:</p>
<pre><code class="language-text">❯ JULIA_CUDA_GC_EARLY&#61;true JULIA_CUDA_HARD_MEMORY_LIMIT&#61;4GiB \
julia --project wip.jl
...
&#91; Info: Epoch 90 train time 0.031s
maybe_collect: collected 1.8 GiB
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 91 train time 0.033s
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 92 train time 0.031s
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 93 train time 0.031s
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 94 train time 0.03s
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 95 train time 0.03s
maybe_collect: collected 1.8 GiB
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 96 train time 0.033s
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 97 train time 0.03s
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 98 train time 0.03s
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 99 train time 0.03s
maybe_collect: collected 1.8 GiB
&#91; Info: Epoch 100 train time 0.03s
&#91; Info: Total time 3.76s</code></pre>
<p>Eager garbage collection is driven by a heuristic that considers the current memory pressure, how much memory was freed during previous collections, and how much time that took. It is possible that the current implementation is not optimal, so if you encounter performance issues, please file an issue.</p>
<h2 id="tracked_memory_allocations">Tracked memory allocations</h2>
<p>When working with multiple GPUs, it is important to differentiate between the device that memory was allocated on, and the device used to execute code. Practically, this meant that users of CUDA.jl had to manually remember that allocating and using <code>CuArray</code> objects &#40;typically&#41; needed to happen with the same device active. The same is true for streams, which are used to order operations executing on a single GPU.</p>
<p>To improve this, <strong>CUDA.jl now keeps track of the device that owns the memory, and the stream last used to access it, enabling the package to &quot;do the right thing&quot; when using that memory</strong> in kernels or with library functionality. This does <strong>not</strong> mean that CUDA.jl will automatically switch the active device: We want to keep the user in control of that, as it often makes sense to access memory from another device, if your system supports it.</p>
<p>Let&#39;s break down what the implications are of this change.</p>
<p><strong>1. Using multiple GPUs</strong></p>
<p>If you have multiple GPUs, it may be possible that direct P2P access between devices is possible &#40;e.g., using NVLink, or just over PCIe&#41;. In this case, CUDA.jl will now automatically configure the system to allow such access, making it possible to seamlessly use memory allocated on one device in kernels executing on a different device:</p>
<pre><code class="language-julia">julia&gt; # Allocate memory on device 0
device&#33;&#40;0&#41;
CuDevice&#40;0&#41;: Tesla V100-PCIE-16GB
julia&gt; a &#61; CuArray&#40;&#91;1&#93;&#41;;julia&gt; # Use on device 1
device&#33;&#40;1&#41;
CuDevice&#40;1&#41;: Tesla V100S-PCIE-32GB
julia&gt; a .&#43; 1;</code></pre>
<p>If P2P access between devices is not possible, CUDA.jl will now raise an error instead of throwing an illegal memory access error as it did before:</p>
<pre><code class="language-julia">julia&gt; # Use on incompatible device 2
device&#33;&#40;2&#41;
CuDevice&#40;2&#41;: NVIDIA GeForce GTX 1080 Ti
julia&gt; a .&#43; 1
ERROR: cannot take the GPU address of inaccessible device memory.You are trying to use memory from GPU 0 on GPU 2.
P2P access between these devices is not possible;
either switch to GPU 0 by calling &#96;CUDA.device&#33;&#40;0&#41;&#96;,
or copy the data to an array allocated on device 2.</code></pre>
<p>As the error message suggests, you can always copy memory between devices using the <code>copyto&#33;</code> function. In this case, CUDA.jl will fall back to staging the copy on the host when P2P access is not possible.</p>
<p><strong>2. Using multiple streams</strong></p>
<p>Streams are used to order operations executing on a single GPU. In CUDA.jl, every Julia task has its own stream, making it very easy to group independent operations together, and make it possible for the GPU to potentially overlap execution of these operations.</p>
<p>Before CUDA.jl v5.4, users had to be careful about synchronizing data used in multiple tasks. It was recommended, for example, to end every data-producing task with an explicit call to <code>synchronize&#40;&#41;</code>, or alternatively make sure to <code>device_synchronize&#40;&#41;</code> at the start of a data-consuming task. Now that CUDA.jl keeps track of the stream used to last access memory, it can automatically synchronize streams when needed:</p>
<pre><code class="language-julia"># Allocate some data
a &#61; CUDA.zeros&#40;4096, 4096&#41;
b &#61; CUDA.zeros&#40;4096, 4096&#41;
#synchronize&#40;&#41; # No longer needed# Perform work on a task
t &#61; @async begin
a * b
#synchronize&#40;&#41; # No longer needed
end# Fetch the results
c &#61; fetch&#40;t&#41;</code></pre>
<p><strong>3. Using capturing APIs</strong></p>
<p>All of the above is implemented by piggybacking on the function that converts memory objects to pointers, in the assumption that this will be the final operation before the memory is used. This is generally true, with one important exception: APIs that capture memory. For example, when recording an operation using the CUDA graph APIs, a memory address may be captured and used later without CUDA.jl being aware of it.</p>
<p>CUDA.jl accounts for this by detecting conversions during stream capture, however, some APIs may not covered yet. If you encounter issues with capturing APIs, let us know, and keep using additional synchronization calls to ensure correctness.</p>
<h2 id="unified_memory_iteration">Unified memory iteration</h2>
<p>Unified memory is a feature of CUDA that allows memory to be accessed from both the CPU and the GPU. We have now greatly <strong>improved the performance of using unified memory with CPU code that iterates over elements</strong> of a <code>CuArray</code>. Although this is typically unwanted, triggering the dreaded &quot;scalar indexing&quot; error when accessing device memory in such a way, it can be useful when incrementaly porting code to the GPU.</p>
<p>Concretely, accessing elements of a unified <code>CuArray</code> on the CPU is much faster now:</p>
<pre><code class="language-julia-repl">julia&gt; # Reference
a &#61; &#91;1&#93;;
julia&gt; @btime &#36;a&#91;&#93;;
1.959 ns &#40;0 allocations: 0 bytes&#41;julia&gt; b &#61; cu&#40;a; unified&#61;true&#41;;julia&gt; # Before
@btime &#36;b&#91;&#93;
2.617 ÎĽs &#40;0 allocations: 0 bytes&#41;;julia&gt; # After
@btime &#36;b&#91;&#93;;
4.140 ns &#40;0 allocations: 0 bytes&#41;</code></pre>
<p>Notice the different unit&#33; This has a massive impact on real-life performance, for example, as demonstrated by calling <code>foldl</code> which does not have a GPU-optimized implementation:</p>
<pre><code class="language-julia-repl">julia&gt; a &#61; cu&#40;rand&#40;1024, 1024&#41;; unified&#61;true&#41;;julia&gt; # Before
@b foldl&#40;&#43;, a&#41;
4.210 s &#40;9 allocs: 208 bytes, without a warmup&#41;julia&gt; # After
@b foldl&#40;&#43;, a&#41;
3.107 ms &#40;9 allocs: 208 bytes&#41;</code></pre>
<p>For completeness, doing this with regular device memory triggers a scalar indexing error:</p>
<pre><code class="language-julia-repl">julia&gt; a &#61; cu&#40;rand&#40;1024, 1024&#41;&#41;;julia&gt; foldl&#40;&#43;, a&#41;
ERROR: Scalar indexing is disallowed.</code></pre>
<p>These changes should make it easier to port applications to the GPU by incrementally moving parts of the codebase to the GPU without having to worry about the performance of accessing memory from the CPU. The only requirement is to use unified memory, e.g., by calling <code>cu</code> with <code>unified&#61;true</code>, or setting the CUDA.jl preference <code>default_memory</code> to use unified memory by default. However, as unified memory comes with a slight cost, and results in synchronous allocation behavior, it is still recommended to switch back to regular device memory when your application has been fully ported to the GPU.</p>
<h2 id="other_changes">Other changes</h2>
<p>To keep this post from becoming even longer, a quick rundown of other changes:</p>
<ul>
<li><p><a href="https://github.com/wsmoses">@wsmoses</a> introduced initial support for automatic differentiation of heterogeneous host/device code using Enzyme.jl. Before, you would have to differentiate through host and device code separately, and manually set up rules for crossing the host/device boundary. Now, you can differentiate through entire applications with ease;</p>
</li>
<li><p><code>CUDA.@profile</code> now <a href="https://github.com/JuliaGPU/CUDA.jl/pull/2339">automatically detects external profilers</a>, so it should not be required to specify <code>external&#61;true</code> anymore when running under NSight;</p>
</li>
<li><p><a href="https://github.com/JuliaGPU/CUDA.jl/pull/2342">Exception output has been improved</a>, only reporting a single error message instead of generating output on each thread, and better forwarding the exception type;</p>
</li>
<li><p>Cached handles from libraries <a href="https://github.com/JuliaGPU/CUDA.jl/pull/2352">will now be freed</a> when under memory pressure;</p>
</li>
<li><p>Tegra devices <a href="https://github.com/JuliaGPU/CUDA.jl/pull/2374">are now supported</a> by our artifacts, obviating the use of a local toolkit;</p>
</li>
<li><p>Support for <a href="https://github.com/JuliaGPU/CUDA.jl/pull/2392">CUDA 12.5</a> has been added, as well as <a href="https://github.com/JuliaGPU/CUDA.jl/pull/2390">initial support for Julia 1.12</a>.</p>
</li>
</ul>
]]></content:encoded>

<pubDate>Tue, 28 May 2024 00:00:00 +0000</pubDate>


<atom:author>
<atom:name>Tim Besard</atom:name>
</atom:author>

</item>

<item>
<title><![CDATA[oneAPI.jl 1.5: Ponte Vecchio support and oneMKL improvements]]></title>
<link>https://juliagpu.org/post/2024-05-24-oneapi_1.5/index.html</link>
Empty file removed previews/PR44/.nojekyll
Empty file.
7 changes: 0 additions & 7 deletions previews/PR44/2019-12-12-new_site/index.html

This file was deleted.

This file was deleted.

7 changes: 0 additions & 7 deletions previews/PR44/2020-07-07-cuda_1.1/index.html

This file was deleted.

7 changes: 0 additions & 7 deletions previews/PR44/2020-07-18-cuda_1.3/index.html

This file was deleted.

Loading

0 comments on commit c5e96be

Please sign in to comment.