forked from jgomezdans/jgomezdans.github.com
-
Notifications
You must be signed in to change notification settings - Fork 0
bogdan124/jgomezdans.github.com
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Fast copying of files over the network - EO & DA Ramblings</title>
<meta name="description" content="">
<meta name="author" content="J Gomez-Dans">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- Le HTML5 shim, for IE6-8 support of HTML elements -->
<!--[if lt IE 9]>
<script src="/theme/html5.js"></script>
<![endif]-->
<!-- Le styles -->
<link href="https://netdna.bootstrapcdn.com/twitter-bootstrap/2.1.1/css/bootstrap.no-icons.min.css" rel="stylesheet">
<link href="/theme/local.css" rel="stylesheet">
<link href="/theme/pygments.css" rel="stylesheet">
<link href="/theme/font-awesome.css" rel="stylesheet">
<link href='https://fonts.googleapis.com/css?family=Gudea:400,400italic|Alegreya+SC' rel='stylesheet' type='text/css'>
</head>
<body>
<header class="blog-header">
<div class="container">
<div class="row-fluid">
<div class="span9">
<a href="" class="brand">EO & DA Ramblings</a>
</div>
<div class="span3" id="blog-nav">
<ul class="nav nav-pills pull-right">
<li class="active" >
<a href="/category/blog.html ">Blog</a>
</ul>
</div>
</div> <!-- End of fluid row-->
</div> <!-- End of Container-->
</header>
<div class="container">
<div class="content">
<div class="row-fluid">
<div class="span10">
<div class='article'>
<div class="row-fluid">
<div class="content-title span9">
<h1>Fast copying of files over the network</h1>
</div>
</div>
<div class="row-fluid">
<div class="span2">
<p>Wed 08 May 2013 </p>
<p style="text-align: left;">
Filed under <a href="/category/blog.html">Blog</a>
</p>
<p style="text-align: left;">
Tags <a href="/tag/unix.html">unix</a> <a href="/tag/tips.html">tips</a> <a href="/tag/meteo.html">meteo</a> </p>
<p>
</p>
</div>
<div class="span8">
<p>A common requirement when dealing with processing large datasets over
multiple networked machines is to have a local staging space: copy the
data on the local disk to improve access speed and not to bog down a
NFS server when all different processes start accessing different files
all at once in the server. In most cases, such insight is typically achieved
post-facto, so you end up needing to copy all files across to your local
staging directories, resulting in pretty much collapsing all NFS traffic
and servers.</p>
<p>If your network allows this, using <tt class="docutils literal">netcat</tt> can be a good and fast option
for copying the files across. If you are on a trusted network, there's no
need to encrypt traffic, so netcat is appropriate (it's just like <tt class="docutils literal">cat</tt>, but
over the network). First, on the target machine, go to the directory where
you want to copy things, and set up a listener</p>
<pre class="literal-block">
nc -l 7000 | gunzip -v| tar -xvf -
</pre>
<p>In the previous example, I'm placing the listener on port 7000 (other ports could
be used). Note that the output is being piped through <tt class="docutils literal">gunzip</tt> into <tt class="docutils literal">tar</tt>: we
expect that the data will be compressed. On the server, we can issue the command
that packs the files we want to transfer and sends them using <tt class="docutils literal">netcat</tt></p>
<pre class="literal-block">
tar c my_directory/ |gzip --fast -v | nc target 7000
</pre>
<p>In here, we <tt class="docutils literal">tar</tt> and <tt class="docutils literal">gzip</tt> (using the fastest compression to minimise overhead), and
pipe stuff into <tt class="docutils literal">netcat</tt>. <tt class="docutils literal">netcat</tt> will output that stuff to a computer named <tt class="docutils literal">target</tt>
on port 7000, which is where our listener is waiting.</p>
<hr />
</div>
</div>
<div class="span10">
<h3>Comments</h3>
</div>
</div>
</div>
</div> </div> </div>
<!--footer-->
<div class="container">
<div class="well" style="background-color: #E9EFF6">
<div id="blog-footer">
<div class="row-fluid">
<div class="social span2" align="center" id="socialist">
<ul class="nav nav-list">
<li class="nav-header">
Social
</li>
<li><a href="http://twitter.com/pepucho666"><i class="icon-twitter" style="color: #1f334b"></i>twitter</a></li>
<li><a href="http://github.com/jgomezdans"><i class="icon-github" style="color: #1f334b"></i>github</a></li>
</ul>
</div>
<div class="site-nav span2" align="center">
<ul class="nav nav-list" id="site-links">
<li class="nav-header">
Site
</li>
<li><a href=""><i class="icon-home" style="color: #1f334b">
</i>Home</a></li>
<li><a href="/archives.html"><i class="icon-list" style="color: #1f334b">
</i>Archives</a></li>
<li><a href="/tags.html"><i class="icon-tags" style="color: #1f334b">
</i>Tags</a></li>
</ul>
</div>
</div> <!--end of fluid row-->
</div> <!--end of blog-footer-->
<hr />
<p align="center"><a href="">EO & DA Ramblings</a>
© J Gomez-Dans
Powered by <a href="https://github.com/getpelican/pelican">Pelican</a> and
<a href="https://twitter.github.com/bootstrap">Twitter Bootstrap</a>.
Icons by <a href="https://fortawesome.github.com/Font-Awesome">Font Awesome</a> and
<a href="https://gregoryloucas.github.com/Font-Awesome-More">Font Awesome More</a></p>
</div> <!--end of well -->
</div> <!--end of container -->
<!--/footer-->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.8.3/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/twitter-bootstrap/2.2.2/js/bootstrap.min.js"></script>
<script>var _gaq=[['_setAccount','UA-25702318-1'],['_trackPageview']];(function(d,t){var g=d.createElement(t),s=d.getElementsByTagName(t)[0];g.src='//www.google-analytics.com/ga.js';s.parentNode.insertBefore(g,s)}(document,'script'))</script>
</body>
</html>About
No description, website, or topics provided.
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published
Languages
- HTML 94.2%
- CSS 5.8%