Skip to content

Creating a local kiwix download server

Jump to bottom
Julian Harty edited this page Dec 20, 2017 · 9 revisions

Background

Kiwix is able to download content from a download server. When we release the app it uses download.kiwix.org. Note: The content is also available from various mirrors, but that's not relevant for this article.

Testing downloads on a local network

For testing purposes we have a build flavour that uses a local minimal replica of the main download server. This wiki page explains what's involved to create a local 'download' server. It doesn't have the full functionality or the full range of content, although you're welcome to expand its capabilities and content, they're beyond the scope of this wiki page.

MVP for testing

The minimum viable product in terms of a download server consists of a web server, configured to listen on port 80 (the default http port) and 3 files.

Possible web servers

There are plenty of web servers to choose from. Suggestions include kiwix-serve, Apache Web Server, and nginx.

I've chosen to use nginx and used the following article as a guide https://www.raspberrypi.org/documentation/remote-access/web-server/nginx.md

3 related files

  1. library_zim.xml contains the master list of files available from this server,
  2. test.zim.meta4 contains meta-data about a particular ZIM file (containing content),
  3. test.zim the file containing the content.

Locations for the files

library_zim.xml -> in /library/ which mirrors http://download.kiwix.org/library/ The other files are a pair and often go in a common folder. library_zim.xml needs to contain the URL to the .meta4 file, in turn that refers to the zim file.

Creating suitable files

We can create files ourselves. Alternatively, and perhaps more practically, we can use some of the existing files from http://download.kiwix.org/ as a template and edit these.

  • Details of what to edit TBC
  • Tool to generate files: kiwix-manage.

Here's an example, reproduced from http://wiki.kiwix.org/wiki/Kiwix-manage

kiwix-manage /var/www/html/library_zim.xml add /var/www/html/zim/test.zim

Note: sudo access is needed by default when writing files to /var/www/html/...

How kiwix-android uses the information and downloads files

Kiwix-Android first downloads the library_zim.xml file. This can be quite large, currently it has over 2300 items listed each includes attributes about the content. The user can choose to download individual items, these are listed in the DOWNLOADING tab in the GUI while being downloaded. Several can be downloaded in parallel.

  • The app uses the url for the .meta4 file listed, strips the .meta4 suffix from the URL and then requests that file directly (rather than downloading the meta4 file to parse its contents to identify the zim file to download).
  • This shortcut means the zim file and the meta4 filenames need to match apart from the extra .meta4 suffix - otherwise the download of the ZIM file will fail.
  • Given this information, potentially the .meta4 files aren't needed for a MVP :)

Structure of each entry in library_zim.xml 

<library version="20110515">
<book id="6e375490-5ee9-5a40-990c-3dca20f1da5f" title="WikiStage" description="WikiStage videos" language="fra" creator="WikiStage" publisher="Kiwix"
favicon="iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAIAAADYYG7QAAAL5UlEQVR4nK1Zbaxl5VV+nrXevc8592PunZkOQ9FpGcQiWFtRsX7EH7WKGkqE1v5opLYpP2gaqglSk36khDQ2sTEqxoiVNlHSWpRYm2AMhJLWiBBBEGSgUFqGYgcGZubOzP06Z+/9vuvxx5l777kzd65D4sr+cXLyvvtd61nrXetZa/PEiRM4QwwBeaE5o2QGRFKSEJUB5hSE0Np6qRCEUkdx/JswMyMgRITcPTbWbyNpy3+LGYtFLqrK7CB5Vcg6oi25Xu4a61AqI8FYU4h9Re4Pmh3WJwyWu1JGo+iyJevAmuekzNkUIplLkeZ3+OJK+ddnu0M/qJYX846dtn9/+ak3edUriytwowgAUTAzaODx7MH+My+VV4+pV6WLLuBbL+Ebp7E47GfkXljY61VIAEVZkdx8xxTv+mb563+qn/m+FQWJEOrarvhR3fhbvPKnubBMZ4mi2Vk+9T2/7e/9G0+k0VCVpSway769fP9v8KNXdUmpk8gSooHbY8X1GCIUYArvvJ2t7RN38PZ7tLtfZvq9QMY4GsSlodrin/lw/ug1PLZke2bLPz9kv/tnq8PlXXNzQydIEwqAprVjy+lXryhfvNmqOtpMeucyybXu7DNkE45GW2U7P4PP3plu/3rs241BldqIEigFOcDA3Eyen+en/6r68r3cuyv+7YDd+MdInDnvDUVoxZQjSrAEexXevDv+5bFy4225TgQEpQLIyjYIbShEoWTt7PkjT/b/8p64YJdphGIRa9YQKF5iVHlpd8+Wz93lLx2KW++MHN5PNupat14or7+wQwxLc+Gu6p6H6q98M3bMdTmTSFIG+H8rFA6ZWcUv3ZutsG/skiCrJtD1sFIHsvXqaFZx05/3Xj7UmxqUpjQQoji0sdrCiMqimx20X73fV4f9KnWEGRNw1kCacJnQq7BwUo+/4P2KTRQXCcWEMQFYUMYIDlL857MlIhtIGgkwb3ohAaANTPfSsy/kgy93varK0Z1dmdMQykblxRMxWjJPZRtUx/aFol8HMH62EZLR5t6rC2aVgO38tUkhUjQWhEwObm/H6xNJJCATCNM5KiRKnabnbVCrFNKEU77Y2E8SMEhkULHaUUScNSJIGCWyqqzZM2elhZtzO30mEYI1ReftrC69MC/lIAiGIE2cJhQphGTmucxctq/ycDK4tdG5RKFjddheuM/3X9AOo0CQztVloeTelet+vShXCpNaaVPOUDGFmylHjtT8ye91e8/nsPW0VVkQDJZq50Kj9/0S56arVm5SnD0rblJIgJVYHOLXfra+7srRqyc75wAwTuR6M6QqaO3Lr8VHfjMuvTg+9t6ysso2B88oCYIq614+lt75Vr/h2mZ51ZNQPEy+zbXfKB2QEQ2sv7iSmeyWO7r7HunNVtEhU2t6EwZbHvIj74lPf7h0q15NtXfdX33ydjfQvBtH2BrkWi34iYv8izfTU0gx0w9FHcA2F5OTfKgw+ob33WKvLPjePfjuC84yiiRbUyhJx2Xv/jn74JWxPDITSZuaH973sH/lHiqVOnuYxngneMnpTftVWX7xcPWFP2h/8WJf7iIYdvaLtlHtGeYsOduRk4MXDo+Ovua9qVWlugplnsqOBT7nfOA/cPcDdVgxSbmA09MzmPWcS1fMCQIi2LHzunvu+dSGiZodRGfWMdfybcJ6Q6HipaavFBbk+R4G/VHbJveSLdZdFlaspKTmjfPZYUItNJSFhp3qhAQwMD5OLkOkqemyI7RSOOhBUB2psDhcZwmjSYJGc4xWicYE5WCiEEEz0IuQEAEFQ5aiqEDCEDAgk26BcHKi0oQIyx6liV6/in7fIpMsYoRcgBsVhBXCqdD4Yk24jEQ0ObclzEnWYSYmRER0oZIVEEgZMA4CwggQgggLarPRFGWQlcBMjb4xUAJMnCJLROlEplaZEaWs3YXJBCIQo9abLpyw0lUMSGGVJzeVcXE5q/O3krFfIjQzjdotInKw06hK2D3D+V4pOYkkFLAzFQLJnJlzcZqAI4u+HNVq4ZGTTNZLwLlz9YmXIgKDnifAXfMzmJ/mscX+3f+ePvS59LX7m5kdgTAXjMTmGDJZ17bMSrXp+HK65l1x49XRH5SvPWi3351ql5ukcweJBJxoA/1enpkt338tPfyE7nvCHn+6O7poR05Ul79l4NFk9hPzuKRMknwBthxYWbal5ebd77I7bo48tJHSJ34bU1Zu/dt0/gw65LNUri1FIdQVXj7O6z9fP/h0WTqGYnm6x72zyCHzAnEyca+bSzCXES+/kNf8cvuOS3u3/k4ZrvQXuxhlHTsR119ll+6P1YZ2zgCNGUwOTCUdOa57H2ywGrt32vzOLlVRcpMjzuRG6wiJ9Bbsp7jtpkgxKsGcO6d64UPk6ZnmHZf07nyxTJEZr1dUM1c7WHKbA96kSCl7phJQTt3XMxACJY8YEhqm4UoaRSFD6JMCszJ/eI98Mxs5J5xIQSFEkTEpIjyFxBBN62RrPXdvxFCQkAaiDzBgszysWrhjFG4WU4Y4MVKGkds1MdtptvFzC0+tyyQnFyR6+taB7qv3e52YlIPeCSilQ/vkQdR8/RC9TtlASKHpyg8eKx/7fP38D+rVrrnhWjt21MKbPbN6+Jnq8afS3FRbBAISBZEYd7RjGuiwCJxJjDCm02FUgE6BJEFtRYvSpk0pVlZMxS66oPzpl/Wm83DlFSWAA//T+/gXGimT6mBu9AijaCAjihXRw6GA5S0HGBQT0HlBGECDSBtbNlaK0tiPE5uJKJjqq6o5anJ2Xf9H9c+/PXZX6YEDykOf7nUt0CspPEQuDasuO8NY57mpkqQhVCNtcZcJK36yVb8/1WNXrC1yx9a0cYIPgRHq19GvfXXYpNTN9KtHn+jaYrMDDAauMFhyNQujurbqZ3489u4JqRw5iseeS20Znd9LQwuckTcVQK9c/iN+4Hvdqyt1ZT6YhltntkXDMgmvSqhyWVLAGQbLg0GaRu6USmndvCcdWZn9hbePbvlQ/Ni+MugXSE3jTx2MT/5N9czTeWrWAqfTL4Kjzv7iJg6b8sCjzYNP1Y88jcMn0+KiopBwMAjDKZ65vs0gWb+m18EgGREASgENBTRAK8N82cV+56diMPClJa42IAjqbW/h330mX/vx+qVD0Z/Km9pviPSusRNL5Sff7Je8Bzdc3bx42B767+rub2kwGEo9KUBCY84+EXciBn0bpLZs1Um7a2HYu+5X8o4pW1goNFQOd7jpxGK8Yap6/5W2kmFnbBwPu1aG7ISjx7WyavvOqz541fDrf5jf+87B0qgkJliMw3vTBC0rpvqYG1jZMteMGYJbsY5Oh41fIZgpwxqpIs8MobErfHUYgJLBTEtds7xapcoHdY4wqJMMNu7l1jcZEi25nb8ToAsUTDDx1JPlOwb4h/tjNKp2TVcKSpRC0tysnVzCP35Dg5RCvr5FpEgQoTh2svMKZhRZk6kS0UXHUw26Tk0TJqYfiMjFhPN2Wy5yFGrticIoKmWm1z75PD5wC7/9StQpQABeVfadQ/aBzw6+fQgzU12UPF7PyIiMCCpHF68sVGQqWYLj1MCBayjb+nXbUMgA0EJ595yCntzczd2SeUrjJ0XyPTvw6AFe/ft68dUY1OGeTi6Waz+lx74Tu2aTSWuLPaWUvEpeVVUVxsMLIopYuG0t20TQzCqymZvVwnE6erkQG1iesqizrs4+U0VXGFEiCkx1Vb0y9OEIGTSud8oSAmBlWFiuDx5uoxvSK26dEdeOmOxcSauSXjma/+u5gqq/Vkc3wtTFUBdFF/0QL9vvTSMAVWUvHcHT320Fc68nZ5Jjf4xna3vmytsuhowGbDMi4mmfFiJUV5zqU+vTwonyLsjMYGiaMmx8TMsl1HXp10YRnNg4gT1oJXzUUpHPdU4NQGMBILN16rTpzYFT2YvmGwdHMEJE0AD4FsdQgEiSLp19xHVaZTYzScT4m8Hanom9hJEkpdiUOs3CnIBzgjBNEieS468wZjqt99pOoVPoYC3hnVICE2cLYwzPjEwJm9nbpo3S2ODTTdxeoc2yba9zZgH9f5L/BXR8JGk83KEfAAAAAElFTkSuQmCC" 
faviconMimeType="image/png" date="2015-07-14" 
url="http://download.kiwix.org/zim/other/wikistage_fr_all_2015-07.zim.meta4" articleCount="337" mediaCount="341" 
size="4886095"/>
</library>

Structure of a .meta4 file

<?xml version="1.0" encoding="UTF-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink">
  <generator>MirrorBrain/2.18.1</generator>
  <origin dynamic="true">http://download.kiwix.org/zim/other/wikistage_fr_all_2015-07.zim.meta4</origin>
  <published>2017-12-14T12:03:58Z</published>
  <publisher>
    <name>Kiwix project</name>
    <url>http://www.kiwix.org</url>
  </publisher>

  <file name="wikistage_fr_all_2015-07.zim">
    <size>5003362189</size>

    <!-- <mtime>1436866639</mtime> -->

    <!-- internal id: 45151 -->
    <hash type="md5">58948ab52eab5b26a438f668beb59cae</hash>
    <hash type="sha-1">0e3dcf1504bee9e03961e30005713d7e3b38b0b2</hash>
    <hash type="sha-256">bcdfba6b47cb54194a4434b1d79f09274e503f17d9e4f0e638db1774d10193fd</hash>
    <pieces length="1048576" type="sha-1">
      <hash>a6a359fe8c222cd2f506ea1001f6c54bfce76674</hash>
      <hash>bd5f9897cd8b7634010ca42831d36aaa17ed13e4</hash>
... 1000's of lines cut ...
    </pieces>


    <!-- Found 4 mirrors: 0 in the same network prefix, 0 in the same autonomous system,
         0 handling this country, 4 in the same region, 0 elsewhere -->

    <!-- Mirrors in the same network (unknown): -->

    <!-- Mirrors in the same AS (unknown): -->

    <!-- Mirrors which handle this country (GB): -->

    <!-- Mirrors in the same continent (EU): -->
    <url location="dk" priority="1">https://mirrors.dotsrc.org/kiwix/zim/other/wikistage_fr_all_2015-07.zim</url>
    <url location="de" priority="2">https://ftp.fau.de/kiwix/zim/other/wikistage_fr_all_2015-07.zim</url>
    <url location="nl" priority="3">https://ftp.nluug.nl/pub/kiwix/zim/other/wikistage_fr_all_2015-07.zim</url>
    <url location="fr" priority="4">http://mirror.download.kiwix.org/zim/other/wikistage_fr_all_2015-07.zim</url>

    <!-- Mirrors in the rest of the world: -->
  </file>
</metalink>

Samples and examples

We have a small repo of test files available at https://github.com/kiwix/testfiles

Additional information

https://github.com/kiwix/kiwix-android/issues/227

Clone this wiki locally