Creating a local kiwix download server

Background

Kiwix is able to download content from a download server. When we release the app it uses download.kiwix.org. Note: The content is also available from various mirrors, but that's not relevant for this article.

Testing downloads on a local network

For testing purposes we have a build flavour that uses a local minimal replica of the main download server. This wiki page explains what's involved to create a local 'download' server. It doesn't have the full functionality or the full range of content, although you're welcome to expand its capabilities and content, they're beyond the scope of this wiki page.

MVP for testing

The minimum viable product in terms of a download server consists of a web server, configured to listen on port 80 (the default http port) and 3 files.

Possible web servers

There are plenty of web servers to choose from. Suggestions include kiwix-serve, Apache Web Server, and nginx.

The 3 essential files

1. library_zim.xml
2. test.zim.meta4
3. testzim.zim

Locations for the files

library_zim.xml -> in /library/ which mirrors http://download.kiwix.org/library/ The other files are a pair and often go in a common folder. library_zim.xml needs to contain the URL to the .meta4 file, in turn that refers to the zim file.

Creating suitable files

We can create files ourselves. Alternatively, and perhaps more practically, we can use some of the existing files from http://download.kiwix.org/ as a template and edit these.

• Details of what to edit TBC
• Tools to generate files also TBC

Structure of each entry in library_zim.xml

<library version="20110515">
<book id="6e375490-5ee9-5a40-990c-3dca20f1da5f" title="WikiStage" description="WikiStage videos" language="fra" creator="WikiStage" publisher="Kiwix"
favicon="iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAIAAADYYG7QAAAL5UlEQVR4nK1Zbaxl5VV+nrXevc8592PunZkOQ9FpGcQiWFtRsX7EH7WKGkqE1v5opLYpP2gaqglSk36khDQ2sTEqxoiVNlHSWpRYm2AMhJLWiBBBEGSgUFqGYgcGZubOzP06Z+/9vuvxx5l777kzd65D4sr+cXLyvvtd61nrXetZa/PEiRM4QwwBeaE5o2QGRFKSEJUB5hSE0Np6qRCEUkdx/JswMyMgRITcPTbWbyNpy3+LGYtFLqrK7CB5Vcg6oi25Xu4a61AqI8FYU4h9Re4Pmh3WJwyWu1JGo+iyJevAmuekzNkUIplLkeZ3+OJK+ddnu0M/qJYX846dtn9/+ak3edUriytwowgAUTAzaODx7MH+My+VV4+pV6WLLuBbL+Ebp7E47GfkXljY61VIAEVZkdx8xxTv+mb563+qn/m+FQWJEOrarvhR3fhbvPKnubBMZ4mi2Vk+9T2/7e/9G0+k0VCVpSway769fP9v8KNXdUmpk8gSooHbY8X1GCIUYArvvJ2t7RN38PZ7tLtfZvq9QMY4GsSlodrin/lw/ug1PLZke2bLPz9kv/tnq8PlXXNzQydIEwqAprVjy+lXryhfvNmqOtpMeucyybXu7DNkE45GW2U7P4PP3plu/3rs241BldqIEigFOcDA3Eyen+en/6r68r3cuyv+7YDd+MdInDnvDUVoxZQjSrAEexXevDv+5bFy4225TgQEpQLIyjYIbShEoWTt7PkjT/b/8p64YJdphGIRa9YQKF5iVHlpd8+Wz93lLx2KW++MHN5PNupat14or7+wQwxLc+Gu6p6H6q98M3bMdTmTSFIG+H8rFA6ZWcUv3ZutsG/skiCrJtD1sFIHsvXqaFZx05/3Xj7UmxqUpjQQoji0sdrCiMqimx20X73fV4f9KnWEGRNw1kCacJnQq7BwUo+/4P2KTRQXCcWEMQFYUMYIDlL857MlIhtIGgkwb3ohAaANTPfSsy/kgy93varK0Z1dmdMQykblxRMxWjJPZRtUx/aFol8HMH62EZLR5t6rC2aVgO38tUkhUjQWhEwObm/H6xNJJCATCNM5KiRKnabnbVCrFNKEU77Y2E8SMEhkULHaUUScNSJIGCWyqqzZM2elhZtzO30mEYI1ReftrC69MC/lIAiGIE2cJhQphGTmucxctq/ycDK4tdG5RKFjddheuM/3X9AOo0CQztVloeTelet+vShXCpNaaVPOUDGFmylHjtT8ye91e8/nsPW0VVkQDJZq50Kj9/0S56arVm5SnD0rblJIgJVYHOLXfra+7srRqyc75wAwTuR6M6QqaO3Lr8VHfjMuvTg+9t6ysso2B88oCYIq614+lt75Vr/h2mZ51ZNQPEy+zbXfKB2QEQ2sv7iSmeyWO7r7HunNVtEhU2t6EwZbHvIj74lPf7h0q15NtXfdX33ydjfQvBtH2BrkWi34iYv8izfTU0gx0w9FHcA2F5OTfKgw+ob33WKvLPjePfjuC84yiiRbUyhJx2Xv/jn74JWxPDITSZuaH973sH/lHiqVOnuYxngneMnpTftVWX7xcPWFP2h/8WJf7iIYdvaLtlHtGeYsOduRk4MXDo+Ovua9qVWlugplnsqOBT7nfOA/cPcDdVgxSbmA09MzmPWcS1fMCQIi2LHzunvu+dSGiZodRGfWMdfybcJ6Q6HipaavFBbk+R4G/VHbJveSLdZdFlaspKTmjfPZYUItNJSFhp3qhAQwMD5OLkOkqemyI7RSOOhBUB2psDhcZwmjSYJGc4xWicYE5WCiEEEz0IuQEAEFQ5aiqEDCEDAgk26BcHKi0oQIyx6liV6/in7fIpMsYoRcgBsVhBXCqdD4Yk24jEQ0ObclzEnWYSYmRER0oZIVEEgZMA4CwggQgggLarPRFGWQlcBMjb4xUAJMnCJLROlEplaZEaWs3YXJBCIQo9abLpyw0lUMSGGVJzeVcXE5q/O3krFfIjQzjdotInKw06hK2D3D+V4pOYkkFLAzFQLJnJlzcZqAI4u+HNVq4ZGTTNZLwLlz9YmXIgKDnifAXfMzmJ/mscX+3f+ePvS59LX7m5kdgTAXjMTmGDJZ17bMSrXp+HK65l1x49XRH5SvPWi3351ql5ukcweJBJxoA/1enpkt338tPfyE7nvCHn+6O7poR05Ul79l4NFk9hPzuKRMknwBthxYWbal5ebd77I7bo48tJHSJ34bU1Zu/dt0/gw65LNUri1FIdQVXj7O6z9fP/h0WTqGYnm6x72zyCHzAnEyca+bSzCXES+/kNf8cvuOS3u3/k4ZrvQXuxhlHTsR119ll+6P1YZ2zgCNGUwOTCUdOa57H2ywGrt32vzOLlVRcpMjzuRG6wiJ9Bbsp7jtpkgxKsGcO6d64UPk6ZnmHZf07nyxTJEZr1dUM1c7WHKbA96kSCl7phJQTt3XMxACJY8YEhqm4UoaRSFD6JMCszJ/eI98Mxs5J5xIQSFEkTEpIjyFxBBN62RrPXdvxFCQkAaiDzBgszysWrhjFG4WU4Y4MVKGkds1MdtptvFzC0+tyyQnFyR6+taB7qv3e52YlIPeCSilQ/vkQdR8/RC9TtlASKHpyg8eKx/7fP38D+rVrrnhWjt21MKbPbN6+Jnq8afS3FRbBAISBZEYd7RjGuiwCJxJjDCm02FUgE6BJEFtRYvSpk0pVlZMxS66oPzpl/Wm83DlFSWAA//T+/gXGimT6mBu9AijaCAjihXRw6GA5S0HGBQT0HlBGECDSBtbNlaK0tiPE5uJKJjqq6o5anJ2Xf9H9c+/PXZX6YEDykOf7nUt0CspPEQuDasuO8NY57mpkqQhVCNtcZcJK36yVb8/1WNXrC1yx9a0cYIPgRHq19GvfXXYpNTN9KtHn+jaYrMDDAauMFhyNQujurbqZ3489u4JqRw5iseeS20Znd9LQwuckTcVQK9c/iN+4Hvdqyt1ZT6YhltntkXDMgmvSqhyWVLAGQbLg0GaRu6USmndvCcdWZn9hbePbvlQ/Ni+MugXSE3jTx2MT/5N9czTeWrWAqfTL4Kjzv7iJg6b8sCjzYNP1Y88jcMn0+KiopBwMAjDKZ65vs0gWb+m18EgGREASgENBTRAK8N82cV+56diMPClJa42IAjqbW/h330mX/vx+qVD0Z/Km9pviPSusRNL5Sff7Je8Bzdc3bx42B767+rub2kwGEo9KUBCY84+EXciBn0bpLZs1Um7a2HYu+5X8o4pW1goNFQOd7jpxGK8Yap6/5W2kmFnbBwPu1aG7ISjx7WyavvOqz541fDrf5jf+87B0qgkJliMw3vTBC0rpvqYG1jZMteMGYJbsY5Oh41fIZgpwxqpIs8MobErfHUYgJLBTEtds7xapcoHdY4wqJMMNu7l1jcZEi25nb8ToAsUTDDx1JPlOwb4h/tjNKp2TVcKSpRC0tysnVzCP35Dg5RCvr5FpEgQoTh2svMKZhRZk6kS0UXHUw26Tk0TJqYfiMjFhPN2Wy5yFGrticIoKmWm1z75PD5wC7/9StQpQABeVfadQ/aBzw6+fQgzU12UPF7PyIiMCCpHF68sVGQqWYLj1MCBayjb+nXbUMgA0EJ595yCntzczd2SeUrjJ0XyPTvw6AFe/ft68dUY1OGeTi6Waz+lx74Tu2aTSWuLPaWUvEpeVVUVxsMLIopYuG0t20TQzCqymZvVwnE6erkQG1iesqizrs4+U0VXGFEiCkx1Vb0y9OEIGTSud8oSAmBlWFiuDx5uoxvSK26dEdeOmOxcSauSXjma/+u5gqq/Vkc3wtTFUBdFF/0QL9vvTSMAVWUvHcHT320Fc68nZ5Jjf4xna3vmytsuhowGbDMi4mmfFiJUV5zqU+vTwonyLsjMYGiaMmx8TMsl1HXp10YRnNg4gT1oJXzUUpHPdU4NQGMBILN16rTpzYFT2YvmGwdHMEJE0AD4FsdQgEiSLp19xHVaZTYzScT4m8Hanom9hJEkpdiUOs3CnIBzgjBNEieS468wZjqt99pOoVPoYC3hnVICE2cLYwzPjEwJm9nbpo3S2ODTTdxeoc2yba9zZgH9f5L/BXR8JGk83KEfAAAAAElFTkSuQmCC" 
faviconMimeType="image/png" date="2015-07-14" 
url="http://download.kiwix.org/zim/other/wikistage_fr_all_2015-07.zim.meta4" articleCount="337" mediaCount="341" 
size="4886095"/>
</library>

Structure of a .meta4 file

<?xml version="1.0" encoding="UTF-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink">
  <generator>MirrorBrain/2.18.1</generator>
  <origin dynamic="true">http://download.kiwix.org/zim/other/wikistage_fr_all_2015-07.zim.meta4</origin>
  <published>2017-12-14T12:03:58Z</published>
  <publisher>
    <name>Kiwix project</name>
    <url>http://www.kiwix.org</url>
  </publisher>

  <file name="wikistage_fr_all_2015-07.zim">
    <size>5003362189</size>

    <!-- <mtime>1436866639</mtime> -->

    <!-- internal id: 45151 -->
    <hash type="md5">58948ab52eab5b26a438f668beb59cae</hash>
    <hash type="sha-1">0e3dcf1504bee9e03961e30005713d7e3b38b0b2</hash>
    <hash type="sha-256">bcdfba6b47cb54194a4434b1d79f09274e503f17d9e4f0e638db1774d10193fd</hash>
    <pieces length="1048576" type="sha-1">
      <hash>a6a359fe8c222cd2f506ea1001f6c54bfce76674</hash>
      <hash>bd5f9897cd8b7634010ca42831d36aaa17ed13e4</hash>
... 1000's of lines cut ...
    </pieces>


    <!-- Found 4 mirrors: 0 in the same network prefix, 0 in the same autonomous system,
         0 handling this country, 4 in the same region, 0 elsewhere -->

    <!-- Mirrors in the same network (unknown): -->

    <!-- Mirrors in the same AS (unknown): -->

    <!-- Mirrors which handle this country (GB): -->

    <!-- Mirrors in the same continent (EU): -->
    <url location="dk" priority="1">https://mirrors.dotsrc.org/kiwix/zim/other/wikistage_fr_all_2015-07.zim</url>
    <url location="de" priority="2">https://ftp.fau.de/kiwix/zim/other/wikistage_fr_all_2015-07.zim</url>
    <url location="nl" priority="3">https://ftp.nluug.nl/pub/kiwix/zim/other/wikistage_fr_all_2015-07.zim</url>
    <url location="fr" priority="4">http://mirror.download.kiwix.org/zim/other/wikistage_fr_all_2015-07.zim</url>

    <!-- Mirrors in the rest of the world: -->
  </file>
</metalink>

Additional information

https://github.com/kiwix/kiwix-android/issues/227