Flaky build process

[bsless]

I've been trying to run the build process, have not been successful yet, but I have come across some hurdles which I addressed as I was going:

• tmp folder must exist before creating files in it.
• flaky download process: I had about 1/3 chance of getting an unexpected EOF as the connection was broken during download
• existing files get downloaded again
• image magic fails mysteriously

I'm now stuck on imagemagic after messing around with its policy file. Managed to convert one file, the rest give:

$ convert  tmp/world.200404.3x21600x21600.B2.png -scale 50% tmp/world.B2.2.png
convert-im6.q16: no images defined `tmp/world.B2.2.png' @ error/convert.c/ConvertImageCommand/3229.

For a more robust download, I added this little helper:

(defn- download
  [url out & {:keys [force? retries]
              :or {force? false
                   retries 3}}]
  (let [target (io/file out)
        exists? (java.io.File/.exists target)]
    (.println *err* (str "Downloading " url " ..."))
    (println url out exists?)
    (when (or (not (java.io.File/.exists target)) force?)
      (try
        (io/copy
         (io/input-stream url)
         target)
        (catch Exception e
          (.println *err* (str "Exception during download" e))
          (if (pos? retries)
            (download url out :force? force? :retries (dec retries))
            (throw e)))))))

[bsless]

Replacing clojure.java.shell with clojure.java.process surfaces the error from convert

[wedesoft]

I think I had to configure the memory limit for "convert". I need to look at the configuration when I get the time.

[wedesoft]

Yes, I also noticed that the downloads from NASA servers have become more unreliable recently. Thanks for looking into this.

[wedesoft]

I have the following section in /etc/ImageMagick-6/policy.xml (Debian Linux). If I remember correctly I had to increase the memory and disk space limit.

<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <policy domain="resource" name="memory" value="8GiB"/>
  <policy domain="resource" name="map" value="512MiB"/>
  <policy domain="resource" name="width" value="32KP"/>
  <policy domain="resource" name="height" value="32KP"/>
  <!-- <policy domain="resource" name="list-length" value="128"/> -->
  <policy domain="resource" name="area" value="128MP"/>
  <policy domain="resource" name="disk" value="8GiB"/>
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <policy domain="resource" name="thread" value="8"/>
 ...
</policymap>

[wedesoft]

Initially I tried to use LWJGL's STB bindings to scale images, but I didn't manage to get it to work on the largest resolution (see commit).