ideas for best practices with binary files or large files like images

[rlanzafame]

Tom and I have been brainstorming how to set up our MUDE book better now that we are moving it from TUD GitLab to GitHub. On the one hand it is a chance to reverse some bad practices like dumping huge binary assets like image files in the repo, as well as having a better way to store large files like a zip containing a notebook, csv or other data for an assignment (e.g., downloaded from a page using the download link replacer).

There are two main reasons for addressing this:

1. the size of the git repo explodes
2. the size of the build artifact (HTML files) becomes large and we will eventually exceed the 1 GB limit for a free GitHub Pages site

For reference, the MUDE book build is currently around 350 MB (unzipped) and includes about 169 MB of figures, 84 MB of ipynb files (including cell outputs), 27 MB of HTML and 2 MB md files (yes, I know this doesn't add up, but you get the idea...the rest is probably files like wheels for the thebe/pyodide feature, py files, etc).

Some obvious things to consider are:

1. forbid addition of binary files to the repo as much as possible
2. use a separate repo or website to store and/or serve the assets from the book using fixed URL's instead of relative paths
3. consider executing notebooks during the build process to generate the notebook output

While these things are relatively straightforward, they also present a burden to newcomers to book-building, as they are already trying to learn about Git, Python, Sphinx, Jupyter Book, etc, etc, etc. Item no. 3 has the added disadvantage of making build time for a big book like MUDE very long.

So far this is what we have considered:

Git Large File Storage: this could work nicely if you don't plan on using the images in more than one place (e.g., book 2 uses image from book 1, then book 1 changes it without telling book 2)

Using a second GitHub repo adds an extra step, but is easy to do. One thing to consider is that there are limits to how often an image can be requested, see here...i don't understand it all, but seems like once you have a few hundred students working on the same set of files at once, you might have issues, and this is definitely something we have in MUDE.

jsDelivr seems like a totally free and reliable way to serve "big" files (less than 20MB) that are already in a github repo, and would prevent github from getting angry if our websites start to get too many requests (see here...i don't understand it all, but I worry that 300 students loading images from the same repo page will eventually lead to issues). So jsDelivr is cool because you can use a jsdelivr URL for files instead of github with predicatable URLs. I found this page and set up an example, it's insanely easy:

• URL generator: https://www.jsdelivr.com/github
• Tom's LFS stored image doesn't work (bummer): https://cdn.jsdelivr.net/gh/Tom-van-Woudenberg/CME4501/book/figures/Tom.jpg
• but the non-LFS image does work! https://cdn.jsdelivr.net/gh/tudelft-books/CME4501@2024/book/figures/Tom.jpg
  so the idea here would be that this is a good solution for images that don't change (don't want to track in git) but also jsDelivr versioning and updating may lead to slow changes

In WWJD fashion we know that Jason uses Dreamhost DreamObject to store assets, see README for policy. We are hesitant to adopt this because it is another platform to manage and for the case where we want to create shared edu resources that are easy to find and reuse, it's more difficult to collaborate. GitHub may be a better option because with PR's contributions can be done the same way as contributing to a book. And a took like jsDelivr solves the problem that GitHub is not intended to be a CDN (even though for our edu purposes it would be uncommon to exceed the request limits)

So at the moment we are thinking of doing this:

1. making a second repository for assets with a few rules for the book repo, for example: not allowed file extensions: jpg, png, gif, jpeg, pdf, xlsx, docx, etc etc; for svg: depends on file size, but also how it is made...if something that is obviously not going to be edited or used in many repo's, put it in the other repo; files generated from page execution will be included in the artifact (and in the repo if outputs are stored), but we may need to develop some strategies for preventing large svg's?
2. use the the "dump everything in one repo with a unique file name" but maybe a few subdirectories like logos, headshot, etc, could be good for organization. also adopting a standard like kebab-case-for-filenames.ext
3. Use existing tools to keep things as small as possible. for example, https://imgbot.net/docs/
4. Use pre-commit hooks to give warnings for book contributors
5. Expect that we will exceed request capacity in the future: jsDelivr will protect against this for free and it is easy to switch back/forth by text search/replace of URL's
6. notebooks are still big. By executing at build time we can keep the repo super super small (but obviously build time becomes huge). This won't address the artifact size, but that could still be managed to a decent size using the external file referencing

[rlanzafame]

@Tom-van-Woudenberg did i capture it all?

[Tom-van-Woudenberg]

Git Large File Storage: this could work nicely if you don't plan on using the images in more than one place (e.g., book 2 uses image from book 1, then book 1 changes it without telling book 2)

This only solves issue 1, using permant github links solves issue 2 but is kinda unrelated to LFS

Using a second GitHub repo adds an extra step, but is easy to do. One thing to consider is that there are limits to how often an image can be requested, see here...i don't understand it all, but seems like once you have a few hundred students working on the same set of files at once, you might have issues, and this is definitely something we have in MUDE.

I don't think these API rate limits apply for calling for opening a single source file in a browser, but I'm not sure

[Tom-van-Woudenberg]

@BSchilperoort, @moorepants

[Tom-van-Woudenberg]

I decided some time back to, in general, put binary assets in an object store (like AWS buckets, etc). This is the standard practice in web development these days. I think it is the appropriate solution, but it adds an extra difficulty for people to upload and manage files there.

Are there object stores which you'd recommend? Do free object stores exist?

Oh, also for notebooks. I have a strict rule that all outputs must be cleared before committing to git. You can do this with a git pre-commit hook to automate it.

In most of our books we push it to github including output because we're too impatient to run all those pages during build.

[moorepants]

Are there object stores which you'd recommend? Do free object stores exist?

They are all about the same, but they do cost money.

But this is something that the university could provide. They can already provide FTP uploads for public websites. You could build that into your system.

In most of our books we push it to github including output because we're too impatient to run all those pages during build.

😄

[Tom-van-Woudenberg]

But this is something that the university could provide. They can already provide FTP uploads for public websites. You could build that into your system.

FTP uploads?

@rlanzafame , should we ask our data steward?

[moorepants]

I think it is more a ICT question. There are some options in the ICT service desk for web services. The PlexSystem gives you an FTP account that serves a website. You could technically fill it with images and have http://teachbooks.tudelft.nl/imagestore/myimage.png.

[rlanzafame]

yeah maybe approaching ICT is a good idea. SURF Drive might be one of the stock answers we get, which is not feasible because the request rate is low (we have reached it several times) and also it is annoying that if the original owner leaves, the materials have to be moved when their account expires. But still, maybe worth asking about because SURF builds these tools for us and hopefully they would be willing to change things or have something else in mind.

[BSchilperoort]

This is a difficult one. I think the main problem can be postponed/mostly avoided with some good practices. However, it does require incorporating all these "good practices" in everyone's way of working.

These will probably get you most of the way:

• reducing image file size:
  • use a higher compression ratio where possible
  • rescale the image to an appropriate size on the page. If it's just a small image you
  • use vector images as much as possible. SVGs also play nice with git (proper diffs). I see the Risk & Reliability book already has a lot of vector images, but also has some PNGs that could easily be vectorized
• As stated before by @moorepants: don't store notebook output (although I know that's not always possible due to the render time, trying to do that for most notebooks already would help).
• Store binary files such as CSVs elsewhere, and incorporate them only on book build if that's desirable (so they're not in git)
  • You can store data on Zenodo or 4TU (e.g. example data, not images)

And for reducing the git history: do any development on the book in branches and use PR's with squash merges. This will reduce how many commits (and probably also changes in binary files) will be in the history.

for @BSchilperoort is this something you have already considered for the hydrology book series?

Not yet 😅

[moorepants]

Store binary files such as CSVs elsewhere

Are you referring to comma separated value files? If so, CSV files aren't binary.

[rlanzafame]

oh true, duh. I liked the Zenodo idea not necessarily for the binary aspect, but if the filesize is big, and also to put the csv in a place that is more aligned with its usage. For example, imagine an interactive page in a book where a figure is made to illustrate the behavior of some process with data. First the student encounters the plot as a static rendering of the notebook output (regardless of whether it is generated prior to or during book build). Second, the student can use the Pyodide interactivity to play with it (imagine an exercise where they adjust parameters of some parametric equation). Third, an assignment outside of the book could use that same dataset. Especially for no. 3 it makes sense to have it on Zenodo, rather than buried in some cryptic collection of files in a github repo, as then the student also is exposed to good practices related to open science, etc., and might be better able to find such resources on their own in the future (and in a course like MUDE it is one of our learning objectives).

[moorepants]

Yes, zenodo is a good place for some data files that can be pulled into CI scripts. For smaller sized data sets I do this when building research papers in CI. CI downloads the data set, runs analysis, produces figures, compiles a website or latex pdf with the figures. This breaks down when the data size grows (that's when a database would be helpful).

But I don't think zenodo is a good option for a binary asset store (like buckets are).

[rlanzafame]

clear, thanks!

[BSchilperoort]

Do you now if there is any local software you can do the svg simplification with? I'd like to add it as a git precommit hook or something similar.

It should be possible with inkscape or "scour", but scour doesn't seem to do path simplification...

Another discussion on this on stackoverflow; https://stackoverflow.com/questions/7299564/automatizing-simplify-path-for-a-svg-file-using-inkscape

[Tom-van-Woudenberg]

@ibcmrocha, you'll like this ;)

[ibcmrocha]

A lot of nice comments above already :). For me basically:

• Always clear notebook outputs before merging. Waiting for them to execute during builds is annoying, but git is grateful. For my ML book I even train some fairly heavy deep learning models every time I compile the book to generate some figures (remember to fix the rng seed if that influences your output!)
• For book pages, the glue functionality is quite useful, it lets you put plots wherever you want at compilation time and avoids saving any outputs to file
• Use svgs as much as possible. With a bit of practice anyone can create nice-looking figures using Inkscape
• Pull anything binary from a server somewhere. I pull datasets from Surfdrive and rasterized figures from a dump github repo I set up just for this

[moorepants]

Always clear notebook outputs before merging. Waiting for them to execute during builds is annoying, but git is grateful. For my ML book I even train some fairly heavy deep learning models every time I compile the book to generate some figures (remember to fix the rng seed if that influences your output!)

In such cases, I typically would store a text solution file and commit it to git. Here is an example from my optimization package documentation which takes minutes to solve the problem. There is a quick check to load a precompted solution if it is present. There are fancier ways to do this, but this is pretty straight forward. Of course, if your solution is many megabytes in size, this may not work.

https://opty.readthedocs.io/stable/examples/advanced/plot_sit_to_stand.html

[rlanzafame]

Thanks! Is rasterizing an image in general a good practice for any situation where the svg will be too large because it is based on many individual data in the figure?

Updating the external resource could perhaps even be automated during the build, if needed (but arranging authentication and permissions seems like more trouble than its worth - easier to just update manually).

[moorepants]

I think it is fine to do anything to reduce the size of files in a git repo, including rasterizing svgs. You can also store any file types in a git repo, but, as this discussion points out, there are downstream consequences depending on what is done with the contents of the git repo and who's "paying" for the compute and storage.

There is also the general concept of separating the source from the presentation. You see this in latex docs, web development, markdown->html, etc. Most best practices try to keep the source separate from the presentation and use compute cycles to generate the presentation when needed. But this breaks down when the compute starts becoming a high cost. So then you try to compute once and cache to disk, only updating when something changes. This is embedded in the long old concept of a Makefile, i.e. only recompute if you have to. But Makefiles also have the negative consequence of reducing time to have office paper sword fights.

There is no golden solution, but as @BSchilperoort points out, I think you have to build in some guidelines for best practices on keeping git repo sizes small and you can even enforce some of the guidelines as rules with CI checks on pull requests that prevent merging certain kinds of changes.

[BSchilperoort]

Is rasterizing an image in general a good practice for any situation where the svg will be too large because it is based on many individual data in the figure

depends! If the paths/shapes are simple, an SVG can be much smaller and will look at any size. If you want nice curvy lines and flourishes, a rasterized image is probably better.

If you manually make something in Inkscape using straight lines, polygons, text (encoded as text not paths), and bezier curves, the file will be small. If you directly take a drawing from a digital notebook the file will likely be much larger, but can be reduced in size by the right tools.

[Tom-van-Woudenberg]

@BSchilperoort , @moorepants @rlanzafame , I had some very nice talks with the DCC (digital competency center) people about this today. I'll forward the info they'll send me when I have it, but the main conclusions:

• TU Delft's FTP service would be the most obvious service to use for this. I only need to check request limits
• Two hacky options might be:
  • dumping files in 4TU research data'
  • dumping files in GitHub release assets
• a potential future free options: Hugging face, although policies and functionalities are changing a lot
• Cloud4research from TU Delft might be able to help setup custom Amazon AWS / Microsoft Azure services if we want something custom

Other insights:

• there's a software policy at TU Delft!
• for software at TU Delft there's a process (approved by legal department) which allows you to have TU Delft disown it's copyright claim (and responsibilities) on software. Steps involve, among others, registration on PURE, adding a waiver to the repo and adding a citation file to the repo.

[rlanzafame]

Regarding the software policy:

Yeah, the software policy can be found on Zenodo here. It shows up in the copyright FAQ for research as well.

To me it's still a bit fuzzy regarding education because the software guidelines explicitly state research (this is typical of most things at the Uni/Library where research and education are referred to separately, and the guidance for education-related things is limited). So my question is: does the same procedure for waiving copyright of TU Delft to research software apply to software created for education purposes? This echoes the question we have about the definition of "book" (no need to go into details here).

on page 15 of the PDF the policy says the following, which doesn't explicitly state research or education, rather funding and CAO agreements, so maybe it's not a problem:

If a project has been solely funded by the government through direct funding, TU Delft owns all of the IP of the project based on the collective labour agreement (CAO VSNU-2019). In that case, it is possible for TU Delft to waive the copyright

[rlanzafame]

Regarding the tools for hosting assets, some comments below. Overall my opinion is that perhaps there are some interesting things to consider for a specific use case (e.g., 100% TU Delft hosted books/websites), but my feeling is that the best option is still to put things in a separate github repo and use jsdelivr to create the hyperlinks in a book. Although it adds baggage in that it is another repo and set of steps, it keeps things just as reliable as the book itself (when hosted on github) and jsdelivr allows us to not worry about rate limits and the URL structure makes it super easy to adapt back to the raw github links, if needed. Archiving versions/releases to Zenodo would be straightforward in that the external content could be included in the zip (though hyperlinks would break if the exernal content is removed---maybe that's something to think of for later).

TU Delft's FTP service would be the most obvious service to use for this. I only need to check request limits

I could not find documentation for it, but my guess is that this is only a good option for books that are also hosted on TU Delft servers, not on GitHub, because a) the rate limits will probably be restrictive, b) same TU server downtime as other TU platforms, and c) it's yet another platform to manage.

dumping files in 4TU research data

Feasible, but these are clunky to update.

dumping files in GitHub release assets

What is the advantage over committing a file? Is there a way to include a file as a release asset without being part of the git history?

Hugging face, although policies and functionalities are changing a lot

will look at it later. website a bit too overwhelming to figure it out in 15 seconds ;)

Cloud4research from TU Delft might be able to help setup custom Amazon AWS / Microsoft Azure services if we want something custom

We met AWS people at SURF edu days and I have contact info for our CEG Cloud4research person. I was asking them about jupyter hub or external server applications (think like the ewatercycle case where a book jumps to a cloud compute server that is set up on demand with some sort of credential). Let me know if you want me to start a conversation there (but I don't see it being feasible for hosting assets).

[Tom-van-Woudenberg]

does the same procedure for waiving copyright of TU Delft to research software apply to software created for education purposes?

I'd say the teachbooks tooling is neither research or education, it's just software whih can be used for both of those purposes (communication). The DCC people suggested this to be applicable too, so I don't see any problems

On FTP:

I could not find documentation for it, but my guess is that this is only a good option for books that are also hosted on TU Delft servers not on GitHub

where the book is hosted is an independent question from where the big files are hosted

a) the rate limits will probably be restrictive

I asked for those rate limits. But a jsdelivr kinda CDN service would provide a similar experience as when those source files are somewhere else right?

c) it's yet another platform to manage.

True, but it seems to be a platform which is meant for this use, while most other options seem a bit 'hacky'

On GH release:

What is the advantage over committing a file? Is there a way to include a file as a release asset without being part of the git history?

Dumping the file in a release is similar to dumping the file on any server. There's no version control so it's not part of the git history. However, this option feels a bit hacky to abuse releases to dump files somewhere.

On Cloud4research

but I don't see it being feasible for hosting assets

The DCC people told be Cloud4Research could help us set up hosting (one of the services of AWS / Azure), although it'd effectively mean we're creating our own service like the FTP-service. So unless rate limits are a big issue for the FTP-service, this seems an option which only means more setup-work.

[Tom-van-Woudenberg]

Mail sent by DCC:

The problem:

• GitHub pages has a size limit of 1GB
• For ease of development, all branches are made available through GitHub pages
• The size of the build artifact is 350 mb, but with multiple branches exceeds the size limit.

Solution:
Avoid embedding of the binary data (images), but reference the files instead.
Option 1: Request an account for the TU Delft ftp-server through the Self Service Portal: FTP-account. For more information, check out the manual. We recommend verifying with the ICT service if this intended use is supported (bandwidth, number of concurrent requests).
Option 2: For a solution that does not depend on TU Delft services, we recommend looking into cloud-based storage solutions from Amazon Web Services. For more information, you can reach out to Cloud4Research at https://tu-delft-ict-innovation.github.io/Cloud4Research/. Their terms specifically mention: “Acceptable use of the service is all use in a scientific or educational research context for research that is (co-) owned by TU Delft. So using it for research projects that TU Delft executes, or in (Master) courses for providing resources to students is the primary intended use of the service.”

Other recommendations:
We discussed licensing of software under an open-source software license. TU Delft created the Guidelines on Research Software: Licensing, Registration and Commercialization, outlining the steps for applying a pre-approved license.
We also recommend adding additional information to the repository:

• CONTRIBUTING guidelines – these guidelines outline how other (external) developers can contribute to your repository. See GitHub docs
• CITATION.cff file: a plain text file with human- and machine-readable citation information for software (and datasets). See General information and GitHub docs for more information.
• A Code of Conduct – outline your community standards: GitHub docs
• WAIVER – states that ‘Technische Universiteit Delft hereby disclaims all copyright interest in the program “Name program” (one line description of the content or function) written by the Author(s). [Name Dean], Dean of [Name Faculty]’

Please find here a Github repo template created by one of our DCC colleagues that helps implementing these steps.
If you have any other code or data-related questions, feel free to book another consultation with us.
Best regards,
On behalf of the DCC,
Maurits and João

[Tom-van-Woudenberg]

@timonidema, opinions from your side here?