Handbook love

A handbook release matching (delayed) the release of DataLad v0.18.0.
This release contains notable changes resulting from a Handbook hackathon in December 2022. It also increases the required version of DataLad to 0.17.

With thanks to many old and new contributors: @mih @christian-monch @asmacdo @mslw @jsheunis @Remi-Gau @yarikoptic, happy Valentines day, and happy International Love Data Week!

Better late than never

Handbook release matching the release of DataLad v0.17.0, which was for the most part only an internal change to a different testing framework (pytest instead of nose), and thus delayed on the handbook side.

It includes contributions the new contributors @Arshitha, @ajveltri22, @complexbrains, @jkosciessa, @candleindark, and @asmacdo, as well as contributions from established contributors @mslw, @sappelhoff, @kimsin98, @adswa, @yarikoptic.

Thank you to everyone contributing ❤️

Spring cleaning!

Handbook release matching the release of DataLad v0.16.0.
Among the most notable changes, this release contains adjustments to API changes introduced in DataLad 0.16 as well as documentation on newly introduced commands and features. In addition, it adds additional content on contributing, writing DataLad extensions, and publishing to s3 buckets.

The changelog below is autogenerated. Please see this repos CHANGELOG for an executive summary.

What's Changed

• docs: add mslw as a contributor for review by @allcontributors in #763
• Refactor the chapter on publishing by @adswa in #762
• Refactor the chapter on publishing by @adswa in #761
• BF: Fix link by @adswa in #764
• [ENH] replace issue template with the new "form" format by @Remi-Gau in #768
• docs: add Remi-Gau as a contributor for infra by @allcontributors in #769
• [FIX] update css unordered list in important note when viewing by @Remi-Gau in #770
• docs: add Remi-Gau as a contributor for code, design by @allcontributors in #772
• Fix two minor typos by @eort in #775
• add author to zenodo by @eort in #777
• docs: add eort as a contributor for bug by @allcontributors in #776
• CI: Fix readthedocs build by @adswa in #774
• Fix typo in section Advanced 1.1 by @eort in #778
• Fix a typo in a command by @mslw in #780
• docs: add mslw as a contributor for bug by @allcontributors in #781
• Computing on clusters: Update chapters with a link to the fairly big workflow by @adswa in #783
• Add a note on how to currently glob for several levels of directories by @adswa in #785
• docs: add AKSoo as a contributor for review by @allcontributors in #786
• New chapter on contributing to datalad by @adswa in #782
• Install: recommend git annex filter-process on windows installations by @adswa in #791
• Maintenance - Speed up sections with imagenette dataset by @adswa in #790
• Mention new recursive glob option in chapter on run by @adswa in #792
• Update Gin as data source workflow by @mslw in #793
• docs: add mslw as a contributor for content by @allcontributors in #795
• Add a section on good practices for creating interoperable file names by @adswa in #794
• Update filenaming.rst - minor typo and url fixes by @mslw in #796
• docs: add Myrskyta as a contributor for bug by @allcontributors in #798
• BF: Makefile - replace leading spaces with tabs by @yarikoptic in #799
• fix typo by @tguiot in #801
• docs: add tguiot as a contributor for bug by @allcontributors in #802
• Workflow: Update package index by @adswa in #804
• Link fixes for failing link check by @adswa in #803
• Revert "Links: Remove link to old benchmarks on previous website" by @adswa in #807
• [ENH] Add new 'Create your own extension' section by @jsheunis in #812
• New content: Mention credential integration with Git by @adswa in #814
• Fix/clarify passing git init options though create to change the default branch by @adswa in #815
• DGPA workshop contents by @adswa in #816
• fix wrong figure name by @jhpb7 in #820
• docs: add jhpb7 as a contributor for bug by @allcontributors in #823
• [ENH] Fix various typos by @jsheunis in #809
• Update dgpa.rst by @eltociear in #825
• docs: add eltociear as a contributor for bug by @allcontributors in #826
• Release-fixes for the chapter on RIA stores by @adswa in #810
• Remove version pinning of Sphinx by @adswa in #827

New Contributors

• @eort made their first contribution in #775
• @mslw made their first contribution in #780
• @tguiot made their first contribution in #801
• @jhpb7 made their first contribution in #820
• @eltociear made their first contribution in #825

Full Changelog: v0.15...v0.16

LaTeX improvements

This release comes 2 months delayed, but belongs to the DataLad 0.15.0 release from September 2021.
The changelog is autogenerated, and hopefully does not contain too many errors.
The majority of work was maintenance, and improvements of the LaTeX backbone of the handbook (huge thanks to @mih for this), rendering the PDF version of the handbook much more beautiful and towards an actually printable book.

What's Changed

• riastores: Avoid deprecated --no-storage-sibling by @kyleam in #641
• Minor changes re code-blocks in PDF by @mih in #647
• Shorten PDF for printed book by @adswa in #648
• Even more adjustments for a printed book by @adswa in #650
• docs: add Remi-Gau as a contributor by @allcontributors in #653
• Python installation instructions by @adswa in #651
• Help: Note about asyncio errors and workaround for Jupyter by @adswa in #646
• More fancy boxes in PDF by @mih in #655
• Further streamlining boxes by @mih in #656
• Numbered, listable and referencable boxes by @mih in #657
• Optional float mode for PDF boxes by @mih in #658
• No vertical lines in a table by @mih in #659
• Bettertables by @mih in #660
• Relax positioning of some boxes, where appropriate by @adswa in #661
• New boxtype importantnote by @mih in #662
• Let figures and tables float. by @mih in #663
• Minor change to align title page with print cover by @mih in #665
• Logo update by @mih in #667
• Disentangle too long and nested Findoutmore's by @adswa in #666
• note -> importantnote by @mih in #664
• Tune footnotes by @mih in #669
• TOC fixes by @mih in #670
• Prevent multiple definition of useless labels by @mih in #671
• New cover art for the Intro part by @mih in #672
• Install: Rework macOS installation by @adswa in #675
• docs: add tamaracha as a contributor by @allcontributors in #676
• In-text references of find-out-mores by @mih in #678
• windowsworkarounds -> windows-wit by @mih in #679
• Adjustments for a printed PDF that work also in html by @adswa in #680
• Refurbish installation instructions by @mih in #682
• BF: Fix a mixed up redirection couple by @adswa in #683
• Beautify the PDF further by @mih in #684
• Make term references accessible in the printed handbook by @mih in #685
• Small update of the extension list by @mih in #686
• Remove quotes from config file displays by @bpoldrack in #689
• On cloning openneuro/openfmri data from the datalad superdataset by @adswa in #690
• Print adjustment chapter 3 onwards by @adswa in #691
• Remove forgotten Windows-workarounds by @adswa in #692
• Add a rough sketch of a DataLad intro for Repronim workshop in Yale by @adswa in #693
• Make the index more comprehensive by @mih in #694
• Add redirector for ?cheatsheet by @yarikoptic in #701
• Adds a Github-actions based CI setup by @adswa in #703
• add redirects for chapters mentioned in ukb manuscript by @adswa in #702
• Another sweep for printing by @adswa in #704
• Fix typo by @mih in #705
• Fix ref tag to close #709 by @jsheunis in #710
• docs: add jsheunis as a contributor by @allcontributors in #711
• GIN: On using Gin as an autoenabled special remote by @adswa in #707
• Mention --assume-ready of datalad-run by @adswa in #699
• ENH: Relative links in sidebar by @effigies in #718
• ENH: Use Zenodo record that resolves to latest version by @effigies in #717
• Add FAQ on GitHub displaying git-annex as the default by @adswa in #722
• docs: add jsheunis as a contributor for content, example by @allcontributors in #726
• Windows weird behavior with file extensions by @adswa in #725
• Mention --dry-run by @adswa in #724
• help: On line endings and autocrfl true configurations for windows users by @adswa in #723
• Add walkthrough for using S3 as a special remote by @jsheunis in #721
• docs: add cni-md as a contributor for bug by @allcontributors in #728
• Fix left-over exit code by @adswa in #727
• Fix inconsistent description of push behavior by @Aksoo in #734
• docs: add AKSoo as a contributor for bug, content by @allcontributors in #735
• Fix typos using codespell by @sappelhoff in #732
• docs: add sappelhoff as a contributor for bug by @allcontributors in #737
• [ENH] add sphinx-copybutton extension by @sappelhoff in #615
• add Stephan as contributor to zenodo file by @jsheunis in #739
• Fix errors in 9.3 back and forth in time by @Aksoo in #740
• MNT: Update outputs to 0.14.4 by @adswa in #742
• Add a monthly cron job to check links in the handbook by @adswa in #743
• Fix broken or permanently redirected links, as discovered by Sphinx's make linkcheck by @adswa in #744
• Update 101-115-symlinks.rst by @lilikapa in #747
• "ultimatively" => "ultimately" by @arokem in #749
• Fixes typo by @arokem in #750
• This is the name of the file that was created earlier on. by @arokem in #751
• "temper" => "tamper" by @arokem in #752
• FIX: GIN siblings - fix typo in code and note that --url should not end with .git by @oesteban in #753
• MAINT: Add @oesteban to zenodo record by @oesteban in #755
• docs: add oesteban as a contributor for data, data, data, bug by @allcontributors in #754
• MNT: Fix link to git-annex docs on GitLab, mention deprecation by @adswa in #756
• Fixlinks by @adswa in #757
• Numbered boxes and references by @mih in #759

New Contributors

• @jsheunis made their first contribution in #710
• @Aksoo made their first contribution in #734
• @oesteban made their first contribution in #753

Full Changelog: v.14.0...v0.15

We're getting usable on Windows!

Handbook release matching the release of DataLad v0.14.0.
Like the software release, this handbook release improves the situation on/for Windows systems starkly from what we had before.
With contributions from Tristan Glatard, Ariel Rokem, Remi Gau, Surya Teja Togaru, Judith Bomba, Konrad Hinsen, Wu Jianxiao, Małgorzata Wierzba, Stefan Appelhoff, and Michael Joseph -- thank you!

Refactoring or deletions

• Overhaul Windows installation instructions #588
• Adjustments for GitHub's user-password deprecation #626, #592

Additions

Introduction

• git-annex installations with custom built git-annex with MagicMime support #603
• A quick-start guide for OpenNeuro #585

Basics

• Disambiguation on configurations #627 with thanks to John Lee for the issue at datalad
• A new section on how to debug and troubleshoot problems - with thanks to Tristan Glatard for the idea and contributions #538

Advanced

• A chapter on large-scale fair processing with parallel datalad-run calls #591
• A new section on configuring subdataset clone candidates and their priority #548
• A new chapter/section that compares the tool DVC to DataLad #569

Usecases

• Addition of a machine-learning application with DataLad #581
• Addition on Human Connectome Project (HCP) AWS credentials (thanks to Michael Joseph) #622
• Addition of a hands-on tutorial for reproducible papers #608, with thanks to Małgorzata Wierzba for feedback and contributions

Miscellaneous additions

• A variety of code lists and introductions #630, #613
• A few new permalinks: git-lfs #624, MPIB intro #614
• A new expandable section "Windows workaround" for Windows-specific notes and explanations #532
• Large amount of Windows adjustments in the Basics #588
• FAQs on copying locked files out of datasets, and on caveats with the BIDS validator - with thanks to Remi Gau #570, #562
• The handbook's GitHub repository received a welcome bot (with thanks to The Turing Way project for CC-BY illustrations), and a "Discussions" Forum
• The handbook's frontpage links to the cheat sheet with a nice illustration #578

DataLad Handbook v0.13

For zenodo archival

0.13 Release for DataLad 0.13

This handbook release matches today's DataLad 0.13 release.

With contributions from Dorian Pustina, Sarah Oliveira, Tristan Glatard,
Hamzah Hamid Baagil, Giulia Ippoliti, Yaroslav Halchenko, Alex Waite,
Dorien Huijser, and Michael Hanke -- thank you!

Refactoring or deletions

• RF: Replace datalad publish with datalad push #412
• RF: The Basics part was split into a Basics and Advanced part #450. The
  chapters "Advanced Options"
  and "Go big or go home"
  have been moved/added there.
• Installation instructions for Windows subsystem for linux have been removed #397

Additions

Introduction

• Installation instructions for rpm-based Linux distributions were added #435
• A "user-type" overview now serves as a guide through the handbook #403

Basics

• A stand-alone section
  on datalad push summarizes all previous publishing-related information #417
• A section for collecting gists (nifty code snippets for various tasks) is added to the
  chapter on help#445
• datalad drop is introduced in the first chapter #463
• Gin's new feature of anonymous read-only access to datasets is
  now mentioned in the chapter on using
  third party infrastructure #456
• The section on getting help
  started to collect and explain common warnings and error messages #418

Advanced

• A new chapter on scaling up with DataLad was added #414
• A section on configuring custom data access was added to the chapter
  "Advanced Options"#440
• The extension overview
  has been updated to a complete overview #477

Usecases

• A new Usecase
  Scaling Up: Managing 80TB and 15 Million files from the HCP release
  was added #225
• Giulia Ippoliti contributed the Usecase
  Using Globus as a data store for the Canadian Open Neuroscience Portal
  (opened in #421, merged as #479

Miscellaneous additions

• Introduction of a system to improve intersphinx linkage between the handbook
  and the technical docs & docstrings of DataLad #377
• Various improvements to the PDF version of the handbook #367
• Major toctree restructuring: Chapter-wise toctrees #367, robustified URLs #457
• Addition of short, README-ready explanations of DataLad datasets for published projects #370
• Redirections are now possible, using a ?<label> element after handbook.datalad.org/r.html #518
• (Almost) complete correspondence between HTML and PDF part, chapter, and section labeling #500

Zenodo archival of v0.12.0

This marks the ZENODO archival of The DataLad Handbook v0.12.0

v0.12 (Jan. 11 2019) -- A good weekend read

Beta stage release matching the release of datalad v0.12.0.

Refactoring or deletions

• RF: Replace datalad install with datalad clone (#326)

Additions

Introduction

• High-level, one page description "What you really need to know" about DataLad
  (#295)

Basics

• The DataLad Cheatsheet (#157)

• Chapter "One step further" with content on advanced dataset nesting (#226)
  and computational reproducibility with the datalad-containers extension (#242)

• Chapter "Further options" with content on DataLad's result hooks (#304),
  an overview on DataLad's extensions (#242), and how to keep clean datasets despite
  untracked contents (#84)

• Chapter "Third party infrastructure" on how to use various hosting services to
  share DataLad datasets, with concrete demonstrations/step-by-step instructions
  of sharing via Dropbox and GIN (#111)

• Section "Frequently Asked Questions" (#239)

• Section "Back and forth in time" on interacting with dataset history with
  Git tools/commands (#106)

• Section "YODA-compliant data analysis project" with an example data science
  project (including Python API) (#226)

• Include datalad download-url in first chapter to emphasize provenance capture
  abilities of DataLad (#294)

Usecases

• Use case "An automatically reproducible analysis of public neuroimaging data" (#205)

• Use case "Building a scalable data storage for scientific computing" (#223)

Miscellaneous additions

• Adjust contents to autorunrecord update to record a flexible
  set of code snippets in "casts" for live demonstrations. Add cast associations
  for existing contents with speakernotes (#219)

• Additional book segment "Code lists from chapters" with code lists used for
  workshops (#273)

• Illustrations from undraw.co (starting in #329)

• Tagged "showroom" repositories with branches reflecting dataset states at different
  book sections (#341)