Complete rewrite of xan join

Fix #198

Cargo.toml

@@ -44,7 +44,6 @@ arrayvec = "0.7.4"
 base64 = "0.22.1"
 bstr = "1.10.0"
 btoi = "0.4.3"
-byteorder = "1"
 bytesize = "1.3.0"
 calamine = "0.24.0"
 colored = "2.0.0"

docs/cmd/join.md

@@ -4,59 +4,74 @@
 ```txt
 Join two sets of CSV data on the specified columns.
 
-The default join operation is an 'inner' join. This corresponds to the
-intersection of rows on the keys specified.
+The default join operation is an "inner" join. This corresponds to the
+intersection of rows on the keys specified. The command is also able to
+perform a left outer join with --left, a right outer join with --right,
+a full outer join with --full and finally a cartesian product/cross join
+with --cross.
 
 By default, joins are done case sensitively, but this can be disabled using
-the --ignore-case flag.
+the -i, --ignore-case flag.
 
 The column arguments specify the columns to join for each input. Columns can
-be selected using the same syntax as the 'xan select' command. Both selections
-must return a same number of columns in proper order.
+be selected using the same syntax as the "xan select" command. Both selections
+must return a same number of columns, for the join keys to be properly aligned.
 
 Note that this command is able to consume streams such as stdin (in which case
-the file name must be '-' to indicate which file will be read from stdin) and
-gzipped files out of the box, but be aware that those file will be entirely
-buffered into memory so the join operation can be done.
+the file name must be "-" to indicate which file will be read from stdin) and
+gzipped files out of the box.
 
-Note that when performing an 'inner' join (the default), it's the second file that
-will be indexed into memory. And when performing an 'outer' join, it will be the file
-that is on the other side of --left/--right.
+# Memory considerations
+
+    - `inner join`: the command does not try to be clever and
+                    always indexes the left file, while the right
+                    file is streamed. Prefer placing the smaller file
+                    on the left.
+    - `left join`:  the command always indexes the right file and streams
+                    the left file.
+    - `right join`: the command always indexes the left file and streams
+                    the right file.
+    - `full join`:  the command does not try to be clever and
+                    always indexes the left file, while the right
+                    file is streamed. Prefer placing the smaller file
+                    on the left.
+    - `cross join`: the command does not try to be clever and
+                    always indexes the left file, while the right
+                    file is streamed. Prefer placing the smaller file
+                    on the left.
 
 Usage:
     xan join [options] <columns1> <input1> <columns2> <input2>
+    xan join [options] --cross <input1> <input2>
     xan join --help
 
 join options:
-    -i, --ignore-case           When set, joins are done case insensitively.
-    --left                      Do a 'left outer' join. This returns all rows in
-                                first CSV data set, including rows with no
-                                corresponding row in the second data set. When no
-                                corresponding row exists, it is padded out with
-                                empty fields.
-    --right                     Do a 'right outer' join. This returns all rows in
-                                second CSV data set, including rows with no
-                                corresponding row in the first data set. When no
-                                corresponding row exists, it is padded out with
-                                empty fields. (This is the reverse of 'outer left'.)
-    --full                      Do a 'full outer' join. This returns all rows in
-                                both data sets with matching records joined. If
-                                there is no match, the missing side will be padded
-                                out with empty fields. (This is the combination of
-                                'outer left' and 'outer right'.)
-    --cross                     USE WITH CAUTION.
-                                This returns the cartesian product of the CSV
-                                data sets given. The number of rows return is
-                                equal to N * M, where N and M correspond to the
-                                number of rows in the given data sets, respectively.
-    --nulls                     When set, joins will work on empty fields.
-                                Otherwise, empty fields are completely ignored.
-                                (In fact, any row that has an empty field in the
-                                key specified is ignored.)
-    --prefix-left <prefix>      Add a prefix to the names of the columns in the
-                                first dataset.
-    --prefix-right <prefix>     Add a prefix to the names of the columns in the
-                                second dataset.
+    --left                       Do an "outer left" join. This returns all rows in
+                                 first CSV data set, including rows with no
+                                 corresponding row in the second data set. When no
+                                 corresponding row exists, it is padded out with
+                                 empty fields. This is the reverse of --right.
+    --right                      Do an "outer right" join. This returns all rows in
+                                 second CSV data set, including rows with no
+                                 corresponding row in the first data set. When no
+                                 corresponding row exists, it is padded out with
+                                 empty fields. This is the reverse of --left.
+    --full                       Do a "full outer" join. This returns all rows in
+                                 both data sets with matching records joined. If
+                                 there is no match, the missing side will be padded
+                                 out with empty fields.
+    --cross                      This returns the cartesian product of the given CSV
+                                 files. The number of rows emitted will be equal to N * M,
+                                 where N and M correspond to the number of rows in the given
+                                 data sets, respectively.
+    -i, --ignore-case            When set, joins are done case insensitively.
+    --nulls                      When set, joins will work on empty fields.
+                                 Otherwise, empty keys are completely ignored, i.e. when
+                                 column selection yield only empty cells.
+    -L, --prefix-left <prefix>   Add a prefix to the names of the columns in the
+                                 first dataset.
+    -R, --prefix-right <prefix>  Add a prefix to the names of the columns in the
+                                 second dataset.
 
 Common options:
     -h, --help                  Display this message

docs/cmd/regex-join.md

@@ -36,20 +36,20 @@ Usage:
     xan regex-join --help
 
 join options:
-    -i, --ignore-case           Make the regex patterns case-insensitive.
-    --left                      Write every row from the first file in the output, with empty
-                                padding cells when no regex pattern from the second file
-                                produced a match.
-    -p, --parallel              Whether to use parallelization to speed up computations.
-                                Will automatically select a suitable number of threads to use
-                                based on your number of cores. Use -t, --threads if you want to
-                                indicate the number of threads yourself.
-    -t, --threads <threads>     Parellize computations using this many threads. Use -p, --parallel
-                                if you want the number of threads to be automatically chosen instead.
-    --prefix-left <prefix>      Add a prefix to the names of the columns in the
-                                searched file.
-    --prefix-right <prefix>     Add a prefix to the names of the columns in the
-                                patterns file.
+    -i, --ignore-case            Make the regex patterns case-insensitive.
+    --left                       Write every row from the first file in the output, with empty
+                                 padding cells when no regex pattern from the second file
+                                 produced a match.
+    -p, --parallel               Whether to use parallelization to speed up computations.
+                                 Will automatically select a suitable number of threads to use
+                                 based on your number of cores. Use -t, --threads if you want to
+                                 indicate the number of threads yourself.
+    -t, --threads <threads>      Parellize computations using this many threads. Use -p, --parallel
+                                 if you want the number of threads to be automatically chosen instead.
+    -L, --prefix-left <prefix>   Add a prefix to the names of the columns in the
+                                 searched file.
+    -R, --prefix-right <prefix>  Add a prefix to the names of the columns in the
+                                 patterns file.
 
 Common options:
     -h, --help                  Display this message

docs/xanzines/8_2025_feb.md

@@ -25,4 +25,5 @@ cargo install xan
 - `xan bins --label`
 - `xan fmt --tabs` & `xan fmt --in-place`
 - Overhauling scales accross commands (`xan plot`, `xan bins --nice`, `xan heatmap`, `xan hist`)
-- `xan slice --skip` alias
+- `xan slice --skip` alias
+- Complete rewrite of `xan join` for better streaming performance.