extension: Human Patch Format (HPF); use cases: customizing programs; applying semantic patches

[timotheecour]

this is another idea i've been curious about for a while; seems related to HAF (at least at implementation + API + use case level)

motivating use case 1: customizing programs

customizing programs via command line arguments (eg via getopt) often need a lot of boilerplate and deep changes to pass these options from the top-level main all the way down to the place where these options are actually used (say in some function).

Wouldn't it be nice if we could just specify on the command line the exact thing we want to change in the source code (which may not have an exposed cmd line option yet)?

This would be quite useful during debugging, development and interactive programming, eg allowing to run a program under a number of different conditions (maybe even in parallel). The alternatives are either inserting cmd line options for what we want to customize (requires adding code which may end up not being useful in the end) or a series of manual git checkout -b experiment ; hack; git checkout master; which is hard to automate.

idea

compile a program with small differences to a set of source files without actually editing the source code; the user specifies a diff (in HPF), the diff is applied on a temp directory and the compiler compiles the modified files.

motivating use case 2: applying semantic patches

standard diff is rigid by design: the patch will fail if the context (including line numbers) don't match what's specified in the diff (by design). HPF allows one to apply a diff even after the source changed a bit; this makes it useful in certain scenarios (eg sharing a change in different branches).
HPF also allows more flexible changes than standard diff, eg: capitalize all occurrences of the word foo\wbar. This is useful for applying refactorings.

the Human Patch Format (HPF)

this is meant as less rigid (no surrounding context is shown, along with line numbers) and more human readable/writable than the standard diff format; it's specified as a set of sed regex replace rules to be applied to a given scope (file, module, function etc).
Unlike standard diff, the HPF is likely to remain valid when surrounding source code changes (eg if lines are added above or below); of course not guaranteed (eg if a line is added that would be affected by a sed statement)

eg of HPF:

--- main.d
# these sed statements are applied in sequence to main.d
s/foo/bar/
#more complex sed example
s/foo\w+/bar2/g
#arbitrary programs can be applied, eg `dfmt`, `clang-format`, `highlight`
@apply dfmt

--- foo.bar
# this block applies to file whose module is `foo.bar` (or function(s) bar in module foo)

example 1

// main.d
void main(){ 10.iota.each(writeln("hello"); }

dmd -run main.d
$ prints hello 10 times
echo "--- main; s/10/100/" | dmd -diff=- -run main.d
$ prints hello 100 times

example 2

cat << EOF | tee dmd -diff=- main.d
--- main # applies a diff on module `main`
s/cols = 14/cols = 100/
s/take(20)/take(1)/

--- foo.util.process # limits scope of edit to function(s) with that name in module `main`
s/return 3/return 1/

-- std.file.dirEntries # also works if in phobos
s/bool followSymlink = true / bool followSymlink = false/
EOF

with these files (adapted from dlang.org)

// file foo/util.d
module foo.util;
auto process(T)(int cols, T chunks){
  return 3 * (cols - chunk.length);
}

// file main.d
module main.d;
void main() {
    import std.algorithm, std.stdio, std.file, std.range;
    enum cols = 14;
    // Split file into 14-byte chunks per row
    thisExePath.File("rb").byChunk(cols).take(20).each!(chunk =>
        // Use range formatting to format the
        // hexadecimal part and align the text part
        writefln!"%(%02X %)%*s  %s"(
            chunk,
            process(cols, chunk), "", // Padding
            chunk.map!(c => // Replace non-printable
                c < 0x20 || c > 0x7E ? '.' : char(c))));
}

implementation

every module affected by a diff is resolved to a file, the file is copied to a temp dir, the diff is applied there, and these files are added to command line so they're compiled in the binary (and override the symbols in the static/shared library)

limitations

• obviously this is not the same as a cmd line option, eg requries recompilation most often (unless what's diff'd is not source files but, say, input files); but with dmd's fast compilation, it's ok
• this isn't meant for replacing cmd line options (eg for production code) since the way the diff is specified may not work after the source code changes significantly
• for lots of changes, git checkout -b feature_branch is preferable
• HPF diff is not reversible unlike regular diff

links

semantic diff seems related, need to read more...
https://stackoverflow.com/questions/523307/semantic-diff-utilities
https://github.com/hoelzro/semantic-diff seems unrelated

API and cmd line

# applies a HPF diff to files specified in `mypatch.hdf`, and dumps them to specified output dir
hpf -diff=mypatch.hdf -od=/tmp/
# same but input files to be modifed are implicitly defined by a HPF
hpf -input-haf=myarchive.haf -diff=mypatch.hdf -od=/tmp/