Remove the GPL as an alternative licence

[isaac-io]

This PR removes the GPL as an alternative licence since we only license Speedb under the Apache 2.0 licence. This requires removing the bundled GNU Parallel as well because it requires having the licence text along for the ride, so we adjust the Makefile to depend on the system-installed binary instead and fall back to running tests serially if not found.

While at it, clean up the Makefile a bit and remove the unneeded dependency on Perl for things that can easily be done using POSIX tools.

Test Plan: run make check and other targets that use/depend on GNU Parallel.

[mrambacher]

Is gnu_parallel really just parallel or have there been modifications to it? I am a little leary of making the change without that confirmation.

For ease of merging later, I would also suggest splitting this into one or two additional PRs. One PR to replace the perl with sed; one to use $PARALLEL instead of gnu_parallel or parallell; and the one you are actually wanting to remove gnu_parallel. The first two are upstreamable and would make it easier for merging later.

[isaac-io]

Is gnu_parallel really just parallel or have there been modifications to it? I am a little leary of making the change without that confirmation.

build_tools/gnu_parallel is mostly GNU Parallel. This is the diff from the point that it was vendored, and it mostly deals with CI in a way that doesn't really matter to us once #103 is integrated. We can always introduce a more "native" solution without dependency on GNU Parallel being installed in the form of something like a slimmed down version of gtest-parallel which would make the CI issue moot and allow for upstreaming the removal of GNU Parallel as well.

diff --git a/build_tools/gnu_parallel b/build_tools/gnu_parallel
index abbf8f100..757b25f11 100755
--- a/build_tools/gnu_parallel
+++ b/build_tools/gnu_parallel
@@ -1561,6 +1561,7 @@ sub save_stdin_stdout_stderr {
 	::die_bug("Can't dup STDERR: $!");
     open $Global::original_stdin, "<&", "STDIN" or
 	::die_bug("Can't dup STDIN: $!");
+    $Global::is_terminal = (-t $Global::original_stderr) && !$ENV{'CIRCLECI'} && !$ENV{'TRAVIS'};
 }

 sub enough_file_handles {
@@ -1840,12 +1841,17 @@ sub start_another_job {
 }
 }

+$opt::min_progress_interval = 0;
+
 sub init_progress {
     # Uses:
     #   $opt::bar
     # Returns:
     #   list of computers for progress output
     $|=1;
+    if (not $Global::is_terminal) {
+      $opt::min_progress_interval = 30;
+    }
     if($opt::bar) {
 	return("","");
     }
@@ -1870,6 +1876,9 @@ sub drain_job_queue {
     }
     my $last_header="";
     my $sleep = 0.2;
+    my $last_left = 1000000000;
+    my $last_progress_time = 0;
+    my $ps_reported = 0;
     do {
         while($Global::total_running > 0) {
             debug($Global::total_running, "==", scalar
@@ -1880,14 +1889,38 @@ sub drain_job_queue {
 		    close $job->fh(0,"w");
 		}
 	    }
-            if($opt::progress) {
+            # When not connected to terminal, assume CI (e.g. CircleCI). In
+            # that case we want occasional progress output to prevent abort
+            # due to timeout with no output, but we also need to stop sending
+            # progress output if there has been no actual progress, so that
+            # the job can time out appropriately (CirecleCI: 10m) in case of
+            # a hung test. But without special output, it is extremely
+            # annoying to diagnose which test is hung, so we add that using
+            # `ps` below.
+            if($opt::progress and
+               ($Global::is_terminal or (time() - $last_progress_time) >= 30)) {
                 my %progress = progress();
                 if($last_header ne $progress{'header'}) {
                     print $Global::original_stderr "\n", $progress{'header'}, "\n";
                     $last_header = $progress{'header'};
                 }
-                print $Global::original_stderr "\r",$progress{'status'};
-		flush $Global::original_stderr;
+                if ($Global::is_terminal) {
+                    print $Global::original_stderr "\r",$progress{'status'};
+                }
+                if ($last_left > $Global::left) {
+                    if (not $Global::is_terminal) {
+                        print $Global::original_stderr $progress{'status'},"\n";
+                    }
+                    $last_progress_time = time();
+                    $ps_reported = 0;
+                } elsif (not $ps_reported and (time() - $last_progress_time) >= 60) {
+                    # No progress in at least 60 seconds: run ps
+                    print $Global::original_stderr "\n";
+                    system("ps", "-wf");
+                    $ps_reported = 1;
+                }
+                $last_left = $Global::left;
+                flush $Global::original_stderr;
             }
 	    if($Global::total_running < $Global::max_jobs_running
 	       and not $Global::JobQueue->empty()) {
@@ -1921,7 +1954,7 @@ sub drain_job_queue {
 	     not $Global::start_no_new_jobs and not $Global::JobQueue->empty());
     if($opt::progress) {
 	my %progress = progress();
-	print $Global::original_stderr "\r", $progress{'status'}, "\n";
+	print $Global::original_stderr $opt::progress_sep, $progress{'status'}, "\n";
 	flush $Global::original_stderr;
     }
 }
@@ -1954,10 +1987,11 @@ sub progress {
     my $eta = "";
     my ($status,$header)=("","");
     if($opt::eta) {
-	my($total, $completed, $left, $pctcomplete, $avgtime, $this_eta) =
-	    compute_eta();
-	$eta = sprintf("ETA: %ds Left: %d AVG: %.2fs  ",
-		       $this_eta, $left, $avgtime);
+      my($total, $completed, $left, $pctcomplete, $avgtime, $this_eta) =
+        compute_eta();
+      $eta = sprintf("ETA: %ds Left: %d AVG: %.2fs  ",
+        $this_eta, $left, $avgtime);
+      $Global::left = $left;
     }
     my $termcols = terminal_columns();
     my @workers = sort keys %Global::host;

For ease of merging later, I would also suggest splitting this into one or two additional PRs. One PR to replace the perl with sed; one to use $PARALLEL instead of gnu_parallel or parallell; and the one you are actually wanting to remove gnu_parallel. The first two are upstreamable and would make it easier for merging later.

I separated the changes in this PR into two commits exactly for this reason. The first commit only deals with cleanup of the Makefile and removal of the dependency on Perl, and the second one deals with the removal of the vendored GNU Parallel and the licence.

[mrambacher]

This generally looks good. Do the Circle or related jobs need to change to install parallel?

Can you post into the PR comments what happens on a machine in which parallel is not installed and if PARALLEL points to something off?

[mrambacher]

What happens here if PARALLEL is empty?

[isaac-io]

You'll get a runtime error, but this is OK because the valgrind_check_0 target is an internal target that's only called from valgrind_check if PARALLEL_OK is set to 1.

[mrambacher]

Isn't it OK for PARALLEL not to be set if J=1?

[isaac-io]

For the parloop and parallel_check (which depend on this new target) this wasn't OK. The original code did the following in that case:

echo "Need to have GNU Parallel and J > 1"; exit 1;	

However, with the latest changes this isn't relevant anymore, since those two targets were removed.

[mrambacher]

This seems off to me. Shouldn't we only care about PARALLEL_OK if J > 1 ? What if J is not set or == 1 does PARALLEL still need to be installed (and PARALLEL_OK == 1)?

[mrambacher]

Set PARALLEL or install GNU parallel?

[isaac-io]

This generally looks good. Do the Circle or related jobs need to change to install parallel?

We don't use CircleCI at the moment, and our GitHub Actions workflows already install GNU Parallel as part of the setup for jobs.

Can you post into the PR comments what happens on a machine in which parallel is not installed and if PARALLEL points to something off?

After the latest changes (pulling the upstream Makefile cleanup commit, which removes the targets that error out in case parallel isn't present) we won't get any error in either case, and the tests will simply run serially instead of in parallel (that's the behaviour of the original code). Here's how the first few lines of the output look on my machine when setting PARALLEL to something bogus (compilation and test output elided):

$ PARALLEL=foo make -j16 check
$DEBUG_LEVEL is 1
Makefile:163: Warning: Compiling in debug mode. Don't use the resulting binary in production
[...]
  GEN      check
===== Running cache_test (Sun 21 Aug 10:13:49 IDT 2022)
[==========] Running 47 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 46 tests from CacheTestInstance/CacheTest
[ RUN      ] CacheTestInstance/CacheTest.UsageTest/0
[       OK ] CacheTestInstance/CacheTest.UsageTest/0 (61 ms)
[...]
[----------] 46 tests from CacheTestInstance/CacheTest (798 ms total)

[----------] 1 test from CacheTestInstance/LRUCacheTest
[ RUN      ] CacheTestInstance/LRUCacheTest.SetStrictCapacityLimit/0
[       OK ] CacheTestInstance/LRUCacheTest.SetStrictCapacityLimit/0 (0 ms)
[----------] 1 test from CacheTestInstance/LRUCacheTest (0 ms total)

[----------] Global test environment tear-down
[==========] 47 tests from 2 test cases ran. (799 ms total)
[  PASSED  ] 47 tests.
===== Running cache_reservation_manager_test (Sun 21 Aug 10:13:50 IDT 2022)
[==========] Running 10 tests from 5 test cases.
[----------] Global test environment set-up.
[----------] 6 tests from CacheReservationManagerTest
[...]
[----------] 6 tests from CacheReservationManagerTest (2 ms total)

[----------] 1 test from CacheReservationManagerIncreaseReservcationOnFullCacheTest
[ RUN      ] CacheReservationManagerIncreaseReservcationOnFullCacheTest.IncreaseCacheReservationOnFullCache
[       OK ] CacheReservationManagerIncreaseReservcationOnFullCacheTest.IncreaseCacheReservationOnFullCache (0 ms)
[----------] 1 test from CacheReservationManagerIncreaseReservcationOnFullCacheTest (0 ms total)

[----------] 1 test from CacheReservationManagerWithDelayedDecreaseTest
[ RUN      ] CacheReservationManagerWithDelayedDecreaseTest.DecreaseCacheReservationWithDelayedDecrease
[       OK ] CacheReservationManagerWithDelayedDecreaseTest.DecreaseCacheReservationWithDelayedDecrease (0 ms)
[----------] 1 test from CacheReservationManagerWithDelayedDecreaseTest (0 ms total)

[----------] 1 test from CacheReservationManagerDestructorTest
[ RUN      ] CacheReservationManagerDestructorTest.ReleaseRemainingDummyEntriesOnDestruction
[       OK ] CacheReservationManagerDestructorTest.ReleaseRemainingDummyEntriesOnDestruction (0 ms)
[----------] 1 test from CacheReservationManagerDestructorTest (0 ms total)

[----------] 1 test from CacheReservationHandleTest
[ RUN      ] CacheReservationHandleTest.HandleTest
[       OK ] CacheReservationHandleTest.HandleTest (0 ms)
[----------] 1 test from CacheReservationHandleTest (0 ms total)

[----------] Global test environment tear-down
[==========] 10 tests from 5 test cases ran. (2 ms total)
[  PASSED  ] 10 tests.
===== Running lru_cache_test (Sun 21 Aug 10:13:50 IDT 2022)
[==========] Running 20 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from LRUCacheTest
[...]
[----------] 3 tests from LRUCacheTest (3 ms total)

[----------] 6 tests from LRUCacheSecondaryCacheTest
[...]
[----------] 6 tests from LRUCacheSecondaryCacheTest (3 ms total)

[----------] 11 tests from DBSecondaryCacheTest
[...]
[----------] 11 tests from DBSecondaryCacheTest (219 ms total)

[----------] Global test environment tear-down
[==========] 20 tests from 3 test cases ran. (225 ms total)
[  PASSED  ] 20 tests.
[...]

[mrambacher]

When running with the Make changes, I find that if PARALLEL is set to something that does not exist or if parallel itself cannot be found, there are no errors printed -- the tests just run serially. If this is the intent, the PR is good to go (please add a comment somewhere as such). If not, the PR can still be committed if the issue is raised/addressed in a separate issue/PR.

[isaac-io]

This is indeed the intended behaviour, as I noted in my previous comment. The original comment above the check rule in the Makefile also describes it as such:

If J != 1 and GNU parallel is installed, run the tests in parallel,
via the check_0 rule above. Otherwise, run them sequentially.