Teach FrattiniSubgroup methods to check for solvability

The generic method and the method using radicals both benefit from
checking solvability, and then redispatching (if solvability was not
known before, thus preventing an endless loop).

The point is that for the former, we are generic anyway, anything may be
either trivial to compute or impossible, so we might as well check for
solvability, which shouldn't be harder than computing all maximal
subgroups (plus, the latter is impossible for huge elementary abelian
groups, while the former is trivial).

The method using radicals automatically computes whether the group is
solvable anyway, so it may just as well use this and redispatch in that
case.

Resolves #710

lib/grp.gi

@@ -1241,6 +1241,9 @@ local m;
     if IsTrivial(G) then
       return G;
     fi;
+    if not HasIsSolvableGroup(G) and IsSolvableGroup(G) then
+       return FrattiniSubgroup(G);
+    fi;
     m := List(ConjugacyClassesMaximalSubgroups(G),C->Core(G,Representative(C)));
     m := Intersection(m);
     if HasIsFinite(G) and IsFinite(G) then

lib/maxsub.gi

@@ -336,13 +336,22 @@ InstallMethod( FrattiniSubgroup, "Using radical",
 [ IsGroup and CanComputeFittingFree ],0,
 function(G)
 local m,f,i;
+  i:=HasIsSolvableGroup(G); # remember if the group knew about its solvability
   if IsTrivial(G) then
     return G;
-  elif Size(RadicalGroup(G))=1 then
+  elif IsTrivial(RadicalGroup(G)) then
     return TrivialSubgroup(G);
   fi;
-  m:=MaximalSubgroupClassesSol(G);
   f:=RadicalGroup(G);
+
+  # computing the radical also determines if the group is solvable; if
+  # it is, and if solvability was not known before, redispatch, to give
+  # methods requiring solvability (e.g. for permutation groups) a chance.
+  if not i and IsSolvableGroup(G) then
+    return FrattiniSubgroup(G);
+  fi;
+
+  m:=MaximalSubgroupClassesSol(G);
   for i in [1..Length(m)] do
     if not IsSubset(m[i],f) then
       f:=Core(G,NormalIntersection(f,m[i]));

tst/testinstall/opers/FrattiniSubgroup.tst

@@ -72,5 +72,13 @@ true
 gap> HasIsNilpotentGroup(FrattiniSubgroup(p));
 true
 
+# test with a fp group which is small and solvable, which may be
+# discovered by GAP as it tries to compute the Frattini subgroup.
+# See issue #710.
+gap> F := FreeGroup("x", "y");; x := F.1;; y := F.2;;
+gap> G := F/[x*y*x^(-1)*y^(-1), x^30, (x*y)^70];;
+gap> FrattiniSubgroup(G);
+Group([  ])
+
 #
 gap> STOP_TEST("FrattiniSubgroup.tst", 1);