Bugs in A_mul_Bc for CholmodSparse

[acroy]

I just noticed some issues with A_mul_Bc and Ac_mul_B in CHOLMOD. If

A = sprand(10, 10, 0.1);
cmA = Base.LinAlg.CHOLMOD.CholmodSparse(A)

B = sprand(10, 10, 0.1);
cmB = Base.LinAlg.CHOLMOD.CholmodSparse(B)

then cmA*cmB' and cmA'*cmB segfault. The reason is that the ccall of cholmod_transpose in
cholmod.jl#L507-L510 and cholmod.jl#L540-L543 misses the second argument which should be 1 in this case (see code for transpose).

There is also a return missing in cholmod.jl#L500.

Another issue arises if A is symmetric. Then only the upper part is stored, but cholmod_aat needs both parts for some reason. Although the error seems to be detected, there is still a segfault.

julia> A = sprand(10, 10, 0.1); A = 0.5*(A + A');
julia> cmA = Base.LinAlg.CHOLMOD.CholmodSparse(A)
CHOLMOD sparse:  :  10-by-10, nz 7, upper.
  nzmax 7, sorted, packed, 
  scalar types: SuiteSparse_long, real, double
  col 0: nz 0 start 0 end 0:
  col 1: nz 0 start 0 end 0:
  col 2: nz 0 start 0 end 0:
  col 3: nz 0 start 0 end 0:
  col 4: nz 0 start 0 end 0:
  col 5: nz 0 start 0 end 0:
  col 6: nz 2 start 0 end 2:
         2: 0.11708
    ...
  col 9: nz 1 start 6 end 7:
         8: 0.3815
  nnz on diagonal: 0
  OK

julia> cmA*cmA'
CHOLMOD error: matrix cannot be symmetric
CHOLMOD error: argument missing

signal (11): Segmentation fault
A_mul_Bc at linalg/cholmod.jl:520

[tkelman]

Closed by #9189?