Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Computation of rank-decompositions in Sage #11754

Closed
nathanncohen mannequin opened this issue Aug 29, 2011 · 63 comments
Closed

Computation of rank-decompositions in Sage #11754

nathanncohen mannequin opened this issue Aug 29, 2011 · 63 comments

Comments

@nathanncohen
Copy link
Mannequin

nathanncohen mannequin commented Aug 29, 2011

Hello everybody !

This patch is an interface between Sage and the software RW, written by Philipp Klaus Krause [1]. It is written in C, and freshly licensed under the GPL2. It is a highly enumerative code that uses a lot of memory, but is still so far the best practical way to obtain optimal decompositions.

This patch creates the module sage.graphs.graph_decompositions and the rankwidth-related files. As it requires some documentation, some words are added on what a rank-decomposition is and where the code is coming from. Doing that, I also added sage.graphs.modular_decompositions to the documentation.

(As I created a module graph_decompositions, it would make sense to move te modular_decomposition there. It thought it best to do it in a distinct patch, this one being large enough already)

Oh. By the way, most of the patch actually contains the copy of the files written by Philipp Klaus Krause.

Nathann

http://pholia.tdi.informatik.uni-frankfurt.de/~philipp/software/rw.shtml

APPLY:

CC: @sagetrac-lsampaio @sagetrac-mvngu

Component: graph theory

Author: Nathann Cohen

Reviewer: David Coudert, Jeroen Demeyer

Merged: sage-5.0.beta6

Issue created by migration from https://trac.sagemath.org/ticket/11754

@nathanncohen nathanncohen mannequin added this to the sage-5.0 milestone Aug 29, 2011
@nathanncohen nathanncohen mannequin added the s: needs review label Aug 29, 2011
@dcoudert
Copy link
Contributor

dcoudert commented Jan 1, 2012

Reviewer: David Coudert

@dcoudert
Copy link
Contributor

dcoudert commented Jan 1, 2012

comment:2

Nathann,

I tried to install the patch with sage-4.8.alpha5 and it's not possible anymore.
Could you update the patch ?

Best,
D.

compote:/path-to-sage/sage-4.8.alpha5/devel/sage-myclone> hg qimport ~/Soft/sage-patchs/trac_11754.patchadding trac_11754.patch to series file
compote:/path-to-sage/sage-4.8.alpha5/devel/sage-myclone> hg qpush
applying trac_11754.patch
patching file doc/en/reference/graphs.rst
Hunk #1 FAILED at 46
1 out of 1 hunks FAILED -- saving rejects to file doc/en/reference/graphs.rst.rej
file sage/graphs/graph_decompositions/__init__.py already exists
1 out of 1 hunks FAILED -- saving rejects to file sage/graphs/graph_decompositions/__init__.py.rej
patching file setup.py
Hunk #1 FAILED at 929
1 out of 1 hunks FAILED -- saving rejects to file setup.py.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working dir
errors during apply, please fix and refresh trac_11754.patch
compote:/path-to-sage/sage-4.8.alpha5/devel/sage-myclone> 

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Jan 1, 2012

comment:3

I thought it would be much worse ! Just obvious conflicts with the vertex_separation stuff... Patch updated ! :-)

Nathann

@dcoudert
Copy link
Contributor

dcoudert commented Jan 2, 2012

comment:4

I can now install the patch correctly.
It works on my computer for graphs with 20 nodes and seems OK for 25 nodes (I stopped computation since its quite long), but not for graphs with 30 nodes (see error message bellow). I assume I don't have enough memory on my computer, but the error message is not explicit enough.
Is it possible to test if enough memory is available to run the algorithm as you did for vertex separation and pathwidth ?
Thanks,
David.

sage: G = graphs.RandomGNM(30,200)
sage: x,H = G.rank_decomposition()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)

/path-to-sage/sage-4.8.alpha5/devel/sage-myclone/<ipython console> in <module>()

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph.pyc in rank_decomposition(self, verbose)
   2999 
   3000         from sage.graphs.graph_decompositions.rankwidth import rank_decomposition
-> 3001         return rank_decomposition(self, verbose = verbose)
   3002 
   3003     ### Matching

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph_decompositions/rankwidth.so in sage.graphs.graph_decompositions.rankwidth.rank_decomposition (sage/graphs/graph_decompositions/rankwidth.c:815)()

RuntimeError: Segmentation fault

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Jan 2, 2012

comment:5

HMmmmmmm.... In the part of the code I wrote the memory allocations are checked, and it would be bad if it came from the original source code. We try not to put our hands in this kind of code, in order to avoid any troubles when the original source codes get updated (we would have to do all our modifications again on the updated version). What we usually do in these cases is report the bug upstream and wait for them to fix it.

The thing is that I was not able to reproduce your bug on my computer. It begins to eat a lot of memory but no segfault, so I do not really know how to debug it ^^;

Are you testing this code on your mac ? I hope it is not one of those integer initializations again :-P

Nathann

@dcoudert
Copy link
Contributor

dcoudert commented Jan 2, 2012

comment:6

The test has been done on a standard PC (32 bits) with 4G of RAM and running linux. Nothing special. The error is almost instantaneous.

Some simple tests.

sage: G = graphs.PetersenGraph()
sage: G.rank_decomposition()
(3, Graph on 19 vertices)
sage: G = graphs.RandomGNM(20,200)
sage: G.rank_decomposition()
(1, Graph on 39 vertices)

Testing with 32 nodes => the patch is working for 31 vertices only !

sage: G = graphs.RandomGNM(32,200)
sage: G.connected_components_number()
1
sage: G.rank_decomposition()
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)

/path-to-sage/sage-4.8.alpha5/devel/sage-myclone/<ipython console> in <module>()

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph.pyc in rank_decomposition(self, verbose)
   2999 
   3000         from sage.graphs.graph_decompositions.rankwidth import rank_decomposition
-> 3001         return rank_decomposition(self, verbose = verbose)
   3002 
   3003     ### Matching

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph_decompositions/rankwidth.so in sage.graphs.graph_decompositions.rankwidth.rank_decomposition (sage/graphs/graph_decompositions/rankwidth.c:677)()

Exception: The rank decomposition cannot be computed on graphs of more than 32 vertices.

The patch contains some test regarding memory requirement

sage: G = graphs.RandomGNM(29,200)
sage: G.connected_components_number()
1
sage: G.rank_decomposition()
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)

/path-to-sage/sage-4.8.alpha5/devel/sage-myclone/<ipython console> in <module>()

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph.pyc in rank_decomposition(self, verbose)
   2999 
   3000         from sage.graphs.graph_decompositions.rankwidth import rank_decomposition
-> 3001         return rank_decomposition(self, verbose = verbose)
   3002 
   3003     ### Matching

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph_decompositions/rankwidth.so in sage.graphs.graph_decompositions.rankwidth.rank_decomposition (sage/graphs/graph_decompositions/rankwidth.c:730)()

Exception: There has been a mistake while converting the Sage graph to a C structure. The memory is probably insufficient (2^(n+1) is a *LOT*).

But the test is not always working, perhaps because I'm on a 32 bits computer...


sage: G = graphs.RandomGNM(30,200)
sage: G.connected_components_number()
1
sage: G.rank_decomposition()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)

/path-to-sage/sage-4.8.alpha5/devel/sage-myclone/<ipython console> in <module>()

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph.pyc in rank_decomposition(self, verbose)
   2999 
   3000         from sage.graphs.graph_decompositions.rankwidth import rank_decomposition
-> 3001         return rank_decomposition(self, verbose = verbose)
   3002 
   3003     ### Matching

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph_decompositions/rankwidth.so in sage.graphs.graph_decompositions.rankwidth.rank_decomposition (sage/graphs/graph_decompositions/rankwidth.c:815)()

RuntimeError: Segmentation fault

I also have a segfault when running on an empty graph. A simple test is certainly missing.

sage: G= Graph()
sage: G.rank_decomposition()
/path-to-sage/sage-4.8.alpha5/local/lib/libcsage.so(print_backtrace+0x3b)[0x4f7967]
/path-to-sage/sage-4.8.alpha5/local/lib/libcsage.so(sigdie+0x17)[0x4f79a7]
...
...
------------------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred in Sage.
This probably occurred because a *compiled* component of Sage has a bug
in it and is not properly wrapped with sig_on(), sig_off(). You might
want to run Sage under gdb with 'sage -gdb' to debug this.
Sage will now terminate.
------------------------------------------------------------------------
local/bin/sage-sage: line 305:  4884 Segmentation fault      (core dumped) sage-ipython "$@" -i

Hope it helps fixing the problems.

Best,

David.

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Jan 8, 2012

comment:7

Hellooooooo David !!

I just updated the patch so that the case n == 0 is checked. This being said, there is nothing I can do with the segfault you meet as I have not been able to reproduce it. I read my code several times and try to pay attention to those memory errors, all the malloc are checked, etc... I am beginning to think the error may be in the original C code and perhaps it is not too hard to spot by adding "printf("Hello\n");" at some different places until we know where the error is coming from.
After this, I can forward the problem to the author and ask for a patch (and I don't expect this to take too much time either).

The thing is that I really can not debug this without being able to reproduce the bug. I tried maaaaaaany times t create a random graph and give it a try but it never happened. Did you tell me that the error occurred right when the function was called ? And do you know if it happens "often" ? How many random graphs do you try before it segfaults ?

Nathann

@dcoudert
Copy link
Contributor

dcoudert commented Jan 8, 2012

comment:8

I just tried the new version of the patch.

It is now OK for empty graphs

sage: G = Graph()
sage: G.rank_decomposition()
(0, Graph on 0 vertices)

I don't know what is the expected result for non-connected graphs. I have the following result:

sage: G = graphs.RandomGNM(10,1)
sage: G.connected_components_number()
9
sage: G.rank_decomposition()
(1, Graph on 19 vertices)

Now, concerning large graphs:

  • The exception message for N==32 could be slightly improve. The current message is "
    Exception: The rank decomposition cannot be computed on graphs of more than 32 vertices.", but is it >32 or >=32 ?

  • N == 31 AND N == 29: I got the correct message: "Exception: There has been a mistake while converting the Sage graph to a C structure. The memory is probably insufficient (2^(n+1) is a LOT)."

  • N == 30: I have a segfault immediatly, that is for the first 30 vertices graph I'm trying:

sage: G = graphs.RandomGNM(30,500)
sage: G.rank_decomposition()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)

/path-to-sage/sage-4.8.alpha5/devel/sage-blop/sage/<ipython console> in <module>()

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph.pyc in rank_decomposition(self, verbose)
   2999 
   3000         from sage.graphs.graph_decompositions.rankwidth import rank_decomposition
-> 3001         return rank_decomposition(self, verbose = verbose)
   3002 
   3003     ### Matching

/path-to-sage/sage-4.8.alpha5/local/lib/python2.6/site-packages/sage/graphs/graph_decompositions/rankwidth.so in sage.graphs.graph_decompositions.rankwidth.rank_decomposition (sage/graphs/graph_decompositions/rankwidth.c:863)()

RuntimeError: Segmentation fault

I tried with other 30 vertices graphs: G = graphs.GridGraph([5,6]), G = graphs.RandomTree(30), G = graphs.CompleteGraph(30), ... and I have exactly the same segfault.

The problem comes from the "if sage_graph_to_matrix(G)" in which you use the " int init_rw_dec(uint_fast8_t n) " function that uses the "int init_rw(uint_fast8_t n)" function.
Most probably, the computation exceeds the type capacity and we get a wrong result.
So, to be on the safe side, I suggest to add a test in the sage_graph_to_matrix function to test if memory allocation is possible.
Or, as you have suggested, you can ask for a patch from the original author.

Best,
D.

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Jan 8, 2012

comment:9

Helloooooo !!!

Patch updated to make the 32 limit clearer :-)

About the segfault : the point is that there is nothing I could "check" to return an exception to avoid the segault unless I know where it comes from. The function you mention contains a total of 5 lines which can produce no segfault, and exclusively consist in memory allocations and checking the memory has been allocated

int init_rw(uint_fast8_t n)
{
 	num_vertices = n;
	adjacency_matrix = malloc(sizeof(subset_t) * n);
	slots = malloc(sizeof(uint_fast8_t) * (1ul << n));
	cslots = 0;
	return((adjacency_matrix && slots) ? 0 : -1);
}

If anything goes wrong, the function return -1 instead of 0. And this function (which is defined in the original C code) is called as you said, by a "if" that is precisely there to ensure the memory allocation went smoothly. If it did not, it also returns and error and the function above (rank_decomposition) raises the exception. I mean, there is no memory allocation that I could check and that is not checked already O_o

Nathann

@dcoudert
Copy link
Contributor

dcoudert commented Jan 8, 2012

comment:10

I added some printf to track the segfault. It occurs in the calculate_level function for ss=1 and i=17. More precisely, the segfault is with the instruction cslots[1 << i] = 0x0;.

void calculate_level(uint_fast8_t ss)
{
        uint_fast8_t i;

        subset_size = ss;

        if(subset_size == 0)
                slots[0] = 0;
        else if(subset_size == 1)
                for(i = 0; i < num_vertices; i++)
                { 
                  printf("ss=%d, i=%d\n",ss,i);
                        slots[1 << i] = cut_rank(1 << i);
                  printf("after cut_rank\n");
                        cslots[1 << i] = 0x0;
                  printf("after cslots\n");
                }
        else
        {
                subset_t i;
                const subset_t end = binomial_coefficient(num_vertices, subset_size);
                for(i = 0; i < end; i++)
                        fill_slot(i);
        }
}

This is definitely a memory allocation problem.

D.

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Jan 9, 2012

comment:11

Ahah !! i=17 ! Now that's something ! I prefer when it is something around a power of two, it makes more sense :-)

Well, for instance I wondered what the result of

1 << (uint_fast8) 17

is in C. Is 1 automatically set to type uint_fast8 ? In this case the result is zero. Is it automatically set to integer ? In this case the result is 2^17. Anyway this should not make much of a difference, as accessing cslots[0] is not big deal -- it is the safest of them all to access !
Oddly the allocation part seems totally safe

cslots = malloc(sizeof(subset_t) * (1ul << n));

Nathann

@dcoudert
Copy link
Contributor

comment:12

I tried with a fresh install of 5.0.beta1 and I still have the problem with 30 :(

D.

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Jan 25, 2012

comment:13

Ok... I gave it another try and really have no idea where this could come from. As the only thing I can guess is that the problem actually come from the original source code, what would you think of merging this patch anyway, with a warning in the documentation (also to ask people experiencing the same bug, if any, to give us information on their hardware), and continue to discuss it with the author ?

The way we are going now, this patch will just be forgotten even though it can actually solve the problem except on your architecture for some mysterious reason O_o

Nathann

@dcoudert
Copy link
Contributor

comment:14

I agree with this proposal, so I give a positive review.

Honestly, I think most of us will never try use the algorithm for N=30 due to very huge computation time ;-)

D.

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Jan 25, 2012

comment:15

Argggg !!! A bit too fast !! The warning was not in the patch when I said that ! Could you check this other patch too ? It contains the message I mentionned and also fixes a warning when building the documentation.

Sorryyyyyyyyyyyyyyyyyy !!

Nathann

@dcoudert
Copy link
Contributor

comment:16

Attachment: trac_11754_warning.patch.gz

I tried the warning patch as well, and the documentation is properly generated with a clear message (pink warning box).

I give a positive review.

D.

@dcoudert
Copy link
Contributor

comment:40

Nathann,

I have added a review patch because 1) the test in rw.h was not correct. We have to define a tag. 2) the test in rw.c was also incorrect due to missing braces.

Now compilation is correct on both my 32 bits and 64 bits computers. The execution is correct, in particular it returns the correct error message when memory is insufficient. Tests are OK, and documentation was already OK.

For me the patch is now good to go.

D.

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Feb 24, 2012

comment:41

Hellooooo !!!

I have added a review patch because 1) the test in rw.h was not correct. We have to define a tag. 2) the test in rw.c was also incorrect due to missing braces.

Hmm... That should be fixed in his code. I will email him about it O_o

Now compilation is correct on both my 32 bits and 64 bits computers. The execution is correct, in particular it returns the correct error message when memory is insufficient. Tests are OK, and documentation was already OK.

One last mail exchange with him, and I'll be glad to see this patch merged ! :-p

Nathann

@dcoudert
Copy link
Contributor

comment:42

I did some more tests with sage-5.0.beta5

  1. on a 32 bits computer with 4Go of RAM, gcc (GCC) 4.6.1 20110908 (Red Hat 4.6.1-9)

    • I tried first to apply only trac_11754.patch + trac_11754_warning.patch + trac_11754-ifndef.patch

    • Install OK, tests, OK, functionality OK, but problem when n = 30: the execution starts as if mallocs were OK, but I don't have enough memory for 29 so it cannot work for 30

    • Now, changing the ugly (unsafe?) test in init_rw from !if(n > MAX_VERTICES || n && !(sizeof(uint_fast8_t) * (1ul << n)))! to !if( ( (n > MAX_VERTICES) ||  (n>0) ) && !(sizeof(uint_fast8_t) * (1ul << n)) )! solves the problem and I get the correct error message !

  2. on a 64 bits computer with 64Go of RAM, gcc (GCC) 4.4.3 20100127 (Red Hat 4.4.3-4)

    • I have an installation error when using only trac_11754.patch + trac_11754_warning.patch + trac_11754-ifndef.patch
    • When patching the ifndef test in rw.h, I solve installation problem, and then every thing seems to work. However, I have no way to control execution with N>=29 due to computation time. So I don't know what's happening when N=30 (I have large enough memory, but computation should be huge).

Altogether, I need the revision patch (as I already identified weeks ago) to have the patch properly installing and operating.

D.

@dcoudert
Copy link
Contributor

comment:43

Wysiwyg problem...

So change the test from

if(n > MAX_VERTICES || n && !(sizeof(uint_fast8_t) * (1ul << n)))

To

if( ( (n > MAX_VERTICES) || (n>0) ) && !(sizeof(uint_fast8_t) * (1ul << n)) )

Honestly, which one is the fastest to read ?

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Feb 24, 2012

comment:44

Ok, the author told me he intended the priority to be different (a parenthesis around "n && !(sizeof(uint_fast8_t) * (1ul << n))", but it makes no difference anyway as we settle the case n=0 differently already.

On my machine everything runs fine with all 4 patches applied. If everything is fine on your computer too I think this patch can go. As for a possible updates in 50 years, we will still have this track ticket lying around :-)

Nathann

@dcoudert
Copy link
Contributor

comment:45

Attachment: trac_11754-ifndef-rev.patch.gz

The author is right ;-)

The previous version was also working for me because the test was true for N> 31, n <0, and N=30, but it was an accident. With the correct version, we have the same result but it is more expected.

I have updated the patch accordingly and its working fine on all my computers: compilation, tests, documentation, functionality, error messages.

For me it's good to go so I give positive review.

D.

PS: I hope the ARM guys will not experience similar problems than for vertex separation...

@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Feb 24, 2012

comment:46

Well, at least not "the same ones", as the author uses the correct types. And I'll do that too from now on :-)

Thank you for the review, by the way ! This is another of the patches that have been waiting for... ages ! :-D

Nathann

@jdemeyer
Copy link

Merged: sage-5.0.beta6

@jdemeyer
Copy link

jdemeyer commented Mar 1, 2012

comment:48

Re-opening this as there some problems with the MANIFEST.in. The following need to be added:

! sage/graphs/graph_decompositions/rankwidth/README
! sage/graphs/graph_decompositions/rankwidth/__init__.py

@jdemeyer jdemeyer reopened this Mar 1, 2012
@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Mar 3, 2012

comment:49

Helloooo !!!

Is this patch what you need ? ^^;

Nathann

@nathanncohen nathanncohen mannequin added the s: needs review label Mar 3, 2012
@nathanncohen
Copy link
Mannequin Author

nathanncohen mannequin commented Mar 3, 2012

comment:50

Attachment: trac_11754-RM.patch.gz

@nathanncohen

This comment has been minimized.

@jdemeyer

This comment has been minimized.

@jdemeyer
Copy link

jdemeyer commented Mar 4, 2012

comment:51

Attachment: 11754_manifest.patch.gz

@jdemeyer
Copy link

jdemeyer commented Mar 4, 2012

Changed reviewer from David Coudert to David Coudert, Jeroen Demeyer

@jdemeyer

This comment has been minimized.

@mwhansen
Copy link
Contributor

comment:53

I think there is an issue with this code as there is a extension module sage.graphs.graph_decompositions.rankwidth as well as a package (with init.py) sage/graphs/decompositions/rankwidth/ . I think that should really be fixed as it's ambiguous which should be used.

@jdemeyer
Copy link

comment:54

To be fixed in a follow-up ticket of course.

@mwhansen
Copy link
Contributor

comment:55

This is now at #12684.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants