Skip to content

Creating Debian Packages from CRAN Sources

Notifications You must be signed in to change notification settings

thehesiod-forks/cran2deb

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

To install:

Please install a series of packages that you will need to
run cran2deb successfully:

	sudo apt-get install pbuilder python-gpgme gnupg-agent

You may install cran2deb manually

	$ cd ..
	$ R CMD INSTALL cran2deb

	copy cran2deb/exec/cran2deb into somewhere in your executable path (e.g.,
	/usr/local/bin, $home/bin)

or prepare a Debian package for it

	fakeroot debian/rules binary
	sudo dpkg -i ../r-*cran2deb*.deb


To configure:

1. You need a web server serving from say, /var/www/cran2deb/

Let ROOT be the value returned by running: cran2deb root
Let SYS be the system you wish to build for (e.g., debian-amd64)
Create user "c2d" who shall have write permissions to the archive.

2. create /etc/cran2deb
   a. copy ROOT/etc/* into /etc/cran2deb/
   b. ensure ROOT/etc/sys/SYS is set up
   c. /etc/cran2deb/archive should be a symlink pointing to /var/www/cran2deb/

    $ ln -s /var/www/cran2deb/ /etc/cran2deb/archive
    $ mkdir /var/www/cran2deb/SYS

   d. modify OTHERMIRROR of /etc/cran2deb/sys/SYS/pbuilderrc.in to point to your webserver.
      As long as the packages on debian.cran.r-project.org are not signed with a
      current key, just rebuild also the architecture-dependent packages or find
      a way for pbuilder to ignore the failing gpg signature check.

      Example:
      OTHERMIRROR='deb http://master.dermacloud.uni-luebeck.de/cran2deb/rep testing main'

      The above URL works for the reprepro tool. For mini-dinstall the URL
      may be slightly different. Please check after the first packages have been
      built that do not depend on other packages external to Debian.

   e. run: cran2deb repopulate

3. cran2deb needs a persistent cache outside of R's control. therefore, create
    /var/cache/cran2deb, writable by whichever user(s) will run cran2deb.

4. add to /etc/rc.local:
	# one mini-dinstall daemon for each apt repo
	for sys in debian-i386 debian-amd64
	do
        	mini-dinstall -c /etc/cran2deb/sys/$sys/mini-dinstall.conf
	done
   and execute.

5. manual change - allow cran2deb user to execute pdebuild as root via sudo
	c2d      ALL=(ALL) SETENV: NOPASSWD: /usr/sbin/pdebuild
   The "SETENV:" is important.

6. run: cran2deb update
   This will also create the pbuilder environment if not already existing.

7. manual changes - create a gpg key for your packages

	gpg --genkey

   and read through 'man gpg-agent' to set it up. Add that key to the
   pbuilder environment so the packages you signed and uploaded are 
   indeed acceptable to the distribution.

	sudo pbuilder  --login --save-after-login \
	     --basetgz /var/cache/pbuilder/base-cran2deb-debian-amd64.tgz

   Once logged in, in a separate shell perform as cran2deb user a

	gpg --export -a

   which should only be one (your) public key. Copy it with the mouse and
   add it to your pbuilder login shell via

	apt-key add -

   (return) to expect the key from stdin. Paste it, press CTRL-D to end
   the input. Another CTRL-D or 'exit' to leave the shell and have the
   base.tgz updated.

   Also, you should allow the arch-independent R packages to be retrieved
   from the public cran2deb effort. This also requires that key to be
   available. From a local shell copy again that key

	gpg --recv-key BFAEA5C2
	gpg --export -a BFAEA5C2

   and use 'apt-key add -' as before.

8. Try building a simple package: cran2deb build zoo
   (The result will be in /var/cache/cran2deb/results/SYS)

9. Check the specification of your pbuilder settings and run
   'cran2deb update'. Do not run 'cran2deb update full' which
   creates a new pbuilder environment and will require the
   adding of keys again.

10. Educate yourself about the schema of the sqlite3 database,
   which is likely to reside at ~/cache/cran2deb.db 
   and is directly accessible via calling from the 
   UNIX command line 'sqlite3 ~/cache/cran2deb.db' .

   The sqlite3 shell will directly attach to the database.
   See the tables first

	sqlite> .tables
	blacklist_packages  debian_dependency   license_override  
	builds              forced_depends      packages          
	database_versions   license_hashes      sysreq_override   

   and then the schema of the one or other table of interest, e.g.

   sqlite> .schema debian_dependency
	CREATE TABLE debian_dependency (
		id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
		alias TEXT NOT NULL,
		build INTEGER NOT NULL,
		debian_pkg TEXT NOT NULL ,
		UNIQUE (alias,build,debian_pkg) );
   sqlite> .schema forced_depends
	CREATE TABLE forced_depends (
		r_name TEXT NOT NULL,
		depend_alias TEXT NOT NULL,
		PRIMARY KEY (r_name,depend_alias) );
   sqlite> .schema blacklist_packages
	CREATE TABLE blacklist_packages (
		package TEXT PRIMARY KEY NOT NULL,
		nonfree INTEGER NOT NULL DEFAULT 0,
		obsolete INTEGER NOT NULL DEFAULT 0,
		broken_dependency INTEGER NOT NULL DEFAULT 0,
		unsatisfied_dependency INTEGER NOT NULL DEFAULT 0,
		breaks_cran2deb INTEGER NOT NULL DEFAULT 0,
		other INTEGER NOT NULL DEFAULT 0,
		explanation TEXT NOT NULL);
   sqlite> .schema builds
	CREATE TABLE builds (
		id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
		system TEXT NOT NULL ,package TEXT NOT NULL,
		r_version TEXT NOT NULL ,deb_epoch INTEGER NOT NULL,
		deb_revision INTEGER NOT NULL ,db_version INTEGER NOT NULL,
		date_stamp TEXT NOT NULL ,time_stamp TEXT NOT NULL,
		scm_revision TEXT NOT NULL ,success INTEGER NOT NULL,
		log TEXT,
		UNIQUE(package,system,r_version,deb_epoch,deb_revision,db_version));
   sqlite> .schema database_versions
	CREATE TABLE database_versions (
		version INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
		version_date INTEGER NOT NULL,
		base_epoch INTEGER NOT NULL );
   sqlite> .schema license_hashes
	CREATE TABLE license_hashes (
		name TEXT NOT NULL,
		sha1 TEXT PRIMARY KEY NOT NULL );
   sqlite> .schema license_override
	CREATE TABLE license_override (
		name TEXT PRIMARY KEY NOT NULL,
		accept INT NOT NULL );
   sqlite> .schema packages
	CREATE TABLE packages (
		package TEXT PRIMARY KEY NOT NULL,
		latest_r_version TEXT );
   sqlite> .schema sysreq_override
	CREATE TABLE sysreq_override (
		depend_alias TEXT NOT NULL,
		r_pattern TEXT PRIMARY KEY NOT NULL );

   The sqlite3 shell allows to modify all those entries. For a
   restart with defaults entries distributed via svn, run "cran2deb
   repopulate". To update the data stored in svn, change to the data
   directory and run "./pull". Caveat, prior to that, make sure the
   license data was read in by you. This is not performed automatically in
   repopulate, a bit to protect us and stress that everyone is reponsible
   for their own respective interpretation of a license's constraints
   and the effect on a redistribution of source and/or binary.

11.Add licenses the way you think it is right. The script in 
   'exec/repopulate' may offer ideas on how to mass-include what
   you find in the svn. For new entries, you may prefer using
   'cran2deb' license that expects to learn from from the command
   line how to treat individual licenses:

      $ cran2deb license accept akima
      I: cran2deb svn: 344 building for debian-amd64 at 2011-02-11 18:46:21
      N: adding akima accept? TRUE

      $ cran2deb license hash_sha1 akima 485316717b03afbd6d20f7810f9123b4445c8660
      I: cran2deb svn: 344 building for debian-amd64 at 2011-02-11 18:46:37
      N: adding hash 485316717b03afbd6d20f7810f9123b4445c8660 for akima

      $ cran2deb license ls
      I: cran2deb svn: 344 building for debian-amd64 at 2011-02-11 18:46:53
      accept akima 
      hash_sha1 akima 485316717b03afbd6d20f7810f9123b4445c8660 

   What happens here is that a license file was read and if that is
   having the same hash value, then it is mapped to the license 'akima'.
   The license 'akima' then again is told to be acceptable. It happened
   here that the license is of the same name as the package that it 
   occurs in.

To think about

1. After several updates of the repository, we have many outdated
   Debian packages in the pbuilder's archive cache. I prefer removing
   them with a skriptlet like

	cd /var/cache/apt/archives \
	&& for i in $(ls | cut -f1 -d_ | uniq -c | egrep -v "^ *1 " | sed -e 's/^\s*[0-9]*\s//')
	   do 
		ls -t ${i}_* | tail -n +2 | xargs -r rm
	   done

2. To investigate which packages are already available in your repository
   and which ones are not, you may add your packages list to your /etc/apt/sources.list
   and perform

	sudo apt-get update
	for i in `grep -i net all_pkgs `
	do
		a=`echo $i|tr 'A-Z' 'a-z'`
		echo "$i:$a"
		apt-cache search r-cran-$a
	done

3. If running on a separate partition, the place reserved for cran2deb is not unlikely
   to exceed your expectation, especially when also running BioConductor. You may
   consider running cran2deb on logical volumes, like by LVM. Together with a file
   system that allows its extension, the process is rather straight forward:

	# lvextend -L+40G /dev/vg0/cran2deb 
	Extending logical volume cran2deb to 115.04 GiB
        Logical volume cran2deb successfully resized
        # xfs_growfs /dev/vg0/cran2deb
        meta-data=/dev/dm-6              isize=256    agcount=21, agsize=983040 blks
                 =                       sectsz=512   attr=2
        data     =                       bsize=4096   blocks=19671040, imaxpct=25
                 =                       sunit=0      swidth=0 blks
        naming   =version 2              bsize=4096   ascii-ci=0
        log      =internal               bsize=4096   blocks=2560, version=2
                 =                       sectsz=512   sunit=0 blks, lazy-count=1
        realtime =none                   extsz=4096   blocks=0, rtextents=0
        data blocks changed from 19671040 to 30156800

   The directory does not even need to unmounted for that with XFS. 


To debug:

$ cran2deb help
will display a short summary of help for each cran2deb command.

The -d option will prevent the cleaning of the unpacked source code.

To improve:

 * The runtime dependencies for Rgraphviz are outdated (there is no libraphviz4
   any more). To get the package run  this was not changed to "alias_run
   libgraphviz libgraphviz-dev" but this can certainly be optimised
   towards something less resource-hungry.


Concerning data/:
This contains scripts necessary to recreate the database should you lose the
database. It's a backup that can be versioned by SVN. There is a script called
pull that, when run from the data directory will recreate all the files from
the database EXCEPT for the licenses. The licenses cannot be recreated because
licenses can be based on one-way hashes.  This process could certainly be
improved. But one gets the hang of it ater a while. So don't despair.

About

Creating Debian Packages from CRAN Sources

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 93.5%
  • R 6.5%