=======================
 Smart Package Manager
=======================
:Author: Gustavo Niemeyer
:Contact: niemeyer@conectiva.com
:Revision: $Rev$
:Date: $Date$

.. contents::

--------
Overview
--------

The **Smart Package Manager** project has the ambitious objective of
creating smart and portable algorithms for solving adequately the
problem of managing software upgrading and installation. This tool
works in all major distributions, and will bring notable advantages
over native tools currently in use (APT, APT-RPM, YUM, URPMI, etc).

From `The Free On-line Dictionary of Computing`::

    smart

        1. <programming> Said of a program that does the {Right Thing}
        in a wide variety of complicated circumstances. (...)


--------------
Project Status
--------------

The development of Smart Package Manager started on May 4th, 2004, and
version 1.0 was released on Aug 14th, 2008, after extended beta testing.

--------
Features
--------

Modular
-------

Smart has been developed with modularity and flexibility in mind. It's
completely backend-based, and package-manager-agnostic. Support is
currently implemented for **RPM**, **DPKG**, and **Slackware**
package management systems, and porting it to new systems should be
very easy.


Smart Transactions
------------------

That's one of the most interesting aspects of Smart Package Manager,
and the one who has motivated calling it `smart`. Computing
transactions respecting the relations involved in the package
management world may become an unpleasant task when thousands of
packages and relations are being considered, or even when just
a few complex relations turn the most obvious choice into the
unwanted one.

While other applications try to find a possible solution to satisfy
the relations involved in some user-requested operation, and
sometimes even fail to do so [1]_, Smart goes beyond it. In the
kernel of Smart Package Manager lives an algorithm that will
not only find a solution, if one is available, but will find
the best solution. This is done by quickly weighting every
possible solution with a pluggable policy, which redefines
the term "best" depending on the operation goal (install,
remove, upgrade, etc).

This behavior has many interesting consequences. In upgrades,
for instance, while precedence is given to newer versions,
intermediate versions may get selected if they bring a
better global result for the system. Packages may even be
reinstalled, if different packages with the same name-version
pair have different relations, and the one not installed
is considered a better option.

Another important goal achieved with the transaction algorithm
is that, even though it is able to check and fix relations in
the whole system, it will work even when there are broken
relations in installed packages. Only relations related to
the operation being made are checked for correctness.

.. [1] Check `Case Studies`_ for real cases where the algorithm
       works better than what is implemented in other applications.

Multiple Interfaces
-------------------

Smart has multiple native and completely integrated interfaces:

- Command line interface, with several useful subcommands: update,
  install, reinstall, upgrade, remove, check, fix, download, search,
  and more.

- Shell interface, with command and argument completion, making it
  easy to perform multiple operations quickly using a local or
  remote terminal.

- Graphic interface, offering the friendliness of visual
  user interaction.

- Command line interface with graphic feedback, allowing one to
  integrate the power of command line with graphic environments.

Besides these interfaces, ksmarttray is also included in the Smart
package. It notifies users about available updates using a KDE
tray icon.

Channels
--------

Channels are the way Smart becomes aware about external repositories
of information. Many different channel types are supported, depending
on the backend and kind of information desired:

- APT-DEB Repository
- APT-RPM Repository
- DPKG Installed Packages
- Mirror Information
- Red Carpet Channel
- RPM Directory
- RPM Header List
- RPM MetaData (YUM)
- RPM Installed Packages
- Slackware Repository
- Slackware Installed Packages
- URPMI Repository

Priority Handling
-----------------

Priorities are a powerful way to easily handle integration
of multiple channels and explicit user setups regarding
preferred package versions.

Basically, packages with higher priorities are considered a
better option to be installed in the system, even when package
versions state otherwise. Priorities may be individually
assigned to all packages in given channels, to all packages
with given names, and to packages with given names inside
given channels.

With custom priority setups, it becomes possible to avoid
unwanted upgrades, force downgrades, select packages in given
channels as preferential, and other kinds of interesting setups.

Autobalancing Mirror System
---------------------------

Smart offers a very flexible mirror support. Mirrors are URLs
that supposedly provide the same contents as are available in
other URLs, named origins. There is no internal restriction on
the kind of information which is mirrored. Once an origin URL
is provided, and one or more mirror URLs are provided, these
mirrors will be considered for any file which is going to be
fetched from an URL starting with the origin URL.

Mirror precedence is dynamically computed based on the history
of downloads of all mirrors available for a given origin URL
(including the origin site itself). The fastest mirrors and
with less errors are chosen. When errors occur, the next
mirror in the queue is tried.

For instance, if a mirror `http://mirror.url/path/` is provided
for the origin `ftp://origin.url/other/path/`, and a file in
`ftp://origin.url/other/path/subpath/somefile` is going to be
fetched, the mirror will be considered for being used, and the
URL `http://mirror.url/path/subpath/somefile` will be used if
the mirror is chosen. Notice that strings are compared and
replaced without any pre-processing, so that it's possible to
use different schemes (ftp, http, etc) in mirror entries, and
even URLs ending in prefixes of directory entries.

Downloading Mechanism
---------------------

Smart has a fast parallel downloading mechanism, allowing multiple
connections to be used for one or more sites. The mechanism
supports:

- Resuming
- Timestamp checking
- Parallel uncompression
- Autodetection of FTP user limit
- Cached file validation

and more.

At the moment, the following schemes are nativelly supported:

- file
- ftp
- http
- https
- scp

Additionally, the following schemes are supported when pycurl is
available:

- ftps
- telnet
- dict
- ldap

Removable Media Support
-----------------------

Smart Package Manager implements builtin support for removable media
(CDROMs, DVDs, etc) in most of the supported channel types. The
following features are currently implemented:

- Mountpoint autodetection
- Support for multiple simultaneous media drives
- Medias may be inserted in any order
- Installed system is guaranteed to maintain correct relations
  between media changes
- Remote removable media support using any of the supported schemes
  (ftp, http, scp, etc)

-------------
Running Smart
-------------

Smart Package Manager may be run in many different ways, depending
on the interface in use and on the intended goal.

The following command would install the `foobar` package, for instance::

	smart install foobar

While the following command would install the `foobar` package, but with
graphic output::

	smart --gui install foobar

To open the graphic interface in interactive mode, one may simply run::

	smart --gui

Similarly, the following command would open the shell interface::

	smart --shell

Extensive help is available for all commands, by using the `--help`
switch::

	smart --help
	smart install --help
	smart channel --help
	...

--------------
Building Smart
--------------

Dependencies
------------

:Core:
	Smart is written in Python, with some core modules rewritten as
	C extensions for memory savings and performance gains. With that
	in mind, the core system of Smart depends on Python 2.3 or
	higher, and a C compiler to build the extensions.

:Graphic Interface:
	The "gtk" graphic interface depends on `pygtk` 2.4 or higher.
	The "qt" graphic interface depends on `pyqt` 3.3 (not 4.x).

:RPM backend:
	The RPM backend depends on the Python `rpm` module of RPM 4.4 or
	higher, due to a limitation which was present in previous versions
	of the `ts.dbMatch()` method, and the availability of the
	`readHeaderFromFD()` function.

	In the `contrib/patches/` subdirectory there are patches for
	previous RPM versions including the missing functionality. There
	are also pre-packaged binary versions which include the patched
	module without requiring changes in other tools.

:DPKG backend:
	There are no extra dependencies besides DPKG itself.

:Slackware backend:
	There are no extra dependencies besides the packaging scripts
	`installpkg`, `upgradepkg` and `removepkg`.

------------
Case Studies
------------

In this section will be described real cases showing `Smart` behavior
in comparison with other tools, or handling unusual situations.

Notice that Smart was not tuned to work in any of these cases, and
the reason it works is because handling unusual situations was the
initial project goal.


Case 1 - APT
------------

This case happened in a real world environment where a weakness in
the algorithm used by `APT` (which is the same used in `APT-RPM`)
turned a simple operation into a problem of obscure results.
Smart Package Manager was used in the same environment to show
its results.

The problem starts when an installation of `xscreensaver` is tried::

  [root@damien:/root] apt-get install xscreensaver
  Reading Package Lists... Done
  Building Dependency Tree... Done
  Some packages could not be installed. This may mean that you have
  requested an impossible situation or if you are using the unstable
  distribution that some required packages have not yet been created
  or been moved out of Incoming.
  
  Since you only requested a single operation it is extremely likely that
  the package is simply not installable and a bug report against
  that package should be filed.
  The following information may help to resolve the situation:
  
  The following packages have unmet dependencies:
    xscreensaver: Depends: libglade-2.0.so.0
                  Depends: libxml2.so.2
  E: Broken packages

The error shown makes the user believe that `libglade-2.0.so.0` and
`libxml2.so.2` are not available. That's not the case::

 [root@damien:/root] apt-get install libxml2
 Reading Package Lists... Done
 Building Dependency Tree... Done
 Some packages could not be installed. This may mean that you have
 requested an impossible situation or if you are using the unstable
 distribution that some required packages have not yet been created
 or been moved out of Incoming.
 
 Since you only requested a single operation it is extremely likely that
 the package is simply not installable and a bug report against
 that package should be filed.
 The following information may help to resolve the situation:
 
 The following packages have unmet dependencies:
   libxml2: Depends: glibc-iconv but it is not going to be installed
 E: Broken packages

Another misguiding error message. Let's go further::

 [root@damien:/root] apt-get install glibc-iconv
 Reading Package Lists... Done
 Building Dependency Tree... Done
 Some packages could not be installed. This may mean that you have
 requested an impossible situation or if you are using the unstable
 distribution that some required packages have not yet been created
 or been moved out of Incoming.
 
 Since you only requested a single operation it is extremely likely that
 the package is simply not installable and a bug report against
 that package should be filed.
 The following information may help to resolve the situation:
 
 The following packages have unmet dependencies:
   glibc-iconv: Depends: glibc-gconvdata (= 2.3.3) but 1:2.3.2-586_1cl is to be installed
 E: Broken packages

Version `2.3.3` is needed, but `1:2.3.2-586_1cl` is to be installed. This
message is mostly correct. The only problem is, "1:2.3.2-586_1cl" is
already installed::

 [root@damien:/root] apt-cache policy glibc-gconvdata
 glibc-gconvdata:
   Installed: 1:2.3.2-586_1cl
   Candidate: 1:2.3.2-586_1cl
   Version Table:
  *** 1:2.3.2-586_1cl 0
         100 RPM Database
      0:2.3.3-69473cl 0
         500 file: conectiva/all pkglist

The problem was found. A package from another repository (586_1cl shows
it's not native, in that specific case) has a higher epoch than the one
available in the usual repository. This clearly shows that the APT
algorithm marks a single version as candidate, and when this is not the
wanted version for some operation, the whole operation is compromised.

When testing `Smart Package Manager` in the same environment, the
expected result is obtained::

 [root@damien:/root] smart install xscreensaver
 Updating cache...              ######################################## [100%]
 
 Computing transaction...
 
 Downgrading packages (1):
   glibc-gconvdata-0:2.3.3-69473cl.i386
 
 Installing packages (4):
   glibc-iconv-0:2.3.3-69473cl.i386
   libglade2-2.4.0-68154cl.i386
   libxml2-2:2.6.13-67598cl.i386
   xscreensaver-4.15-69825cl.i386

 Confirm changes (Y/n)?

Smart correctly selected `glibc-gconvdata` for downgrading as the only
possibility of performing the user requested operation.

Case 2 - APT & YUM
------------------

This is another real case, and is being reproduced in a controlled
environment for tests with YUM, APT-RPM, and Smart.

The issue is, a package named `A` requires package `BCD` explicitly, and
RPM detects implicit dependencies between `A` and `libB`, `libC`, and `libD`.
Package `BCD` provides `libB`, `libC`, and `libD`, but additionally there
is a package `B` providing `libB`, a package `C` providing `libC`, and
a package `D` providing `libD`.

In other words, there's a package `A` which requires four different symbols,
and one of these symbols is provided by a single package `BCD`, which happens
to provide all symbols needed by `A`. There are also packages `B`, `C`, and `D`,
that provide some of the symbols required by `A`, but can't satisfy all
dependencies without `BCD`.

The expected behavior for an operation asking to install `A` is obviously
selecting `BCD` to satisfy `A`'s dependencies, on the other hand, `YUM` and
APT fail to deliver that as a guaranteed operation, as is shown
below.

First, let's see how YUM deals with the problem::

  [root@burma ~]% yum install A
  Setting up Install Process
  Setting up Repo:  localpub
  repomd.xml                100% |=========================|  951 B    00:00
  Reading repository metadata in from local files
  localpub  : ################################################## 5/5
  Resolving Dependencies
  --> Populating transaction set with selected packages. Please wait.
  ---> Downloading header for A to pack into transaction set.
  A-1.0-1cl.i386.rpm        100% |=========================| 1.0 kB    00:00
  ---> Package A.i386 0:1.0-1cl set to be installed
  --> Running transaction check
  --> Processing Dependency: libD for package: A
  --> Processing Dependency: libC for package: A
  --> Processing Dependency: libB for package: A
  --> Processing Dependency: BCD for package: A
  --> Restarting Dependency Resolution with new changes.
  --> Populating transaction set with selected packages. Please wait.
  ---> Downloading header for D to pack into transaction set.
  D-1.0-1cl.i386.rpm        100% |=========================| 1.0 kB    00:00
  ---> Package D.i386 0:1.0-1cl set to be installed
  ---> Downloading header for C to pack into transaction set.
  C-1.0-1cl.i386.rpm        100% |=========================| 1.0 kB    00:00
  ---> Package C.i386 0:1.0-1cl set to be installed
  ---> Downloading header for B to pack into transaction set.
  B-1.0-1cl.i386.rpm        100% |=========================| 1.0 kB    00:00
  ---> Package B.i386 0:1.0-1cl set to be installed
  ---> Downloading header for BCD to pack into transaction set.
  BCD-1.0-1cl.i386.rpm      100% |=========================| 1.0 kB    00:00
  ---> Package BCD.i386 0:1.0-1cl set to be installed
  --> Running transaction check
  
  Dependencies Resolved
  Transaction Listing:
    Install: A.i386 0:1.0-1cl
  
  Performing the following to resolve dependencies:
    Install: B.i386 0:1.0-1cl
    Install: BCD.i386 0:1.0-1cl
    Install: C.i386 0:1.0-1cl
    Install: D.i386 0:1.0-1cl
  Is this ok [y/N]:


YUM selected **all** packages for installation, even though `BCD`
alone would satisfy `A`'s dependencies.

Let's see how APT deals with that::

  [root@burma ~]% apt-get install A
  Reading Package Lists... Done
  Building Dependency Tree... Done
  The following extra packages will be installed:
    B BCD
  The following NEW packages will be installed:
    A B BCD
  0 upgraded, 3 newly installed, 0 removed and 0 not upgraded.
  Need to get 0B/4055B of archives.
  After unpacking 0B of additional disk space will be used.
  Do you want to continue? [Y/n] n

As a coincidence, APT did a better job, and selected only `B` and
`BCD` to satisfy `A`'s dependency, which is still not right.

Now, let's see how Smart would solve the problem::

  [root@burma ~]% smart install A
  Updating cache...               ######################################## [100%]
  
  Computing transaction...
  
  Installing packages (2):
    A-1.0-1cl@i386     BCD-1.0-1cl@i386
  
  2.7kb of package files are needed.
  
  Confirm changes (Y/n)?

Smart correctly selected only `BCD`, since it's necessary anyway, and
solves all dependencies.


Case 3 - APT & YUM
------------------

That's another interesting case which was tested with APT-RPM and YUM.

In this case, there's a package `A` version 1.0 installed in the
system, and there are two versions available for upgrading: 1.5 and 2.0.
Version 1.5 may be installed without problems, but version 2.0 has a
dependency on `B`, which is not available anywhere.

In this case, the best possibility is upgrading to 1.5, since upgrading
to 2.0 is not an option.

Let's see how APT reacts to this situation::

  [root@burma ~]% apt-get upgrade A
  Reading Package Lists... Done
  Building Dependency Tree... Done
  The following packages have been kept back
    A
  0 upgraded, 0 newly installed, 0 removed and 1 not upgraded.

APT seems to refuse to upgrade `A`, even though version 1.5 might be
installed without problems.

What happens when forcing APT to install `A`::

  [root@burma ~]% apt-get install A
  Reading Package Lists... Done
  Building Dependency Tree... Done
  Some packages could not be installed. This may mean that you have
  requested an impossible situation or if you are using the unstable
  distribution that some required packages have not yet been created
  or been moved out of Incoming.
  
  Since you only requested a single operation it is extremely likely that
  the package is simply not installable and a bug report against
  that package should be filed.
  The following information may help to resolve the situation:
  
  The following packages have unmet dependencies:
    A: Depends: B but it is not installable
  E: Broken packages

It really refuses to install the newest version, and doesn't consider
the possibility of using version 1.5.

Now, let's see how YUM would handle it.

:Update: This test case was showing the wrong results for YUM, since it
         was using a cached version of the package header that didn't
         present the missing dependency. I apologise for showing invalid
         results.

::

  [root@burma ~]% yum update
  Setting up Update Process
  Setting up Repo:  localpub
  repomd.xml                100% |=========================|  951 B    00:00
  Reading repository metadata in from local files
  primary.xml.gz            100% |=========================|  809 B    00:00
  MD Read   : ################################################## 3/3
  localpub  : ################################################## 3/3
  Resolving Dependencies
  --> Populating transaction set with selected packages. Please wait.
  ---> Downloading header for A to pack into transaction set.
  A-2.0-1cl.i386.rpm        100% |=========================| 1.3 kB    00:00
  ---> Package A.i386 0:2.0-1cl set to be updated
  --> Running transaction check
  --> Processing Dependency: B for package: A
  --> Finished Dependency Resolution
  Error: missing dep: B for pkg A

Just like APT, YUM selected version 2.0 and didn't consider the
availability of an intermediate version.

Now, let's see how Smart would behave in the same situation::

  [root@burma ~]% smart upgrade
  Loading cache...
  Updating cache...               ######################################## [100%]
  
  Computing transaction...
  
  Upgrading packages (1):
    A-1.5-1cl@i386
  
  1.3kb of package files are needed.
  
  Confirm changes (Y/n)?

Smart correctly selects the intermediate version 1.5, which is the
only viable possibility given the current options.


Case 4 - APT
------------

This case presents the following situation: there's a package `A`,
installed in the system, which depends on `libfoo`, currently
being provided by `B` 1.0. What happens if `B` is upgraded to
version 2.0, but `libfoo` is moved to be provided by package `C`?

The expected behavior would be to upgrade `B` to version 2.0,
and install `C` to satisfy `A`'s dependency.

That's not what happens with APT::

  [root@burma ~]% apt-get dist-upgrade
  Reading Package Lists... Done
  Building Dependency Tree... Done
  Calculating Upgrade... Done
  The following packages will be upgraded
    B
  The following packages will be REMOVED:
    A
  1 upgraded, 0 newly installed, 1 removed and 0 not upgraded.
  Need to get 0B/1321B of archives.
  After unpacking 0B of additional disk space will be used.
  Do you want to continue? [Y/n]

Let's see Smart in the same situation::

  [root@burma ~]% smart upgrade
  Loading cache...
  Updating cache...               ######################################## [100%]
  
  Computing transaction...
  
  Upgrading packages (1):
    B-2.0-1cl@i386
  
  Installing packages (1):
    C-2.0-1cl@i386
  
  2.6kB of package files are needed.
  
  Confirm changes (Y/n)?

Smart correctly selected package `C` for installation as a viable
possibility of leaving `A` installed in the system while upgrading
`B`.

-------
Credits
-------

This is the credit section, where people and institutions that
have somehow contributed to the project are mentioned.

:Conectiva, Inc.:
	Funded the creation of Smart, and its development up to
	August of 2005.

:Canonical Ltd:
	Funded Smart development up to November of 2009.

:Unity Linux:
	Smart development and deployment support.

:Wanderlei Cavassin:
	Conectiva's research & development coordinator, who believed
	the project was viable and encouraged the author to work on it.

:Ednilson Miura & Herton Ronaldo Krzesinski:
	Conectiva employees, helped setting up many distributions
	for tests whenever necessary.

:Andreas Hasenack:
	Conectiva employee, helped as being the first brave pre-alpha
	tester, and contributed with many ideas, discussions, etc.

:Arnaldo Carvalho de Melo:
	Conectiva board member, helped with the "channel of mirrors" idea
	and by encouraging the author to build a generic channel
	information method.

:Others @ Conectiva:
	Many other people in Conectiva helped with ideas and
	alpha-testing in general during the pre-release period of
	Smart development.

:Guilerme Manika & Ruda Moura:
	Ancient Conectiva employees, now board members of the Haxent company,
	helped by testing Smart extensively in Fedora, reporting many
	bugs and suggesting changes. They have also created the Smart
	FAQ_.

:APT-RPM & Debian:
	Experience on packaging and ideas for a better framework were
	developed while the author of Smart worked as the `APT-RPM`
	maintainer.

:Jeff Johnson:
	Contributed as being the RPM maintainer itself, and in
	many discussions regarding packaging theory in general.

:Seth Vidal:
	YUM author, and member of the Duke University, contributed
	to Smart with the development of the XML `MetaData` repository
	format and discussions about it.

:Michael Vogt:
	Currently the maintainer of the Synaptic, used to co-maintain it
	with the author of Smart. Many of his ideas ended up being adopted
	in Smart as a consequence.

:Sebastian Heinlein:
	Author of the package icons for Synaptic, that were mercilessly
	stolen to be used in Smart's graphic interface.

:TaQ/PiterPunk at #slackware-br:
	These guys helped Smart development by explaining details of
	Slackware practices regarding packaging.

:Matt Zimmerman:
	Debian/Ubuntu developer and co-maintainer of the APT software,
	helped by shining some light regarding details of the `DPKG`
	pre-depends ordering expectations.

:Mauricio Teixeira:
    FAQ maintenance, YaST2 channel maintainer, "tracker cleaner",
    general suggestions and code contributions.

:Jonathan Rocker:
	Documentation help.

.. _FAQ: http://zorked.net/smart/FAQ.html