Clamp background

Clamp is part of the jythontools project (https://github.com/jythontools). Although Jython integrates well with Java, Clamp improves this support by enabling precise generation of the Java bytecode used to wrap Python classes. In a nutshell, this means such clamped classes can be used as modern Java classes.

Clamp integrates with setuptools. Clamped packages are installed into site-packages. Clamp can also take an entire Jython installation, including site-packages, and wrap it into a single jar.

Clamp thereby provides the following benefits:

• JVM frameworks and containers can readily work with clamped code, oblivious of its source

• This is especially true of those frameworks that need single jar support

• Developers can stay as much in Python as possible. Clamp simply requires that any clamped Python classes inherit from a Java base class and/or extend Java interfaces. We may relax this restriction in the future.

• Clamp uses a SQLAlchemy-like DSL that is declarative, using metaclasses and other metaprogramming techniques. We also plan to extend this DSL substantially in the future.

Clamp example: Clamped

Please see the "Clamped" project on how to use this package. This project provides crucial documentation on how to use Clamp by going through an example in detail in the README: https://github.com/jimbaker/clamped

The Clamped README also details some aspects of the bytecode generation and how it enables direct Java usage.

Lastly, there is a talk on Clamp available (source). Note that this talk goes more into the implementation of Clamp, including how we use metaprogramming.

Important caveats

Clamp has proven to be useful in production environments, but it needs additional work before we can announce a finalized, stable release (a 1.0 in other words). In particular, it should be possible to run Clamp on Windows environments. In addition, we need a robust test suite, in addition to the functional testing we are doing by hand. Once we have such a test suite, we plan to post on PyPI.

Expect to see more updates once we complete the release of Jython 2.7.0, which has been keeping us from spending more time on this project.

Usage

Installation

Start by installing Jython 2.7. Clamp does not currently work on Windows, so the most recent beta 4 will work for you. Get it at the Jython website. You will want to bootstrap pip (this next step will be part of the Jython installer by the final release):

$ jython -m ensurepip

With this step, the pip command is now available in $JYTHON_HOME/bin/pip. You may want to alias $JYTHON_HOME/bin/pip as jpip, or you can use pyenv to use it alongside CPython's pip. Your choice. In the example below, we use jpip to keep it unambiguous which one you are using:

jpip install git+https://github.com/jythontools/clamp.git

Now Clamp is installed.

Setuptools integration

The clamp project uses setuptools integration. You simply need to add one keyword, clamp, as well as depend on the Clamp package:

import ez_setup
ez_setup.use_setuptools()

from setuptools import setup, find_packages


setup(
    name = "clamped",
    version = "0.1",
    packages = find_packages(),
    install_requires = ["clamp"],
    clamp = {
        "modules": ["clamped"]
    }
)

At a minimum, you need to specify with modules any modules you wish to clamp. This will result in clamp attempting to import each module, then saving any generated Java bytecode for clamped classes into a constructed jar.

Example clamped class

Your class (currently) needs to implement Java interfaces and/or extend a Java class - this inheritance scheme ensures that Java code knows how to use your clamped class. Your clamped class also needs to be imported by one of the modules you specified with clamp.modules, so that Clamp can generate the necessary proxy bytecode.

Your class also needs to use a base class generated by clamp_base to provide the mapping to a specific Java package namespace. You can choose an arbitrarily nested prefix, such as com.example.bar.baz.foo:

from java.io import Serializable
from java.util.concurrent import Callable

from clamp import clamp_base

BarBase = clamp_base("bar")


class BarClamp(BarBase, Callable, Serializable):

    def __init__(self):
        print "Being init-ed", self

    def call(self):
        print "foo"
        return 42

From Java, your class is now available and directly importable as bar.clamped.BarClamp - the package prefix + the module name (possibly nested) + the class name. See the Clamped example project for more usage info.

clamp command

The clamp command performs the following operations:

• Constructs a jar for all clamped classes specified by clamp.modules, per the above setup.py.

• Copies into site-packages any jars embedded in the clamped package. So these are usually jars that your Python code depends upon - "third-party jars". Note at this time, Maven and other package managers are not yet supported - you have to explicitly embed any necessary jars.

• Registers both types of jars (constructed, embdded) in jar.pth so that they are available for import. (By using a pth file, we ensure that they are referenceable on sys.path.)

You should run the clamp command after running the install command:

$ jython27 setup.py install
$ jython27 setup.py clamp

Currently this results in a layout in site-packages as follows. Ideally, the jars would be placed in the egg (unzipped), but setuptools does not like files it does not control in eggs. Regardless this layout is certainly subject to change to make it better:

$ tree site-packages/
site-packages/
├── README
├── clamp-0.4-py2.7.egg
├── clamped
│   └── clamped
│       └── javalib
│           └── baz-4.2.jar
├── clamped-0.1-py2.7.egg
│   ├── EGG-INFO
│   │   ├── PKG-INFO
│   │   ├── SOURCES.txt
│   │   ├── dependency_links.txt
│   │   ├── not-zip-safe
│   │   ├── requires.txt
│   │   └── top_level.txt
│   └── clamped
│       ├── __init__$py.class
│       ├── __init__.py
│       └── data
│           └── example.txt
├── easy-install.pth
├── jar.pth
├── jars
│   └── clamped-0.1.jar
├── setuptools-2.1-py2.7.egg
└── setuptools.pth

What is not possible - without extremely invasive (and brittle) code - is to monkeypatch the install command for users of the Clamp package. The Paver project is one possible alternative.

build_jar command

This command is not normally needed (as of 0.4), since the clamp command subsumes this functionality.

To create a jar for a clamped module in site-packages/jars and register this new jar in site-packages/jar.pth:

$ jython27 setup.py build_jar

singlejar command

Use the singlejar command to create a single jar version of the current Jython installation. (This will include virtualenv environments, but note that virtualenv support for Jython 2.7 needs some additional work. If you are building Jython from source at this time, just use that directory for now.) This setup.py custom command will use the project name as the base for the jar:

$ jython27 setup.py singlejar

To create a single jar version of the current Jython installation, you can also run this script, which is installed in Jython's bin directory. By default, the jar is named jython-single.jar

$ bin/singlejar # same, but outputs jython-single.jar, 

A number of options are supported:

bin/singlejar --help
usage: singlejar [-h] [--output PATH] [--classpath CLASSPATH] [--runpy PATH]

create a singlejar of all Jython dependencies, including clamped jars

optional arguments:
  -h, --help            show this help message and exit
  --output PATH, -o PATH
                        write jar to output path
  --classpath CLASSPATH
                        jars to include in addition to Jython runtime and
                        site-packages jars
  --runpy PATH, -r PATH
                        path to __run__.py to make a runnable jar

TODO

• Add support for variadic constructors of clamped classes. This means that in Java, using code can simply perform new BarClamp(x, y, ...); in Python, BarClamp(x, y, ...).

• Provide basic support for annotations.

• Annotation magic. It would be nice to import annotations into Python, use as class decorators and function decorators, and then still compile a Java class that works.

• Instance fields support, comparable to __slots__, but baked into the emitted Java class. Such support would directly enable emitted clases to be used as POJOs by using Java code. Clamp should use __slots__ if available. However, without further information, this would mean emitting fields of type Object. So there should be also some way of constraining the types of emitted instance fields in ClampProxyMaker. Likely this should be as simple as a new slots keyword when creating a proxymaker that simply maps fields to Java types.

• Map Python descriptors to Java's convention of getters/setters. Note that __delete__ is not a mappable idea!

• Add support for resolving external jars with Maven.

• Standalone jar support in Jython itself does not currently support .pth files and consequently site-packages. Clamp works around this by packaging everything in Lib/, but this is not desirable due to possible collisions. This means the possibility of subtle changes in class loader resolution, compared to what Jython offers with sys.path.

  Moreover, it would be nice if jars in site-packages could simply be included directly without unpacking.

• The singlejar command should generate Jython cache info on all included files and bundle in the generated uber jar. It's not clear how readily this precaching can be done on a per-jar basis with build_jar, but cache data is per jar; see {python.cachedir}/packages/*.pkc; the corresponding code in Jython's internals is in org.python.core.packagecache.

• Testing and placement in PyPI. Due to the bytecode construction, writing unit tests for this type of functionality seems to be nontrivial, but still very much needed to move this from an initial spike to not being in a pre-alpha stage.

Known issues

It's not feasible to use __new__ in your Python classes that are clamped. Why not? Java expects that constructing an object for a given class returns an object of that class! The solution is simple: call a factory function, in Python or Java, to return arbitrary objects. This is just a simple, but fundamental, mismatch between Python and Java in its object model.

A related issue is that you cannot change the __class__ of an instance of clamped class.

Variadic constructors

Clamp currently supports no-arg constructors of clamped classes, as seen in the generated code below for a Jython proxy:

    public BarClamp() {
        super();
        this.__initProxy__(Py.EmptyObjects);
    }

Note that it should be a simple matter to add variadic constructors, eg BarClamp(Object... args), by using the underlying support in __initProxy__, also generated in Jython proxies:

    public void __initProxy__(final Object[] array) {
        Py.initProxy((PyProxy)this, "clamped", "BarClamp", array);
    }

This should be as simple as using ClassFile.addMethod to generate the following code:

    public BarClamp(Object[] args) {
        super(args);
        this.__initProxy__(args);
    }

__initProxy__ will in turn take care of boxing any args as PyObject args.

Supporting Java annotations

Java annotations are widely used in contemporary Java code. Following an example in the Apache Quartz documentation, in Quartz one might write the following in Java:

@PersistJobDataAfterExecution
@DisallowConcurrentExecution
public class ColorJob implements Job {
    ...
}

Compiled usage of such annotations is very simple: they simply are part of the metadata of the class. As metadata, they are then used for metaprogramming at the Java level, eg, to support introspection or bytecode rewriting.

It would seem that class decorators would be the natural analogue to writing this in Jython:

@PersistJobDataAfterExecution
@DisallowConcurrentExecution
class ColorJob(Job):
    ...

But there are a few problems. First, Java annotations are interfaces. To solve, clamp can support a module, let's call it clamp.magic, which when imported, will intercept any subsequent imports of Java class/method annotations and turn them into class decorators/function decorators. This requires the top-level script of clamp.magic to insert an appropriate meta importer to sys.meta_path, as described in PEP 302.

Next, class decorators are applied after type construction in Python. The solution is for such class decorators to transform (rewrite) the bytecode for generated Java class to add any desired annotations, then save it under the original class name. Such transformations can be readily done with the ASM package by using an AnnotationVisitor, as documented in section 4.2 of the ASM user guide.

Lastly, saving under the original class name requires a little more work, because currently all generated classes in Clamp are directly written using JarOutputStream; simply resaving will result in a ZipException of "duplicate entry". This simply requires deferring the write of a module, including any supporting Java classes, until the top-level script of the module has completed.

Mapping method annotations to function decorators should likewise be straightforward. Field annotations currently would only correspond to static fields, which has direct support in Clamp - there's no Python syntax equivalent.