diff --git a/README.txt b/README.txt index bd67551..cc286dc 100644 --- a/README.txt +++ b/README.txt @@ -1,277 +1,277 @@ -Pyrolite - Python Remote Objects "light" and Pickle for Java/.NET - - Pyrolite is written by Irmen de Jong (irmen@razorvine.net). - This software is distributed under the terms written in the file `LICENSE`. - - -Contents: - 1. INTRODUCTION - 2. THE LIBRARY - 3. TYPE MAPPINGS - 4. EXCEPTIONS - 5. SECURITY WARNING - 6. RECOMMENDED DEPENDENCY: SERPENT SERIALIZER - 7. DOWNLOAD COMPILED BINARIES - - -1. INTRODUCTION ---------------------- - -This library allows your Java or .NET program to interface very easily with -the Python world. It uses the Pyro protocol to call methods on remote objects. -(See https://github.com/irmen/Pyro4). Pyrolite contains and uses a feature -complete pickle protocol implementation to exchange data with Pyro/Python. - -Pyrolite only implements part of the client side Pyro library, hence its name -'lite'... But because Pyrolite has no dependencies, it is a much lighter way -to use Pyro from Java/.NET than a solution with jython+pyro or IronPython+Pyro -would provide. So if you don't need Pyro's full feature set, and don't require -your Java/.NET code to host Pyro objects itself, Pyrolite may be a good choice -to connect java or .NET and python. - - -Java packages: net.razorvine.pickle, net.razorvine.pyro -.NET namespaces: Razorvine.Pickle, Razorvine.Pyro - -Small piece of example code in Java: - - import net.razorvine.pyro.*; - - NameServerProxy ns = NameServerProxy.locateNS(null); - PyroProxy remoteobject = new PyroProxy(ns.lookup("Your.Pyro.Object")); - Object result = remoteobject.call("pythonmethod", 42, "hello", new int[]{1,2,3}); - String message = (String)result; // cast to the type that 'pythonmethod' returns - System.out.println("result message="+message); - remoteobject.close(); - ns.close(); - -Same piece of example code in C#: - - using Razorvine.Pyro; - - using( NameServerProxy ns = NameServerProxy.locateNS(null) ) - { - using( PyroProxy something = new PyroProxy(ns.lookup("Your.Pyro.Object")) ) - { - object result = something.call("pythonmethod", 42, "hello", new int[]{1,2,3}); - string message = (string)result; // cast to the type that 'pythonmethod' returns - Console.WriteLine("result message="+message); - } - } - -More examples can be found in the examples directory. You could also study the -unit tests. These include a lot of code dealing with just the pickle subsystem -as well. - - -2. THE LIBRARY ---------------------- - -The library consists of 2 parts: a thin version of the client side part of -Pyro, and a feature complete implementation of Python's pickle protocol, -including memoization. It is fully compatible with pickles from Python 2.x and -Python 3.x, and you can use it idependently from the rest of the library, to -read and write Python pickle structures. - -Pickle protocol version support: -Pyrolite can read all pickle protocol versions (1 to 4, so this includes -the latest additions made in Python 3.4). -Pyrolite always writes pickles in protocol version 2. There are no plans on -including protocol version 1 support. Protocols 3 and 4 contain some nice new -features which may eventually be utilized, but for now, only version 2 is used. - - -The source archive contains the full source, and also unit test code and a -couple of example programs in the java/test/ directory. - -Pyrolite speaks Pyro4 protocol version 46 only (Pyro 4.22 and newer). -The java library requires java 1.6 or newer. -The .net library requires .net runtime 3.5. -The Java source was developed using Eclipse. -The C#/.NET source was developed using mono, monodevelop and sharpdevelop. - - - -3. TYPE MAPPINGS ---------------------- - -Pyrolite does the following type mappings: - -PYTHON ----> JAVA ------- ---- -None null -bool boolean -int int -long long or BigInteger (depending on size) -string String -unicode String -complex net.razorvine.pickle.objects.ComplexNumber -datetime.date java.util.Calendar -datetime.datetime java.util.Calendar -datetime.time java.util.Calendar -datetime.timedelta net.razorvine.pickle.objects.TimeDelta -float double (float isn't used) -array.array array of appropriate primitive type (char, int, short, long, float, double) -list java.util.List -tuple Object[] -set java.util.Set -dict java.util.Map -bytes byte[] -bytearray byte[] -decimal BigDecimal -custom class Map (dict with class attributes including its name in "__class__") -Pyro4.core.URI net.razorvine.pyro.PyroURI -Pyro4.core.Proxy net.razorvine.pyro.PyroProxy -Pyro4.errors.* net.razorvine.pyro.PyroException -Pyro4.utils.flame.FlameBuiltin net.razorvine.pyro.FlameBuiltin -Pyro4.utils.flame.FlameModule net.razorvine.pyro.FlameModule -Pyro4.utils.flame.RemoteInteractiveConsole net.razorvine.pyro.FlameRemoteConsole - -The unpickler simply returns an Object. Because Java is a statically typed -language you will have to cast that to the appropriate type. Refer to this -table to see what you can expect to receive. - - -JAVA ----> PYTHON ------ ------ -null None -boolean bool -byte int -char str/unicode (length 1) -String str/unicode -double float -float float -int int -short int -BigDecimal decimal -BigInteger long -any array array if elements are primitive type (else tuple) -Object[] tuple (cannot contain self-references) -byte[] bytearray -java.util.Date datetime.datetime -java.util.Calendar datetime.datetime -Enum the enum value as string -java.util.Set set -Map, Hashtable dict -Vector, Collection list -Serializable treated as a JavaBean, see below. -JavaBean dict of the bean's public properties + __class__ for the bean's type. -net.razorvine.pyro.PyroURI Pyro4.core.URI -net.razorvine.pyro.PyroProxy cannot be pickled. - - -PYTHON ----> C# ------- ---- -None null -bool bool -int int -long long (c# doesn't have BigInteger so there's a limit on the size) -string string -unicode string -complex Razorvine.Pickle.Objects.ComplexNumber -datetime.date DateTime -datetime.datetime DateTime -datetime.time TimeSpan -datetime.timedelta TimeSpan -float double -array.array array (all kinds of element types supported) -list ArrayList (of objects) -tuple object[] -set HashSet -dict Hashtable (key=object, value=object) -bytes ubyte[] -bytearray ubyte[] -decimal decimal -custom class IDictionary (dict with class attributes including its name in "__class__") -Pyro4.core.URI Razorvine.Pyro.PyroURI -Pyro4.core.Proxy Razorvine.Pyro.PyroProxy -Pyro4.errors.* Razorvine.Pyro.PyroException -Pyro4.utils.flame.FlameBuiltin Razorvine.Pyro.FlameBuiltin -Pyro4.utils.flame.FlameModule Razorvine.Pyro.FlameModule -Pyro4.utils.flame.RemoteInteractiveConsole Razorvine.Pyro.FlameRemoteConsole - -The unpickler simply returns an object. Because C# is a statically typed -language you will have to cast that to the appropriate type. Refer to this -table to see what you can expect to receive. TIP: if you are using C# 4.0 you -can use the 'dynamic' type in some places to avoid excessive type casting. - - - C# ----> PYTHON ------- ------- -null None -boolean bool -byte byte -sbyte int -char str/unicode (length 1) -string str/unicode -double float -float float -int/short/sbyte int -uint/ushort/byte int -decimal decimal -byte[] bytearray -primitivetype[] array -object[] tuple (cannot contain self-references) -DateTime datetime.datetime -TimeSpan datetime.timedelta -Enum just the enum value as string -HashSet set -Map, Hashtable dict -Collection list -Enumerable list -object with public properties dictionary of those properties + __class__ -anonymous class type dictonary of the public properties -Razorvine.Pyro.PyroURI Pyro4.core.URI -Razorvine.Pyro.PyroProxy cannot be pickled. - - -4. EXCEPTIONS ---------------------- - -Pyrolite also maps Python exceptions that may occur in the remote object. It -has a rather simplistic approach: - -*all* exceptions, including the Pyro ones (Pyro4.errors.*), are converted to -PyroException objects. PyroException is a normal Java or C# exception type, -and it will be thrown as a normal exception in your program. The message -string is taken from the original exception. The remote traceback string is -available on the PyroException object in the _pyroTraceback field. - - -5. SECURITY WARNING ---------------------- - -If you use Pyrolite to talk to a Pyro server it will use pickle as -serialization protocol. THIS MEANS YOUR PYRO SERVER CAN BE VULNERABLE TO -REMOTE ARBITRARY CODE EXECUTION (because of the well known security problem -with the pickle protocol). - -The current version of Pyrolite is only able to talk to Pyro when using the -pickle protocol. Because pickle is not enabled by default in recent Pyro -versions, you will have to configure Pyro to allow the use of pickle. See the -Pyro documentation on how to do this. A future Pyrolite version may improve -this by allowing other serializers. - -Note: your .NET or Java client code is perfectly safe. The unpickler -implementation in Pyrolite doesn't randomly construct arbitrary objects and is -safe to use for parsing data from the network. - - -6. RECOMMENDED DEPENDENCY FOR PYRO: SERPENT SERIALIZER ------------------------------------------------------- - -The default serializer is set to serpent. Unless you change the configuration -to use pickle instead, Pyrolite will require the Razorvine.Serpent assembly or -the serpent jar to be available. If you do not supply this library, Pyrolite -will still work but only with the built-in pickle serializer. Serpent is a -separate project, and the library is not included in the Pyrolite project. - -You can find the Serpent project at: https://github.com/irmen/Serpent -You need version 1.5 of Serpent, or newer. - - -7. DOWNLOAD COMPILED BINARIES ------------------------------ - -Precompiled binaries (java jar, .net assembly dll) are here: -http://irmen.home.xs4all.nl/pyrolite/ +Pyrolite - Python Remote Objects "light" and Pickle for Java/.NET + + Pyrolite is written by Irmen de Jong (irmen@razorvine.net). + This software is distributed under the terms written in the file `LICENSE`. + + +Contents: + 1. INTRODUCTION + 2. THE LIBRARY + 3. TYPE MAPPINGS + 4. EXCEPTIONS + 5. SECURITY WARNING + 6. RECOMMENDED DEPENDENCY: SERPENT SERIALIZER + 7. DOWNLOAD COMPILED BINARIES + + +1. INTRODUCTION +--------------------- + +This library allows your Java or .NET program to interface very easily with +the Python world. It uses the Pyro protocol to call methods on remote objects. +(See https://github.com/irmen/Pyro4). Pyrolite contains and uses a feature +complete pickle protocol implementation to exchange data with Pyro/Python. + +Pyrolite only implements part of the client side Pyro library, hence its name +'lite'... But because Pyrolite has no dependencies, it is a much lighter way +to use Pyro from Java/.NET than a solution with jython+pyro or IronPython+Pyro +would provide. So if you don't need Pyro's full feature set, and don't require +your Java/.NET code to host Pyro objects itself, Pyrolite may be a good choice +to connect java or .NET and python. + + +Java packages: net.razorvine.pickle, net.razorvine.pyro +.NET namespaces: Razorvine.Pickle, Razorvine.Pyro + +Small piece of example code in Java: + + import net.razorvine.pyro.*; + + NameServerProxy ns = NameServerProxy.locateNS(null); + PyroProxy remoteobject = new PyroProxy(ns.lookup("Your.Pyro.Object")); + Object result = remoteobject.call("pythonmethod", 42, "hello", new int[]{1,2,3}); + String message = (String)result; // cast to the type that 'pythonmethod' returns + System.out.println("result message="+message); + remoteobject.close(); + ns.close(); + +Same piece of example code in C#: + + using Razorvine.Pyro; + + using( NameServerProxy ns = NameServerProxy.locateNS(null) ) + { + using( PyroProxy something = new PyroProxy(ns.lookup("Your.Pyro.Object")) ) + { + object result = something.call("pythonmethod", 42, "hello", new int[]{1,2,3}); + string message = (string)result; // cast to the type that 'pythonmethod' returns + Console.WriteLine("result message="+message); + } + } + +More examples can be found in the examples directory. You could also study the +unit tests. These include a lot of code dealing with just the pickle subsystem +as well. + + +2. THE LIBRARY +--------------------- + +The library consists of 2 parts: a thin version of the client side part of +Pyro, and a feature complete implementation of Python's pickle protocol, +including memoization. It is fully compatible with pickles from Python 2.x and +Python 3.x, and you can use it idependently from the rest of the library, to +read and write Python pickle structures. + +Pickle protocol version support: +Pyrolite can read all pickle protocol versions (1 to 4, so this includes +the latest additions made in Python 3.4). +Pyrolite always writes pickles in protocol version 2. There are no plans on +including protocol version 1 support. Protocols 3 and 4 contain some nice new +features which may eventually be utilized, but for now, only version 2 is used. + + +The source archive contains the full source, and also unit test code and a +couple of example programs in the java/test/ directory. + +Pyrolite speaks Pyro4 protocol version 47 only (Pyro 4.26). +The java library requires java 1.6 or newer. +The .net library requires .net runtime 3.5. +The Java source was developed using Eclipse. +The C#/.NET source was developed using mono, monodevelop and sharpdevelop. + + + +3. TYPE MAPPINGS +--------------------- + +Pyrolite does the following type mappings: + +PYTHON ----> JAVA +------ ---- +None null +bool boolean +int int +long long or BigInteger (depending on size) +string String +unicode String +complex net.razorvine.pickle.objects.ComplexNumber +datetime.date java.util.Calendar +datetime.datetime java.util.Calendar +datetime.time java.util.Calendar +datetime.timedelta net.razorvine.pickle.objects.TimeDelta +float double (float isn't used) +array.array array of appropriate primitive type (char, int, short, long, float, double) +list java.util.List +tuple Object[] +set java.util.Set +dict java.util.Map +bytes byte[] +bytearray byte[] +decimal BigDecimal +custom class Map (dict with class attributes including its name in "__class__") +Pyro4.core.URI net.razorvine.pyro.PyroURI +Pyro4.core.Proxy net.razorvine.pyro.PyroProxy +Pyro4.errors.* net.razorvine.pyro.PyroException +Pyro4.utils.flame.FlameBuiltin net.razorvine.pyro.FlameBuiltin +Pyro4.utils.flame.FlameModule net.razorvine.pyro.FlameModule +Pyro4.utils.flame.RemoteInteractiveConsole net.razorvine.pyro.FlameRemoteConsole + +The unpickler simply returns an Object. Because Java is a statically typed +language you will have to cast that to the appropriate type. Refer to this +table to see what you can expect to receive. + + +JAVA ----> PYTHON +----- ------ +null None +boolean bool +byte int +char str/unicode (length 1) +String str/unicode +double float +float float +int int +short int +BigDecimal decimal +BigInteger long +any array array if elements are primitive type (else tuple) +Object[] tuple (cannot contain self-references) +byte[] bytearray +java.util.Date datetime.datetime +java.util.Calendar datetime.datetime +Enum the enum value as string +java.util.Set set +Map, Hashtable dict +Vector, Collection list +Serializable treated as a JavaBean, see below. +JavaBean dict of the bean's public properties + __class__ for the bean's type. +net.razorvine.pyro.PyroURI Pyro4.core.URI +net.razorvine.pyro.PyroProxy cannot be pickled. + + +PYTHON ----> C# +------ ---- +None null +bool bool +int int +long long (c# doesn't have BigInteger so there's a limit on the size) +string string +unicode string +complex Razorvine.Pickle.Objects.ComplexNumber +datetime.date DateTime +datetime.datetime DateTime +datetime.time TimeSpan +datetime.timedelta TimeSpan +float double +array.array array (all kinds of element types supported) +list ArrayList (of objects) +tuple object[] +set HashSet +dict Hashtable (key=object, value=object) +bytes ubyte[] +bytearray ubyte[] +decimal decimal +custom class IDictionary (dict with class attributes including its name in "__class__") +Pyro4.core.URI Razorvine.Pyro.PyroURI +Pyro4.core.Proxy Razorvine.Pyro.PyroProxy +Pyro4.errors.* Razorvine.Pyro.PyroException +Pyro4.utils.flame.FlameBuiltin Razorvine.Pyro.FlameBuiltin +Pyro4.utils.flame.FlameModule Razorvine.Pyro.FlameModule +Pyro4.utils.flame.RemoteInteractiveConsole Razorvine.Pyro.FlameRemoteConsole + +The unpickler simply returns an object. Because C# is a statically typed +language you will have to cast that to the appropriate type. Refer to this +table to see what you can expect to receive. TIP: if you are using C# 4.0 you +can use the 'dynamic' type in some places to avoid excessive type casting. + + + C# ----> PYTHON +------ ------- +null None +boolean bool +byte byte +sbyte int +char str/unicode (length 1) +string str/unicode +double float +float float +int/short/sbyte int +uint/ushort/byte int +decimal decimal +byte[] bytearray +primitivetype[] array +object[] tuple (cannot contain self-references) +DateTime datetime.datetime +TimeSpan datetime.timedelta +Enum just the enum value as string +HashSet set +Map, Hashtable dict +Collection list +Enumerable list +object with public properties dictionary of those properties + __class__ +anonymous class type dictonary of the public properties +Razorvine.Pyro.PyroURI Pyro4.core.URI +Razorvine.Pyro.PyroProxy cannot be pickled. + + +4. EXCEPTIONS +--------------------- + +Pyrolite also maps Python exceptions that may occur in the remote object. It +has a rather simplistic approach: + +*all* exceptions, including the Pyro ones (Pyro4.errors.*), are converted to +PyroException objects. PyroException is a normal Java or C# exception type, +and it will be thrown as a normal exception in your program. The message +string is taken from the original exception. The remote traceback string is +available on the PyroException object in the _pyroTraceback field. + + +5. SECURITY WARNING +--------------------- + +If you use Pyrolite to talk to a Pyro server it will use pickle as +serialization protocol. THIS MEANS YOUR PYRO SERVER CAN BE VULNERABLE TO +REMOTE ARBITRARY CODE EXECUTION (because of the well known security problem +with the pickle protocol). + +The current version of Pyrolite is only able to talk to Pyro when using the +pickle protocol. Because pickle is not enabled by default in recent Pyro +versions, you will have to configure Pyro to allow the use of pickle. See the +Pyro documentation on how to do this. A future Pyrolite version may improve +this by allowing other serializers. + +Note: your .NET or Java client code is perfectly safe. The unpickler +implementation in Pyrolite doesn't randomly construct arbitrary objects and is +safe to use for parsing data from the network. + + +6. RECOMMENDED DEPENDENCY FOR PYRO: SERPENT SERIALIZER +------------------------------------------------------ + +The default serializer is set to serpent. Unless you change the configuration +to use pickle instead, Pyrolite will require the Razorvine.Serpent assembly or +the serpent jar to be available. If you do not supply this library, Pyrolite +will still work but only with the built-in pickle serializer. Serpent is a +separate project, and the library is not included in the Pyrolite project. + +You can find the Serpent project at: https://github.com/irmen/Serpent +You need version 1.5 of Serpent, or newer. + + +7. DOWNLOAD COMPILED BINARIES +----------------------------- + +Precompiled binaries (java jar, .net assembly dll) are here: +http://irmen.home.xs4all.nl/pyrolite/ diff --git a/dotnet/Pyrolite.Tests/Pyro/MessageTests.cs b/dotnet/Pyrolite.Tests/Pyro/MessageTests.cs index 0c8d7e4..65f3afa 100644 --- a/dotnet/Pyrolite.Tests/Pyro/MessageTests.cs +++ b/dotnet/Pyrolite.Tests/Pyro/MessageTests.cs @@ -198,14 +198,35 @@ public void testRecvAnnotations() } [Test] - [ExpectedException(typeof(PyroException), ExpectedMessage="invalid protocol version: 25390")] - public void testProtocolVersion() + [ExpectedException(typeof(PyroException), ExpectedMessage="invalid protocol version: 25455")] + public void testProtocolVersionKaputt() { byte[] msg = new Message(Message.MSG_RESULT, new byte[0], this.serializer_id, 0, 1, null).to_bytes().Take(Message.HEADER_SIZE).ToArray(); msg[4] = 99; // screw up protocol version in message header + msg[5] = 111; // screw up protocol version in message header Message.from_header(msg); } + [Test] + [ExpectedException(typeof(PyroException), ExpectedMessage="invalid protocol version: 46")] + public void testProtocolVersionsNotSupported1() + { + byte[] msg = new Message(Message.MSG_RESULT, new byte[0], this.serializer_id, 0, 1, null).to_bytes().Take(Message.HEADER_SIZE).ToArray(); + msg[4] = 0; + msg[5] = 46; + Message.from_header(msg); + } + + [Test] + [ExpectedException(typeof(PyroException), ExpectedMessage="invalid protocol version: 48")] + public void testProtocolVersionsNotSupported2() + { + byte[] msg = new Message(Message.MSG_RESULT, new byte[0], this.serializer_id, 0, 1, null).to_bytes().Take(Message.HEADER_SIZE).ToArray(); + msg[4] = 0; + msg[5] = 48; + Message.from_header(msg); + } + [Test] public void testHmac() { diff --git a/dotnet/Pyrolite/Properties/AssemblyInfo.cs b/dotnet/Pyrolite/Properties/AssemblyInfo.cs index 1fce8ce..f778808 100644 --- a/dotnet/Pyrolite/Properties/AssemblyInfo.cs +++ b/dotnet/Pyrolite/Properties/AssemblyInfo.cs @@ -28,4 +28,4 @@ // // You can specify all the values or you can use the default the Revision and // Build Numbers by using the '*' as shown below: -[assembly: AssemblyVersion("3.0.*")] +[assembly: AssemblyVersion("3.1.*")] diff --git a/dotnet/Pyrolite/Pyro/Config.cs b/dotnet/Pyrolite/Pyro/Config.cs index 787b498..275024b 100644 --- a/dotnet/Pyrolite/Pyro/Config.cs +++ b/dotnet/Pyrolite/Pyro/Config.cs @@ -21,8 +21,8 @@ public enum SerializerType { public static bool SERPENT_SET_LITERALS = false; // set to true if talking to Python 3.2 or newer public static SerializerType SERIALIZER = SerializerType.serpent; - public const int PROTOCOL_VERSION = 46; // Pyro 4.22 and newer, cannot be modified - public const string PYROLITE_VERSION="3.0"; + public const int PROTOCOL_VERSION = 47; // Pyro 4.26 + public const string PYROLITE_VERSION="3.1"; } } diff --git a/java/src/net/razorvine/pickle/Unpickler.java b/java/src/net/razorvine/pickle/Unpickler.java index cbf0ff1..e627247 100644 --- a/java/src/net/razorvine/pickle/Unpickler.java +++ b/java/src/net/razorvine/pickle/Unpickler.java @@ -1,723 +1,722 @@ -package net.razorvine.pickle; - -import java.io.ByteArrayInputStream; -import java.io.IOException; -import java.io.InputStream; -import java.lang.reflect.Method; -import java.math.BigDecimal; -import java.math.BigInteger; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Hashtable; -import java.util.Map; - -import net.razorvine.pickle.objects.AnyClassConstructor; -import net.razorvine.pickle.objects.ArrayConstructor; -import net.razorvine.pickle.objects.ByteArrayConstructor; -import net.razorvine.pickle.objects.ClassDictConstructor; -import net.razorvine.pickle.objects.ComplexNumber; -import net.razorvine.pickle.objects.DateTimeConstructor; -import net.razorvine.pickle.objects.SetConstructor; - -/** - * Unpickles an object graph from a pickle data inputstream. Supports all pickle protocol versions. - * Maps the python objects on the corresponding java equivalents or similar types. - * This class is NOT threadsafe! (Don't use the same pickler from different threads) - * - * See the README.txt for a table of the type mappings. - * - * @author Irmen de Jong (irmen@razorvine.net) - */ -public class Unpickler { - - private final int HIGHEST_PROTOCOL = 4; - - private Map memo; - protected UnpickleStack stack; - private InputStream input; - private static Map objectConstructors; - - static { - objectConstructors = new HashMap(); - objectConstructors.put("__builtin__.complex", new AnyClassConstructor(ComplexNumber.class)); - objectConstructors.put("builtins.complex", new AnyClassConstructor(ComplexNumber.class)); - objectConstructors.put("array.array", new ArrayConstructor()); - objectConstructors.put("array._array_reconstructor", new ArrayConstructor()); - objectConstructors.put("__builtin__.bytearray", new ByteArrayConstructor()); - objectConstructors.put("builtins.bytearray", new ByteArrayConstructor()); - objectConstructors.put("__builtin__.bytes", new ByteArrayConstructor()); - objectConstructors.put("__builtin__.set", new SetConstructor()); - objectConstructors.put("builtins.set", new SetConstructor()); - objectConstructors.put("datetime.datetime", new DateTimeConstructor(DateTimeConstructor.DATETIME)); - objectConstructors.put("datetime.time", new DateTimeConstructor(DateTimeConstructor.TIME)); - objectConstructors.put("datetime.date", new DateTimeConstructor(DateTimeConstructor.DATE)); - objectConstructors.put("datetime.timedelta", new DateTimeConstructor(DateTimeConstructor.TIMEDELTA)); - objectConstructors.put("decimal.Decimal", new AnyClassConstructor(BigDecimal.class)); - } - - /** - * Create an unpickler. - */ - public Unpickler() { - memo = new HashMap(); - } - - /** - * Register additional object constructors for custom classes. - */ - public static void registerConstructor(String module, String classname, IObjectConstructor constructor) { - objectConstructors.put(module + "." + classname, constructor); - } - - /** - * Read a pickled object representation from the given input stream. - * - * @return the reconstituted object hierarchy specified in the file. - */ - public Object load(InputStream stream) throws PickleException, IOException { - stack = new UnpickleStack(); - input = stream; - try { - while (true) { - short key = PickleUtils.readbyte(input); - if (key == -1) - throw new IOException("premature end of file"); - dispatch(key); - } - } catch (StopException x) { - return x.value; - } - } - - /** - * Read a pickled object representation from the given pickle data bytes. - * - * @return the reconstituted object hierarchy specified in the file. - */ - public Object loads(byte[] pickledata) throws PickleException, IOException { - return load(new ByteArrayInputStream(pickledata)); - } - - /** - * Close the unpickler and frees the resources such as the unpickle stack and memo table. - */ - public void close() { - if(stack!=null) stack.clear(); - if(memo!=null) memo.clear(); - if(input!=null) - try { - input.close(); - } catch (IOException e) { - } - } - - private class StopException extends RuntimeException { - private static final long serialVersionUID = 6528222454688362873L; - - public StopException(Object value) { - this.value = value; - } - - public Object value; - } - - /** - * Process a single pickle stream opcode. - */ - protected void dispatch(short key) throws PickleException, IOException { - switch (key) { - case Opcodes.MARK: - load_mark(); - break; - case Opcodes.STOP: - Object value = stack.pop(); - stack.clear(); - throw new StopException(value); - case Opcodes.POP: - load_pop(); - break; - case Opcodes.POP_MARK: - load_pop_mark(); - break; - case Opcodes.DUP: - load_dup(); - break; - case Opcodes.FLOAT: - load_float(); - break; - case Opcodes.INT: - load_int(); - break; - case Opcodes.BININT: - load_binint(); - break; - case Opcodes.BININT1: - load_binint1(); - break; - case Opcodes.LONG: - load_long(); - break; - case Opcodes.BININT2: - load_binint2(); - break; - case Opcodes.NONE: - load_none(); - break; - case Opcodes.PERSID: - throw new InvalidOpcodeException("opcode not implemented: PERSID"); - case Opcodes.BINPERSID: - throw new InvalidOpcodeException("opcode not implemented: BINPERSID"); - case Opcodes.REDUCE: - load_reduce(); - break; - case Opcodes.STRING: - load_string(); - break; - case Opcodes.BINSTRING: - load_binstring(); - break; - case Opcodes.SHORT_BINSTRING: - load_short_binstring(); - break; - case Opcodes.UNICODE: - load_unicode(); - break; - case Opcodes.BINUNICODE: - load_binunicode(); - break; - case Opcodes.APPEND: - load_append(); - break; - case Opcodes.BUILD: - load_build(); - break; - case Opcodes.GLOBAL: - load_global(); - break; - case Opcodes.DICT: - load_dict(); - break; - case Opcodes.EMPTY_DICT: - load_empty_dictionary(); - break; - case Opcodes.APPENDS: - load_appends(); - break; - case Opcodes.GET: - load_get(); - break; - case Opcodes.BINGET: - load_binget(); - break; - case Opcodes.INST: - throw new InvalidOpcodeException("opcode not implemented: INST"); - case Opcodes.LONG_BINGET: - load_long_binget(); - break; - case Opcodes.LIST: - load_list(); - break; - case Opcodes.EMPTY_LIST: - load_empty_list(); - break; - case Opcodes.OBJ: - throw new InvalidOpcodeException("opcode not implemented: OBJ"); - case Opcodes.PUT: - load_put(); - break; - case Opcodes.BINPUT: - load_binput(); - break; - case Opcodes.LONG_BINPUT: - load_long_binput(); - break; - case Opcodes.SETITEM: - load_setitem(); - break; - case Opcodes.TUPLE: - load_tuple(); - break; - case Opcodes.EMPTY_TUPLE: - load_empty_tuple(); - break; - case Opcodes.SETITEMS: - load_setitems(); - break; - case Opcodes.BINFLOAT: - load_binfloat(); - break; - - // protocol 2 - case Opcodes.PROTO: - load_proto(); - break; - case Opcodes.NEWOBJ: - load_newobj(); - break; - case Opcodes.EXT1: - throw new InvalidOpcodeException("opcode not implemented: EXT1"); - case Opcodes.EXT2: - throw new InvalidOpcodeException("opcode not implemented: EXT2"); - case Opcodes.EXT4: - throw new InvalidOpcodeException("opcode not implemented: EXT4"); - case Opcodes.TUPLE1: - load_tuple1(); - break; - case Opcodes.TUPLE2: - load_tuple2(); - break; - case Opcodes.TUPLE3: - load_tuple3(); - break; - case Opcodes.NEWTRUE: - load_true(); - break; - case Opcodes.NEWFALSE: - load_false(); - break; - case Opcodes.LONG1: - load_long1(); - break; - case Opcodes.LONG4: - load_long4(); - break; - - // Protocol 3 (Python 3.x) - case Opcodes.BINBYTES: - load_binbytes(); - break; - case Opcodes.SHORT_BINBYTES: - load_short_binbytes(); - break; - - // Protocol 4 (Python 3.4+) - case Opcodes.BINUNICODE8: - load_binunicode8(); - break; - case Opcodes.SHORT_BINUNICODE: - load_short_binunicode(); - break; - case Opcodes.BINBYTES8: - load_binbytes8(); - break; - case Opcodes.EMPTY_SET: - load_empty_set(); - break; - case Opcodes.ADDITEMS: - load_additems(); - break; - case Opcodes.FROZENSET: - load_frozenset(); - break; - case Opcodes.MEMOIZE: - load_memoize(); - break; - case Opcodes.FRAME: - load_frame(); - break; - case Opcodes.NEWOBJ_EX: - load_newobj_ex(); - break; - case Opcodes.STACK_GLOBAL: - load_stack_global(); - break; - - default: - throw new InvalidOpcodeException("invalid pickle opcode: " + key); - } - } - - void load_build() { - Object args=stack.pop(); - Object target=stack.peek(); - try { - Method setStateMethod=target.getClass().getMethod("__setstate__", args.getClass()); - setStateMethod.invoke(target, args); - } catch (Exception e) { - throw new PickleException("failed to __setstate__()",e); - } - } - - void load_proto() throws IOException { - short proto = PickleUtils.readbyte(input); - if (proto < 0 || proto > HIGHEST_PROTOCOL) - throw new PickleException("unsupported pickle protocol: " + proto); - } - - void load_none() { - stack.add(null); - } - - void load_false() { - stack.add(false); - } - - void load_true() { - stack.add(true); - } - - void load_int() throws IOException { - String data = PickleUtils.readline(input, true); - Object val; - if (data.equals(Opcodes.FALSE.substring(1))) - val = false; - else if (data.equals(Opcodes.TRUE.substring(1))) - val = true; - else { - String number=data.substring(0, data.length() - 1); - try { - val = Integer.parseInt(number, 10); - } catch (NumberFormatException x) { - // hmm, integer didn't work.. is it perhaps an int from a 64-bit python? so try long: - val = Long.parseLong(number, 10); - } - } - stack.add(val); - } - - void load_binint() throws IOException { - int integer = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); - stack.add(integer); - } - - void load_binint1() throws IOException { - stack.add((int)PickleUtils.readbyte(input)); - } - - void load_binint2() throws IOException { - int integer = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 2)); - stack.add(integer); - } - - void load_long() throws IOException { - String val = PickleUtils.readline(input); - if (val != null && val.endsWith("L")) { - val = val.substring(0, val.length() - 1); - } - BigInteger bi = new BigInteger(val); - stack.add(PickleUtils.optimizeBigint(bi)); - } - - void load_long1() throws IOException { - short n = PickleUtils.readbyte(input); - byte[] data = PickleUtils.readbytes(input, n); - stack.add(PickleUtils.decode_long(data)); - } - - void load_long4() throws IOException { - int n = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); - byte[] data = PickleUtils.readbytes(input, n); - stack.add(PickleUtils.decode_long(data)); - } - - void load_float() throws IOException { - String val = PickleUtils.readline(input, true); - stack.add(Double.parseDouble(val)); - } - - void load_binfloat() throws IOException { - double val = PickleUtils.bytes_to_double(PickleUtils.readbytes(input, 8),0); - stack.add(val); - } - - void load_string() throws IOException { - String rep = PickleUtils.readline(input); - boolean quotesOk = false; - for (String q : new String[] { "\"", "'" }) // double or single quote - { - if (rep.startsWith(q)) { - if (!rep.endsWith(q)) { - throw new PickleException("insecure string pickle"); - } - rep = rep.substring(1, rep.length() - 1); // strip quotes - quotesOk = true; - break; - } - } - - if (!quotesOk) - throw new PickleException("insecure string pickle"); - - stack.add(PickleUtils.decode_escaped(rep)); - } - - void load_binstring() throws IOException { - int len = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); - byte[] data = PickleUtils.readbytes(input, len); - stack.add(PickleUtils.rawStringFromBytes(data)); - } - - void load_binbytes() throws IOException { - int len = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); - stack.add(PickleUtils.readbytes(input, len)); - } - - void load_binbytes8() throws IOException { - long len = PickleUtils.bytes_to_long(PickleUtils.readbytes(input, 8),0); - stack.add(PickleUtils.readbytes(input, len)); - } - - void load_unicode() throws IOException { - String str=PickleUtils.decode_unicode_escaped(PickleUtils.readline(input)); - stack.add(str); - } - - void load_binunicode() throws IOException { - int len = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); - byte[] data = PickleUtils.readbytes(input, len); - stack.add(new String(data,"UTF-8")); - } - - void load_binunicode8() throws IOException { - long len = PickleUtils.bytes_to_long(PickleUtils.readbytes(input, 8),0); - byte[] data = PickleUtils.readbytes(input, len); - stack.add(new String(data,"UTF-8")); - } - - void load_short_binunicode() throws IOException { - int len = PickleUtils.readbyte(input); - byte[] data = PickleUtils.readbytes(input, len); - stack.add(new String(data,"UTF-8")); - } - - void load_short_binstring() throws IOException { - short len = PickleUtils.readbyte(input); - byte[] data = PickleUtils.readbytes(input, len); - stack.add(PickleUtils.rawStringFromBytes(data)); - } - - void load_short_binbytes() throws IOException { - short len = PickleUtils.readbyte(input); - stack.add(PickleUtils.readbytes(input, len)); - } - - void load_tuple() { - ArrayList top = stack.pop_all_since_marker(); - stack.add(top.toArray()); - } - - void load_empty_tuple() { - stack.add(new Object[0]); - } - - void load_tuple1() { - stack.add(new Object[] { stack.pop() }); - } - - void load_tuple2() { - Object o2 = stack.pop(); - Object o1 = stack.pop(); - stack.add(new Object[] { o1, o2 }); - } - - void load_tuple3() { - Object o3 = stack.pop(); - Object o2 = stack.pop(); - Object o1 = stack.pop(); - stack.add(new Object[] { o1, o2, o3 }); - } - - void load_empty_list() { - stack.add(new ArrayList(0)); - } - - void load_empty_dictionary() { - stack.add(new HashMap(0)); - } - - void load_empty_set() { - stack.add(new HashSet()); - } - - void load_list() { - ArrayList top = stack.pop_all_since_marker(); - stack.add(top); // simply add the top items as a list to the stack again - } - - void load_dict() { - ArrayList top = stack.pop_all_since_marker(); - HashMap map = new HashMap(top.size()); - for (int i = 0; i < top.size(); i += 2) { - Object key = top.get(i); - Object value = top.get(i + 1); - map.put(key, value); - } - stack.add(map); - } - - void load_frozenset() { - ArrayList top = stack.pop_all_since_marker(); - HashSet set = new HashSet(); - set.addAll(top); - stack.add(set); - } - - void load_additems() { - ArrayList top = stack.pop_all_since_marker(); - @SuppressWarnings("unchecked") - HashSet set = (HashSet) stack.pop(); - set.addAll(top); - stack.add(set); - } - - void load_global() throws IOException { - String module = PickleUtils.readline(input); - String name = PickleUtils.readline(input); - load_global_sub(module, name); - } - - void load_stack_global() { - String name = (String) stack.pop(); - String module = (String) stack.pop(); - load_global_sub(module, name); - } - - void load_global_sub(String module, String name) { - IObjectConstructor constructor = objectConstructors.get(module + "." + name); - if (constructor == null) { - // check if it is an exception - if(module.equals("exceptions")) { - // python 2.x - constructor=new AnyClassConstructor(PythonException.class); - } else if(module.equals("builtins") || module.equals("__builtin__")) { - if(name.endsWith("Error") || name.endsWith("Warning") || name.endsWith("Exception") - || name.equals("GeneratorExit") || name.equals("KeyboardInterrupt") - || name.equals("StopIteration") || name.equals("SystemExit")) - { - // it's a python 3.x exception - constructor=new AnyClassConstructor(PythonException.class); - } - else - { - // return a dictionary with the class's properties - constructor=new ClassDictConstructor(module, name); - } - } else { - // return a dictionary with the class's properties - constructor=new ClassDictConstructor(module, name); - } - } - stack.add(constructor); - } - - - void load_pop() { - stack.pop(); - } - - void load_pop_mark() { - Object o = null; - do { - o = stack.pop(); - } while (o != stack.MARKER); - stack.trim(); - } - - void load_dup() { - stack.add(stack.peek()); - } - - void load_get() throws IOException { - int i = Integer.parseInt(PickleUtils.readline(input), 10); - if(!memo.containsKey(i)) throw new PickleException("invalid memo key"); - stack.add(memo.get(i)); - } - - void load_binget() throws IOException { - int i = PickleUtils.readbyte(input); - if(!memo.containsKey(i)) throw new PickleException("invalid memo key"); - stack.add(memo.get(i)); - } - - void load_long_binget() throws IOException { - int i = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); - if(!memo.containsKey(i)) throw new PickleException("invalid memo key"); - stack.add(memo.get(i)); - } - - void load_put() throws IOException { - int i = Integer.parseInt(PickleUtils.readline(input), 10); - memo.put(i, stack.peek()); - } - - void load_binput() throws IOException { - int i = PickleUtils.readbyte(input); - memo.put(i, stack.peek()); - } - - void load_long_binput() throws IOException { - int i = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); - memo.put(i, stack.peek()); - } - - void load_memoize() { - memo.put(memo.size(), stack.peek()); - } - - void load_append() { - Object value = stack.pop(); - @SuppressWarnings("unchecked") - ArrayList list = (ArrayList) stack.peek(); - list.add(value); - } - - void load_appends() { - ArrayList top = stack.pop_all_since_marker(); - @SuppressWarnings("unchecked") - ArrayList list = (ArrayList) stack.peek(); - list.addAll(top); - list.trimToSize(); - } - - void load_setitem() { - Object value = stack.pop(); - Object key = stack.pop(); - @SuppressWarnings("unchecked") - Map dict = (Map) stack.peek(); - dict.put(key, value); - } - - void load_setitems() { - HashMap newitems = new HashMap(); - Object value = stack.pop(); - while (value != stack.MARKER) { - Object key = stack.pop(); - newitems.put(key, value); - value = stack.pop(); - } - - @SuppressWarnings("unchecked") - Map dict = (Map) stack.peek(); - dict.putAll(newitems); - } - - void load_mark() { - stack.add_mark(); - } - - void load_reduce() { - Object[] args = (Object[]) stack.pop(); - IObjectConstructor constructor = (IObjectConstructor) stack.pop(); - stack.add(constructor.construct(args)); - } - - void load_newobj() { - load_reduce(); // for Java we just do the same as class(*args) instead of class.__new__(class,*args) - } - - void load_newobj_ex() { - HashMap kwargs = (HashMap) stack.pop(); - Object[] args = (Object[]) stack.pop(); - IObjectConstructor constructor = (IObjectConstructor) stack.pop(); - if(kwargs.size()==0) - stack.add(constructor.construct(args)); - else - throw new PickleException("newobj_ex with keyword arguments not supported"); - } - - void load_frame() throws IOException { - // for now we simply skip the frame opcode and its length - PickleUtils.readbytes(input, 8); - } -} +package net.razorvine.pickle; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Method; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; + +import net.razorvine.pickle.objects.AnyClassConstructor; +import net.razorvine.pickle.objects.ArrayConstructor; +import net.razorvine.pickle.objects.ByteArrayConstructor; +import net.razorvine.pickle.objects.ClassDictConstructor; +import net.razorvine.pickle.objects.ComplexNumber; +import net.razorvine.pickle.objects.DateTimeConstructor; +import net.razorvine.pickle.objects.SetConstructor; + +/** + * Unpickles an object graph from a pickle data inputstream. Supports all pickle protocol versions. + * Maps the python objects on the corresponding java equivalents or similar types. + * This class is NOT threadsafe! (Don't use the same pickler from different threads) + * + * See the README.txt for a table of the type mappings. + * + * @author Irmen de Jong (irmen@razorvine.net) + */ +public class Unpickler { + + private final int HIGHEST_PROTOCOL = 4; + + private Map memo; + protected UnpickleStack stack; + private InputStream input; + private static Map objectConstructors; + + static { + objectConstructors = new HashMap(); + objectConstructors.put("__builtin__.complex", new AnyClassConstructor(ComplexNumber.class)); + objectConstructors.put("builtins.complex", new AnyClassConstructor(ComplexNumber.class)); + objectConstructors.put("array.array", new ArrayConstructor()); + objectConstructors.put("array._array_reconstructor", new ArrayConstructor()); + objectConstructors.put("__builtin__.bytearray", new ByteArrayConstructor()); + objectConstructors.put("builtins.bytearray", new ByteArrayConstructor()); + objectConstructors.put("__builtin__.bytes", new ByteArrayConstructor()); + objectConstructors.put("__builtin__.set", new SetConstructor()); + objectConstructors.put("builtins.set", new SetConstructor()); + objectConstructors.put("datetime.datetime", new DateTimeConstructor(DateTimeConstructor.DATETIME)); + objectConstructors.put("datetime.time", new DateTimeConstructor(DateTimeConstructor.TIME)); + objectConstructors.put("datetime.date", new DateTimeConstructor(DateTimeConstructor.DATE)); + objectConstructors.put("datetime.timedelta", new DateTimeConstructor(DateTimeConstructor.TIMEDELTA)); + objectConstructors.put("decimal.Decimal", new AnyClassConstructor(BigDecimal.class)); + } + + /** + * Create an unpickler. + */ + public Unpickler() { + memo = new HashMap(); + } + + /** + * Register additional object constructors for custom classes. + */ + public static void registerConstructor(String module, String classname, IObjectConstructor constructor) { + objectConstructors.put(module + "." + classname, constructor); + } + + /** + * Read a pickled object representation from the given input stream. + * + * @return the reconstituted object hierarchy specified in the file. + */ + public Object load(InputStream stream) throws PickleException, IOException { + stack = new UnpickleStack(); + input = stream; + try { + while (true) { + short key = PickleUtils.readbyte(input); + if (key == -1) + throw new IOException("premature end of file"); + dispatch(key); + } + } catch (StopException x) { + return x.value; + } + } + + /** + * Read a pickled object representation from the given pickle data bytes. + * + * @return the reconstituted object hierarchy specified in the file. + */ + public Object loads(byte[] pickledata) throws PickleException, IOException { + return load(new ByteArrayInputStream(pickledata)); + } + + /** + * Close the unpickler and frees the resources such as the unpickle stack and memo table. + */ + public void close() { + if(stack!=null) stack.clear(); + if(memo!=null) memo.clear(); + if(input!=null) + try { + input.close(); + } catch (IOException e) { + } + } + + private class StopException extends RuntimeException { + private static final long serialVersionUID = 6528222454688362873L; + + public StopException(Object value) { + this.value = value; + } + + public Object value; + } + + /** + * Process a single pickle stream opcode. + */ + protected void dispatch(short key) throws PickleException, IOException { + switch (key) { + case Opcodes.MARK: + load_mark(); + break; + case Opcodes.STOP: + Object value = stack.pop(); + stack.clear(); + throw new StopException(value); + case Opcodes.POP: + load_pop(); + break; + case Opcodes.POP_MARK: + load_pop_mark(); + break; + case Opcodes.DUP: + load_dup(); + break; + case Opcodes.FLOAT: + load_float(); + break; + case Opcodes.INT: + load_int(); + break; + case Opcodes.BININT: + load_binint(); + break; + case Opcodes.BININT1: + load_binint1(); + break; + case Opcodes.LONG: + load_long(); + break; + case Opcodes.BININT2: + load_binint2(); + break; + case Opcodes.NONE: + load_none(); + break; + case Opcodes.PERSID: + throw new InvalidOpcodeException("opcode not implemented: PERSID"); + case Opcodes.BINPERSID: + throw new InvalidOpcodeException("opcode not implemented: BINPERSID"); + case Opcodes.REDUCE: + load_reduce(); + break; + case Opcodes.STRING: + load_string(); + break; + case Opcodes.BINSTRING: + load_binstring(); + break; + case Opcodes.SHORT_BINSTRING: + load_short_binstring(); + break; + case Opcodes.UNICODE: + load_unicode(); + break; + case Opcodes.BINUNICODE: + load_binunicode(); + break; + case Opcodes.APPEND: + load_append(); + break; + case Opcodes.BUILD: + load_build(); + break; + case Opcodes.GLOBAL: + load_global(); + break; + case Opcodes.DICT: + load_dict(); + break; + case Opcodes.EMPTY_DICT: + load_empty_dictionary(); + break; + case Opcodes.APPENDS: + load_appends(); + break; + case Opcodes.GET: + load_get(); + break; + case Opcodes.BINGET: + load_binget(); + break; + case Opcodes.INST: + throw new InvalidOpcodeException("opcode not implemented: INST"); + case Opcodes.LONG_BINGET: + load_long_binget(); + break; + case Opcodes.LIST: + load_list(); + break; + case Opcodes.EMPTY_LIST: + load_empty_list(); + break; + case Opcodes.OBJ: + throw new InvalidOpcodeException("opcode not implemented: OBJ"); + case Opcodes.PUT: + load_put(); + break; + case Opcodes.BINPUT: + load_binput(); + break; + case Opcodes.LONG_BINPUT: + load_long_binput(); + break; + case Opcodes.SETITEM: + load_setitem(); + break; + case Opcodes.TUPLE: + load_tuple(); + break; + case Opcodes.EMPTY_TUPLE: + load_empty_tuple(); + break; + case Opcodes.SETITEMS: + load_setitems(); + break; + case Opcodes.BINFLOAT: + load_binfloat(); + break; + + // protocol 2 + case Opcodes.PROTO: + load_proto(); + break; + case Opcodes.NEWOBJ: + load_newobj(); + break; + case Opcodes.EXT1: + throw new InvalidOpcodeException("opcode not implemented: EXT1"); + case Opcodes.EXT2: + throw new InvalidOpcodeException("opcode not implemented: EXT2"); + case Opcodes.EXT4: + throw new InvalidOpcodeException("opcode not implemented: EXT4"); + case Opcodes.TUPLE1: + load_tuple1(); + break; + case Opcodes.TUPLE2: + load_tuple2(); + break; + case Opcodes.TUPLE3: + load_tuple3(); + break; + case Opcodes.NEWTRUE: + load_true(); + break; + case Opcodes.NEWFALSE: + load_false(); + break; + case Opcodes.LONG1: + load_long1(); + break; + case Opcodes.LONG4: + load_long4(); + break; + + // Protocol 3 (Python 3.x) + case Opcodes.BINBYTES: + load_binbytes(); + break; + case Opcodes.SHORT_BINBYTES: + load_short_binbytes(); + break; + + // Protocol 4 (Python 3.4+) + case Opcodes.BINUNICODE8: + load_binunicode8(); + break; + case Opcodes.SHORT_BINUNICODE: + load_short_binunicode(); + break; + case Opcodes.BINBYTES8: + load_binbytes8(); + break; + case Opcodes.EMPTY_SET: + load_empty_set(); + break; + case Opcodes.ADDITEMS: + load_additems(); + break; + case Opcodes.FROZENSET: + load_frozenset(); + break; + case Opcodes.MEMOIZE: + load_memoize(); + break; + case Opcodes.FRAME: + load_frame(); + break; + case Opcodes.NEWOBJ_EX: + load_newobj_ex(); + break; + case Opcodes.STACK_GLOBAL: + load_stack_global(); + break; + + default: + throw new InvalidOpcodeException("invalid pickle opcode: " + key); + } + } + + void load_build() { + Object args=stack.pop(); + Object target=stack.peek(); + try { + Method setStateMethod=target.getClass().getMethod("__setstate__", args.getClass()); + setStateMethod.invoke(target, args); + } catch (Exception e) { + throw new PickleException("failed to __setstate__()",e); + } + } + + void load_proto() throws IOException { + short proto = PickleUtils.readbyte(input); + if (proto < 0 || proto > HIGHEST_PROTOCOL) + throw new PickleException("unsupported pickle protocol: " + proto); + } + + void load_none() { + stack.add(null); + } + + void load_false() { + stack.add(false); + } + + void load_true() { + stack.add(true); + } + + void load_int() throws IOException { + String data = PickleUtils.readline(input, true); + Object val; + if (data.equals(Opcodes.FALSE.substring(1))) + val = false; + else if (data.equals(Opcodes.TRUE.substring(1))) + val = true; + else { + String number=data.substring(0, data.length() - 1); + try { + val = Integer.parseInt(number, 10); + } catch (NumberFormatException x) { + // hmm, integer didn't work.. is it perhaps an int from a 64-bit python? so try long: + val = Long.parseLong(number, 10); + } + } + stack.add(val); + } + + void load_binint() throws IOException { + int integer = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); + stack.add(integer); + } + + void load_binint1() throws IOException { + stack.add((int)PickleUtils.readbyte(input)); + } + + void load_binint2() throws IOException { + int integer = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 2)); + stack.add(integer); + } + + void load_long() throws IOException { + String val = PickleUtils.readline(input); + if (val != null && val.endsWith("L")) { + val = val.substring(0, val.length() - 1); + } + BigInteger bi = new BigInteger(val); + stack.add(PickleUtils.optimizeBigint(bi)); + } + + void load_long1() throws IOException { + short n = PickleUtils.readbyte(input); + byte[] data = PickleUtils.readbytes(input, n); + stack.add(PickleUtils.decode_long(data)); + } + + void load_long4() throws IOException { + int n = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); + byte[] data = PickleUtils.readbytes(input, n); + stack.add(PickleUtils.decode_long(data)); + } + + void load_float() throws IOException { + String val = PickleUtils.readline(input, true); + stack.add(Double.parseDouble(val)); + } + + void load_binfloat() throws IOException { + double val = PickleUtils.bytes_to_double(PickleUtils.readbytes(input, 8),0); + stack.add(val); + } + + void load_string() throws IOException { + String rep = PickleUtils.readline(input); + boolean quotesOk = false; + for (String q : new String[] { "\"", "'" }) // double or single quote + { + if (rep.startsWith(q)) { + if (!rep.endsWith(q)) { + throw new PickleException("insecure string pickle"); + } + rep = rep.substring(1, rep.length() - 1); // strip quotes + quotesOk = true; + break; + } + } + + if (!quotesOk) + throw new PickleException("insecure string pickle"); + + stack.add(PickleUtils.decode_escaped(rep)); + } + + void load_binstring() throws IOException { + int len = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); + byte[] data = PickleUtils.readbytes(input, len); + stack.add(PickleUtils.rawStringFromBytes(data)); + } + + void load_binbytes() throws IOException { + int len = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); + stack.add(PickleUtils.readbytes(input, len)); + } + + void load_binbytes8() throws IOException { + long len = PickleUtils.bytes_to_long(PickleUtils.readbytes(input, 8),0); + stack.add(PickleUtils.readbytes(input, len)); + } + + void load_unicode() throws IOException { + String str=PickleUtils.decode_unicode_escaped(PickleUtils.readline(input)); + stack.add(str); + } + + void load_binunicode() throws IOException { + int len = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); + byte[] data = PickleUtils.readbytes(input, len); + stack.add(new String(data,"UTF-8")); + } + + void load_binunicode8() throws IOException { + long len = PickleUtils.bytes_to_long(PickleUtils.readbytes(input, 8),0); + byte[] data = PickleUtils.readbytes(input, len); + stack.add(new String(data,"UTF-8")); + } + + void load_short_binunicode() throws IOException { + int len = PickleUtils.readbyte(input); + byte[] data = PickleUtils.readbytes(input, len); + stack.add(new String(data,"UTF-8")); + } + + void load_short_binstring() throws IOException { + short len = PickleUtils.readbyte(input); + byte[] data = PickleUtils.readbytes(input, len); + stack.add(PickleUtils.rawStringFromBytes(data)); + } + + void load_short_binbytes() throws IOException { + short len = PickleUtils.readbyte(input); + stack.add(PickleUtils.readbytes(input, len)); + } + + void load_tuple() { + ArrayList top = stack.pop_all_since_marker(); + stack.add(top.toArray()); + } + + void load_empty_tuple() { + stack.add(new Object[0]); + } + + void load_tuple1() { + stack.add(new Object[] { stack.pop() }); + } + + void load_tuple2() { + Object o2 = stack.pop(); + Object o1 = stack.pop(); + stack.add(new Object[] { o1, o2 }); + } + + void load_tuple3() { + Object o3 = stack.pop(); + Object o2 = stack.pop(); + Object o1 = stack.pop(); + stack.add(new Object[] { o1, o2, o3 }); + } + + void load_empty_list() { + stack.add(new ArrayList(0)); + } + + void load_empty_dictionary() { + stack.add(new HashMap(0)); + } + + void load_empty_set() { + stack.add(new HashSet()); + } + + void load_list() { + ArrayList top = stack.pop_all_since_marker(); + stack.add(top); // simply add the top items as a list to the stack again + } + + void load_dict() { + ArrayList top = stack.pop_all_since_marker(); + HashMap map = new HashMap(top.size()); + for (int i = 0; i < top.size(); i += 2) { + Object key = top.get(i); + Object value = top.get(i + 1); + map.put(key, value); + } + stack.add(map); + } + + void load_frozenset() { + ArrayList top = stack.pop_all_since_marker(); + HashSet set = new HashSet(); + set.addAll(top); + stack.add(set); + } + + void load_additems() { + ArrayList top = stack.pop_all_since_marker(); + @SuppressWarnings("unchecked") + HashSet set = (HashSet) stack.pop(); + set.addAll(top); + stack.add(set); + } + + void load_global() throws IOException { + String module = PickleUtils.readline(input); + String name = PickleUtils.readline(input); + load_global_sub(module, name); + } + + void load_stack_global() { + String name = (String) stack.pop(); + String module = (String) stack.pop(); + load_global_sub(module, name); + } + + void load_global_sub(String module, String name) { + IObjectConstructor constructor = objectConstructors.get(module + "." + name); + if (constructor == null) { + // check if it is an exception + if(module.equals("exceptions")) { + // python 2.x + constructor=new AnyClassConstructor(PythonException.class); + } else if(module.equals("builtins") || module.equals("__builtin__")) { + if(name.endsWith("Error") || name.endsWith("Warning") || name.endsWith("Exception") + || name.equals("GeneratorExit") || name.equals("KeyboardInterrupt") + || name.equals("StopIteration") || name.equals("SystemExit")) + { + // it's a python 3.x exception + constructor=new AnyClassConstructor(PythonException.class); + } + else + { + // return a dictionary with the class's properties + constructor=new ClassDictConstructor(module, name); + } + } else { + // return a dictionary with the class's properties + constructor=new ClassDictConstructor(module, name); + } + } + stack.add(constructor); + } + + + void load_pop() { + stack.pop(); + } + + void load_pop_mark() { + Object o = null; + do { + o = stack.pop(); + } while (o != stack.MARKER); + stack.trim(); + } + + void load_dup() { + stack.add(stack.peek()); + } + + void load_get() throws IOException { + int i = Integer.parseInt(PickleUtils.readline(input), 10); + if(!memo.containsKey(i)) throw new PickleException("invalid memo key"); + stack.add(memo.get(i)); + } + + void load_binget() throws IOException { + int i = PickleUtils.readbyte(input); + if(!memo.containsKey(i)) throw new PickleException("invalid memo key"); + stack.add(memo.get(i)); + } + + void load_long_binget() throws IOException { + int i = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); + if(!memo.containsKey(i)) throw new PickleException("invalid memo key"); + stack.add(memo.get(i)); + } + + void load_put() throws IOException { + int i = Integer.parseInt(PickleUtils.readline(input), 10); + memo.put(i, stack.peek()); + } + + void load_binput() throws IOException { + int i = PickleUtils.readbyte(input); + memo.put(i, stack.peek()); + } + + void load_long_binput() throws IOException { + int i = PickleUtils.bytes_to_integer(PickleUtils.readbytes(input, 4)); + memo.put(i, stack.peek()); + } + + void load_memoize() { + memo.put(memo.size(), stack.peek()); + } + + void load_append() { + Object value = stack.pop(); + @SuppressWarnings("unchecked") + ArrayList list = (ArrayList) stack.peek(); + list.add(value); + } + + void load_appends() { + ArrayList top = stack.pop_all_since_marker(); + @SuppressWarnings("unchecked") + ArrayList list = (ArrayList) stack.peek(); + list.addAll(top); + list.trimToSize(); + } + + void load_setitem() { + Object value = stack.pop(); + Object key = stack.pop(); + @SuppressWarnings("unchecked") + Map dict = (Map) stack.peek(); + dict.put(key, value); + } + + void load_setitems() { + HashMap newitems = new HashMap(); + Object value = stack.pop(); + while (value != stack.MARKER) { + Object key = stack.pop(); + newitems.put(key, value); + value = stack.pop(); + } + + @SuppressWarnings("unchecked") + Map dict = (Map) stack.peek(); + dict.putAll(newitems); + } + + void load_mark() { + stack.add_mark(); + } + + void load_reduce() { + Object[] args = (Object[]) stack.pop(); + IObjectConstructor constructor = (IObjectConstructor) stack.pop(); + stack.add(constructor.construct(args)); + } + + void load_newobj() { + load_reduce(); // for Java we just do the same as class(*args) instead of class.__new__(class,*args) + } + + void load_newobj_ex() { + HashMap kwargs = (HashMap) stack.pop(); + Object[] args = (Object[]) stack.pop(); + IObjectConstructor constructor = (IObjectConstructor) stack.pop(); + if(kwargs.size()==0) + stack.add(constructor.construct(args)); + else + throw new PickleException("newobj_ex with keyword arguments not supported"); + } + + void load_frame() throws IOException { + // for now we simply skip the frame opcode and its length + PickleUtils.readbytes(input, 8); + } +} diff --git a/java/src/net/razorvine/pickle/objects/package-info.java b/java/src/net/razorvine/pickle/objects/package-info.java index 4cd39e1..bc12f38 100644 --- a/java/src/net/razorvine/pickle/objects/package-info.java +++ b/java/src/net/razorvine/pickle/objects/package-info.java @@ -2,7 +2,7 @@ * Object constructors and other utility classes for the pickle package. * * @author Irmen de Jong (irmen@razorvine.net) - * @version 3.0 + * @version 3.1 * @see net.razorvine.pickle */ package net.razorvine.pickle.objects; diff --git a/java/src/net/razorvine/pickle/package-info.java b/java/src/net/razorvine/pickle/package-info.java index 5d727bf..6e6c2ec 100644 --- a/java/src/net/razorvine/pickle/package-info.java +++ b/java/src/net/razorvine/pickle/package-info.java @@ -10,7 +10,7 @@ * functionality. * * @author Irmen de Jong (irmen@razorvine.net) - * @version 3.0 + * @version 3.1 */ package net.razorvine.pickle; diff --git a/java/src/net/razorvine/pyro/Config.java b/java/src/net/razorvine/pyro/Config.java index 39e3a5f..f402c3d 100644 --- a/java/src/net/razorvine/pyro/Config.java +++ b/java/src/net/razorvine/pyro/Config.java @@ -2,11 +2,12 @@ import java.io.Serializable; + /** * Minimalistic holders for Pyro config items. * * @author Irmen de Jong (irmen@razorvine.net) - * @version 3.0 + * @version 3.1 */ public final class Config implements Serializable { private static final long serialVersionUID = 198635706890570066L; @@ -16,9 +17,8 @@ public final class Config implements Serializable { public static int NS_PORT = 9090; public static int NS_BCPORT = 9091; - public final static int PROTOCOL_VERSION = 46; // Pyro 4.22 and newer. Cannot be changed - - public final static String PYROLITE_VERSION = "3.0"; + public final static int PROTOCOL_VERSION = 47; // Pyro 4.26 + public final static String PYROLITE_VERSION = "3.1"; public enum SerializerType { pickle, diff --git a/java/src/net/razorvine/pyro/package-info.java b/java/src/net/razorvine/pyro/package-info.java index ca21b2d..ec03e29 100644 --- a/java/src/net/razorvine/pyro/package-info.java +++ b/java/src/net/razorvine/pyro/package-info.java @@ -12,7 +12,7 @@ * Note that Pyrolite only supports Pyro4. * * @author Irmen de Jong (irmen@razorvine.net) - * @version 3.0 + * @version 3.1 * @see net.razorvine.pickle */ package net.razorvine.pyro; diff --git a/java/test/net/razorvine/pyro/test/MessageTests.java b/java/test/net/razorvine/pyro/test/MessageTests.java index 5278bcf..e8820e6 100644 --- a/java/test/net/razorvine/pyro/test/MessageTests.java +++ b/java/test/net/razorvine/pyro/test/MessageTests.java @@ -14,8 +14,8 @@ import net.razorvine.pyro.*; import net.razorvine.pyro.serializer.*; - import static org.junit.Assert.*; + import org.junit.After; import org.junit.Before; import org.junit.Test; @@ -190,12 +190,46 @@ public void testRecvAnnotations() throws IOException assertTrue(msg.annotations.containsKey("HMAC")); } - @Test(expected=PyroException.class) - public void testProtocolVersion() + @Test + public void testProtocolVersionKaputt() + { + byte[] msg = getHeaderBytes(new Message(Message.MSG_RESULT, new byte[0], this.ser.getSerializerId(), 0, 1, null).to_bytes()); + msg[4] = 99; // screw up protocol version in message header + msg[5] = 111; // screw up protocol version in message header + try { + Message.from_header(msg); + fail("should crash"); + } catch (PyroException x) { + assertEquals("invalid protocol version: 25455", x.getMessage()); + } + } + + @Test + public void testProtocolVersionsNotSupported1() { byte[] msg = getHeaderBytes(new Message(Message.MSG_RESULT, new byte[0], this.ser.getSerializerId(), 0, 1, null).to_bytes()); - msg[4] = 99; // screw up protocol version in message header - Message.from_header(msg); + msg[4] = 0; + msg[5] = 46; + try { + Message.from_header(msg); + fail("should crash"); + } catch (PyroException x) { + assertEquals("invalid protocol version: 46", x.getMessage()); + } + } + + @Test + public void testProtocolVersionsNotSupported2() + { + byte[] msg = getHeaderBytes(new Message(Message.MSG_RESULT, new byte[0], this.ser.getSerializerId(), 0, 1, null).to_bytes()); + msg[4] = 0; + msg[5] = 48; + try { + Message.from_header(msg); + fail("should crash"); + } catch (PyroException x) { + assertEquals("invalid protocol version: 48", x.getMessage()); + } } @Test