Windows support

[dgrunwald]

I actually started writing rust-cpython on Windows, but had to give up due to strange issues when linking against the official python binaries. Now that the x86_64-pc-windows-msvc target exists, we should give it another try.

[novocaine]

I've started working on at least getting the build script to be correct.

[dgrunwald]

Have you had any success linking against the official python27.dll on Windows?

[novocaine]

python27-sys appeared to compile fine, but there may still be issues that I didn't see because I didn't try to get rust-cpython itself to work yet.

some interesting comments here about what .rlibs actually contain: rust-lang/rust#25820

so would make sense that the issues would only come up when trying to actually create an executable, which I didn't do.

also some chatter here: https://mail.mozilla.org/pipermail/rust-dev/2014-July/010702.html

[novocaine]

okay, so once I moved the python.org's python27.dll into C:\Program Files\Rust nightly 1.2\bin\rustlib\x86_64-pc-windows-gnu\lib, the linker was able to locate but it rejects it with this when running cargo test:

note: ld: skipping incompatible C:\Program Files\Rust nightly 1.2\bin\rustlib\x86_64-pc-windows-gnu\lib/python27.dll when searching for -lpython27
ld: skipping incompatible C:\Program Files\Rust nightly 1.2\bin\rustlib\x86_64-pc-windows-gnu\lib/python27.dll when searching for -lpython27
ld: skipping incompatible C:/Program Files/Rust nightly 1.2/bin/rustlib/x86_64-pc-windows-gnu/bin/../lib/gcc/x86_64-w64-mingw32/4.9.1/../../../../lib/python27.dll when searching for -lpython27
ld: skipping incompatible C:/Program Files/Rust nightly 1.2/bin/rustlib/x86_64-pc-windows-gnu/bin/../lib/gcc/x86_64-w64-mingw32/4.9.1/../../../python27.dll when searching for -lpython27
ld: skipping incompatible C:\Program Files\Rust nightly 1.2\bin\rustlib\x86_64-pc-windows-gnu\lib/python27.dll when searching for -lpython27
ld: skipping incompatible C:\Program Files\Rust nightly 1.2\bin\rustlib\x86_64-pc-windows-gnu\lib/python27.dll when searching for -lpython27
ld: skipping incompatible C:/Program Files/Rust nightly 1.2/bin/rustlib/x86_64-pc-windows-gnu/bin/../lib/gcc/x86_64-w64-mingw32/4.9.1/../../../../lib/python27.dll when searching for -lpython27
ld: skipping incompatible C:/Program Files/Rust nightly 1.2/bin/rustlib/x86_64-pc-windows-gnu/bin/../lib/gcc/x86_64-w64-mingw32/4.9.1/../../../python27.dll when
searching for -lpython27
ld: cannot find -lpython27

I suspect this is because I have a 64-bit rust with a 32-bit python, but the error message is opaque.

[dgrunwald]

I tried using a x86_64-pc-windows-msvc rustc, but I'm getting some missing symbols:

python27_sys-c1b521149f9904e3.o : error LNK2019: unresolved external symbol PyByteArray_Type referenced in function _ZN15bytearrayobject17PyByteArray_Check20h8e69401868e4196bTDbE
python27_sys-c1b521149f9904e3.o : error LNK2019: unresolved external symbol PyFrozenSet_Type referenced in function _ZN9setobject22PyFrozenSet_CheckExact20h7aaa0fe8c1c5aefaaTbE
python27_sys-c1b521149f9904e3.o : error LNK2019: unresolved external symbol PySet_Type referenced in function _ZN9setobject19PyAnySet_CheckExact20ha41476a56e29f32exTbE
python27_sys-c1b521149f9904e3.o : error LNK2019: unresolved external symbol PyClass_Type referenced in function _ZN11classobject13PyClass_Check20h19fdd48fccf38ff578bE
python27_sys-c1b521149f9904e3.o : error LNK2019: unresolved external symbol PyInstance_Type referenced in function _ZN11classobject16PyInstance_Check20h66ff2a8ae590f7abu9bE
python27_sys-c1b521149f9904e3.o : error LNK2019: unresolved external symbol PyCapsule_Type referenced in function _ZN9pycapsule20PyCapsule_CheckExact20h92843e2bc63353cdwicE
c:\work\rust\cpython\python27-sys\target\debug\python27_sys-c1b521149f9904e3.exe : fatal error LNK1120: 6 unresolved externals

But those symbols are exported by the .lib:

python27-sys>dumpbin /exports "c:\Program Files\Python27\libs\python27.lib" | grep PyByteArray_Type
                  PyByteArray_Type

[dgrunwald]

The missing symbols seem to always be global variables.

Looks like this is caused by missing __dllspec(dllimport) support in rust: rust-lang/rust#7196

[yunge]

Hi,

I get some issues when try the test code in windows, so windows support is not finished yet?

E:\dev\msys32\home\fu.cargo\registry\src\github.com-121aea75f9ef2ce2\cpython-0.0.3\src/python.rs:156: undefined reference to `_Py_NoneStruct'

[dgrunwald]

For the msys-version of rust (x86_64-pc-windows-gnu), I have no idea how to get the msys linker to link against pythonXY.dll.

For the msvc-version of rust (x86_64-pc-windows-msvc), I think it's currently impossible to link against pythonXY.dll due to Rust's lack of __declspec(dllimport). The linker seems to find functions but not global variables, but global variables like _Py_NoneStruct are pretty critical for python.

I've given up on Windows support for now. Feel free to dig deeper into the linker troubles.

[yunge]

Ok, I'll play it with linux. Anyway, thanks for the cool project!

[novocaine]

rust-lang/rust#25350

[dgrunwald]

The #[linked_from] attribute might help here.

[Eh2406]

Is this still "given up" on?

[dgrunwald]

Is this still "given up" on?

Yes.

Here's a small test case demonstrating where I'm stuck:

#![feature(linked_from)]

use std::ffi;
use std::os::raw::c_char;

type PyObject = i32;

#[linked_from = "python27"]
extern "C" {
    fn Py_Initialize() -> ();
    static mut _Py_TrueStruct: PyObject;
    fn PyObject_Str(o: *mut PyObject) -> *mut PyObject;
    fn PyString_AsString(string: *mut PyObject) -> *mut c_char;
}

fn main() {
    unsafe {
        Py_Initialize();
        println!("{:?}", ffi::CStr::from_ptr(PyString_AsString(PyObject_Str(&mut _Py_TrueStruct))));
    }
}

Compile:

> rustc use_python.rs -L "C:\Program Files\Python27\libs" -l python27
....
note: use_python.0.o : error LNK2019: unresolved external symbol _Py_TrueStruct referenced in function _ZN4main20he3711d09b9574a24FaaE
use_python.exe : fatal error LNK1120: 1 unresolved externals

Note that the function calls don't give linker errors, only the static variables do.

Explicitly telling rustc that python27 is a .dll (using -l dylib=python27) results in the same error.

Note that the .lib and .dll do contain this symbol:

> "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\dumpbin.exe" /exports "C:\Program Files\Python27\libs\python27.lib" | grep _Py_True
                  _Py_TrueStruct

> "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\dumpbin.exe" /exports "C:\Windows\System32\python27.dll" | grep _Py_True
       1032  407 002872E0 _Py_TrueStruct

> "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\dumpbin.exe" /symbols use_python.0.o | grep Py
03C 00000000 UNDEF  notype       External     | Py_Initialize
03D 00000000 UNDEF  notype       External     | _Py_TrueStruct
03E 00000000 UNDEF  notype       External     | PyObject_Str
03F 00000000 UNDEF  notype       External     | PyString_AsString

The equivalent C program:

#include "stdio.h"

typedef struct PyObject PyObject;

extern __declspec(dllimport) void Py_Initialize();
extern __declspec(dllimport) PyObject _Py_TrueStruct;
extern __declspec(dllimport) PyObject* PyObject_Str(PyObject* o);
extern __declspec(dllimport) char* PyString_AsString(PyObject* o);

int main()
{
    Py_Initialize();
    printf("%s", PyString_AsString(PyObject_Str(&_Py_TrueStruct)));
}

Compile:

> cl test.c /link "c:\Program Files\Python27\libs\python27.lib"
> test.exe
True
> dumpbin /symbols test.obj | grep Py
016 00000000 UNDEF  notype       External     | __imp_Py_Initialize
017 00000000 UNDEF  notype       External     | __imp_PyObject_Str
018 00000000 UNDEF  notype       External     | __imp_PyString_AsString
030 00000000 UNDEF  notype       External     | __imp__Py_TrueStruct

Note that if I leave out the __declspec(dllimport) attribute, cl produces the same kind of object file as rustc, and linking fails with:

test.obj : error LNK2019: unresolved external symbol _Py_TrueStruct referenced in function main

So the problem is that I can't find a way to convince rustc to mark the imported symbols as dllimport.

[Eh2406]

Just some notes to save some time next time I look into this.

I searched rust git for dllimport and read it all. Looks like this is best traced at:

rust-lang/rfcs#1061: checkbox 2. Support dllimport properly.
rust-lang/rust#27438: is a long discussion that is hard to follow. The only useful thing is that winapi uuid.lib may be in the same boat.
rust-lang/rfcs#1296: is the current RFC.

If any of those move we may get unstuck.

[retep998]

The behavior of dllimport with regards to statics in Rust is like this the last time I checked.

When referencing an extern static from within the same crate, dllimport is not applied.
When referencing an extern static from a different crate than the one it was defined it, dllimport is applied.

So basically dllimport is being applied based on whether it is referenced across a crate boundary even though that has absolutely nothing to do with how the static is being linked in. Try not to rely on this behavior though and just push Rust to get it properly fixed.

[vadimcn]

@dgrunwald: Your code linked just fine using i686-pc-windows-gnu toolchain. Gnu ld is actually better for linking to dlls because it performs the necessary magic automatically.

Re i686-pc-windows-msvc, I am not sure why #[linked_from...] didn't work, but you can try this:

extern "C" {
    ...
    #[link_name = "\x01__imp___Py_TrueStruct"]
    static imp_Py_TrueStruct: *mut PyObject;
    ...
}

fn main() {
    unsafe {
        Py_Initialize();
        println!("{:?}", ffi::CStr::from_ptr(PyString_AsString(PyObject_Str(imp_Py_TrueStruct))));
    }
}

Not pretty, but works.

[PJB3005]

Any hope of this happening in the near future? I'd absolutely love to use this for a project but if Windows support isn't a thing I'll have to pass.

[Eh2406]

It is still waiting on rust. The rfc is accepted rust-lang/rfcs#1717 and the trcking ishuw is at rust-lang/rust#37403

I don't think anyone has started implementation.

[PJB3005]

Alright, thanks for the info.

Here's hoping I suppose.

[Eh2406]

This works on the new nightlies!!!

use std::ffi;
use std::os::raw::c_char;

type PyObject = i32;

#[link(name="python27", kind="dylib")]
extern "C" {
    fn Py_Initialize() -> ();
    static mut _Py_TrueStruct: PyObject;
    fn PyObject_Str(o: *mut PyObject) -> *mut PyObject;
    fn PyString_AsString(string: *mut PyObject) -> *mut c_char;
}

fn main() {
    unsafe {
        Py_Initialize();
        println!("{:?}",
                 ffi::CStr::from_ptr(PyString_AsString(PyObject_Str(&mut _Py_TrueStruct))));
    }
}

I am so excited!!! What are the next steps to getting rust-cpython working on windows? How can we help?

[dgrunwald]

I'm on it; if everything works as well as my initial tests indicate, I should be able to add Windows support today.

[Eh2406]

Thank you so much! I am stced to try this out! how do I tell cargo where to find my python lib and that XY in this case is 27?

[dgrunwald]

The default is Python 3.x, to target Python 2.x, you'll need to use:

[dependencies.cpython]
git = "https://github.com/dgrunwald/rust-cpython.git"
default-features = false
features = ["python27-sys"]

If you have python2 or python in PATH, the build script will invoke it to determine the location of the .lib file (and also the values of some compilation options that affect the ABI).

[Eh2406]

Wow! As usual on this project things are easier and smarter than I thout. :-)

I missed that it checked the path to find python in reading the readme/docs. (same for the 27 feature fag) Can we make it more obvious in readme/docs how cleverly it just figures things out? Or is it there and I just missed it?