Test availability paridy with PySide2

[mottosso]

Goal

Check that all bindings provide identical members as PySide2; the reference binding.

Motivation

It has happened that one binding was missing access to certain classes that should have been available due to a typo or forgetfulness. A "one test to rule them all" for availability should iron these out.

Implementation

I'd imagine running a simple dir(PySide2.QtWidgets) and checking that against what's available elsewhere. It would be nice to also include argument signatures for these. Possibly that we can achieve this via the Python inspect module.

The number one goal is for each binding to provide the same members and argument signatures (differences break cross-binding compatibility). If we can also reach a state where they all provide everything that PySide2 has in an easy manner, then that would be great too, but it isn't as important and is unlikely to ever happen. Simply because some of the features provided are ad-hoc (due to trying to convert C++ methodologies to Python).

[ethanhs]

There are several issues with PySide2's bindings of QtQuick and QtOpengl, but I believe the QtWidgets should be complete or nearly there.

[mottosso]

Thanks @ethanhs, good to see you here. Have you by any chance changed your username recently? IronManMark20? :)

[ethanhs]

@mottosso I think its been a few months, but yes. :) At some point I want to add a UI to a binding generator I've been playing around with, and came across this library. I'm surprised I didn't notice it earlier, this solves a lot of headaches I've been solving myself.

[fredrikaverpil]

There are several issues with PySide2's bindings of QtQuick and QtOpengl, but I believe the QtWidgets should be complete or nearly there.

That is correct. For completeness, I've added the changelog from https://wiki.qt.io/PySide2 below:

4 August 2016

• Additional Pyside examples under review
• Fixed warnings coming from Shiboken

21 July 2016

• main Pyside 2 example port done
• additional examples to fill gaps are being ported (as per prio list)
• CI patches running, but still gaps (should be done by next week)
• about 86% of auto test working (failing count of test 80+ auto test)
• Project Test Status

14 July 2016

• Pyside 2 examples ported
• OpenGL & SVG not working
• QtQuick 2 is in strange situation (QtQml depends on QtQuick)
• Python 3 realed Unicode handling not working with Qt
• QMessageBox hangs
• no documentation for any example
• lots of warnings when building wrapper
• results: https://paste.kde.org/pgje2toyh/mlbqic
• COIN setup for pyside https://codereview.qt-project.org/#/c/158336/
• Qt 5.7 still blocked due to missing C++11 support in shiboken
• food for ideas: https://steveire.wordpress.com/

7 July 2016

• automated CI testing
• patches for Pyside and Qt CI side required
• Testing somewhat more complicated due to closed nature of Qt CI
• script to port examples to Qt 5 -> https://bugreports.qt.io/issues/?jql=labels%20%3D%20exampleport5
• all examples to remain BSD licensed

[mottosso]

Just a heads up, QtPy are/have been solving this issue as well.

• Auto generate methods that have been deprecated for Qt5 stuff spyder-ide/qtpy#68

What works for them/us should work for us/them!

[fredrikaverpil]

Ohh, nice!

[mottosso]

Implemented!

[mottosso]

For future reference, because members of bindings are currently discovered through ugly hacks that are also incompatible with Python 3, involving the import * mechanism, have a look at whether we can leverage the unix strings program.

$ strings QtGui.x86_64-linux-gnu.so | grep ^PySide2.QtGui.
PySide2.QtGui.QTextTableFormat.setCellSpacing
PySide2.QtGui.QTextTableFormat.setCellPadding
PySide2.QtGui.QTextTableFormat.setColumnWidthConstraints
PySide2.QtGui.QToolBarChangeEvent
PySide2.QtGui.QTouchDevice.DeviceType
PySide2.QtGui.QTouchDevice.setType
PySide2.QtGui.QTouchDevice.setMaximumTouchPoints
PySide2.QtGui.QTouchDevice.Capabilities
PySide2.QtGui.QTouchDevice.setCapabilities
PySide2.QtGui.QTouchDevice.setName
PySide2.QtGui.QTouchDevice.CapabilityFlag
PySide2.QtGui.QTouchDevice
PySide2.QtGui.TouchPoint(): too many arguments
PySide2.QtGui.TouchPoint(): got multiple values for keyword argument 'id'.
PySide2.QtGui.QTouchEvent::TouchPoint
PySide2.QtGui.TouchPoint.setVelocity
PySide2.QtGui.TouchPoint.setStartScreenPos
PySide2.QtGui.TouchPoint.setStartScenePos
PySide2.QtGui.TouchPoint.setStartPos
PySide2.QtGui.TouchPoint.setStartNormalizedPos
PySide2.QtGui.TouchPoint.setScreenRect
PySide2.QtGui.TouchPoint.setScreenPos
PySide2.QtGui.TouchPoint.setSceneRect
...

It should return absolutely all members of any binding, as-is, from the source of where they come from.

Potential problems include:

1. strings not being available on Windows
2. Post-processing happening after the .so has been loaded into Python

[ethanhs]

@mottosso while it is not a compatible version, there is a re-implementation of strings in sysinternals. I suppose one could download it via powershell or the like, then use that.

But I don't see why you can't just read the file as a binary in Python, and get strings that way. there is an example here

[mottosso]

Thanks Ethan!

To be honest, I just stumbled upon strings today and have never seen anything like it. I knew data was stored separately in binaries and that you can read them with any hex editor, but not that it would be coherent enough to actually get data out of as cleanly as this and figured it might even be clean enough to build upon.

It's good to know the discovery wasn't a fluke and that there are others who know about it!

But what do you think, does it make sense? Is it reliable enough? Does it produce different results per build, per platform, or per differing build flags? I get the feeling that are quite a large number of variables to the equation resulting in what comes out of it.

On the other hand, it also feels real and undeniable. As if the information one can get out of it has nothing to hide behind.

[ethanhs]

I think it should be reliable enough. Ideally, each binding generator (Shiboken, SIP, and any others) would produce a text file that has all the functions, classes, etc that are being exported. But I don't think that will happen, so the binary reading seems to be the best approach to me.

But what do you think, does it make sense? Is it reliable enough? Does it produce different results per build, per platform, or per differing build flags? I get the feeling that are quite a large number of variables to the equation resulting in what comes out of it.

Sadly, yes, there can be quite a variety. First of all, things like SSL won't be wrapped if one does not pass the openssl libraries. Also, at least for PySide (1 and 2), there are different typesystem files (files which define what is to be wrapped) which have some platform differences. There will definitely be differences per build/platform. Also, it is not too difficult to remove some parts from being built.

I think the best idea would be to scan the binaries of the library in use (PySide, PyQt, etc) at runtime, and check compatibility then, or perhaps as an installation step?

[mottosso]

I think the best idea would be to scan the binaries of the library in use (PySide, PyQt, etc) at runtime, and check compatibility then, or perhaps as an installation step?

That does sound the most bullet proof, but I'm not sure about the idea of throwing errors at installation time.. :( What is the end-user supposed to do about it?

[ethanhs]

Well, many programs fail at installation. If we have sanity checks that raise exceptions, and warnings for other potential issues, then we should be fine. We don't want the program run if it fails sanity checks during installation, right?

[mottosso]

Hm, I'm not too sure.. I'm trying to think of an installation that ran tests at all, and based the success of the installation on it. Could you be thinking of builds?

With builds it'd be ok, because (1) the user it likely tech savvy and (2) errors are likely related to something within the users control; such as missing dependencies.

But here, neither of those are necessarily true. Just to clarify, is this what we're saying?

mottosso:~ $ pip install Qt.py
Downloading/unpacking Qt.py
  Downloading Qt.py-0.4.0-py2.py3-none-any.whl (12kB): 12kB downloaded
Installing collected packages: Qt.py
FAIL
mottosso:~ $ 

[ethanhs]

Yes, and your points are well taken, I suppose I was thinking of non-wheel distribution. Anyway, perhaps on import sanity checks would happen? That seems the most reasonable.

[mottosso]

Ah, that is also a good idea - one I hadn't considered - and sorry to keep shooting you down, but doesn't that imply a run-time overhead?

[ethanhs]

Well, you only have to do it once I suppose, right? Then it wouldn't be as much of an issue?

[mottosso]

Let's play it out.

A) Say it takes 1+ second to perform this test (fair, perhaps optimistic, assuming we'll do a binary file lookup and comparison)
B) When a user runs a tool from within a Python-embedded environment (e.g. Maya), a 1+ second freeze occurs.

Another scenario is when a user runs the tool standalone, like from the command-line. That way, "once" turns into "every time".

[ethanhs]

Well, I suppose threading may help with locking? But I don't know too much about Python in Maya. And I suppose, one could just write a file after the first run into the install dir, and the check wouldn't be run if that file exists...
There is probably a better way of doing this... I'm going to sleep on it.

[mottosso]

Appreciate the ideas @ethanhs! I feel really bad shooting you down! I will also keep these suggestions in mind and see if there's any way to separate the bad from the good.

To reiterate:

• Our end goal is to ensure that the end-user has the correct members on import Qt ("correct" being those from PySide2).
• The problem we're having is that different users may end up having a version of a binding compiled differently, which makes it impossible for us to ensure up-front that Qt.py is consistent.

The way it's handled currently is to assume one binding (one provided by a custom PPA in Ubuntu 16.04) and only check for members relative this one and other bindings also provided via said packaging mechanism.