Python 3.6 socket error

[anikey-m]

Hi SDK team,
I upgrade Python to version 3.6 and got following traceback:

2017-01-18 18:33:20,806 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Paho MQTT Client init.
2017-01-18 18:33:20,807 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - ClientID: ThingShadowEcho
2017-01-18 18:33:20,808 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Protocol: MQTTv3.1.1
2017-01-18 18:33:20,808 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Register Paho MQTT Client callbacks.
2017-01-18 18:33:20,809 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - mqttCore init.
2017-01-18 18:33:20,810 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: queueSize = 0
2017-01-18 18:33:20,811 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: dropBehavior = Drop Newest
2017-01-18 18:33:20,812 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for draining interval: 0.1 sec
2017-01-18 18:33:20,812 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load CAFile from: VeriSign-Class 3-Public-Primary-Certification-Authority-G5.pem
2017-01-18 18:33:20,813 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Key from: my-private.pem.key
2017-01-18 18:33:20,814 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Cert from: my-certificate.pem.crt
2017-01-18 18:33:20,814 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: baseReconnectTime = 1 sec
2017-01-18 18:33:20,815 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: maximumReconnectTime = 32 sec
2017-01-18 18:33:20,816 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: minimumConnectTime = 20 sec
2017-01-18 18:33:20,816 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum connect/disconnect timeout to be 10 second.
2017-01-18 18:33:20,817 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum MQTT operation timeout to be 5 second
2017-01-18 18:33:20,819 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connection type: TLSv1.2 Mutual Authentication
Traceback (most recent call last):
  File "ThingShadowEcho.py", line 142, in <module>
    myAWSIoTMQTTShadowClient.connect()
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/MQTTLib.py", line 808, in connect
    return self._AWSIoTMQTTClient.connect(keepAliveIntervalSecond)
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/MQTTLib.py", line 403, in connect
    return self._mqttCore.connect(keepAliveIntervalSecond)
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/core/protocol/mqttCore.py", line 290, in connect
    self._pahoClient.connect(self._host, self._port, keepAliveInterval)  # Throw exception...
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/core/protocol/paho/client.py", line 655, in connect
    return self.reconnect()
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/core/protocol/paho/client.py", line 809, in reconnect
    self._sock.setblocking(0)
OSError: [Errno 9] Bad file descriptor

The error is repeated on all samples from this repository. It seems that the problem is within the PAHO library. This problem seems to have been solved by a bugfix (issue).
As I understand in other Python 3 versions this code does not produce exception, but ssl-socket still in blocking mode.

[liuszeng]

Hi @anikey-m ,

We were able to duplicate the issue. Thank you very much for pointing this out.

As a work around for now, you can manually update the lines (line 806 and line 807) as follows:

...
self._sock = sock

if self._ssl and not self._useSecuredWebsocket:
    self._ssl.setblocking(0)  # For X.509 cert mutual auth.
elif not self._ssl:
    self._sock.setblocking(0)  # For plain socket
else:
    pass  # For MQTT over WebSocket

return self._send_connect(self._keepalive, self._clean_session)

You should then be able to use the SDK as usual.

We will incorporate the fix into the next release after it goes through our internal testing/release process. Sorry for the inconvenience.

Thanks,
Liusu

[mcintyremason]

Tried this patch in both 2.7.10 and 3.6.0, and no resolve. For whatever reason, the 2.7.13 version works just fine. That is the current version that installs through brew, 'brew install python'.

Any word on the patch, or an update for 3.6?

[liuszeng]

Hi @mcintyremason ,

Thank you very much for your interest in AWS IoT Python SDK.

Sorry that the work-around is not working for you. One thing you may want to check is that you made changes to the correct SDK source files. If you have different versions of Python installed, you will need to go to the site-packages directory for that specific Python version. On macOS, for example, the Python 3.6 installation directory should look similar to this:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/AWSIoTPythonSDK

Can you share the traceback when you are having the issue using the SDK with Python 3.6? It would be really helpful for us to detect some undiscovered issues or do further investigation for you.

Thanks,
Liusu

[mcintyremason]

I believe this has everything you're asking for. If there's anything else I can provide, please let me know.

[liuszeng]

Hi @mcintyremason ,

Thank you very much for providing the information.

Can you verify again that the patch has been applied to the following SDK source file?

/usr/local/lib/python3.6/site-packages/AWSIoTPythonSDK/core/protocol/paho/client.py

Thanks,
Liusu

[mcintyremason]

I must have applied the patch in the wrong place before. It was missing from the file you mentioned in the file above. I added the patch, and everything seems to be working as expected.

Thank you!
Mason

[heri16]

This is bad and should be fixed. Can't believe it has been so many months since this issue is opened and AWS is still providing such poor quality libraries and support. This patch should be released to pip asap! @liuszeng

[LunkwillAndFook]

I spun wheels for several hours before finally deciding that the SDK just didn't work, at which point I found this issue. This was reported back in January. How is this not patched and released to pip yet?

[lightblu]

+1

Btw, this is already fixed in upstream paho:
https://github.com/eclipse/paho.mqtt.python/blob/master/src/paho/mqtt/client.py#L854

[daviian]

But when will it be fixed in the sdk!?
Only a few lines of code and still not fixed after months.

[liuszeng]

Addressed in v1.1.2.