fix the bytearray has not .encode() error

[wwqgtxx]

No description provided.

[miguelgrinberg]

@wwqgtxx what payload did you send from the client that ended up being a bytearray on the server side?

[wwqgtxx]

if the websocket opcode is 0x2

[wwqgtxx]

As we know,the socket.io protocol alow the binary data send by two way.One is send a base64encode data which start with 'b4',the other one is send a raw bytes data which start with b'0x04'.If want to send a websocket raw bytes data, It need to set websocket opcode to 0x2,than the python-engineio had receive the bytearray data from client.

[miguelgrinberg]

Sorry, but it seems you are confusing websocket with Socket.IO. My question was which Socket.IO client sends packets that on the server appear as byte arrays. All the tests I've done with the official Socket.IO clients send bytes, not byte arrays.

If you are using a plain websocket client and expecting Socket.IO on the server will understand that then that's the problem, these two protocols are not compatible, you need to use a Socket.IO client.

[wwqgtxx]

I am trying to make python's socketIO-client support binary data send,and add this pr to socketIO-client
invisibleroads/socketIO-client#144
I had test that if I send a base64 encode binay data, everythings will be fine.But it well waste many space because of base64.So I try to send the same package like python-socketio and it cause the python-engineio recevied the bytearray data.

the data send and receive like this (log by client):

13:17:00{SocketIOTunnel}transports.py[line:166]<send_packet> pid:10096 DummyThread-1 INFO : 451-["data", {"_placeholder": true, "num": 0}]
13:17:00{SocketIOTunnel}transports.py[line:166]<send_packet> pid:10096 DummyThread-1 INFO : b'\x04\xe2\x94\xe1}\x1a\x00\x00\x00\xf0\x0b\xe2\x0b\xb1\xf7\x02\x15\x7f\x1b\xd8\xb66\xac\x19\xf3ZD\x97R\xd8]\x9d\x91\x11\x99\xd1{'

13:17:01{SocketIOTunnel}transports.py[line:156]<recv_packet> pid:10096 DummyThread-5 INFO : 451-["data",{"_placeholder":true,"num":0}]
13:17:01{SocketIOTunnel}transports.py[line:156]<recv_packet> pid:10096 DummyThread-5 INFO : b'\x04\xb6\\\xaaH\x14\x01\x00\x00\xf0\xff\x06\xf0m\x81C\x01\x15\x1c}_v\x1b\x0e\xe0\xf4\x9a\x1a\x83W8\xf459!\xeb\xcf\xd8q)#Ph\x18\xede\xdd\x175\xda\xc1\x1c\rK\x93\xdf/\x06\x8eP\xa1\xed\xf8?U@<&G&n(=\xeaW<R\x1ed\xa5+\xf7\xe7\xb3EC\xd0\xacb\xcf\x0e~3.q\xd8\xbat"\xe9\xf7i\x9e\x7f\xed8\xe4\xce\xc5\xff\xe3\xfcfwL=]\n`\x87\xf9\xd53\tE\xefw\xef\xbc\xb2\x9d7\xea\x9c\xa8\xca\xa5k\xb1\xe4\x97j\x91\xa8\xf8M:\xdb\x80\xdec\xa9\\\xdc\x15n\x8e\xcc\x9b\x96\x8f\x98\x8a\xfd \xe7M\xd5S\xbd6\x03j\x99\xcb[5\xact\x00\x9c\xe7s.F.\xc3\xa1\xcaQpA\xf2(\xd4\x0b$DL\x8e\xc2\x96l\x15\xe1\xaf\xa9-\x9d\x000[[\xf73\xdf\x03\x1f9\x1eB\xea\x80\xd7\x86\xdc\xf4\x8e\xa2\xbb\x7f(_;\xcd\xfc\xb6\x0b\x0e\xd7[\xaf\x8e\x0e\xc5(\x01D\x90\x11\x9c\x1b;\x82\x1e\xa58\n.\x7f\xce\xd9\x8bGl\xd7\'mww\x01\xd8\x08DE\xfa\xeb\x0eu\xc4,\xac\x99\xfe\'\xae\xbc\xba'

[miguelgrinberg]

Ah, okay, that makes sense. So this is the python Socket.IO client.

On the server, are you using eventlet, gevent or uwsgi?

[wwqgtxx]

I use gevent.

[wwqgtxx]

I found this problem cause by geventsebsocket:

    def read_message(self):
        """
        Return the next text or binary message from the socket.

        This is an internal method as calling this will not cleanup correctly
        if an exception is called. Use `receive` instead.
        """
        opcode = None
        message = bytearray()

        while True:
            header, payload = self.read_frame()
            f_opcode = header.opcode

            if f_opcode in (self.OPCODE_TEXT, self.OPCODE_BINARY):
                # a new frame
                if opcode:
                    raise ProtocolError("The opcode in non-fin frame is "
                                        "expected to be zero, got "
                                        "{0!r}".format(f_opcode))

                # Start reading a new message, reset the validator
                self.utf8validator.reset()
                self.utf8validate_last = (True, True, 0, 0)

                opcode = f_opcode

            elif f_opcode == self.OPCODE_CONTINUATION:
                if not opcode:
                    raise ProtocolError("Unexpected frame with opcode=0")

            elif f_opcode == self.OPCODE_PING:
                self.handle_ping(header, payload)
                continue

            elif f_opcode == self.OPCODE_PONG:
                self.handle_pong(header, payload)
                continue

            elif f_opcode == self.OPCODE_CLOSE:
                self.handle_close(header, payload)
                return

            else:
                raise ProtocolError("Unexpected opcode={0!r}".format(f_opcode))

            if opcode == self.OPCODE_TEXT:
                self.validate_utf8(payload)

            message += payload

            if header.fin:
                break

        if opcode == self.OPCODE_TEXT:
            self.validate_utf8(message)
            return str(message, "utf-8")
        else:
            return message

because it recevie a OPCODE_BINARY so it will return a bytearray without decode to str

[wwqgtxx]

Above geventwebsocket is from package karellen-geventws because I use python3.5.
But I see the origin geventwebsocket do the same things in the code

    def read_message(self):
        """
        Return the next text or binary message from the socket.

        This is an internal method as calling this will not cleanup correctly
        if an exception is called. Use `receive` instead.
        """
        opcode = None
        message = ""

        while True:
            header, payload = self.read_frame()
            f_opcode = header.opcode

            if f_opcode in (self.OPCODE_TEXT, self.OPCODE_BINARY):
                # a new frame
                if opcode:
                    raise ProtocolError("The opcode in non-fin frame is "
                                        "expected to be zero, got "
                                        "{0!r}".format(f_opcode))

                # Start reading a new message, reset the validator
                self.utf8validator.reset()
                self.utf8validate_last = (True, True, 0, 0)

                opcode = f_opcode

            elif f_opcode == self.OPCODE_CONTINUATION:
                if not opcode:
                    raise ProtocolError("Unexpected frame with opcode=0")

            elif f_opcode == self.OPCODE_PING:
                self.handle_ping(header, payload)
                continue

            elif f_opcode == self.OPCODE_PONG:
                self.handle_pong(header, payload)
                continue

            elif f_opcode == self.OPCODE_CLOSE:
                self.handle_close(header, payload)
                return

            else:
                raise ProtocolError("Unexpected opcode={0!r}".format(f_opcode))

            if opcode == self.OPCODE_TEXT:
                self.validate_utf8(payload)

            message += payload

            if header.fin:
                break

        if opcode == self.OPCODE_TEXT:
            self.validate_utf8(message)
            return message
        else:
            return bytearray(message)

[miguelgrinberg]

Okay, that explains it. Thanks for the detailed analysis. I did not find this problem myself because I rarely use gevent-websocket for my testing, since it has never been ported to Python 3. I should give the 3.5 fork a try, I guess.

[miguelgrinberg]

@wwqgtxx Thanks, the PR is now merged. I have added a couple other places where binary types were being checked, plus some unit tests.

[wwqgtxx]

Only a suggest that you can try to use ws4py package to support the gevent on both python2 and python3.
I had read this package's doc and think it can provide all the operater that python-engineio will used.

[miguelgrinberg]

Thanks, I will take a look at ws4py and see if I can integrate it.