"HAP PDU not recognized or supported" when trying to lock/unlock the Yale (Homekit enabled BLE) lock

[GitHub-0614]

Hi, I am trying to use this to control a Yale smart lock, a Homekit enabled smart lock using BLE.
Everything works fine (pair, list_pairings, get_accessories, get_characteristic, etc.) until I try to lock/unlock the lock with the put_characteristic command.
According to the DEBUG log, a status code of 1 is returned, indicating "The request failed as the HAP PDU was not recognized or supported".
Any idea how to fix this?
Thanks.

More details:
get the accessories:
root@linux:/home/me/Desktop# python3 -m homekit.get_accessories -f Yale -a Yale
1.900: >Unknown Service: 00001530-1212-EFDE-1523-785FEABCD123<
1.901: (DFU Control Point) >Unknown Characteristic 00001531-1212-EFDE-1523-785FEABCD123< [pw,hd]
1.902: (Service Signature) >service-signature< [hd]
1.41: >battery<
1.44: (Charging_state) >charging-state< [pr]
1.45: (Status_low_battery) >status-lo-batt< [pr,evc,evd]
1.43: (Battery_level) >battery-level< [pr]
1.141: >lock-management<
1.149: (Lock_last_known_action) >lock-mechanism.last-known-action< [pr]
1.148: (Administrator_only_access) >administrator-only-access< [pr,pw]
1.146: (Audio_feedback) >audio-feedback< [pr,pw]
1.147: (Lock_management_auto_security_timeout) >lock-management.auto-secure-timeout< [pr,pw]
1.143: (Lock_control_point) >lock-management.control-point< [pw]
1.144: (Version) >version< [pr]
1.145: (Logs) >logs< [pr]
1.152: >lock-mechanism<
1.156: (Name) >name< [pr]
1.155: (Lock_target_state) >lock-mechanism.target-state< [tw,pr,pw,evc,evd]
1.154: (Lock_current_state) >lock-mechanism.current-state< [pr,evc,evd]
1.9: >pairing<
1.13: (Pairing Pairings) >pairing.pairings< [pr,pw]
1.12: (Pairing Features) >pairing.features< [r]
1.11: (Pair Verify) >pairing.pair-verify< [r,w]
1.10: (Pair Setup) >pairing.pair-setup< [r,w]
1.7: >service<
1.8: (Protocol Version) >version< [pr]
1.1: >accessory-information<
1.15: (Hardware Revision) >hardware.revision< [pr]
1.14: (Firmware Revision) >firmware.revision< [pr]
1.6: (Serial Number) >serial-number< [pr]
1.2: (Name) >name< [pr]
1.5: (Model) >model< [pr]
1.4: (Manufacturer) >manufacturer< [pr]
1.3: (Identify) >identify< [w,pw]

read the Lock_target_state:
root@linux:/home/me/Desktop# python3 -m homekit.get_characteristic -f Yale -a Yale -c 1.155
{
"1.155": {
"value": 1
}
}

try to write the Lock_target_state:
root@linux:/home/me/Desktop# python3 -m homekit.put_characteristic -f Yale -a Yale -c 1.155 0 --log DEBUG
2020-01-31 12:59:48,664 init.py:0431 DEBUG connecting to device
2020-01-31 12:59:48,938 device.py:0093 DEBUG waiting for services to be resolved
2020-01-31 12:59:49,943 device.py:0103 DEBUG enumerating resolved services
2020-01-31 12:59:50,174 init.py:0433 DEBUG connected to device
2020-01-31 12:59:50,209 tlv.py:0134 DEBUG sending [
6: (1 bytes/<class 'bytearray'>) 0x01
3: (32 bytes/<class 'bytes'>) b'\x9fW4r8`\xdc\xb6E\xde\x8a\x0b\xa2\x93\n\xbb$8i\x06\x91\xc6\xa8G|\xb4\xc4\x03\x1e\x03Rw'
]

2020-01-31 12:59:50,210 tlv.py:0117 DEBUG receiving [
6: (1 bytes/<class 'bytearray'>) 0x01
3: (32 bytes/<class 'bytearray'>) 0x9f5734723860dcb645de8a0ba2930abb2438690691c6a8477cb4c4031e035277
]

2020-01-31 12:59:50,210 init.py:0610 DEBUG entering write function [
6: (1 bytes/<class 'bytearray'>) 0x01
3: (32 bytes/<class 'bytearray'>) 0x9f5734723860dcb645de8a0ba2930abb2438690691c6a8477cb4c4031e035277
]

2020-01-31 12:59:50,210 tlv.py:0134 DEBUG sending [
9: (1 bytes/<class 'bytearray'>) 0x01
1: (37 bytes/<class 'bytearray'>) 0x06010103209f5734723860dcb645de8a0ba2930abb2438690691c6a8477cb4c4031e035277
]

2020-01-31 12:59:50,211 init.py:0620 DEBUG sent 0002160b002a00090101012506010103209f5734723860dcb645de8a0ba2930abb2438690691c6a8477cb4c4031e035277
2020-01-31 12:59:51,212 init.py:0625 DEBUG reading characteristic
2020-01-31 12:59:51,388 init.py:0633 DEBUG control field: 2, tid: 16, status: 0, length: 142
2020-01-31 12:59:51,388 init.py:0641 DEBUG received 0216008e00018c06010203206c2468a2b85eb0fb04285598a30f63acf471c75c070b9c28680219ddfd60f5200565c8ae1963ff78b7a4fc6e2f6ac2b82978c1e14ed55644a47d9d029ef1f9ecfc19996b686303a638638b7fe0db7a61e5a7a9df456084ff4ebfa1538b562da45517cea67c0e9b03238e8784c146e535cb64bd5abe31ef581aca06e3523a15cebcdccc346bf33d
2020-01-31 12:59:51,389 init.py:0642 DEBUG decode 018c06010203206c2468a2b85eb0fb04285598a30f63acf471c75c070b9c28680219ddfd60f5200565c8ae1963ff78b7a4fc6e2f6ac2b82978c1e14ed55644a47d9d029ef1f9ecfc19996b686303a638638b7fe0db7a61e5a7a9df456084ff4ebfa1538b562da45517cea67c0e9b03238e8784c146e535cb64bd5abe31ef581aca06e3523a15cebcdccc346bf33d
2020-01-31 12:59:51,389 tlv.py:0117 DEBUG receiving [
1: (140 bytes/<class 'bytearray'>) 0x06010203206c2468a2b85eb0fb04285598a30f63acf471c75c070b9c28680219ddfd60f5200565c8ae1963ff78b7a4fc6e2f6ac2b82978c1e14ed55644a47d9d029ef1f9ecfc19996b686303a638638b7fe0db7a61e5a7a9df456084ff4ebfa1538b562da45517cea67c0e9b03238e8784c146e535cb64bd5abe31ef581aca06e3523a15cebcdccc346bf33d
]

2020-01-31 12:59:51,390 tlv.py:0117 DEBUG receiving [
6: (1 bytes/<class 'bytearray'>) 0x02
3: (32 bytes/<class 'bytearray'>) 0x6c2468a2b85eb0fb04285598a30f63acf471c75c070b9c28680219ddfd60f520
5: (101 bytes/<class 'bytearray'>) 0xc8ae1963ff78b7a4fc6e2f6ac2b82978c1e14ed55644a47d9d029ef1f9ecfc19996b686303a638638b7fe0db7a61e5a7a9df456084ff4ebfa1538b562da45517cea67c0e9b03238e8784c146e535cb64bd5abe31ef581aca06e3523a15cebcdccc346bf33d
]

2020-01-31 12:59:51,390 init.py:0645 DEBUG leaving write function [
6: (1 bytes/<class 'bytearray'>) 0x02
3: (32 bytes/<class 'bytearray'>) 0x6c2468a2b85eb0fb04285598a30f63acf471c75c070b9c28680219ddfd60f520
5: (101 bytes/<class 'bytearray'>) 0xc8ae1963ff78b7a4fc6e2f6ac2b82978c1e14ed55644a47d9d029ef1f9ecfc19996b686303a638638b7fe0db7a61e5a7a9df456084ff4ebfa1538b562da45517cea67c0e9b03238e8784c146e535cb64bd5abe31ef581aca06e3523a15cebcdccc346bf33d
]

2020-01-31 12:59:51,394 tlv.py:0117 DEBUG receiving [
1: (17 bytes/<class 'bytearray'>) 0x34333a41373a46433a38333a46333a4239
10: (64 bytes/<class 'bytearray'>) 0xf56fbb18db517fbba5e48c4265595222c3d14a9a88a13b0eb9961ab096ed9a461f103b95d2ac47cea4c1fc90ac3edf6591349c4ab179f7bbae5b84630ad6d608
]

2020-01-31 12:59:51,400 tlv.py:0134 DEBUG sending [
1: (36 bytes/<class 'bytes'>) b'fe42eb2e-0ebd-4be6-8d34-6bee20e964ec'
10: (64 bytes/<class 'bytes'>) b"\x07\x00Hp\x8b\xa2'\xe9\xa7\xccz\x99\x9c\x87o~\xde+:%\xbf\x17\xfd\x1c\x90\x12\xe0\xbaG\xf4\xfd\x0e.\xd7=\x8f\xc4\xe4s\xf2\xd6@\xc8\x91\x9f\xf0\xc9blw)\x80S\x18\xcdb\x87\xe4\x14\x8eLl\xa5\x01"
]

2020-01-31 12:59:51,401 tlv.py:0134 DEBUG sending [
6: (1 bytes/<class 'bytearray'>) 0x03
5: (120 bytes/<class 'bytearray'>) 0xfc96f288a7a769fb265db002e1c0b6039ef707034d87c2752bdd04dbc6523440df6e49f447a853fe2584eb8ca514fbcf70a30a27f10504afd3f4d28cb8157f1584d18dd2b9a65ae2db3dbc4d3cb3c703c918538847806b20edaac343edab9db035a2b2514385ab358d71ba5ca09513f1179651ef0f2fba33
]

2020-01-31 12:59:51,401 tlv.py:0117 DEBUG receiving [
6: (1 bytes/<class 'bytearray'>) 0x03
5: (120 bytes/<class 'bytearray'>) 0xfc96f288a7a769fb265db002e1c0b6039ef707034d87c2752bdd04dbc6523440df6e49f447a853fe2584eb8ca514fbcf70a30a27f10504afd3f4d28cb8157f1584d18dd2b9a65ae2db3dbc4d3cb3c703c918538847806b20edaac343edab9db035a2b2514385ab358d71ba5ca09513f1179651ef0f2fba33
]

2020-01-31 12:59:51,402 init.py:0610 DEBUG entering write function [
6: (1 bytes/<class 'bytearray'>) 0x03
5: (120 bytes/<class 'bytearray'>) 0xfc96f288a7a769fb265db002e1c0b6039ef707034d87c2752bdd04dbc6523440df6e49f447a853fe2584eb8ca514fbcf70a30a27f10504afd3f4d28cb8157f1584d18dd2b9a65ae2db3dbc4d3cb3c703c918538847806b20edaac343edab9db035a2b2514385ab358d71ba5ca09513f1179651ef0f2fba33
]

2020-01-31 12:59:51,402 tlv.py:0134 DEBUG sending [
9: (1 bytes/<class 'bytearray'>) 0x01
1: (125 bytes/<class 'bytearray'>) 0x0601030578fc96f288a7a769fb265db002e1c0b6039ef707034d87c2752bdd04dbc6523440df6e49f447a853fe2584eb8ca514fbcf70a30a27f10504afd3f4d28cb8157f1584d18dd2b9a65ae2db3dbc4d3cb3c703c918538847806b20edaac343edab9db035a2b2514385ab358d71ba5ca09513f1179651ef0f2fba33
]

2020-01-31 12:59:51,402 init.py:0620 DEBUG sent 0002bb0b008200090101017d0601030578fc96f288a7a769fb265db002e1c0b6039ef707034d87c2752bdd04dbc6523440df6e49f447a853fe2584eb8ca514fbcf70a30a27f10504afd3f4d28cb8157f1584d18dd2b9a65ae2db3dbc4d3cb3c703c918538847806b20edaac343edab9db035a2b2514385ab358d71ba5ca09513f1179651ef0f2fba33
2020-01-31 12:59:52,403 init.py:0625 DEBUG reading characteristic
2020-01-31 12:59:52,498 init.py:0633 DEBUG control field: 2, tid: bb, status: 0, length: 5
2020-01-31 12:59:52,498 init.py:0641 DEBUG received 02bb0005000103060104
2020-01-31 12:59:52,498 init.py:0642 DEBUG decode 0103060104
2020-01-31 12:59:52,499 tlv.py:0117 DEBUG receiving [
1: (3 bytes/<class 'bytearray'>) 0x060104
]

2020-01-31 12:59:52,499 tlv.py:0117 DEBUG receiving [
6: (1 bytes/<class 'bytearray'>) 0x04
]

2020-01-31 12:59:52,499 init.py:0645 DEBUG leaving write function [
6: (1 bytes/<class 'bytearray'>) 0x04
]

2020-01-31 12:59:52,500 init.py:0476 DEBUG pair_verified, keys:
c2a: 45a14d05b2bb3ac0ba43cb314f2aee9491b8d8550534b454f3a56271ea2a449a
a2c: b41abd3794ef9d9f1f67c850557d7c1dc1031a33a4f060e17338325dc45e539a
2020-01-31 12:59:52,500 init.py:0209 DEBUG value: 0 format: uint8
2020-01-31 12:59:52,501 tlv.py:0134 DEBUG sending [
1: (1 bytes/<class 'bytes'>) b'\x00'
]

2020-01-31 12:59:52,501 init.py:0498 DEBUG body: b'\x03\x00\x01\x01\x00'
2020-01-31 12:59:52,502 init.py:0501 DEBUG data: bytearray(b'\x00\x02\xeb\x9b\x00\x03\x00\x01\x01\x00')
2020-01-31 12:59:52,503 init.py:0507 DEBUG cipher and mac 7bd361746e4c8d1b324c403ebf6661a6764374dc66007cd836d9
2020-01-31 12:59:52,503 init.py:0510 DEBUG write resulted in: None
2020-01-31 12:59:53,505 init.py:0517 DEBUG reading characteristic
2020-01-31 12:59:53,645 init.py:0523 DEBUG read: 1618e502c1a371f4c0c11b1c2543b096d00d46
2020-01-31 12:59:53,649 init.py:0528 DEBUG decrypted: 02eb01
2020-01-31 12:59:53,649 init.py:0534 DEBUG parse sig read response 02eb01
2020-01-31 12:59:53,649 init.py:0538 DEBUG control field 2
2020-01-31 12:59:53,649 init.py:0540 DEBUG transaction id 235 (expected was 235)
2020-01-31 12:59:53,649 init.py:0544 DEBUG status code 1 (The request failed as the HAP PDU was not recognized or supported.)
2020-01-31 12:59:53,649 init.py:0485 DEBUG closing session
descriptor 'init' requires a 'Exception' object but received a 'int'
2020-01-31 12:59:56,178 put_characteristic.py:0071 DEBUG descriptor 'init' requires a 'Exception' object but received a 'int'
Traceback (most recent call last):
File "/root/.local/lib/python3.6/site-packages/homekit/put_characteristic.py", line 68, in
results = pairing.put_characteristics(characteristics, do_conversion=True)
File "/root/.local/lib/python3.6/site-packages/homekit/controller/ble_impl/init.py", line 376, in put_characteristics
raise e
File "/root/.local/lib/python3.6/site-packages/homekit/controller/ble_impl/init.py", line 365, in put_characteristics
response = self.session.request(fc, cid, HapBleOpCodes.CHAR_WRITE, body)
File "/root/.local/lib/python3.6/site-packages/homekit/controller/ble_impl/init.py", line 550, in request
raise RequestRejected(status, HapBleStatusCodes[status])
File "/root/.local/lib/python3.6/site-packages/homekit/exceptions.py", line 250, in init
Exception.init(message)
TypeError: descriptor 'init' requires a 'Exception' object but received a 'int'
2020-01-31 12:59:56,204 init.py:0485 DEBUG closing session

[a3135134]

The problem may be caused by wrong OpCode. This characteristic requires a tw (Timed Write Procedure) operation. But in homekit\controller\ble_impl_init_.py line 365, only HapBleOpCodes.CHAR_WRITE is used.
Please see HAP doc section 7.3.5 for details. Will you support more Procedures?

[GitHub-0614]

I agree with @a3135134 .
As per HAP Specification：

7.3.5.4 HAP Characteristic Timed Write Procedure
The HAP characteristic timed write procedure will write to characteristic that require time sensitive actions. An example is security class characteristics (Lock Target State, Target Door State, etc.). The accessory must indicate the characteristic requires the timed write in the HAP Characteristics Properties descriptor. An accessory must support timed writes to all characteristics even if the characteristic property does not require it.

The HAP characteristic timed write procedure shall be used to securely execute a write command within a specified TTL. The accessory must start the TTL timer after sending the HAP-Characteristic-Timed-Write-Response. The scheduled request shall be executed only if the accessory receives the HAP-Characteristic-ExecuteWrite-Request before its TTL timer expires.

[GitHub-0614]

@jlusiardi @a3135134
I managed to change the codes to successfully send lock/unlock commands.
see the codes below (homekit/controller/ble_impl/init.py):

def put_characteristics(self, characteristics, do_conversion=False):
...
# procedure(1) send CHAR_TIMED_WRITE requst
value = TLV.encode_list([(1, self._convert_from_python(aid, cid, value))])
# the value of TTL is bit tricky, a larger number is prefered
# otherwise, the procedure(2) would fail because of time out
TTLvalue = TLV.encode_list([(TLV.kTLVHAPParamTTL, bytearray(b'\x0f'))])
body = (len(value)+len(TTLvalue)).to_bytes(length=2, byteorder='little') + value + TTLvalue
try:
fc, fc_info = self.session.find_characteristic_by_iid(cid)
response = self.session.request(fc, cid, HapBleOpCodes.CHAR_TIMED_WRITE, body)
logger.debug('response %s', response)
except:
...

        # procedure (2) send CHAR_EXEC_WRITE request
        body = None
     try:
            response = self.session.request(fc, cid, HapBleOpCodes.CHAR_EXEC_WRITE, body)
    except:
           ...

Just a rough implementation to only send lock/unlock commands with the "put_characteristic" command.
Perhaps you could consider adding more options/parameters to the "put_characteristic" command for specifying timed-write for devices (e.g., lock) that require it.

[jlusiardi]

Hi @GitHub-0614,

Just to be sure, what the timed write does here: will it change the target-state of the lock for 0x0f* 100ms = 1,5s (to open most probably)? Whick lock do you own exactly, perhaps I can get one myself.

I am not really sure if I got your change correctly. Can you please create a diff?

Is the requirement, to add an option to set the kind of write for some characteristics? Or can we figure out which kind of write to use?

[GitHub-0614]

Hi @jlusiardi
Different characteristics require different types of write request. As far as I know, there are at least two types (and probably the 2 most common types): Characteristic Write and Characteristic Timed Write.

1. Characteristic Write
This is what you already did in your codes (line 365, homekit/controller/ble_impl/init.py):
response = self.session.request(fc, cid, HapBleOpCodes.CHAR_WRITE, body)

CHAR_WRITE works for characteristics that require "paired write" (pw) only, such as the power on/off characteristic of a smart bulb. For example, the characteristic 1.15 of my SYL Bluetooth bulb as below:

1.20: >lightbulb<
1.16: () >name< [pr]
1.15: () >on< [pr,pw,evc,evd]
1.25: () >saturation< [pr,pw,evc,evd]
1.24: () >hue< [pr,pw,evc,evd]
1.23: () >brightness< [pr,pw,evc,evd]
1.19: () >service-signature< [r]

2. Characteristic Timed Write
There are other characteristics that require "timed write" (tw), for example, 1.155 of my Yale lock.

1.152: >lock-mechanism<
1.156: (Name) >name< [pr]
1.155: (Lock_target_state) >lock-mechanism.target-state< [tw,pr,pw,evc,evd]
1.154: (Lock_current_state) >lock-mechanism.current-state< [pr,evc,evd]

To change/write the Lock_target_state, which requires tw, the current "put_characteristic" command won't work, because it always sends CHAR_WRITE request, regardless of the target characteristic.

To do that, as per HAP, we need to use a "HAP Characteristic Timed Write Procedure", which contains 2 steps.
Step 1: send a HAP-Characteristic-Timed-Write-Request, with the parameters of value and TTL
Step 2: within the TTL in step, send a HAP-Characteristic-Execute-Write-Request

The value indicates to lock or unlock the lock.
The TTL indicates the HAP-Characteristic-Execute-Write-Request must be received by the lock within the time.
After both steps are successfully done, the Lock_target_state will be changed as instructed.
Note that, if the lock doesn't receive a HAP-Characteristic-Execute-Write-Request until TTL expires, the write to Lock_target_state would fail. This is what I meant in the former reply:

the value of TTL is bit tricky, a larger number is preferred
otherwise, the procedure(2) would fail because of time out

[GitHub-0614]

This is the modified codes.
__init__timed_write.py.txt

[GitHub-0614]

I only modified it so that I can use "put_characterisitc" command to unlock/lock my lock.
However, to make "put_characterisitc" command support all characteristics (e.g., both pw and tw), more modification is needed.
We can let the user choose, by adding more parameters to the command, such as -tw, -ttl, etc.

Or, we can automatically choose which procedure (e.g, the CHAR_WRITE procedure or the CHAR_TIMED_WRITE procedure) to use based on the target characteristic, since we can know whether it requires pw or tw by getting its features from something like "get_accessorries", e.g.,:

1.155: (Lock_target_state) >lock-mechanism.target-state< [tw,pr,pw,evc,evd]

1.15: () >on< [pr,pw,evc,evd]

[GitHub-0614]

The lock I used:
The "Apple Homekit enabled" version

https://www.amazon.com/Yale-Security-YRD256-iM1-605-Assure-HomeKit/dp/B073V7SG1H/ref=sr_1_1_sspa?keywords=yale+assure+lock+sl&qid=1579815104&sr=8-1-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUEyT0EwOEFKSkE1S05UJmVuY3J5cHRlZElkPUEwMTY5Mzc3MTlGT05MM1pSRU4xQSZlbmNyeXB0ZWRBZElkPUEwNzIxNTY0MTVFUVUxQk8zRjBCViZ3aWRnZXROYW1lPXNwX2F0ZiZhY3Rpb249Y2xpY2tSZWRpcmVjdCZkb05vdExvZ0NsaWNrPXRydWU=

[belkinhappyzj]

Hi jlusiardi , may I know if we can add support for "CHAR_TIMED_WRITE procedure"? In my case, I also need to use this timed write method and it seems there are many HK devices in the field contains key CHAR which support timed write only (a CHAR_write in this case would be rejected), thanks.

[jlusiardi]

Oh I did forget about this issue completly. So you would be able to test the specific time writes if i make changes to the code?

[belkinhappyzj]

Oh I did forget about this issue completly. So you would be able to test the specific time writes if i make changes to the code?

Hi jlusiardi, sure thing, I can test it for you once you are done code change, thank you.

[jlusiardi]

Hi,
ok, I'll try to have a look at it, but I cannot promise anything soon.
Joachim

[los93sol]

I can verify that Level bolt requires timed write as well.