Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Base64 validation issue when handling binary payload sent with SENML_JSON format #1548

Closed
vshorin opened this issue Nov 27, 2023 · 3 comments
Closed

Base64 validation issue when handling binary payload sent with SENML_JSON format #1548

vshorin opened this issue Nov 27, 2023 · 3 comments
Labels
bug Dysfunctionnal behavior core Impact core of Leshan

Comments

@vshorin
Copy link

vshorin commented Nov 27, 2023

Version(s)

v2.0.0-M11 and up

Which components

leshan-core

Tested With

leshan-client-demo

What happened

When sending object 19 with a "send" operation from the leshan client, the server returns [BAD_REQUEST(400)] : Invalid Payload. if SENML_JSON is used and the Base64 version of the payload contains "w" in the end. If SENML_CBOR is used on the same operation, the server returns [CHANGED(204)]..
The server is using org.eclipse.leshan.core.util.base64.DefaultBase64Decoder method validateIsCanonical. There seems to be a typo here:

leshan/leshan-core/src/main/java/org/eclipse/leshan/core/util/base64/DefaultBase64Decoder.java

Line 185 in fa6f7d2

validChar = new char[] { 'A', 'Q', 'g', 'W' };

and

leshan/leshan-core/src/main/java/org/eclipse/leshan/core/util/base64/DefaultBase64Decoder.java

Line 197 in fa6f7d2

validChar = new char[] { 'A', 'Q', 'g', 'W', 'E', 'U', 'k', '0', 'I', 'Y', 'o', '4', 'M', 'c', 's', '8' };

where validChar array contains "W", but it seems that it should have contained "w" instead.

How to reproduce

  1. Have a leshan client that has object 19 defined
  2. Resource 19/0/0 should have value that ends with padded "w" when base64 encoded, like "ABCDEFG" => "QUJDREVGRw==" or "ABCDL" => "QUJDREw=":
public class FakeBinaryObject extends BaseInstanceEnabler {
...
    private byte[] value;

    public FakeBinaryObject() {
        this.value = "ABCDEFG".getBytes();
        // this.value = "ABCDL".getBytes();
    }

    @Override
    public synchronized ReadResponse read(LwM2mServer server, int resourceId) {
        LOG.info("Read on Fake Binary Object resource /{}/{}/{}", getModel().id, getId(), resourceId);
        switch (resourceId) {
        case RESOURCE_DATA:
            return ReadResponse.success(resourceId, this.value);
...
        default:
            return super.read(server, resourceId);
        }
    }
...
  1. Connect to the server
  2. Use send commands with object 19:
send -c SENML_CBOR current-value 19
send -c SENML_JSON current-value 19

Relevant Output

client logs:

send -c SENML_CBOR current-value 19
           InteractiveCommands 2023-11-27 12:47:45,850 [INFO] Sending Data to coaps://127.0.0.1:5684[LWM2M_SERVER 1111] using SENML_CBOR(112).
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/0
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/1
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/2
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/3
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/4
              FakeBinaryObject 2023-11-27 12:47:45,853 [INFO] Read on Fake Binary Object resource /19/0/5
           InteractiveCommands 2023-11-27 12:47:45,888 [INFO] Data sent successfully to coaps://127.0.0.1:5684[LWM2M_SERVER 1111] [CHANGED(204)].
send -c SENML_JSON current-value 19
           InteractiveCommands 2023-11-27 12:47:51,702 [INFO] Sending Data to coaps://127.0.0.1:5684[LWM2M_SERVER 1111] using SENML_JSON(110).
              FakeBinaryObject 2023-11-27 12:47:51,702 [INFO] Read on Fake Binary Object resource /19/0/0
              FakeBinaryObject 2023-11-27 12:47:51,702 [INFO] Read on Fake Binary Object resource /19/0/1
              FakeBinaryObject 2023-11-27 12:47:51,702 [INFO] Read on Fake Binary Object resource /19/0/2
              FakeBinaryObject 2023-11-27 12:47:51,703 [INFO] Read on Fake Binary Object resource /19/0/3
              FakeBinaryObject 2023-11-27 12:47:51,703 [INFO] Read on Fake Binary Object resource /19/0/4
              FakeBinaryObject 2023-11-27 12:47:51,703 [INFO] Read on Fake Binary Object resource /19/0/5
           InteractiveCommands 2023-11-27 12:47:51,734 [INFO] Send data to coaps://127.0.0.1:5684[LWM2M_SERVER 1111] failed [BAD_REQUEST(400)] : Invalid Payload.

server logs:

org.eclipse.leshan.core.request.exception.InvalidRequestException: Invalid payload in [CON-POST   MID=47999, Token=4C65C148D04AAFBD, OptionSet={"Uri-Path":"dp", "Content-Format":"application/senml+json"}, acked "[{"bn":"/19/","n":"0/0","vd":"QU".. 197 bytes] from [Identity /127.0.0.1:60725[psk=test]]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:102) [leshan-server-cf-2.0.0-M11.jar:?]
        at org.eclipse.californium.core.CoapResource.handleRequest(CoapResource.java:261) [californium-core-3.8.0.jar:?]
        at org.eclipse.leshan.core.californium.LwM2mCoapResource.handleRequest(LwM2mCoapResource.java:57) [leshan-core-cf-2.0.0-M11.jar:?]
        at org.eclipse.californium.core.server.ServerMessageDeliverer.deliverRequest(ServerMessageDeliverer.java:133) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.BaseCoapStack$StackTopAdapter.receiveRequest(BaseCoapStack.java:204) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.AbstractLayer.receiveRequest(AbstractLayer.java:84) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.AbstractLayer.receiveRequest(AbstractLayer.java:84) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.BlockwiseLayer.receiveRequest(BlockwiseLayer.java:568) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.ReliabilityLayer.receiveRequest(ReliabilityLayer.java:296) [californium-core-3.8.0.jar:?]     
        at org.eclipse.californium.core.network.stack.AbstractLayer.receiveRequest(AbstractLayer.java:84) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.stack.BaseCoapStack.receiveRequest(BaseCoapStack.java:126) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.CoapEndpoint$1.receiveRequest(CoapEndpoint.java:309) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.core.network.UdpMatcher$2.run(UdpMatcher.java:289) [californium-core-3.8.0.jar:?]
        at org.eclipse.californium.elements.util.SerialExecutor$1.run(SerialExecutor.java:293) [element-connector-3.8.0.jar:?]
        at org.eclipse.californium.core.network.CoapEndpoint$6.run(CoapEndpoint.java:1366) [californium-core-3.8.0.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.eclipse.leshan.core.node.codec.CodecException: Unable to decode nodes : ...
        at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.decodeTimestampedNodes(LwM2mNodeSenMLDecoder.java:214) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decodeTimestampedNodes(DefaultLwM2mDecoder.java:223) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:81) ~[leshan-server-cf-2.0.0-M11.jar:?]
        ... 17 more
Caused by: org.eclipse.leshan.senml.SenMLException: Unable to parse SenML JSON.
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonJacksonEncoderDecoder.fromSenML(SenMLJsonJacksonEncoderDecoder.java:99) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.decodeTimestampedNodes(LwM2mNodeSenMLDecoder.java:198) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decodeTimestampedNodes(DefaultLwM2mDecoder.java:223) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:81) ~[leshan-server-cf-2.0.0-M11.jar:?]
        ... 17 more
Caused by: org.eclipse.leshan.core.util.json.JsonException: Node vd with value 'QUJDREVGRw' is not in valid Base64 format.
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonRecordSerDes.deserialize(SenMLJsonRecordSerDes.java:181) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonRecordSerDes.deserialize(SenMLJsonRecordSerDes.java:33) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.util.json.JacksonJsonSerDes.deserialize(JacksonJsonSerDes.java:65) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]        
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonJacksonEncoderDecoder.fromSenML(SenMLJsonJacksonEncoderDecoder.java:96) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.decodeTimestampedNodes(LwM2mNodeSenMLDecoder.java:198) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decodeTimestampedNodes(DefaultLwM2mDecoder.java:223) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:81) ~[leshan-server-cf-2.0.0-M11.jar:?]
        ... 17 more
Caused by: org.eclipse.leshan.core.util.base64.InvalidBase64Exception: Base64 string QUJDREVGRw is not in canonical form.
        at org.eclipse.leshan.core.util.base64.DefaultBase64Decoder.validateIsCanonical(DefaultBase64Decoder.java:209) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.util.base64.DefaultBase64Decoder.validateEncodedData(DefaultBase64Decoder.java:102) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.util.base64.DefaultBase64Decoder.decode(DefaultBase64Decoder.java:68) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]     
        at org.eclipse.leshan.core.util.base64.DefaultBase64Decoder.decode(DefaultBase64Decoder.java:63) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]     
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonRecordSerDes.deserialize(SenMLJsonRecordSerDes.java:178) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonRecordSerDes.deserialize(SenMLJsonRecordSerDes.java:33) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.util.json.JacksonJsonSerDes.deserialize(JacksonJsonSerDes.java:65) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]        
        at org.eclipse.leshan.senml.json.jackson.SenMLJsonJacksonEncoderDecoder.fromSenML(SenMLJsonJacksonEncoderDecoder.java:96) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.decodeTimestampedNodes(LwM2mNodeSenMLDecoder.java:198) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decodeTimestampedNodes(DefaultLwM2mDecoder.java:223) ~[leshan-core-2.0.0-M11.jar:2.0.0-M11]
        at org.eclipse.leshan.server.californium.send.SendResource.handlePOST(SendResource.java:81) ~[leshan-server-cf-2.0.0-M11.jar:?]
        ... 17 more
@vshorin vshorin added the bug Dysfunctionnal behavior label Nov 27, 2023
@sbernard31 sbernard31 added the core Impact core of Leshan label Nov 27, 2023
sbernard31 added a commit that referenced this issue Nov 27, 2023
@sbernard31
GH-1548: Fix DefaultBase64Decoder rejecting payload finishing by 'W'
aab18a4
@sbernard31
Copy link
Contributor

sbernard31 commented Nov 27, 2023 •
edited
Loading

@vshorin very good catch, very well described !

Thx a lot for taking time to report this 🙏

This is now fixed in master (commit aab18a4) and this will be available in next release 2.0.0-M15. (not yet planned, we just release the 2.0.0-M14)

Waiting the workaround is to copy/paster the fixed version and configure your client/server/bsserver like :

Map<ContentFormat, NodeDecoder> decoders = new HashMap<>(DefaultLwM2mDecoder.getDefaultNodeDecoders(false));
decoders.put(ContentFormat.TEXT, new LwM2mNodeTextDecoder(new DefaultLwM2mLinkParser(),
        new FixedDefaultBase64Decoder(DecoderAlphabet.BASE64, DecoderPadding.REQUIRED)));
decoders.put(ContentFormat.SENML_JSON,
        new LwM2mNodeSenMLDecoder(new SenMLJsonJacksonEncoderDecoder(false,
                new FixedDefaultBase64Decoder(DecoderAlphabet.BASE64URL, DecoderPadding.FORBIDEN),
                new DefaultBase64Encoder(EncoderAlphabet.BASE64URL, EncoderPadding.WITHOUT)), true));
decoders.put(ContentFormat.JSON,
        new LwM2mNodeJsonDecoder(new LwM2mJsonJacksonEncoderDecoder(), new DefaultLwM2mLinkParser(),
                new FixedDefaultBase64Decoder(DecoderAlphabet.BASE64, DecoderPadding.REQUIRED)));
builder.setDecoder(new DefaultLwM2mDecoder(decoders, DefaultLwM2mDecoder.getDefaultPathDecoder()));

The fix should also be deployed on the sandbox : https://leshan.eclipseprojects.io/
Let me know if it works for you now and so we can close this issue.

(I think this bug exists since 2.0.0-M10 : ce98924#diff-d6e9393e636f661cde99688c1bbfedbd664fb43c7aefddd6a7a06eb32ff78826R158-R185)

@sbernard31
Copy link
Contributor

@cyril2maq, @JaroslawLegierski you could be interested by that ☝️

@vshorin
Copy link
Author

vshorin commented Nov 28, 2023

@sbernard31 wow, that was fast! Thanks a lot for your work! The workaround is fine for now, so I'll close.

@vshorin vshorin closed this as completed Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Dysfunctionnal behavior core Impact core of Leshan
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sbernard31 @vshorin