OSCore observe seconde times occur MAC validation failed

[KeithLaiKB]

when I try the case about OSCORE of californium observe
(my case is in https://github.com/KeithLaiKB/java_test_learn_cf_oscore)

step1
start the server program

step2
start the client program in the same local machine

step2.1
cancel the observer relationship

step2.2
shutdown the client (and then the client program finished)

step3
when I start the client program in the same local machine again

then the client console would appear

[main] INFO org.eclipse.californium.core.network.config.NetworkConfig - loading properties from file E:\java_learn_californium_bug_workspace\java_test_learn_cf_oscore\oscore_client_workspace1\com.learn.californium.tryoscore.client\Californium.properties
[main] INFO org.eclipse.californium.core.network.RandomTokenGenerator - using tokens of 8 bytes in length
[main] INFO org.eclipse.californium.core.network.RandomTokenGenerator - using tokens of 8 bytes in length
[main] INFO org.eclipse.californium.ban - Started.
[main] INFO org.eclipse.californium.core.network.CoapEndpoint - coap CoapEndpoint uses udp plain
[main] INFO org.eclipse.californium.core.network.stack.BlockwiseLayer - coap BlockwiseLayer uses MAX_MESSAGE_SIZE=1024, PREFERRED_BLOCK_SIZE=512, BLOCKWISE_STATUS_LIFETIME=300000, MAX_RESOURCE_BODY_SIZE=8192, BLOCKWISE_STRICT_BLOCK2_OPTION=false
[main] INFO org.eclipse.californium.core.network.CoapEndpoint - coap Endpoint [coap://0.0.0.0:0] requires an executor to start, using default single-threaded daemon executor
[main] INFO org.eclipse.californium.elements.UDPConnector - UDPConnector starts up 1 sender threads and 1 receiver threads
[main] INFO org.eclipse.californium.elements.UDPConnector - UDPConnector listening on 0.0.0.0/0.0.0.0:52281, recv buf = 65536, send buf = 65536, recv packet size = 2048
[main] INFO org.eclipse.californium.core.network.CoapEndpoint - coap Started endpoint at coap://0.0.0.0:52281
[main] INFO org.eclipse.californium.core.network.EndpointManager - created implicit endpoint coap://0.0.0.0:52281 for coap
[:CoapEndpoint-UDP-0.0.0.0:0#1] INFO org.eclipse.californium.oscore.ObjectSecurityContextLayer - Request: CON-GET MID= -1, Token=2550EE23F7489CDD, OptionSet={"Observe":0, "Object-Security":0x, "Uri-Path":["oscore","observe2"]}, no payload
[:CoapEndpoint-UDP-0.0.0.0:0#1] INFO org.eclipse.californium.oscore.ObjectSecurityLayer - Request: CON-GET MID= -1, Token=2550EE23F7489CDD, OptionSet={"Observe":0, "Object-Security":0x, "Uri-Path":["oscore","observe2"]}, no payload
[:CoapEndpoint-UDP-0.0.0.0:0#1] INFO org.eclipse.californium.oscore.HashMapCtxDB - Token exists, but this could be a refresh if not there is a problem
[:CoapEndpoint-UDP-0.0.0.0:0#1] INFO org.eclipse.californium.oscore.ObjectSecurityLayer - Incoming response is OSCORE protected
[:CoapEndpoint-UDP-0.0.0.0:0#1] INFO org.eclipse.californium.oscore.ResponseDecryptor - Removes E options from outer options which are not allowed there
[:CoapEndpoint-UDP-0.0.0.0:0#1] INFO org.eclipse.californium.oscore.OptionJuggle - Removing inner only E options from the outer options
org.eclipse.californium.scandium.dtls.cipher.InvalidMacException: MAC validation failed
at org.eclipse.californium.scandium.dtls.cipher.CCMBlockCipher.decrypt(CCMBlockCipher.java:406)
at org.eclipse.californium.scandium.dtls.cipher.CCMBlockCipher.decrypt(CCMBlockCipher.java:335)
at org.eclipse.californium.cose.EncryptCommon.AES_CCM_Decrypt(EncryptCommon.java:134)
at org.eclipse.californium.cose.EncryptCommon.decryptWithKey(EncryptCommon.java:74)
at org.eclipse.californium.cose.Encrypt0Message.decrypt(Encrypt0Message.java:138)
at org.eclipse.californium.oscore.Decryptor.decryptAndDecode(Decryptor.java:142)
at org.eclipse.californium.oscore.ResponseDecryptor.decrypt(ResponseDecryptor.java:99)
at org.eclipse.californium.oscore.ObjectSecurityLayer.prepareReceive(ObjectSecurityLayer.java:116)
at org.eclipse.californium.oscore.ObjectSecurityLayer.receiveResponse(ObjectSecurityLayer.java:341)
at org.eclipse.californium.core.network.stack.AbstractLayer.receiveResponse(AbstractLayer.java:89)
at org.eclipse.californium.core.network.stack.BaseCoapStack.receiveResponse(BaseCoapStack.java:143)
at org.eclipse.californium.core.network.CoapEndpoint$1.receiveResponse(CoapEndpoint.java:306)
at org.eclipse.californium.core.network.UdpMatcher$4.run(UdpMatcher.java:457)
at org.eclipse.californium.elements.util.SerialExecutor$1.run(SerialExecutor.java:290)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)

[boaks]

@rikard-sics

Hi Rikard,

are you still on OSCORE?
If so, could you check that?

[rikard-sics]

@rikard-sics

Hi Rikard,

are you still on OSCORE?
If so, could you check that?

Hello, yes indeed, I am still on OSCORE.

when I try the case about OSCORE of californium observe
(my case is in https://github.com/KeithLaiKB/java_test_learn_cf_oscore)

Thank you for the feedback and for testing the OSCORE functionality. This week I am on vacation, I will return to work next week and can give more in-depth feedback then.

I'd have to test your code to understand what happens in more detail, but one question I had is if you are using the same Token for the first request from the client and the request after you restarted the client?

Normally if you restart the client and not the server, the server will take following requests as replays since the client starts over from sequence number 0. But here since Observe is involved or if the same Token is used again something else seems to be happening (which seems incorrect). One possible thing to test is to use the Appendix B.2/context rederivation functionality on the client to generate a new context with the server, that may solve the issue (although I should still look into what is happening here).

[KeithLaiKB]

@rikard-sics
Hi Rikard,
are you still on OSCORE?
If so, could you check that?

Hello, yes indeed, I am still on OSCORE.

when I try the case about OSCORE of californium observe
(my case is in https://github.com/KeithLaiKB/java_test_learn_cf_oscore)

Thank you for the feedback and for testing the OSCORE functionality. This week I am on vacation, I will return to work next week and can give more in-depth feedback then.

I'd have to test your code to understand what happens in more detail, but one question I had is if you are using the same Token for the first request from the client and the request after you restarted the client?

Normally if you restart the client and not the server, the server will take following requests as replays since the client starts over from sequence number 0. But here since Observe is involved or if the same Token is used again something else seems to be happening (which seems incorrect). One possible thing to test is to use the Appendix B.2/context rederivation functionality on the client to generate a new context with the server, that may solve the issue (although I should still look into what is happening here).

The initial case is that, when I restart the client with the different token, it shows MAC validation failed,
and as for using same token, and I update this case in the repository(https://github.com/KeithLaiKB/java_test_learn_cf_oscore), but it does not work as well

In addition, As for Appendix B.2/context rederivation,I found ctx.setContextRederivationPhase(PHASE.CLIENT_INITIATE);
but in ContextRederivation.PHASE, there are
INACTIVE, CLIENT_INITIATE, SERVER_INITIATE, SERVER_PHASE_1, SERVER_PHASE_2, SERVER_PHASE_3, CLIENT_PHASE_1, CLIENT_PHASE_2, CLIENT_PHASE_3;
however there not comments of this things in californium for use, according to Appendix B.2/context rederivation description.

Could you please add a test case that restarting observe relationship in client of thiese things

[rikard-sics]

In addition, As for Appendix B.2/context rederivation,I found ctx.setContextRederivationPhase(PHASE.CLIENT_INITIATE);
but in ContextRederivation.PHASE, there are
INACTIVE, CLIENT_INITIATE, SERVER_INITIATE, SERVER_PHASE_1, SERVER_PHASE_2, SERVER_PHASE_3, CLIENT_PHASE_1, CLIENT_PHASE_2, CLIENT_PHASE_3;
however there not comments of this things in californium for use, according to Appendix B.2/context rederivation description.

To try out your code using Appendix B.2 you can add in the client code (just after creating the context):
ctx.setContextRederivationEnabled(true);
ctx.setContextRederivationPhase(PHASE.CLIENT_INITIATE);
And then on the server side (just after creating the context):
ctx.setContextRederivationEnabled(true);
By adding that, Appendix B.2 will be ran and a new context established every time the client communicates with the server for the first time.

I will try to take some time during the week to test out your code without Appendix B.2 to understand better what is happening there. Seems to me the server should detect the request from the client after reboot as a replay, but is for some reason not doing so.

[boaks]

@rikard-sics

I would like to release 3.0.0-RC1 next week.
Are fixes required for this issue? If so, would it be possible for you, to contribute them begin of next week?

[rikard-sics]

I would like to release 3.0.0-RC1 next week.
Are fixes required for this issue? If so, would it be possible for you, to contribute them begin of next week?

I do believe usage of Appendix B.2 would solve this issue, as restarting the client but not the server will otherwise cause problems (due to replays). Basically restarting the client but not the server should not work (without Appendix B.2 or B.1).

However, to be certain I will test the code in the first post and come back with if any changes are practically required. I will try to determine that as soon as possible. If changes are needed I can contribute them during the beginning of next week, yes.

[KeithLaiKB]

I would like to release 3.0.0-RC1 next week.
Are fixes required for this issue? If so, would it be possible for you, to contribute them begin of next week?

I do believe usage of Appendix B.2 would solve this issue, as restarting the client but not the server will otherwise cause problems (due to replays). Basically restarting the client but not the server should not work (without Appendix B.2 or B.1).

However, to be certain I will test the code in the first post and come back with if any changes are practically required. I will try to determine that as soon as possible. If changes are needed I can contribute them during the beginning of next week, yes.

hi, would you like to add some comments or combination tests for the stuff in ContextRederivation.PHASE:
INACTIVE, CLIENT_INITIATE, SERVER_INITIATE, SERVER_PHASE_1, SERVER_PHASE_2, SERVER_PHASE_3, CLIENT_PHASE_1, CLIENT_PHASE_2, CLIENT_PHASE_3; according to Appendix B.2/context rederivation description. Because there is no comments for these values to use.

[boaks]

If changes are needed I can contribute them during the beginning of next week, yes.

Just to mention, there will be also releases after RC1 :-).

[rikard-sics]

hi, would you like to add some comments or combination tests for the stuff in ContextRederivation.PHASE:
INACTIVE, CLIENT_INITIATE, SERVER_INITIATE, SERVER_PHASE_1, SERVER_PHASE_2, SERVER_PHASE_3, CLIENT_PHASE_1, CLIENT_PHASE_2, CLIENT_PHASE_3; according to Appendix B.2/context rederivation description. Because there is no comments for these values to use.

Yes, this could probably be documented better. However my intention is that all these details should remain only for the internal code of OSCORE. The goal was to make it easy for an application developer to use Appendix B.2. All code you should need for that is in the message I posted earlier:
#1707 (comment)

So there should be no need to use for instance SERVER_PHASE_2 in the application code. However if you wish to learn more about this you can check the code of ContextRederivation.java. In there I have some comments that describe the phases and the procedure. Search for instance for "handle" and you will find comments near the phase checks that describe what part of the Appendix B.2 procedure they refer to.
https://github.com/eclipse/californium/blob/master/cf-oscore/src/main/java/org/eclipse/californium/oscore/ContextRederivation.java#L407

There is also a JUnit test for the Context rederivation/Appendix B.2 procedure here:
https://github.com/eclipse/californium/blob/master/cf-oscore/src/test/java/org/eclipse/californium/oscore/ContextRederivationTest.java

[rikard-sics]

Just to mention, there will be also releases after RC1 :-).

True :)

I will try to look into this code ASAP and estimate if and what changes are needed. If they seem like way too much to do quickly (which I don't expect) I can report that here.

[rikard-sics]

Just to mention, there will be also releases after RC1 :-).

As I have not had time to review the code in the first post in detail, I will have to wait with any potential updates needed for a later release.

hi, would you like to add some comments or combination tests for the stuff in ContextRederivation.PHASE:
INACTIVE, CLIENT_INITIATE, SERVER_INITIATE, SERVER_PHASE_1, SERVER_PHASE_2, SERVER_PHASE_3, CLIENT_PHASE_1, CLIENT_PHASE_2, CLIENT_PHASE_3; according to Appendix B.2/context rederivation description. Because there is no comments for these values to use.

@KeithLaiKB if you have the opportunity could you try the suggestion of using Appendix B.2 as I describe in #1707 (comment)?

[boaks]

Any progress? Or plans to fix that in the next days?

If not, and with the current feedback/experience from 3.0.0-RC2, I would like to release the 3.0.0 tomorrow, 3. November, as already announced in the Eclipse/Californium - Release Plan.

[rikard-sics]

Any progress? Or plans to fix that in the next days?

If not, and with the current feedback/experience from 3.0.0-RC2, I would like to release the 3.0.0 tomorrow, 3. November, as already announced in the Eclipse/Californium - Release Plan.

Apologies for the late response regarding this. I am on vacation until the end of this week. Let me come back with an update on this (and issue #1748) during next week.

[boaks]

@rikard-sics

I'm not sure about the state of this issue. Is it fixed with the PR for the other issue? Or is it still pending?

[rikard-sics]

I'm not sure about the state of this issue. Is it fixed with the PR for the other issue? Or is it still pending?

The other PR would not affect this issue. Although I believe usage of Appendix B.2 (as currently implemented) would fix the problem.

Let me have a final look at this during next week and see if it indeed can be closed.

[rikard-sics]

After internal discussions I have determined that this seems to need a fix in the code.

Even if use of Appendix B.2 solves the issue, the response from the server after the client has restarted should be a "replay detected" error message, and not the exception "InvalidMacException: MAC validation failed".

I will start putting together a fix for this.

[rikard-sics]

I will start putting together a fix for this.

I am continuing my work on this and should be able to create a PR in a few days time.

[boaks]

@KeithLaiKB

Is it possible for you to verify the PR?

[boaks]

The bugfix is on master.

[boaks]

@KeithLaiKB

Just in the case, you still have this issue, don't hesitate to report that.