Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ledger API and sync #2715

Merged
merged 149 commits into from
Feb 6, 2024
Merged
Show file tree
Hide file tree
Changes from 146 commits
Commits
Show all changes
149 commits
Select commit Hold shift + click to select a range
3e48aab
Initial Ledger API
sergeuz May 23, 2023
e5b4d53
Remove atomic ledger operations
sergeuz May 24, 2023
d1751c4
Add a schema for local ledger storage
sergeuz May 25, 2023
df16216
Use a virtual environment to compile Protobuf definitions
sergeuz May 25, 2023
6039b03
System API for Ledger
sergeuz May 26, 2023
4768539
Minor refactoring
sergeuz May 29, 2023
eaf6abb
Implement system Ledger API (WIP)
sergeuz May 30, 2023
1d80ae6
Compile ledger protocol files; update the submodule
sergeuz May 31, 2023
83afecb
Define error codes for filesystem operations; minor fixes
sergeuz May 31, 2023
0f57677
Allow specifying the size of the Protobuf message to read
sergeuz Jun 1, 2023
c7405a5
System ledger API (WIP)
sergeuz Jun 1, 2023
a83ca40
Regenerate protocol definitions
sergeuz Jun 2, 2023
a145f95
Fix reference counting
sergeuz Jun 2, 2023
5e3b6cd
Save/load the ledger info
sergeuz Jun 2, 2023
f7e358c
Update protocol definitions
sergeuz Jun 2, 2023
fa22a55
Require the ledger scope to be specified explicitly
sergeuz Jun 2, 2023
5723d3d
Add some filesystem utilities
sergeuz Jun 5, 2023
aa66787
Add a stream interface for accessing page data; minor fixes
sergeuz Jun 5, 2023
0a94b40
Minor renamings
sergeuz Jun 5, 2023
1f4b4f1
Notify the app layer when the page is changed by the system
sergeuz Jun 5, 2023
9c1f84d
Export the system ledger API
sergeuz Jun 5, 2023
fdbe0f5
Import ArduinoJson in the Wiring library; fix compiler errors in the …
sergeuz Jun 6, 2023
dd9516d
Add a function for rewinding a ledger stream
sergeuz Jun 7, 2023
5b01b82
Serialize/deserialize page data
sergeuz Jun 7, 2023
bc49f17
Fix linker errors
sergeuz Jun 7, 2023
67a23c9
Use C++17 for building the Wiring library
sergeuz Jun 7, 2023
8d36d31
Implement basic ledger entry operations
sergeuz Jun 8, 2023
d0c8c5d
Add serialization to a JSON string
sergeuz Jun 8, 2023
ee44e8e
Bugfixes
sergeuz Jun 8, 2023
afa389e
New ledger API
sergeuz Jun 14, 2023
8ffe63d
Update the prototype ledger API
sergeuz Jun 16, 2023
5179e77
Implement the Map class
sergeuz Jun 16, 2023
553adea
Add basic tests for the Map container; minor fixes
sergeuz Jun 20, 2023
7664167
Add an interator-based API to the Map class; minor refactoring
sergeuz Jun 21, 2023
a58828f
Add Variant class
sergeuz Jun 22, 2023
a26d290
Minor fixes
sergeuz Jun 22, 2023
6c38379
Make a null variant not convertible to anything but std::monostate
sergeuz Jun 22, 2023
c1b3800
Add support for unsigned integer types; implement comparison operator…
sergeuz Jun 23, 2023
629f328
Fix comparison operators; minor refactoring
sergeuz Jun 26, 2023
7ed8485
Add OutputStringStream
sergeuz Jun 27, 2023
90d00da
Work around missing functions from <charconv> in GCC 10; minor fixes
sergeuz Jun 27, 2023
7c1e09f
Add methods for converting Variant to/from JSON
sergeuz Jun 28, 2023
a95d476
Refactor the system ledger API
sergeuz Jun 30, 2023
bee836e
Remove ArduinoJson from dependencies
sergeuz Jul 3, 2023
86cba51
Refactor the application ledger API; add API docs; minor fixes
sergeuz Jul 4, 2023
94b1019
Limit the maximum size of ledger data; bugfixes
sergeuz Jul 4, 2023
3d379fd
Add API docs; minor fixes
sergeuz Jul 5, 2023
8aef864
Implement the MERGE mode
sergeuz Jul 6, 2023
1b8ea2c
Make the key of a map entry immutable
sergeuz Jul 6, 2023
8186dcf
Fix a bug in String::concat()
sergeuz Jul 6, 2023
32b2e91
Fix compiler warnings
sergeuz Jul 6, 2023
cec7226
Add wiring tests for Ledger
sergeuz Jul 6, 2023
e1ec87e
Fix overload resolution when printing a String
sergeuz Jul 6, 2023
5f261e2
Refactor the internal ledger classes so that file handles are never c…
sergeuz Jul 6, 2023
0aa8304
Minor fixes
sergeuz Jul 6, 2023
2eab6b9
Fix reference counting
sergeuz Jul 6, 2023
cfd6acd
Various fixes after a self-review
sergeuz Jul 7, 2023
0e4ca45
Address review comments
sergeuz Jul 7, 2023
b05b0dc
Add tests for concurrent reading/writing; bugfixes
sergeuz Jul 7, 2023
cad8461
Disable the Ledger on the Tracker and GCC platforms
sergeuz Jul 7, 2023
4ec4b41
Add a benchmark test app for the Ledger
sergeuz Jul 10, 2023
a797b0a
Add more performance tests
sergeuz Jul 10, 2023
702fe60
Move the fixed-size header to the end of a ledger file so that it can…
sergeuz Jul 10, 2023
4d1f956
Try recovering some consistent state when flushing the staged data fails
sergeuz Jul 10, 2023
8af2d1b
Tracker: Save on flash space by not compiling in message strings for …
sergeuz Jul 12, 2023
c643954
Add a test; minor fixes
sergeuz Jul 12, 2023
d6e0c1a
Wrap the C-style callback in an std::function
sergeuz Jul 12, 2023
12ec94a
Update the protocol submodule; regenerate protocol files; minor fixes
sergeuz Jul 18, 2023
80cabba
Define possible states when synchronizing one or more ledgers
sergeuz Jul 21, 2023
8cb5b53
Initial state machine for ledger synchronization
sergeuz Jul 25, 2023
1f21c50
Refactoring
sergeuz Jul 26, 2023
3e68380
Define the new CoAP API
sergeuz Jul 28, 2023
fb270da
Initial implementation of the new CoAP API
sergeuz Aug 14, 2023
f84be3a
Refactoring
sergeuz Aug 16, 2023
895920a
Shut down the new CoAP channel when the connection is closed; support…
sergeuz Aug 17, 2023
4907639
Mark the new CoAP channel as open after the handshake
sergeuz Aug 17, 2023
b762084
Bugfixes
sergeuz Aug 18, 2023
a1d4ace
Update submodule refs; regenerate protocol files
sergeuz Aug 23, 2023
4e55f9b
Use the new CoAP API for sending/receiving ledger requests
sergeuz Aug 23, 2023
ab69a74
Add missing request handlers; refactoring
sergeuz Aug 30, 2023
7651680
Handle reset info requests; support notifications for multiple ledger…
sergeuz Sep 5, 2023
46fe5c5
Bugfixes
sergeuz Sep 6, 2023
22de801
Allow rewinding a ledger reader; bugfixes
sergeuz Sep 12, 2023
395ffdf
Refactoring
sergeuz Sep 13, 2023
f5d3144
Minor fixes
sergeuz Sep 14, 2023
0826556
Encode ledger data as CBOR
sergeuz Sep 14, 2023
a5a0a82
Decode ledger data as CBOR
sergeuz Sep 18, 2023
c6bce08
Make Stream::readBytes virtual
sergeuz Sep 19, 2023
6d66612
Use Wiring streams for encoding/decoding Variant as CBOR
sergeuz Sep 19, 2023
79f56bc
Add unit tests; fix error handling
sergeuz Sep 20, 2023
942c669
Increase the maximum number of arguments supported by PP_COUNT and PP…
sergeuz Sep 22, 2023
6398874
Minor fixes
sergeuz Sep 22, 2023
10d21f9
Initial support for CoAP blockwise transfer
sergeuz Sep 22, 2023
c038451
Refactoring
sergeuz Oct 4, 2023
ff7ae3f
Require the server to use a ETag option with all blockwise responses
sergeuz Oct 4, 2023
5396230
Allow cancelling a request by ID; refactoring
sergeuz Oct 4, 2023
6d62959
Refactoring
sergeuz Oct 5, 2023
8931639
Bugfixes
sergeuz Oct 6, 2023
b7d7f02
Disable error messages on nRF52 platforms to free some flash space
sergeuz Oct 12, 2023
0364a7d
Add ledger_get_names()
sergeuz Oct 23, 2023
0e7af28
Ledger test app; minor bugfixes
sergeuz Oct 24, 2023
2f9f32a
Add docs; minor fixes
sergeuz Oct 25, 2023
40bc132
Store an opaque scope ID along with ledger data
sergeuz Nov 8, 2023
a64f41d
Add batch mode to some commands
sergeuz Nov 8, 2023
d212a25
Bugfixes
sergeuz Nov 8, 2023
3576404
Add a Request-Tag option to blockwise requests
sergeuz Nov 13, 2023
1fadb1e
Take over the message buffer when starting a new request or response …
sergeuz Nov 15, 2023
c082128
Minor fixes
sergeuz Nov 15, 2023
e98287c
Store a counter that gets incremented every time the ledger is changed
sergeuz Nov 15, 2023
99b40c0
Retry synchronization after a delay
sergeuz Nov 15, 2023
198359f
Add Particle.useLedgers() method; minor fixes
sergeuz Nov 17, 2023
d9142af
Minor fixes
sergeuz Dec 6, 2023
9cab5a9
Update submodule refs
sergeuz Dec 6, 2023
12e80f5
Bugfixes
sergeuz Dec 7, 2023
0873de2
Disable mbedTLS server support
sergeuz Dec 8, 2023
0627355
Merge remote-tracking branch 'origin/develop' into ledger-sync-merged…
sergeuz Dec 11, 2023
71bfd6e
Fix merge artifacts
sergeuz Dec 11, 2023
71f816d
Fix build for M SoM
sergeuz Dec 11, 2023
130ce55
Temporarily disable Ledger on Tracker
sergeuz Dec 11, 2023
fcdeb14
Fix reporting of timestamps
sergeuz Dec 11, 2023
9e4842f
Fix sync context initialization
sergeuz Dec 11, 2023
2cac4cd
Treat empty ledger data as an empty map
sergeuz Dec 12, 2023
284f077
Simplify callback management
sergeuz Dec 12, 2023
c1b6f51
[third_party] mbedtls: update submodule reference
avtolstoy Dec 12, 2023
3292ada
[tracker] Enable Ledger / disable MBEDTLS_SSL_DISABLE_PARSE_CERTIFICATE
avtolstoy Dec 12, 2023
97cf0f5
minor
avtolstoy Dec 12, 2023
1c48ae0
Add some debug utilities
sergeuz Dec 13, 2023
c62a364
Fix a double free bug
sergeuz Dec 14, 2023
21b53b1
Fix a deadlock
sergeuz Dec 14, 2023
eb271bf
Suppress the error message when getting the data of an empty ledger
sergeuz Dec 15, 2023
ec8a149
Use lazy initialization when accessing the ledger manager
sergeuz Dec 15, 2023
00a2bc3
Fix a couple more deadlocks
sergeuz Dec 18, 2023
cdcddd9
Fix sending responses to ledger requests
sergeuz Dec 19, 2023
b7ab859
Fix wiring tests
sergeuz Dec 21, 2023
2cca849
Basic integration tests for Ledger
sergeuz Dec 21, 2023
f64c08f
Fix integration tests
sergeuz Jan 12, 2024
34087ed
Add a utility class for timers executed in the system thread
sergeuz Jan 12, 2024
ef6c3e8
Minor fix
sergeuz Jan 16, 2024
87bf163
Use RTOS timers for scheduling ledger operations; minor renamings
sergeuz Jan 16, 2024
3c71e0c
Synchronize the ledger right away if it's not being throttled
sergeuz Jan 17, 2024
73546a7
Bugfixes
sergeuz Jan 18, 2024
aa5fe34
Remove ledger data when resetting the device to factory default settings
sergeuz Jan 19, 2024
4f990ef
Comment out the factory reset changes for now
sergeuz Jan 19, 2024
f57c206
Reduce delays in integration tests
sergeuz Jan 22, 2024
c0c352c
Add integration tests
sergeuz Jan 24, 2024
07308ad
Fix GCC platform build
sergeuz Jan 24, 2024
5932588
Address review comments; bugfixes
sergeuz Feb 2, 2024
3b46e83
Merge remote-tracking branch 'origin/develop' into ledger-sync-merged…
sergeuz Feb 6, 2024
7a89c8d
Use placement new when allocating non-POD structures in the system pool
sergeuz Feb 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
522 changes: 522 additions & 0 deletions communication/inc/coap_api.h

Large diffs are not rendered by default.

154 changes: 154 additions & 0 deletions communication/inc/coap_channel_new.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
/*
* Copyright (c) 2023 Particle Industries, Inc. All rights reserved.
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation, either
* version 3 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, see <http://www.gnu.org/licenses/>.
*/

#include <memory>
#include <cstdint>

#include "message_channel.h"
#include "coap_api.h"
#include "coap.h" // For token_t

#include "system_tick_hal.h"

#include "ref_count.h"

namespace particle::protocol {

class CoapMessageDecoder;
class Protocol;

namespace experimental {

// This class implements the new experimental protocol API that allows the system to interact with
sergeuz marked this conversation as resolved.
Show resolved Hide resolved
// the server at the CoAP level. It's meant to be used through the functions defined in coap_api.h
class CoapChannel {
public:
enum Result {
HANDLED = 1 // Returned by the handle* methods
};

~CoapChannel();

// Methods called by the new CoAP API (coap_api.h)

int beginRequest(coap_message** msg, const char* uri, coap_method method, int timeout);
int endRequest(coap_message* msg, coap_response_callback respCallback, coap_ack_callback ackCallback,
coap_error_callback errorCallback, void* callbackArg);

int beginResponse(coap_message** msg, int code, int requestId);
int endResponse(coap_message* msg, coap_ack_callback ackCallback, coap_error_callback errorCallback,
void* callbackArg);

int writePayload(coap_message* msg, const char* data, size_t& size, coap_block_callback blockCallback,
coap_error_callback errorCallback, void* callbackArg);
int readPayload(coap_message* msg, char* data, size_t& size, coap_block_callback blockCallback,
coap_error_callback errorCallback, void* callbackArg);
int peekPayload(coap_message* msg, char* data, size_t size);

void destroyMessage(coap_message* msg);

void cancelRequest(int requestId);

int addRequestHandler(const char* uri, coap_method method, coap_request_callback callback, void* callbackArg);
void removeRequestHandler(const char* uri, coap_method method);

int addConnectionHandler(coap_connection_callback callback, void* callbackArg);
void removeConnectionHandler(coap_connection_callback callback);

// Methods called by the old protocol implementation

void open();
void close(int error = SYSTEM_ERROR_COAP_CONNECTION_CLOSED);

int handleCon(const Message& msg);
int handleAck(const Message& msg);
int handleRst(const Message& msg);

int run();

static CoapChannel* instance();

private:
// Channel state
enum class State {
CLOSED,
OPENING,
OPEN,
CLOSING
};

enum class MessageType {
REQUEST, // Regular or blockwise request carrying request data
BLOCK_REQUEST, // Blockwise request retrieving a block of response data
RESPONSE // Regular or blockwise response
};

enum class MessageState {
NEW, // Message created
READ, // Reading payload data
WRITE, // Writing payload data
WAIT_ACK, // Waiting for an ACK
WAIT_RESPONSE, // Waiting for a response
WAIT_BLOCK, // Waiting for the next message block
DONE // Message exchange completed
};

struct CoapMessage;
struct RequestMessage;
struct ResponseMessage;
struct RequestHandler;
struct ConnectionHandler;

CoapChannel(); // Use instance()

Message msgBuf_; // Reference to the shared message buffer
ConnectionHandler* connHandlers_; // List of registered connection handlers
RequestHandler* reqHandlers_; // List of registered request handlers
RequestMessage* sentReqs_; // List of requests awaiting a response from the server
RequestMessage* recvReqs_; // List of requests awaiting a response from the device
ResponseMessage* blockResps_; // List of responses for which the next message block is expected to be received
CoapMessage* unackMsgs_; // List of messages awaiting an ACK from the server
Protocol* protocol_; // Protocol instance
State state_; // Channel state
uint32_t lastReqTag_; // Last used request tag
int lastMsgId_; // Last used internal message ID
int curMsgId_; // Internal ID of the message stored in the shared buffer
int sessId_; // Counter incremented every time a new session with the server is started
int pendingCloseError_; // If non-zero, the channel needs to be closed
bool openPending_; // If true, the channel needs to be reopened

int handleRequest(CoapMessageDecoder& d);
int handleResponse(CoapMessageDecoder& d);
int handleAck(CoapMessageDecoder& d);

int prepareMessage(const RefCountPtr<CoapMessage>& msg);
int updateMessage(const RefCountPtr<CoapMessage>& msg);
int sendMessage(RefCountPtr<CoapMessage> msg);
void clearMessage(const RefCountPtr<CoapMessage>& msg);

int sendAck(int coapId, bool rst = false);

int handleProtocolError(ProtocolError error);

void releaseMessageBuffer();

system_tick_t millis() const;
};

} // namespace experimental

} // namespace particle::protocol
102 changes: 102 additions & 0 deletions communication/inc/coap_util.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
/*
* Copyright (c) 2022 Particle Industries, Inc. All rights reserved.
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation, either
* version 3 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, see <http://www.gnu.org/licenses/>.
*/

#pragma once

#include "coap_api.h"
#include "logging.h"

namespace particle {

/**
* Smart pointer for CoAP message instances.
*/
class CoapMessagePtr {
public:
CoapMessagePtr() :
msg_(nullptr) {
}

explicit CoapMessagePtr(coap_message* msg) :
msg_(msg) {
}

CoapMessagePtr(const CoapMessagePtr&) = delete;

CoapMessagePtr(CoapMessagePtr&& ptr) :
msg_(ptr.msg_) {
ptr.msg_ = nullptr;
}

~CoapMessagePtr() {
coap_destroy_message(msg_, nullptr);
}

coap_message* get() const {
return msg_;
}

coap_message* release() {
auto msg = msg_;
msg_ = nullptr;
return msg;
}

void reset(coap_message* msg = nullptr) {
coap_destroy_message(msg_, nullptr);
msg_ = msg;
}

CoapMessagePtr& operator=(const CoapMessagePtr&) = delete;

CoapMessagePtr& operator=(CoapMessagePtr&& ptr) {
coap_destroy_message(msg_, nullptr);
msg_ = ptr.msg_;
ptr.msg_ = nullptr;
return *this;
}

explicit operator bool() const {
return msg_;
}

friend void swap(CoapMessagePtr& ptr1, CoapMessagePtr& ptr2) {
auto msg = ptr1.msg_;
ptr1.msg_ = ptr2.msg_;
ptr2.msg_ = msg;
}

private:
coap_message* msg_;
};

namespace protocol {

/**
* Log the contents of a CoAP message.
*
* @param level Logging level.
* @param category Logging category.
* @param data Message data.
* @param size Message size.
* @param logPayload Whether to log the payload data of the message.
*/
void logCoapMessage(LogLevel level, const char* category, const char* data, size_t size, bool logPayload = false);

} // namespace protocol

} // namespace particle
1 change: 1 addition & 0 deletions communication/inc/protocol_defs.h
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ enum ProtocolError
IO_ERROR_SOCKET_SEND_FAILED = 33,
IO_ERROR_SOCKET_RECV_FAILED = 34,
IO_ERROR_REMOTE_END_CLOSED = 35,
COAP_ERROR = 36,
// NOTE: when adding more ProtocolError codes, be sure to update toSystemError() in protocol_defs.cpp
UNKNOWN = 0x7FFFF
};
Expand Down
1 change: 1 addition & 0 deletions communication/src/build.mk
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ CPPSRC += $(TARGET_SRC_PATH)/coap_message_decoder.cpp
CPPSRC += $(TARGET_SRC_PATH)/coap_util.cpp
CPPSRC += $(TARGET_SRC_PATH)/firmware_update.cpp
CPPSRC += $(TARGET_SRC_PATH)/description.cpp
CPPSRC += $(TARGET_SRC_PATH)/coap_channel_new.cpp

# ASM source files included in this build.
ASRC +=
Expand Down
6 changes: 6 additions & 0 deletions communication/src/coap_channel.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,11 @@
#undef LOG_COMPILE_TIME_LEVEL

#include "coap_channel.h"
#include "coap_channel_new.h"
#include "service_debug.h"
#include "messages.h"
#include "communication_diagnostic.h"
#include "system_error.h"

namespace particle { namespace protocol {

Expand Down Expand Up @@ -64,6 +66,9 @@ void CoAPMessageStore::message_timeout(CoAPMessage& msg, Channel& channel)
if (msg.is_request()) {
LOG(ERROR, "CoAP message timeout; ID: %d", (int)msg.get_id());
g_unacknowledgedMessageCounter++;
// XXX: This will cancel _all_ messages with a timeout error, not just the timed out one.
// That's not ideal but should be okay while we're transitioning to the new CoAP API
experimental::CoapChannel::instance()->close(SYSTEM_ERROR_COAP_TIMEOUT);
channel.command(MessageChannel::CLOSE);
}
}
Expand Down Expand Up @@ -141,6 +146,7 @@ ProtocolError CoAPMessageStore::receive(Message& msg, Channel& channel, system_t
}
if (msgtype==CoAPType::RESET) {
LOG(WARN, "Received RST message; discarding session");
experimental::CoapChannel::instance()->handleRst(msg);
if (coap_msg) {
coap_msg->notify_delivered_nak();
}
Expand Down
Loading