Dynamic initialisation/data types?

[JoeEarlam]

Hi all

I'm trying to shift data from multiple micros into a series of circular buffers. The number of sources is 0<N<16, and the data is an array of uint16_t values, which could be 1<N<32 (or more). Since the data sources change, I've coded the length of the data (and the number of samples each packet consists of) into the comms protocol that connects the micros together. With my current understanding, I'm using a huge static buffer for each data source, which isn't great for memory usage.

Is there a way to start and dynamically assign buffers once the code is running?

Currently I'm initialising the buffers like this:

#define BUFFLEN 100   //number of samples in circular buffer

struct cardDataStruct
{
  uint16_t data[32];    //single sample (NOT a single  data transaction, which carries multiple samples in a single message)
};

struct circularStruct
{
  CircularBuffer<cardDataStruct, BUFFLEN>circBuff;   //<type, length> name
};

circularStruct cardData[9];

However, I would like to initialise it once I have all the data, so that my declaration looks more like this (please excuse the pseudocode, I know it wont compile if I do this):

#define BUFFLEN 10   //number of samples in circular buffer

typedef dataStruct 
{
  uint16_t dataSample[sampleWidth];  
};

dataStruct dataSources[numSources];

for(uint8_t i = 0; i < numSources; i++)
{
    CircularBuffer<dataSource[i], BUFFLEN> circularBuffer[i];   //<type, length> name
}

1. Does this make sense?
2. Is this possible?
3. Is there a better way to do this?

Thanks!

[digamesystems]

Hi, Joe.

You didn't talk too much about the communication protocol you are using to exchange data between the microprocessors. Is this a networked application, or are you using something like I2C to exchange data between the micros?

It sort of sounds like you are developing a system where you have a number of message queues to buffer data, but I'm not clear on that, yet.

Could you provide some more detail on the application?

You might want to take a look at how MQTT is set up. MQTT uses a central processor that has a buffer for messages and interested parties can subscribe to messages they care about. MQTT is designed around TCP networks, but you might get some inspiration if you are trying to build something similar in spirit using another bus-based protocol.

Take care

[JoeEarlam]

Hi there! I'm using modbus to slurp data from a bunch of Pi Picos. I'm queuing a number of samples on a circular buffer on each pico (as many as I can fit into a single modbus transaction), then the master Pico requests that data and adds it to its own circular buffer. The idea is that the system will sit collecting data, and you can request the past X seconds of data with a command to the master.

If I'm able to dynamically assign circular buffer parameters in the code after I've detected the slaves, I can detect how many slaves are connected, how many data points per sample they provide, and scale the buffers appropriately to save as much data as possible.

Thanks!

[rlogiacco]

Functional speaking I would go for one single shared buffer where all sensors' data get appended and then processed, but without revolutionizing the approach, what might be achievable is to have circular buffers being statically sized and have their processing occur at dynamically manged frequencies to account for higher usage.

This way you can process more frequently the buffers which get filled up more quickly (under dimensioned) to avoid data loss.
To account for differences in data size you might want to use a dynamic approach: store in the buffers only the pointers to the data structures, this will make your approach more complex because you will have to allocate and deallocate the memory for each data block on your own, but will give you the flexibility on what you store in the buffer and the memory usage will increase dynamically while the buffers get filled up.

I'm referencing the Object example, which stores references instead of data.

[JoeEarlam]

Thanks for the reply, that gives me some stuff to think about. However, considering my limited programming aptitude I think I'll stick to static buffers for now, and experiment with dynamic buffers later on. I'd forgotten how much memory the Pico has, so I can afford to assign huge chunks of memory I might not use!

Thanks!

[rlogiacco]

If you feel this has been replied please mark the response you believe best fulfills your question.
Thanks