How to handle memory issues

[ramdhan1989]

Hi,
I try to prepare data for my model by transforming original data format into sliding window approach shited 1 step. However, I have memory issues when creating sliding window dataset it makes data size become bigger. Is it still valuable to do resampling become hourly data? if it is better to keep data frequency as is, what should I use to handle memory issues.

thank you

[ricardoevvargas]

Hi, @ramdhan1989. Remember that Data Science is very rich in terms of strategies and that each one can have positive or negative consequences.

Some examples:

• You can increasing the step used to slide the window. You can even eliminate overlapping. Caution: at the limit this can make it difficult to extract/identify features;
• You can increase the size of the window. Caution: patterns with fast dynamics can “disappear”;
• You can downsample the time series. Caution: this can make it difficult to learn/detect patterns with fast dynamics;
• For each instance, you can extract characteristics from the samples obtained and then remove the raw data for that instance from memory.

Various other strategies can be suggested by other members of the community. In the meantime, try to address this issue creatively (very important for Data Science).

[ricardoevvargas]

As this issue is not specific to 3W Dataset, it will be converted into a discussion. Contributions are welcome.