Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce the memory size of AggregateFunction state #1189

Closed
sundy-li opened this issue Jul 26, 2021 · 1 comment · Fixed by #1214
Closed

Reduce the memory size of AggregateFunction state #1189

sundy-li opened this issue Jul 26, 2021 · 1 comment · Fixed by #1214
Assignees
Labels
C-improvement Category: improvement

Comments

@sundy-li
Copy link
Member

sundy-li commented Jul 26, 2021

Summary

This test code proves that DataValue has memory size 64 bytes, because DataValue is an enum struct which uses the maximum memory in each enum.

 println!("memory size -> {:?}", std::mem::size_of::<DataValue>());   // 64 
 println!("memory size -> {:?}", std::mem::size_of::<Option<u64>>()); // 16

We should introduce a generic state implementation to improve memory and efficiency.

@sundy-li sundy-li added the C-improvement Category: improvement label Jul 26, 2021
@sundy-li sundy-li self-assigned this Jul 26, 2021
@sundy-li
Copy link
Member Author

Example for aggregate sum state.

struct AggregateSumState<T> {
    pub value: Option<T>,
}

impl<'a, T> GetState<'a, AggregateSumState<T>> for AggregateSumState<T> {}

impl<T> AggregateSumState<T>
where T: std::ops::Add<Output = T> + Clone + Copy + serde::Serialize + serde::de::DeserializeOwned
{
    #[inline(always)]
    fn add(&mut self, other: &Option<T>) {
        match (&self.value, other) {
            (Some(a), Some(b)) => self.value = Some(a.add(*b)),
            (None, c) => self.value = c.clone(),
            _ => {}
        }
    }

    #[inline(always)]
    fn merge(&mut self, other: &Self) {
        match (&self.value, &other.value) {
            (Some(a), Some(b)) => self.value = Some(a.add(*b)),
            (None, c) => self.value = c.clone(),
            _ => {}
        }
    }

    pub fn serialize(&self, writer: &mut Vec<u8>) -> Result<()> {
        serde_json::to_writer(writer, &self.value)?;
        Ok(())
    }

    pub fn deserialize<R: std::io::Read>(&mut self, reader: R) -> Result<()> {
        self.value = serde_json::from_reader(reader)?;
        Ok(())
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-improvement Category: improvement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant