Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Stack overflow in parquet RowGroupReader with groups_filter #1206

Closed
sigurdteigen opened this issue Aug 4, 2022 · 0 comments · Fixed by #1210
Closed

Stack overflow in parquet RowGroupReader with groups_filter #1206

sigurdteigen opened this issue Aug 4, 2022 · 0 comments · Fixed by #1210
Labels
bug Something isn't working

Comments

@sigurdteigen
Copy link

Hi. Parsing parquet files with a large number of row groups and using the row groups_filter causes a stack overflow for me. The reason seems to be the recursive call to self._next() in the following code.

return self._next();

impl<R: Read + Seek> RowGroupReader<R> {

    ...

    #[inline]
    fn _next(&mut self) -> Result<Option<RowGroupDeserializer>> {
        ...

        let current_row_group = self.current_group;
        let row_group = &self.row_groups[current_row_group];
        if let Some(groups_filter) = self.groups_filter.as_ref() {
            if !(groups_filter)(current_row_group, row_group) {
                self.current_group += 1;
                return self._next();
            }
        }
        ...
    }
}
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants