Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Commit

Permalink
feat: Add nested_column_iter_to_arrays to deserialize inner columns
Browse files Browse the repository at this point in the history
  • Loading branch information
b41sh committed Oct 23, 2023
1 parent 346c866 commit 32a3104
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 1 deletion.
19 changes: 19 additions & 0 deletions src/io/parquet/read/deserialize/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -214,3 +214,22 @@ where
.map(|x| x.map(|x| x.1)),
))
}

/// Basically the same as `column_iter_to_arrays`, with the addition of the `init` parameter
/// to read the inner columns of the nested type directly, instead of reading the entire nested type.
pub fn nested_column_iter_to_arrays<'a, I: 'a>(
columns: Vec<I>,
types: Vec<&PrimitiveType>,
field: Field,
init: Vec<InitNested>,
chunk_size: Option<usize>,
num_rows: usize,
) -> Result<ArrayIter<'a>>
where
I: Pages,
{
Ok(Box::new(
nested::columns_to_iter_recursive(columns, types, field, init, num_rows, chunk_size)?
.map(|x| x.map(|x| x.1)),

Check warning on line 233 in src/io/parquet/read/deserialize/mod.rs

View check run for this annotation

Codecov / codecov/patch

src/io/parquet/read/deserialize/mod.rs#L220-L233

Added lines #L220 - L233 were not covered by tests
))
}

Check warning on line 235 in src/io/parquet/read/deserialize/mod.rs

View check run for this annotation

Codecov / codecov/patch

src/io/parquet/read/deserialize/mod.rs#L235

Added line #L235 was not covered by tests
2 changes: 1 addition & 1 deletion src/io/parquet/read/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ use crate::{array::Array, error::Result};
use crate::types::{i256, NativeType};
pub use deserialize::{
column_iter_to_arrays, create_list, create_map, get_page_iterator, init_nested, n_columns,
InitNested, NestedArrayIter, NestedState, StructIterator,
nested_column_iter_to_arrays, InitNested, NestedArrayIter, NestedState, StructIterator,
};
pub use file::{FileReader, RowGroupReader};
pub use row_group::*;
Expand Down

0 comments on commit 32a3104

Please sign in to comment.