Parquet: Verify 32-bit CRC checksum when decoding pages #6290
Conversation
There was a problem hiding this comment.
I agree with @tustvold that this code needs to have some tests to ensure we don't break the feature in the future
Also I think the feature flag should be documented here https://crates.io/crates/parquet
|
FYI: https://github.com/apache/parquet-testing/tree/master/data |
|
Thanks for the pointers. I added the tests and documented the feature flag. Please take a look |
|
I am depressed about the large review backlog in this crate. We are looking for more help from the community reviewing PRs -- see #6418 for more |
| @@ -215,3 +218,4 @@ harness = false | |||
|
|
|||
| [lib] | |||
| bench = false | |||
|
|
|||
| @@ -82,4 +83,4 @@ The `parquet` crate provides the following features which may be enabled in your | |||
|
|
|||
| ## License | |||
|
|
|||
| Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0. | |||
| Licensed under the Apache License, Version 2.0: <http://www.apache.org/licenses/LICENSE-2.0>. | |||
There was a problem hiding this comment.
Are the angle brackets necessary?
| return Err(ParquetError::General( | ||
| "Page CRC checksum mismatch".to_string(), | ||
| )); |
There was a problem hiding this comment.
| return Err(ParquetError::General( | |
| "Page CRC checksum mismatch".to_string(), | |
| )); | |
| return Err(general_err!("Page CRC checksum mismatch")); |
| @@ -0,0 +1,55 @@ | |||
| use std::path::PathBuf; | |||
There was a problem hiding this comment.
Please add the apache license notification.
Closes #6289
Please let me know if we should expose this in the reader APIs instead of a crate feature