stop panic in MetadataLoader on invalid data

[samuelcolvin]

Which issue does this PR close?

None

Rationale for this change

While messing with our object_store implementation I sent invalid data to MetadataEncoder and got a panic:

thread 'tokio-runtime-worker' panicked at /Users/samuel/.cargo/git/checkouts/arrow-rs-3b86e19e889d5acc/ee2f75a/parquet/src/arrow/async_reader/metadata.rs:74:40:
attempt to subtract with overflow
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
libunwind: stepWithCompactEncoding - invalid compact unwind encoding

Looking at the code I'm pretty sure it's because suffix_len is <8 causing the subtract to fail.

What changes are included in this PR?

return an error instead of panicing. I don't know how to reproduce this in a test, hence just the change. LMK if this requires a test?

Are there any user-facing changes?

no.

[Xuanwo]

This change looks good to me initially, but I'm wondering about the logic behind it.

First of all, it should fail at file_size < 8, right? We need to clarify this in our API documentation regarding the validation of file_size.

Additionally, it's possible that users provide an incorrect file size. The result of fetch.fetch(footer_start..file_size).await?; seems incorrect. We should specify that fetch must handle this correctly, returning an error if the range of footer_start..file_size fails to read. We can add a check here: if suffix_len != file_size - footer_start, we can raise an error like EOF.

What do you think? My motivation is that we should return the error at the correct place instead of here, where it is just a side effect.

---

I believe this error could be reproduced by an invalid file size or a malfunctioning fetch that returns a smaller size than expected.

[jhorstmann]

Maybe it is also possible to hit this error if size_hint is smaller than 8? A minimum value for size_hint could make sense, or returning an error if it is too small.

[samuelcolvin]

ye I think @jhorstmann was right, the error almost certainly happened because the prefetch value was invalid, not because fetch returned the wrong size of slice.

I've updated to check prefetch size and added a test.

[etseidl]

Given that it's a "hint", I think it's ok to ignore an invalid size and instead set size_hint to 8 if necessary (although I suppose the docs for fetch_parquet_metadata would need to reflect this). I'd make a similar argument for a prefetch value larger than file_size.

[samuelcolvin]

I can just set it to None in both cases if that's agreed?

[etseidl]

I'd prefer to not penalize a user for over specifying the prefetch. I think of it as a budget... you can use up to 2MB of prefetch. I can see angry AWS customers complaining that a metadata fetch that should have been a single GET is now 3 😅.

[samuelcolvin]

Okay, I'm lost on what you would suggest. Feel free to take this, or let me know concretely what you'd suggest.

[etseidl]

I'm thinking along the lines of

        // If a size hint is provided, read more than the minimum size
        // to try and avoid a second fetch.
        let footer_start = if let Some(size_hint) = prefetch {
            // check for hint smaller than footer
            let size_hint = std::cmp::max(size_hint, FOOTER_SIZE);
            // check for hint larger than the file
            let size_hint = std::cmp::min(size_hint, file_size);
            file_size.saturating_sub(size_hint)
        } else {
            file_size - FOOTER_SIZE
        };

This guards against the error you're seeing, but allows providing a hint larger than the file without triggering extra I/O (as would happen if the hint were simply set to None).

[samuelcolvin]

LGTM, except we don't need the let size_hint = std::cmp::min(size_hint, file_size); since we're using file_size.saturating_sub(size_hint) directly afterwards.

Changed.

[etseidl]

One last nit. Love the reduction in magic numbers! Thanks @samuelcolvin.

[etseidl]

This is defined elsewhere in the parquet crate.

use crate::file::FOOTER_SIZE;

[samuelcolvin]

done.