when asserting that the size of a restored partition is close to the
original partition in topic recovery tests, we use the default segment
size and a multiplier of 1.5 to tolerate errors where the last segment
of the ntp is not uploaded to tiered storage because it is open.

sometimes the segment is larger than 1.5x the default size causing an
assertion failure in the test. this change stores the last segment size
for each partition and uses it as the tolerance size in assertions.

when comparing segment sizes which are restored from s3, the restored
size is usually smaller than the original size because of the presence
of config batches in the original which are removed from restored
segment.

with this change we try to estimate the size of these omitted config
batches to account for the difference we will tolerate in original and
restored segments.

this is an approximate comparison, it computes the max offsets for a ntp
and uses it to come up with a diff assuming 1024 bytes size of a config
batch - which is actually smaller, close to 512 bytes.

Instead of guessing the segment batch sizes from counts in
manifest.json, we parse segment files downloaded from minio/s3 and
accumulate the sizes of non data batches to account for differences in
data on s3 vs restored partitions.

we use the cloud_storage_max_upload_sec setting to force uploading open
segments so that we can get a closer size of the recovered segment
compared to original segment. this allows us to do away with the large
tolerance value.

as we force upload to s3 frequently, when shutting down the node we
should wait for S3 to have all data uploaded. previously we waited until
the data was close enough (a diff of default_segment_size * 1.5 was
acceptable).

files on disk can be larger than actual data uploaded to cloud storage.
it appears this is due to block allocation of segments, in this case the
maximum diff can be 4095 bytes, if there is an extra block allocated for
one byte on disk. so we adjust the margin of error to 4096 bytes to
account for these differences.