mirrorring depends on non-required MD5sum field of the Binary Package Index

[steven-aerts]

The MD5sum field is a non-required field in the Binary Package Index.

Aplty however cannot mirror packages where this field is abscent. It gives the following error message: "unable to compute pool location for filename 'test.deb', MD5 is missing".

This situation appears for example when you want to mirror a debian repository hosted on bintray.

[smira]

Yes, aptly requires MD5sum, do you have example Packages index with that behavior?

[steven-aerts]

Random bintray link: http://dl.bintray.com:80/sbt/debian/Packages

[smira]

As aptly requires MD5sum, most probably it would have to calculate it on its own.

[lusis]

Why not just use the SHA instead of the MD5 as an option.

[smira]

If different package index would have MD5Sum for the same package, we would have split-brain: two entries for the same package.

[linuxluser]

This bug has broken the ability for me to use google-cloud-sdk (example Packages file with missing MD5sum field).

Given that MD5sum is no longer required by the repository format spec (in fact, it's discouraged), I believe the responsibility should fall on aptly to conform to the spec. Specifically, instead of requiring MD5, require SHA256. Here's what the spec reads:

At least one field providing a SHA2 hash shall be provided. Providing SHA256 is highly recommended ...
Clients may not use the MD5Sum and SHA1 fields for security purposes, and must require a SHA256 or a SHA512 field.

If I'm reading this correctly, in this context, aptly would be considered a 'client' and thus should not use MD5sum or SHA1. It also means that even if there are repositories that only provide md5sum field for some reason (old, etc), Apt clients ought not use them anyway.

[hihouhou]

Hello, same problem :-(
aptly mirror update cloud-sdk-jessie-main
Downloading http://packages.cloud.google.com/apt/dists/cloud-sdk-jessie/InRelease...
gpgv: Signature made Fri 30 Sep 2016 01:42:54 AM UTC using RSA key ID A7317B0F
gpgv: Good signature from "Google Cloud Packages Automatic Signing Key gc-team@google.com"
Downloading & parsing package files...
Downloading http://packages.cloud.google.com/apt/dists/cloud-sdk-jessie/main/binary-amd64/Packages.gz...
Building download queue...
ERROR: unable to update: unable to compute pool location for filename google-cloud-sdk_128.0.1-0_all_7b87862bc5988f6446415fd911874a54e55d3c09cc58c9313eb16d6cf7174023.deb, MD5 is missing

[kmasaryk]

I'm also encountering this issue with repos hosted on Artifactory. Artifactory does not use the deprecated MD5 sums.

[linuxluser]

From my short look at the sources, it seems that Aptly is storing the package files in a global pool by their MD5 hash (source code). There is nothing particularly wrong with doing this, however, it seems like this hash is not computed locally but, instead, the MD5 hash is taken from the Packages file (source code). If there is no MD5 hash, then we get the errors.

If this is a correct analysis, it seems like the best approach would be to simply do the MD5 computation for each package. Or maybe use the upstream hash and only compute it locally as a fallback if the MD5, specifically, isn't given. This seems like the simplest solution which wouldn't break things, even if it incurs an efficiency hit.

The other approach, of using SHA256 for example, has the drawbacks of 1) breaking existing pools (at best when you update a mirror, all packages will be re-downloaded and you'll never have access to your history again), and 2) when SHA256 goes away in the future, we are left with the same problem all over again.

Because MD5 is being used for uniquely identifying files and not being used for security, I see no reason why not to continue using MD5. However, the "case of the missing MD5Sum field" must be accounted for.

** UPDATED to provide source code links.

[camelpunch]

@smira any pointers on how the test fixtures could be changed, or added to, to cover the case where MD5Sum isn't present in Packages? I suppose a new example could be added to https://github.com/smira/aptly/blob/master/system/fixture.sh ? Not sure how these are consumed.

[ghost]

Is there a workaround for this?

[smira]

@jsilverman-blispay not yet, working on huge refactoring which would address that, but looks like it hits a lot of code in aptly unfortunately

[smira]

Progress in #539

[linuxluser]

Thanks for getting to this! I know it's a lot of work. Let me know if I can help test implementations.

For those wondering, the current workaround is to ...

1. Create a local repository in aptly (aptly repo create ... myrepo)
2. Download all the *.deb files from upstream mirror manually
3. Add all of the downloaded *.deb files to your aptly repository (aptly repo add myrepo *.deb)

An example of step #2:

#!/bin/sh -e
BaseURI=https://packages.cloud.google.com/apt
PackageURIs=$(curl -s $BaseURI/dists/cloud-sdk-jessie/main/binary-amd64/Packages | sed -n 's#Filename: \(.*\)#-O '${BaseURI}'/\1#p')
curl $PackageURIs

... roughly.

[smira]

@linuxluser right now implementation seems to be fairly stable in #539. There are a lot of changes all across the core of aptly, but good thing that aptly can handle packages even without checksums at all or with any combination of missing checksums. Packages published from aptly though should now be coming with full checksum set always.