Size check mismatch errors on certain mirror updates #376

david-gregory-inmar · 2016-04-07T19:18:58Z

In the past few days, two of the Amazon EC2 mirrors in my repo set are being stubborn when I try to update them. I get the following messages:

ERROR: unable to update: http://us-east-1.clouds.archive.ubuntu.com/ubuntu/dists/precise-updates/main/binary-i386/Packages.bz2: size check mismatch 1045491 != 1045248
ERROR: unable to update: http://us-east-1.clouds.archive.ubuntu.com/ubuntu/dists/trusty-updates/main/binary-i386/Packages.bz2: size check mismatch 720276 != 712806

If I retry the updates for these two many times, eventually they will complete successfully. I'm not seeing any problems with the remote repos themselves when I try and update an Ubuntu instance directly from them, so I don't think the problem is with Amazon. However, I'm mirroring twenty-two other remote repos and Aptly has no trouble with any of them.

smira · 2016-04-07T19:38:10Z

I don't know the exact reason for that, but this could happen when pulling from the mirror which is syncing currently. In that case Release file signatures (and size info) doesn't match index files, and this error is raised. This is quite common problem with apt repos (when sync is finished, all the files match Release info, and problem goes away).

david-gregory-inmar · 2016-04-07T20:51:44Z

Okay, thanks for replying. This is a relatively new behavior so I thought I'd run it by you. I'll keep testing against those mirrors, if things change I'll post an update here. Maybe it's just Amazon changing the way/time they do their sync...

notpeter · 2016-05-25T18:08:48Z

I have a test case which reproduces this issue reliably for me. Can someone else see if they can duplicate this too?

aptly mirror create pagerduty http://packages.pagerduty.com/pdagent deb/
aptly mirror update pagerduty

Repeat the second command until it breaks (usually takes 2-3 times). Eventually you will get:

ERROR: unable to update: http://packages.pagerduty.com/pdagent/deb/Packages.gz: size check mismatch 2324 != 973

I think the issue may have something to do with picking parsing of this repo's Release file:

MD5Sum:
 2a8a571579f60da022470ce300e731af            11432 Packages
 9a52acd73b01cef65bf8227adaaf026f             2324 Packages.gz
 db77040214a23ab564e3c3a09768dfea             2446 deb/Packages
 5799d88860e10251afb6737995811b3d              973 deb/Packages.gz
 20ba00172df962f951c70828c9fa8123              944 deb/Release

Note the fact that the two sizes (2324 and 973) are correct for two different Packages.gz files.

Btw, I confirmed this wasn't a CDN issue by confirming that all six of the CloudFront IPs which service this repo all consistently return the correct files as requested.

smira · 2016-05-26T13:15:23Z

Looks like a but in aptly, it does postifx match there and it should have been choosing longest match, but it is matching on the first one, and it depends on hash map order, it would match either right one or wrong one.

smira · 2017-05-19T19:42:09Z

I've found a bug which leads to this behaviour, going to fix it for 1.1.0

nikunj12 · 2017-08-15T11:10:07Z

I have seen same error (size mismatch) with aptly 1.1
root@bng-lnx-repo01:/repo# aptly version
aptly version: 1.1.0+1+ga584b2e

Log

root@bng-lnx-repo01:/repo# aptly mirror show ubuntu1604-main2
Name: ubuntu1604-main2
Archive Root URL: http://127.0.0.1:8080/ubuntu/
Distribution: xenial
Components: main, restricted, universe, multiverse
Architectures: amd64, i386
Download Sources: no
Download .udebs: no
Filter: libolecf-dev_20160107-1_amd64 (even I have use filter it tried to download same pkg)
Filter With Deps: no
Last update: never

Information from release file:
Acquire-By-Hash: yes
Architectures: amd64 arm64 armhf i386 powerpc ppc64el s390x
Codename: xenial
Components: main restricted universe multiverse
Date: Thu, 21 Apr 2016 23:23:46 UTC
Description: Ubuntu Xenial 16.04

Label: Ubuntu
Origin: Ubuntu
Suite: xenial
Version: 16.04

aptly mirror update

nikunj12 · 2017-08-15T11:12:04Z

you can see 127.0.01 because I have configure apt-mirror to download the repo locally and use it with aptly.

there are many other pkgs have mismatch errors

smira · 2017-08-15T18:40:32Z

@nikunj12 this is different error from the one we had in this issue (it's about package, not about index).

Would you mind finding matching entry in Packages index and verifying that package downloaded really matches the index?

nikunj12 · 2017-08-17T06:51:48Z

In my case apart from the packages index, other packages are showing mismatch.

even I use the filter to ignore those its failing at same packages .

nikunj12 · 2017-08-17T06:53:12Z

following packages shows mismatch
http://127.0.0.1:8080/ubuntu/pool/main/i/icu/libicu55_55.1-7_i386.deb: size check mismatch 7070440 != 7758562
http://127.0.0.1:8080/ubuntu/pool/universe/g/gcc-i686-linux-android/gcc-i686-linux-android_9_i386.deb: size check mismatch 10499978 != 18193232
http://127.0.0.1:8080/ubuntu/pool/universe/libo/libolecf/libolecf-dev_20160107-1_amd64.deb: size check mismatch 151502 != 353022
http://127.0.0.1:8080/ubuntu/pool/universe/libe/libesedb/libesedb-dbg_20151213-1_i386.deb: size check mismatch 296042 != 346434
http://127.0.0.1:8080/ubuntu/pool/universe/f/festvox-suopuhe-mv/festvox-suopuhe-mv_20041119-2_all.deb: size check mismatch 2468519 != 3020742
http://127.0.0.1:8080/ubuntu/pool/main/o/openjdk-8/openjdk-8-dbg_8u77-b03-3ubuntu3_i386.deb: size check mismatch 112951097 != 128455428
http://127.0.0.1:8080/ubuntu/pool/universe/g/gcc-snapshot/gcc-snapshot_20160415-0ubuntu1_i386.deb: size check mismatch 76449637 != 179706218

smira · 2017-08-17T19:29:57Z

@nikunj12 so what happens is:

aptly downloads Packages file you see in the log, reads them and gets information about checksums and file sizes
aptly starts downloading files and checks that size/checksum match entries in Packages index

So what I'm asking you to do (and as it's your apt-mirror, I can't do that) is to cross-verify Packages index entries and errors above. Also what would you get downloading file with curl/wget?

gittygoo · 2018-04-17T03:10:38Z

I'm getting the same problems on 1.2, seems everytime udebs are involved aptly does not download the complete file. Downloading direct via wget on the machine works flawlessly

smira · 2018-04-17T18:29:31Z

@gittygoo do you have a mirror which reproduces that?

gittygoo · 2018-04-17T18:30:47Z

yes but its an internal mirror

smira · 2018-04-17T21:20:11Z

@gittygoo are you able to share part of Packages index which shows package files and size and error message? If that is size mismatch error as above, I want to figure out where wrong size comes from

gittygoo · 2018-04-17T23:44:31Z

where can i find these? here is an example of the returned output
http://archive.ubuntu.com/ubuntu/pool/main/l/linux/firewire-core-modules-3.13.0-54-generic-di_3.13.0-54.91_amd64.udeb: size check mismatch 453 != 68232 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-lts-vivid/nic-shared-modules-3.19.0-25-generic-di_3.19.0-25.26~14.04.1_amd64.udeb: size check mismatch 453 != 482762 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-lts-vivid/serial-modules-3.19.0-26-generic-di_3.19.0-26.28~14.04.1_amd64.udeb: size check mismatch 453 != 84444 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-lts-xenial/fs-secondary-modules-4.4.0-59-generic-di_4.4.0-59.80~14.04.1_amd64.udeb: size check mismatch 453 != 578908 http://archive.ubuntu.com/ubuntu/pool/main/l/linux/speakup-modules-3.13.0-141-generic-di_3.13.0-141.190_amd64.udeb: size check mismatch 453 != 55050 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-lts-utopic/irda-modules-3.16.0-67-generic-di_3.16.0-67.87~14.04.1_amd64.udeb: size check mismatch 453 != 180518 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-lts-utopic/vlan-modules-3.16.0-76-generic-di_3.16.0-76.98~14.04.1_amd64.udeb: size check mismatch 453 != 26908 http://archive.ubuntu.com/ubuntu/pool/main/l/linux/plip-modules-3.13.0-92-generic-di_3.13.0-92.139_amd64.udeb: size check mismatch 453 != 9328 http://archive.ubuntu.com/ubuntu/pool/main/l/linux/ppp-modules-3.13.0-109-generic-di_3.13.0-109.156_amd64.udeb: size check mismatch 453 != 25908 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-lts-xenial/vlan-modules-4.4.0-28-generic-di_4.4.0-28.47~14.04.1_amd64.udeb: size check mismatch 453 != 27036 http://archive.ubuntu.com/ubuntu/pool/main/l/linux/nic-usb-modules-3.13.0-54-generic-di_3.13.0-54.91_amd64.udeb: size check mismatch 453 != 294868 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-lts-xenial/vlan-modules-4.4.0-72-generic-di_4.4.0-72.93~14.04.1_amd64.udeb: size check mismatch 453 != 27056 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-signed-lts-xenial/kernel-signed-image-4.4.0-53-generic-di_4.4.0-53.74~14.04.1_amd64.udeb: size check mismatch 453 != 6691566 http://archive.ubuntu.com/ubuntu/pool/main/l/linux/nic-shared-modules-3.13.0-36-generic-di_3.13.0-36.63_amd64.udeb: size check mismatch 453 != 442838 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-lts-vivid/irda-modules-3.19.0-80-generic-di_3.19.0-80.88~14.04.1_amd64.udeb: size check mismatch 453 != 228918 http://archive.ubuntu.com/ubuntu/pool/main/l/linux/pcmcia-modules-3.13.0-85-generic-di_3.13.0-85.129_amd64.udeb: size check mismatch 453 != 65060 http://archive.ubuntu.com/ubuntu/pool/main/l/linux/vlan-modules-3.13.0-44-generic-di_3.13.0-44.73_amd64.udeb: size check mismatch 453 != 27000 http://archive.ubuntu.com/ubuntu/pool/main/l/linux-lts-xenial/input-modules-4.4.0-92-generic-di_4.4.0-92.115~14.04.1_amd64.udeb: size check mismatch 453 != 287110

gittygoo · 2018-04-17T23:53:30Z

I just realized this might be because some of the traffic is being blocked by the ubuntu achive servers.
When i try to get a file with lynx i get this:
`Traffic blocked because of exceed session quota

Traffic blocked because of exceed per IP shaper session quota. Please contact the system administrator.
Your session quota is:1000, further traffic will be blocked.`

and after some time it unblocks the traffic again. Strange is that i am only using 2 concurrent downloads

notpeter · 2018-04-18T01:37:33Z

I have not checked the source recently, but this sounds like aptly may not be properly confirming the http response code is 200 OK (or your intercepting proxy is not properly returning a 4xx code). When I originally ran into this issue I found my upstream mirror was returning 500 server errors for ~1% of requests. Aptly was trying to parse the nginx error text as if it were the content (and correctly rejecting it). Perhaps worth considering some sort of retry (with exponential decaying delay) to enable aptly to recover from the occasional load balancer hiccup or spurious tcp reset.

smira · 2018-04-18T18:35:24Z

aptly checks for response status code to be within 200-299, so I think this proxy is broken, and aptly automatically retries on network errors.

smira added the bug label May 26, 2016

smira added this to the v0.9.8 milestone May 26, 2016

smira removed this from the v0.9.8 milestone Mar 27, 2017

smira added the 1.1.0-Maybe label Apr 14, 2017

smira added 1.1.0 and removed 1.1.0-Maybe labels May 19, 2017

smira mentioned this issue May 22, 2017

Fix checksum matching from Release file #574

Merged

6 tasks

smira closed this as completed in #574 May 23, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Size check mismatch errors on certain mirror updates #376

Size check mismatch errors on certain mirror updates #376

david-gregory-inmar commented Apr 7, 2016

smira commented Apr 7, 2016

david-gregory-inmar commented Apr 7, 2016

notpeter commented May 25, 2016 •

edited

Loading

smira commented May 26, 2016

smira commented May 19, 2017

nikunj12 commented Aug 15, 2017

nikunj12 commented Aug 15, 2017

smira commented Aug 15, 2017

nikunj12 commented Aug 17, 2017

nikunj12 commented Aug 17, 2017

smira commented Aug 17, 2017

gittygoo commented Apr 17, 2018 •

edited

Loading

smira commented Apr 17, 2018

gittygoo commented Apr 17, 2018

smira commented Apr 17, 2018

gittygoo commented Apr 17, 2018

gittygoo commented Apr 17, 2018

notpeter commented Apr 18, 2018 via email

smira commented Apr 18, 2018

Size check mismatch errors on certain mirror updates #376

Size check mismatch errors on certain mirror updates #376

Comments

david-gregory-inmar commented Apr 7, 2016

smira commented Apr 7, 2016

david-gregory-inmar commented Apr 7, 2016

notpeter commented May 25, 2016 • edited Loading

smira commented May 26, 2016

smira commented May 19, 2017

nikunj12 commented Aug 15, 2017

nikunj12 commented Aug 15, 2017

smira commented Aug 15, 2017

nikunj12 commented Aug 17, 2017

nikunj12 commented Aug 17, 2017

smira commented Aug 17, 2017

gittygoo commented Apr 17, 2018 • edited Loading

smira commented Apr 17, 2018

gittygoo commented Apr 17, 2018

smira commented Apr 17, 2018

gittygoo commented Apr 17, 2018

gittygoo commented Apr 17, 2018

notpeter commented Apr 18, 2018 via email

smira commented Apr 18, 2018

notpeter commented May 25, 2016 •

edited

Loading

gittygoo commented Apr 17, 2018 •

edited

Loading