"panic: unable to find data.tar.* part in ..."

[ghost]

While publishing, aptly gets a panic when processing a deb which misses "data.tar".

Detailed Description

In my case I had the deb-file http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10.16/linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all.deb which got downloaded by a helper-script and imported by aptly. So this is more an issue of a damaged/broken package provided, but maybe aptly can except such cases.

Log

#        publish_exists "sprd-xenial" "sprd-xenial" \
#        && log "info" "* Update publish 'sprd-xenial'" \
#        && aptly "${CONFIG_ARG}" publish update "${GPG_KEY_ARG}" "sprd-xenial" "sprd-xenial"
sprd-aptly update-sprd: Fri Jul 14 13:53:45 UTC 2017: * Update publish 'sprd-xenial'
Loading packages...
Generating metadata files and linking package files...
panic: unable to find data.tar.* part in /data/mirror/apt/new/xenial/pool/b2/aa/linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all.deb [recovered]
        panic: unable to find data.tar.* part in /data/mirror/apt/new/xenial/pool/b2/aa/linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all.deb

goroutine 1 [running]:
github.com/smira/aptly/cmd.Run.func1(0xc420287f70)
        /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/run.go:17 +0x167 
panic(0xb458c0, 0xc4244401d0)
        /usr/local/Cellar/go/1.8/libexec/src/runtime/panic.go:489 +0x2cf 
github.com/smira/aptly/deb.(*Package).CalculateContents(0xc42048ac00, 0xfcf400, 0xc420268000, 0xc420053120, 0x1, 0x0)
        /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package.go:428 +0x191 
github.com/smira/aptly/deb.(*PackageCollection).loadContents(0xc42000a280, 0xc42048ac00, 0xfcf400, 0xc420268000, 0x1, 0x0, 0xc420286ea8)
        /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package_collection.go:184 +0x17a 
github.com/smira/aptly/deb.(*Package).Contents(0xc42048ac00, 0xfcf400, 0xc420268000, 0xb, 0xc420f16170, 0xb)
        /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package.go:411 +0x7c
github.com/smira/aptly/deb.(*ContentsIndex).Push(0xc42197ce10, 0xc42048ac00, 0xfcf400, 0xc420268000, 0xc4218eae58, 0xc420f16170)
        /Users/smira/Documents/go/src/github.com/smira/aptly/deb/contents.go:30 +0x5a
github.com/smira/aptly/deb.(*PublishedRepo).Publish.func2(0xc42048ac00, 0x0, 0x0)
        /Users/smira/Documents/go/src/github.com/smira/aptly/deb/publish.go:586 +0x798 
github.com/smira/aptly/deb.(*PackageList).ForEachIndexed(0xc420437380, 0xc420287900, 0xc42197c9c0, 0x0)
        /Users/smira/Documents/go/src/github.com/smira/aptly/deb/list.go:171 +0x69
github.com/smira/aptly/deb.(*PublishedRepo).Publish(0xc420156000, 0xfcf400, 0xc420268000, 0xfc63c0, 0xc4201b7760, 0xc42005a600, 0xfd1020, 0xc42025bbc0, 0xfd18c0, 0xc42025d590, ...)
        /Users/smira/Documents/go/src/github.com/smira/aptly/deb/publish.go:611 +0xd09 
github.com/smira/aptly/cmd.aptlyPublishUpdate(0xc4201cf0e0, 0xc4201d0ea0, 0x2, 0x2, 0xfc7c80, 0xc420175380)
        /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/publish_update.go:62 +0x680
github.com/smira/aptly/vendor/github.com/smira/commander.(*Command).Dispatch(0xc4201cf0e0, 0xc4201d0ea0, 0x2, 0x2, 0x1, 0xc4201f9e38)
        /Users/smira/Documents/go/src/github.com/smira/aptly/vendor/github.com/smira/commander/commands.go:305 +0x243
github.com/smira/aptly/vendor/github.com/smira/commander.(*Command).Dispatch(0xc4201cf320, 0xc4201d0e90, 0x3, 0x3, 0xc4201f9e01, 0xc4200001a0)
        /Users/smira/Documents/go/src/github.com/smira/aptly/vendor/github.com/smira/commander/commands.go:283 +0x48b
github.com/smira/aptly/vendor/github.com/smira/commander.(*Command).Dispatch(0xc4201cfb00, 0xc4201d0e80, 0x4, 0x4, 0xc4201d96c0, 0xc4201d0e80)
        /Users/smira/Documents/go/src/github.com/smira/aptly/vendor/github.com/smira/commander/commands.go:283 +0x48b
github.com/smira/aptly/cmd.Run(0xc4201cfb00, 0xc420066010, 0x6, 0x6, 0xc420000101, 0x0)
        /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/run.go:41 +0x12a
main.main()
        /Users/smira/Documents/go/src/github.com/smira/aptly/main.go:20 +0xda
# Then the publish of this repo gets not updated.

Possible Implementation

Maybe the publish or pre-publish steps should do a sanity check of the processed package. Now a corrupt package prevents publishing/updating a repo. If the package is just skipped and a warning is given to the user, but the rest of the publish would go through.

Additional notes

dpkg also has issues processing the deb:

root@m572:/tmp# dpkg-deb -R ./linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all.deb ./linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all/
dpkg-deb: error: unexpected end of file in archive member header in ./linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all.deb
root@m572:/tmp# 
root@m572:/tmp# dpkg -x linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all.deb ./linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all/
dpkg-deb: error: unexpected end of file in archive member header in linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all.deb

root@m572:/tmp/linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all# ar x linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all.deb 
root@m572:/tmp/linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all# ls -l
total 756
-rw-r--r-- 1 root root 382274 Jul 18 15:54 control.tar.gz
-rw-r--r-- 1 root root      4 Jul 18 15:54 debian-binary
-rw-r--r-- 1 root root 382406 Jul 18 15:53 linux-headers-4.10.16-041016_4.10.16-041016.201706210231_all.deb

Your Environment

aptly 1.0.1

If you are missing information, please let me know.

[smira]

Thanks for your report, it should be fixed in upcoming 1.1.0 (or nightly build) in #534

(Please note, that going back from nightly to 1.0.1 is not quite possible if you try to add packages or mirror update something).

[ghost]

I may can try with a subset of my setup with the nightly build (or wait for the next release).
Thanks for the fast response!