Unable to import file: file already exists

[maratsh]

I try to replace package with same package (but different by content). Also, i have over 4000+ packages that aptly added well in one transaction (not listed here).

This is error.

aptly repo add -force-replace=true xbtos-imx6ulevk-dev /var/repo/tmp/jenkins-xbt-image-20/xbtos-imx6ulevk-dev/cortexa7hf-vfp-neon/libgnutls-dbg_3.3.17.1-r0_armel.deb
Loading packages...
[!] Unable to import file /var/repo/tmp/jenkins-xbt-image-20/xbtos-imx6ulevk-dev/cortexa7hf-vfp-neon/libgnutls-dbg_3.3.17.1-r0_armel.deb into pool: unable to import into pool: file /var/repo/aptly/pool/55/bb/libgnutls-dbg_3.3.17.1-r0_armel.deb already exists
[!] Some files were skipped due to errors:
  /var/repo/tmp/jenkins-xbt-image-20/xbtos-imx6ulevk-dev/cortexa7hf-vfp-neon/libgnutls-dbg_3.3.17.1-r0_armel.deb

aptly package  show -with-references=true -with-files=true libgnutls-dbg_3.3.17.1-r0_armel
Package: libgnutls-dbg
Priority: optional
Section: devel
Maintainer: Poky <poky@yoctoproject.org>
Architecture: armel
Version: 3.3.17.1-r0
Provides: gnutls-dbg
Recommends: nettle-dbg, libgmp-dbg, libstdc++-dbg, libgcc-s-dbg, libz-dbg, libc6-dbg
Filename: libgnutls-dbg_3.3.17.1-r0_armel.deb
Size: 2198478
MD5sum: 138d6b6f4bc063e563aa93ebdd71f568
SHA1: 2aa132ed4e67b6d28e9a5cf2f02f171ef14f99c6
SHA256: a7795602bc916a4b05d0c0f291fecd323ab3ab5230692d13b50563677851e5eb
Description: GNU Transport Layer Security Library - Debugging files
 GNU Transport Layer Security Library.  This package contains ELF symbols
 and related sources for debugging purposes.
Oe: gnutls
Homepage: http://www.gnu.org/software/gnutls/
Packagearch: cortexa7hf-vfp-neon

Files in the pool:
  /var/repo/aptly/pool/13/8d/libgnutls-dbg_3.3.17.1-r0_armel.deb

References to package:
  snapshot [xbtos-imx6ulevk-dev-2015.12.19-b1]: Snapshot from local repo [xbtos-imx6ulevk-dev]: xbterminal-firmware

ls -la /var/repo/aptly/pool/55/bb/libgnutls-dbg_3.3.17.1-r0_armel.deb 
-rw-r--r-- 2 repo repo 2198670 Dec 16 15:24 /var/repo/aptly/pool/55/bb/libgnutls-dbg_3.3.17.1-r0_armel.deb

It looks like aptly cleanup does not do "Deleting unreferenced files " - File /var/repo/aptly/pool/55/bb/libgnutls-dbg_3.3.17.1-r0_armel.deb is not listed in aptly database

aptly db cleanup
Loading mirrors, local repos, snapshots and published repos...
Loading list of all packages...
Deleting unreferenced packages (0)...
Building list of files referenced by packages...
Building list of files in package pool...
Deleting unreferenced files (0)...
Compacting database...
ls -la /var/repo/aptly/pool/55/bb/libgnutls-dbg_3.3.17.1-r0_armel.deb 
-rw-r--r-- 2 repo repo 2198670 Dec 16 15:24 /var/repo/aptly/pool/55/bb/libgnutls-dbg_3.3.17.1-r0_armel.deb

[smira]

@Deutsche that is really interesting... this is actually first time I hear that, that's design problem of aptly. It can't handle duplicate packages with same first two bytes of MD5. The file you have in the pool and the file you're trying to import have the same first two bytes of MD5 but different size.

[maratsh]

It's a pity! It is can be fixed?
why not use full md5 for path?

[richard-scott]

I also get that when trying to make a snapshot from a debian mirror:

root@localhost:~/sbin/aptly# aptly publish -skip-signing --distribution=jessie snapshot debian-1454272343 debian
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: error linking file to /root/.aptly/public/debian/pool/main/0/0ad/0ad_0.0.17-1_amd64.deb: file already exists and is different

The original mirror was setup with this:

aptly mirror create -architectures=amd64 -with-sources debian http://ftp.ticklers.org/debian/ jessie main
aptly mirror update debian
aptly snapshot create debian-1454272343 from mirror debian

Do you think it is also an MD5 clash?

[smira]

@richard-scott no, http://www.aptly.info/doc/feature/duplicate/

When aptly is publishing repository, it would give an error if conflicting package files 
(same name, but different content) are put together in one package pool. Package 
pool is shared by all published repositories with the same component and prefix. 
The same applies to switching snapshots or updating published repositories:
 if previous state and new state contain two conflicting packages, aptly would 
give an error. If you’re completely sure that this update operation is correct, 
you can use flag -force-overwrite to disable check for conflicting package files.

[richard-scott]

ok, I've deleted my .aptly folder and the .aptly.conf file.
resetup the mirror again:

# aptly mirror create -architectures=amd64 -with-sources debian-backports-mirror http://ftp.ticklers.org/debian-backports/ squeeze-backports main

Updated the mirror:

aptly mirror update debian-backports-mirror

(I think I am being quite restricive on what I want to mirror.)

I have then created the snapshot:

aptly snapshot create debian-backports-mirror-1454338256 from mirror debian-backports-mirror

and attempted to publish it:

aptly publish -skip-signing --distribution=jessie snapshot debian-backports-mirror-1454338256 debian-backports
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: error linking file to /root/.aptly/public/debian-backports/pool/main/b/bind9/bind9_9.8.4.dfsg.P1.orig.tar.gz: file already exists and is different

I am unable to find a duplicate file:

cd ~/.aptly/pool
find -name bind9_9.8.4.dfsg.P1.orig.tar.gz
./96/f5/bind9_9.8.4.dfsg.P1.orig.tar.gz

I don't want to use -force-overwrite as there are no duplicate package files that I can find on disk so don't feel that is the right way forward.

Can you see what am I doing wrong or is this a bug?

[richard-scott]

Hmmm, deleted it all again and it is complaining about a totally different file:

aptly publish -skip-signing --distribution=jessie snapshot debian-backports-mirror-1454341261 debian-backports
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: error linking file to /root/.aptly/public/debian-backports/pool/main/a/adblock-plus/adblock-plus_2.1-1~bpo60+1.debian.tar.gz: file already exists and is different

same thing tho, only one copy of that file anywhere:

# find /root/.aptly -name adblock-plus_2.1-1~bpo60+1.debian.tar.gz
/root/.aptly/public/debian-backports/pool/main/a/adblock-plus/adblock-plus_2.1-1~bpo60+1.debian.tar.gz
#

I can't understand why an official debian repo would have duplicate files in there as this would conflict with their packaging guidelines.

It's almost as if there are two threads in aptly publishing the snapshot and the 2nd thread is incorrectly processing the same links as the 1st thread and this then causes the thing to abort.

[smira]

@richard-scott your find result looks strange, you should have at least two files named adblock-plus_2.1-1~bpo60+1.debian.tar.gz - one in the aptly pool and another one in public directory.

There are no 2 threads doing publishing. aptly does the check using the fact that files are hard-linked from aptly pool to public directory. So if aptly discovers something which is not a hard link to the file it expects, it refuses to overwrite it. If you did something (like copying files) without preserving hardlinks, you might see this problem.

[richard-scott]

I think that this may have been caused by my use of SSHFS for the /root/.aptly folder? I wanted to keep that folder off my VM so when I rebuilt it I didn't have to start all over again.

i'm not sure if that really is the root cause as I am able to mirror, make snapshots and publish other repos... odd.

Anyway, I'm now running it with NFS as that mountpoint and things seem better so far.

[smira]

aptly is using hard links, if your FS fails to provide them in correct way, you might have any kind of problems going forward.

[smira]

See #506

[smira]

@maratsh Should be fixed with #539

Sorry it took too long