How to re-generate correct MANIFEST file?

[xingao0803]

How to re-generate correct MANIFEST file?

Detailed Description

Our aptly server was shutdown suddenly. When we restarted it, seems aptly db crashed. The MANIFEST file only has 0 bytes.
And we always get error like:
ERROR: can't open database: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-1197260]
We run "aptly db recover" to recover db, but still got the same issue.

How can I recover the db and re-generate the correct MANIFEST file?

Thanks.

Context

Possible Implementation

Your Environment

[xingao0803]

The error in LOG file like:
=============== Nov 2, 2017 (UTC) ===============
08:56:04.755149 log@legend F·NumFile S·FileSize N·Entry C·BadEntry B·BadBlock Ke·KeyError D·DroppedEntry L·Level Q·SeqNum T·TimeElapsed

[smira]

@syndtr could you please chime in?

I believe you can try removing corrupted MANIFEST file (which is part of error message) and try running aptly db recover which should rebuild MANIFEST from SSTables. But please make sure you have backup of db/ first!

[syndtr]

Hmm, this is strange DB.Recover should also creates new MANIFEST file. @xingao0803 is it works if you remove the corrupted MANIFEST file first as @smira sugested?

[smira]

I wonder if related to #654

[mevatron]

I'm also seeing this issue as soon as I upgrade from 1.1.1 to 1.2.0.

I tried removing the MANIFEST file and regenerating it with aptly db recover

But, it fails with

syncDir: sync /media/xen/apt/db: invalid argument

Any ideas I could additionally try?

[smira]

@mevatron what kind of filesystem is that? It complains on fsync() I believe?

[mevatron]

@smira This is a cifs mounted filesystem.

[smira]

I don't think this is supposed to work with goleveldb. Do you have a chance to try to copy to local filesystem and run aptly db recover?

[mevatron]

@smira I just rsync'd the apt folder down locally and I am able to run aptly db recover successfully. So, currently I have a CI system and a developer box that both publish packages to the aptly repo, and they have been interacting with it using a network filesystem. What would be the best way to cover this use case do you think?

Thanks for all the assistance!

[smira]

@mevatron aptly doesn't support network filesystems neither for DB nor for pool directory, as this is going to cause issues. the best way of course it use aptly as a service and interact via API

[mevatron]

@smira Thanks I didn't realize I could interact with it via REST API. I will definitely convert my workflow over to that!