Misbehaving --dryrun flag in az storage blob delete-batch

[elibarzilay]

(Ubuntu, azure-cli 2.0.27, and showing storage as 2.0.25.)

I've tried playing with the new delete-batch, and could not figure out
how to use --if-modified-since or --if-unmodified-since -- no matter
what I did, it seems to always insist on deleting everything. After
lots of fiddling around, I used a new container just to experiment with,
and to my surprise, it looks like --dryrun just ignores these flags,
but when it's not used, there are not ignored (but have other, unrelated
issues).

If this is correct then this is a very bad problem, since it is common
to use a dry run flag to see what will be deleted, and ignoring these
flags means that I can end up deleting blobs unexpectedly.

[williexu]

The parameters are used to limit the operations according to whether the resource has been modified or not.
None of the batch commands take these parameters into account upon --dryrun.

--dryrun will simply list all blobs under consideration for deletion/upload.
This will always be a superset of the actual blobs that would be deleted/uploaded.
Only once the actual command is run will the properties be applied on a blob by blob basis.

[elibarzilay]

@williexu This is pretty bad in two ways:

1. It is a definition of "dry run" that is very different from any other
   use of the term. This should be made very clear in both the -h
   text and in the output that it produces because it's so unexpected.

2. There should be some way for people to do a proper dry run.
   Consider writing a script that purges old files: it is very common to
   write such a script in a dry run mode to verify that it would delete
   the correct blobs, and then just remove the dry run flag.

   If I understand it correctly, there is no way other than either
   creating a dummy container for testing (which is a major PITA, since
   there is no way to change the last-modified time, so it's a tedious
   process of copying files, waiting some time, etc).

[williexu]

@seguler If we want to do this, it would have to be done through server support for batch-commands.
The CLI simply passes the modified parameters to the service to process and cannot be done for a dryrun.

[arbaiti]

Hello,
I need to do cleanup of dump files on a blob container, and i don't know how to use this command.
I should delete dump older than 14 days.
Please if you have an example ?
Regards,

[williexu]

@arbaiti if you still need an example:

az storage blob delete-batch --source backups --if-unmodified-since `date -d "14 days  ago" '+%Y-%m-%dT%H:%MZ'`

[williexu]

Changing this issue to a service attention feature, as the CLI can only do so much to facilitate a dry-run if the service does not support this option.

[amishra-dev]

Hi,
We are discussing this feature request between the Azure Cli and the Azure Storage team. We will update this bug with what we finalize.

[Juliehzl]

Any update here? @amishra-dev

[zezha-msft]

Hi @Juliehzl, please correct me if I'm wrong, does the delete-batch command perform a listing operation and then invoke delete command on each blob?

If yes, I would suggest to implement this feature on the client side, since the last-modified-time of each blob is available and the dry run can make the comparison. AFAIK, no dry run command exists on the service end, and there doesn't seem to be any such plan.

[Juliehzl]

@zezha-msft you are really right. But for the list operations, list_blobs in SDK doesn't accept precondition arguments, such as if_modified_since.
In this way, we cannot collect blobs with precondition arguments on the client side. If you know some ways to do that, please let me know and we can do implement the feature in CLI.

[zezha-msft]

Hi @Juliehzl, the last-modified-time is returned for every blob with listing operations, so the comparison can be made on the client side.

Example:

if if_modified_since is not None and if_modified_since is earlier than blob_lmt:
    continue to delete
...

[zezha-msft]

It doesn't make much sense for the service to provide a dry-run operation, since the same result is also not guaranteed when the user eventually makes the real call.

Example:

1. Cx makes dry-run request against service
2. Service says the blob would be deleted if this was a real request
3. Cx makes real request against service
4. The blob now has a lease (or any other state), and delete fails

The service cannot provide any additional guarantee than the client side, so I see no reason why dry-run has to be done on the service side.

[yonzhan]

move to S166.