Unable to delete Object in empty dir with s3.DeleteObject

[svenwltr]

Solution

Set the aws.Config.DisableRestProtocolURICleaning member to true, to use adjacent slashes, trailing slashes, or dot slash(./) in key names.

svc := s3.New(sess, &aws.Config{
   	DisableRestProtocolURICleaning: aws.Bool(true),
})
out, err := svc.GetObject(&s3.GetObjectInput {
   	Bucket: aws.String("bucketname"),
    	Key: aws.String("//foo//bar//moo"),
})

---

Version of AWS SDK for Go?

v1.13.17

Version of Go (go version)?

go version go1.9.2 linux/amd64

What issue did you see?

It is not possible to delete an Object in an "empty dir" with s3.DeleteObject, since the SDK converts // to / in the request path.

Steps to reproduce

1. Create Object: aws s3 cp holidays s3://aws-empty-object-test/foo//bar/holidays

upload: ./holidays to s3://aws-empty-object-test/foo//bar/holidays

2. Verify object is there: aws s3 ls s3://aws-empty-object-test/foo//bar/.

2018-03-21 09:25:17        142 holidays

3. Delete Object: ./bug-test foo//bar/holidays (see example below).

{
  Bucket: "aws-empty-object-test",
  Key: "foo//bar/holidays"
}
sending AWS request:
    > DELETE /aws-empty-object-test/foo/bar/holidays HTTP/1.1
    > Host: aws-empty-object-test.s3.eu-west-1.amazonaws.com
    > Authorization: <hidden>
    > User-Agent: aws-sdk-go/1.13.17 (go1.9.2; linux; amd64)
    > X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    > X-Amz-Date: 20180321T084011Z

Note that the path is /aws-empty-object-test/foo/bar/holidays instead of /aws-empty-object-test/foo//bar/holidays.

4. Verify object is still there: aws s3 ls s3://aws-empty-object-test/foo//bar/.

2018-03-21 09:25:17        142 holidays

Verification of the example code

1. Create Object: aws s3 cp holidays s3://aws-empty-object-test/blub/holidays

upload: ./holidays to s3://aws-empty-object-test/foo//bar/holidays

2. Verify object is there: aws s3 ls s3://aws-empty-object-test/blub/.

2018-03-21 09:40:42        142 holidays

3. Delete Object: ./bug-test blub/holidays (see example below).

{
  Bucket: "aws-empty-object-test",
  Key: "blub/holidays"
}
sending AWS request:
    > DELETE /aws-empty-object-test/blub/holidays HTTP/1.1
    > Host: aws-empty-object-test.s3.eu-west-1.amazonaws.com
    > Authorization: <hidden>
    > User-Agent: aws-sdk-go/1.13.17 (go1.9.2; linux; amd64)
    > X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    > X-Amz-Date: 20180321T084432Z

4. Verify object is still there: aws s3 ls s3://aws-empty-object-test/blub/.

Example Code

package main

import (
	"fmt"
	"os"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/request"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
	sess, err := session.NewSessionWithOptions(session.Options{
		Config: aws.Config{
			Region: aws.String("eu-west-1"),
		},
		SharedConfigState: session.SharedConfigEnable,
		Profile:           "default",
	})
	if err != nil {
		panic(err)
	}

	sess.Handlers.Send.PushFront(func(r *request.Request) {
		fmt.Printf("sending AWS request:\n%s\n", DumpRequest(r.HTTPRequest))
	})

	svc := s3.New(sess)
	params := &s3.DeleteObjectInput{
		Bucket: aws.String("aws-empty-object-test"),
		Key:    aws.String(os.Args[1]),
	}
	fmt.Println(params)
	_, err = svc.DeleteObject(params)
	if err != nil {
		panic(err)
	}
}

package main

import (
	"bytes"
	"net/http"
	"net/http/httputil"
	"regexp"

	"github.com/rebuy-de/aws-nuke/pkg/util"
)

var (
	RESecretHeader = regexp.MustCompile(`(?m:^([^:]*(Auth|Security)[^:]*):.*$)`)
)

func HideSecureHeaders(dump []byte) []byte {
	return RESecretHeader.ReplaceAll(dump, []byte("$1: <hidden>"))
}

func DumpRequest(r *http.Request) string {
	dump, err := httputil.DumpRequest(r, true)
	if err != nil {
		panic(err)
	}

	dump = bytes.TrimSpace(dump)
	dump = HideSecureHeaders(dump)
	dump = util.IndentBytes(dump, []byte("    > "))
	return string(dump)
}

[xibz]

Hello @svenwltr, thank you for reaching out to us. Try setting DisableRestProtocolURICleaning in the aws.Config. That should preserve the extra /. Please let us know if that resolves your issue.

[svenwltr]

Thank you @xibz, this indeed works. I hope changing this does not cause any other side effects.

I guess it is hard to fix it in general for DeleteObjectInput, but it might be helpful to add a note to the docs, since "empty directories" are possible in S3.

[diehlaws]

Thanks for the feedback @svenwltr and I'm glad to hear this config option works as expected for you. Since this use case applies to other API calls in addition to DeleteObject we'll be adding a note about this config option at the top of the documentation page for the S3 package.