SignatureDoesNotMatch with multipart upload to Ceph radosgw

[tehwalris]

We are using minio-py to upload files to our S3 server (Ceph radosgw). Uploading files over 5MiB fails with SignatureDoesNotMatch.

The following code is enough to reproduce the issue with Python 3.6.6, Ceph 12.2.8 and minio-py 4.0.8:

import os
from minio import Minio

client = Minio('rgw.vis.ethz.ch',
                access_key='<our access key>',
                secret_key='<our secret key>',
                secure=True)

fname = '<path to some local file larger than 5MiB>'
with open(fname, 'rb') as file_data:
        file_stat = os.stat(fname)
        client.put_object('<our bucket name>', '<some remote path>', file_data,
                        file_stat.st_size, content_type='<some content type>')

It fails with the following stacktrace:

Traceback (most recent call last):
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/api.py", line 1555, in _stream_put_object
    upload_result = pool.result()
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/thread_pool.py", line 92, in result
    raise self.exceptions_queue.get()
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/thread_pool.py", line 57, in run
    result = func(*args, **kargs)
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/api.py", line 1502, in _upload_part_routine
    upload_id, part_number, sse=sse)
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/api.py", line 1491, in _do_put_object
    content_sha256=sha256_hex
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/api.py", line 1832, in _url_open
    object_name).get_exception()
minio.error.SignatureDoesNotMatch: SignatureDoesNotMatch: message: The request signature we calculated does not match the signature you provided.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 14, in <module>
    file_stat.st_size, content_type='application/pdf')
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/api.py", line 782, in put_object
    data, length, metadata=metadata, sse=sse)
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/api.py", line 1561, in _stream_put_object
    upload_id)
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/api.py", line 1618, in _remove_incomplete_upload
    headers={})
  File "/home/philippe/temp/minio-py-debug/.venv/lib/python3.6/site-packages/minio/api.py", line 1832, in _url_open
    object_name).get_exception()
minio.error.SignatureDoesNotMatch: SignatureDoesNotMatch: message: The request signature we calculated does not match the signature you provided.

The issue does not occur with files smaller than 5 MiB. The issue does not occur when using Minio as the S3 server. The issue does not occur when uploading files to radosgw through the Minio CLI.

I have filed a corresponding issue in Ceph.

[kannappanr]

@tehwalris in the case of minio-py, anything above 5MiB is uploaded using multi-part upload. In the case of mc, the size is 64MiB. So, if you are trying to upload say an object of size 6MiB, it is not using multi-part upload in the case of mc. You might have to try to upload an object of size say 70MiB

You can try that with mc and with the debug flag turned on.

[tehwalris]

@kannappanr I've tried with 500MiB files to the same Ceph/radosgw server using the Minio CLI. That works fine. Here's the debug output for a (working) example of that.

[kannappanr]

@tehwalris Can you also please run the following code and paste the debug output here?

import os
import sys
from minio import Minio

client = Minio('rgw.vis.ethz.ch',
                access_key='<our access key>',
                secret_key='<our secret key>',
                secure=True)

fname = '<path to some local file larger than 5MiB>'
with open(fname, 'rb') as file_data:
        file_stat = os.stat(fname)
	client.trace_on(sys.stderr)
        client.put_object('<our bucket name>', '<some remote path>', file_data,
                        file_stat.st_size, content_type='<some content type>')

[tehwalris]

I ran the code you gave with the same server, bucket and file as the working CLI example. Here's the output.

[harshavardhana]

Most probably the bug is in Ceph server

PUT /vis-visit-staging/visionen/gzds/test.pdf?partNumber=3&uploadId=2%7EOAFGozUlbdCeflyvZp1Ofm9A7qZxNew HTTP/1.1
User-Agent: Minio (Linux; x86_64) minio-py/4.0.8
Content-Length: 5242880
Content-Md5: /Nn7Fl1uETASDHSVUt9AHw==
Host: rgw.vis.ethz.ch
X-Amz-Date: 20181212T090720Z
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
Authorization: AWS4-HMAC-SHA256 Credential=JNG80IALJP9QQZQC9D77/20181212/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-md5;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=324295fb1abb42b9f7853b21ec7fb1874f62c548c02cd2dd27cf502f9b71663c

Can you send the output of mc here ?

[harshavardhana]

The issue is uploadid is not encoded in case of mc

PUT /vis-visit-staging/visionen/gzds/test.pdf?partNumber=1&uploadId=2~XG6srDcyzowhxw7f2brGJyy1fXlvfUs HTTP/1.1
Host: rgw.vis.ethz.ch
User-Agent: Minio (linux; amd64) minio-go/v6.0.6 mc/2018-11-16T14:52:27Z
Content-Length: 67108864
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181211/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20181211T210033Z
Accept-Encoding: gzip

mcli: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Date: Tue, 11 Dec 2018 21:01:53 GMT
Etag: "364c6941389baad1e940ccfbad52096e"
X-Amz-Request-Id: tx00000000000000030baa9-005c102571-190eaa9-default

Because of this perhaps minio-py signature calculation is wrong

[tehwalris]

Can you send the output of mc here ?

@harshavardhana Which output do you need exactly? It looks like you have the mc log that I previously uploaded.

[harshavardhana]

@harshavardhana Which output do you need exactly? It looks like you have the mc log that I previously uploaded.

Not needed anymore

[stumelius]

I'm seeing similar behavior (incorrect content-md5) when trying to upload files larger than 5 MB to DigitalOcean Spaces or Scaleway Object Storage. As a workaround, I just commented out the Content-Md5 header in _complete_multipart_upload:

headers = {
    'Content-Length': len(data),
    'Content-Type': 'application/xml',
    #'Content-Md5': md5_base64,
}

[kannappanr]

Should be fixed by #798. Closing this as fixed. please let us know if you are seeing this issue still.