Model Management API doesn't work with S3 presigned URL

[Iron-Stark]

Context

I am trying to use the management API with a S3 presigned URL to download and register a new model. This is the code snippet:

   def create_presigned_url(self, bucket_name, object_name, expiration=3600):
        """Generate a presigned URL to share an S3 object

        :param bucket_name: string
        :param object_name: string
        :param expiration: Time in seconds for the presigned URL to remain valid
        :return: Presigned URL as string. If error, returns None.
        """

        # Generate a presigned URL for the S3 object
        s3_client = boto3.client('s3')
        try:
            response = s3_client.generate_presigned_url('get_object',
                                                        Params={'Bucket': bucket_name,
                                                                'Key': object_name},
                                                        ExpiresIn=expiration)
        except ClientError as e:
            logging.error(e)
            return None

        # The response contains the presigned URL
        return response
presigned_uri = self.create_presigned_url(bucket_name, object_name)
response = requests.post("http://127.0.0.1:8081/models?url={}".format(presigned_uri))

Doing a normal GET on the presigned URI works but it doesn't work when used with the model management API. I tried all the workarounds mentioned in #669 but those are not giving any results as well. Can you please recommend what to do to make it work. Listing the errors I get in various situations:

Using the raw presigned URI:

{
  "code": 400,
  "type": "DownloadArchiveException",
  "message": "Failed to download archive from: https://log-analyzer-torchserve-mar.s3.amazonaws.com/test_service/stage/uw1/21_10_22_23_25/anomaly_detection_1634945079.8530345.mar?AWSAccessKeyId=****"
}

Replacing & with %26:

{
  "code": 400,
  "type": "DownloadArchiveException",
  "message": "Failed to download archive from: https://log-analyzer-torchserve-mar.s3.amazonaws.com/test_service/stage/uw1/21_10_26_00_23/anomaly_detection_1635207765.7685282.mar?AWSAccessKeyId=****&Signature=****=&x-amz-security-token=****&Expires=1635283752"
}

Using the torchserve 0.4.2-cpu docker image.

Your Environment

• Installed using source? [yes/no]: no
• Are you planning to deploy it using docker container? [yes/no]: yes
• Is it a CPU or GPU environment?: CPU
• Using a default/custom handler? [If possible upload/share custom handler/model]: custom but unrelated to the issue.
• What kind of model is it e.g. vision, text, audio?: text
• Are you planning to use local models from model-store or public url being used e.g. from S3 bucket etc.?
  [If public url then provide link.]: S3 bucket

Expected Behavior

It should be able to download the model from S3 and register it.

Current Behavior

Returning a 400

Steps to Reproduce

1. Just upload a mar file to a s3 bucket
2. Create the presigned uri using the given code and then call the managment API to try to register the model.
   ...

[lxning]

@Iron-Stark can you try make the mar file public in s3 to see if it work?

[Iron-Stark]

@lxning Thank you for your response. :-) Creating public s3 files is disabled in our corporate accounts. Also the s3 presigned uri has the key and token embedded in it as parameter, so anyone with that url can download the object irrespective of it being a public object or not. I will try your suggestion out using a personal AWS account but I don't believe it will solve the issue.

[mrcolo]

any news on this one? having the same problem for 0.5.0

[lxning]

@mrcolo your s3 url contains aws acceskey and password. it is not safe to include them in url. I guess that's why TS failed to downloads the model due to security policy.

Please follow the link to access encrypted model.

[zacnam]

Still having issues with this - however we have narrowed down the cause of the issue to the following FileUtils.copyURLToFile within the HttpUtils.copyURLToFile function

As we're not currently using S3-SSE-KMS, the function defaults to the following Apache function, which can throw an error from a number of IO issues.

We will try with S3-SSE-KMS enabled and see if that fixes it.

[LewisDavidWong]

I have encountered the exact same issue, I suspect it is due to TorchServe not handling s3 v4 signature URLs as previously mentioned in issue 669. I’m not sure why this issue was closed as it was not resolved.

[AndreasKleineberg]

I am currently facing the same problem. Download via browser or using curl works fine, but torchserve cannot retrieve the model. I always get an HTTP 400 error.

[rmmr]

Same here, quite the dealbreaker

[dgcnz]

Any updates on this issue?

[cfculhane]

Also facing this error - would be great if this could be fixed!