Add system tests for topic/subscription IAM policy get/set methods.

[tseaver]

The tests uncover a wart in the API: setIamPermissions takes an extra wrapper element (policy) around the actual Policy resource. I don't know where to report that issue.

The new system tests fail repeatedly for my system account with 503s: adding retries (interspersed with time.sleep(1)) doesn't seem to help. @tmatsuo can you comment? (Note that I have made that account an owner of my project).

[dhermes]

Mostly looks fine though we should resolve the 503s before I gave an LGTM. Assigned to @tmatsuo for now.

[dhermes]

@tmatsuo Bump

[tmatsuo]

Which test and API are repeatedly failing? Any logs? Detailed message? Does it always fail or sometimes succeed? Can you show the retry code?

[tseaver]

@tmatsuo the new system tests being added in this PR fail with 503s at the following points.

• https://github.com/GoogleCloudPlatform/gcloud-python/pull/1654/files#diff-5e9dc96fb356f1cbb692593c8a9d2988R180
• https://github.com/GoogleCloudPlatform/gcloud-python/pull/1654/files#diff-5e9dc96fb356f1cbb692593c8a9d2988R193

Wrapping the set_iam_policy calls with retry logic didn't help, so I removed it.

[tmatsuo]

Does it always fail? If so, is it possible to show the actual JSON request?

[tseaver]

@tmatsuo Just before the projects.topics.setIamPolicy API call:

(Pdb) l
298             """
299             client = self._require_client(client)
300             path = '%s:setIamPolicy' % (self.path,)
301             resource = policy.to_api_repr()
302             wrapped = {'policy': resource}
303  ->         resp = client.connection.api_request(
304                 method='POST', path=path, data=wrapped)
305             return Policy.from_api_repr(resp)
(pdb) pp path
u'/projects/citric-celerity-697/topics/test-iam-policy-topic1459191212607:setIamPolicy'
(Pdb) pp wrapped
{'policy': {'bindings': [{'members': ['user:jgeewax@google.com'],
                          'role': 'roles/reader'}],
            'etag': u'ACAB'}}

At the return:

(Pdb) pp response
{'-content-encoding': 'gzip',
 'alt-svc': 'quic=":443"; ma=2592000; v="31,30,29,28,27,26,25"',
 'alternate-protocol': '443:quic,p=1',
 'cache-control': 'private',
 'content-length': '162',
 'content-type': 'application/json; charset=UTF-8',
 'date': 'Mon, 28 Mar 2016 18:55:54 GMT',
 'server': 'ESF',
 'status': '503',
 'transfer-encoding': 'chunked',
 'vary': 'Origin, X-Origin, Referer',
 'x-content-type-options': 'nosniff',
 'x-frame-options': 'SAMEORIGIN',
 'x-xss-protection': '1; mode=block'}
(Pdb) pp content
'{\n  "error": {\n    "code": 503,\n    "message": "The service was unable to fulfill your request. Please try again. [code=8a75]",\n    "status": "UNAVAILABLE"\n  }\n}\n'
(Pdb) pp url
'https://pubsub.googleapis.com/v1/projects/citric-celerity-697/topics/test-iam-policy-topic1459191212607:setIamPolicy'

[tseaver]

@tmatsuo the projects.subscriptions.getIamPolicy call is now repeatable returning a 404 (waiting is not helping):

gcloud.exceptions.NotFound: 404 Resource not found (resource=test-iam-policy-sub-1459192208068). (GET https://pubsub.googleapis.com/v1/projects/citric-celerity-697/subscriptions/test-iam-policy-sub-1459192208068:getIamPolicy)

[tmatsuo]

According to https://cloud.google.com/pubsub/access_control

I don't think roles/reader is a valid role. Can you try it with roles/viewer or roles/pubsub.viewer ?
The error message is not helpful at all, which should be improved though.

[tseaver]

@tmatsuo After updating the role strings, I'm still getting the 503 from projects.topics.setIamPolicy. I have checked that the payload for the request now has the correct role:

-> resp = client.connection.api_request(
(Pdb) pp wrapped
{'policy': {'bindings': [{'members': ['user:jgeewax@google.com'],
                          'role': 'roles/viewer'}],
            'etag': u'ACAB'}}

FWIW: A 503 is a terrible status code for "you gave me bad data"; that status is supposed to mean "my backend went away unexpectedly, try again later." A better status would be a 40x (probably just 400 "Bad Request").

[tmatsuo]

@tseaver

Indeed, the HTTP status code is terrible. I think the product team is working on it.

Maybe I found out the cause. I suspect the account jgeewax@google.com doesn't exist. Can you try using an existing account like tmatsuo@google.com?

[tseaver]

@tmatsuo Indeed, fixing the e-mail address makes the setIamPolicy calls succeed. Now I just need to figure out why the teardown code blows up for the subscription case.

[tseaver]

@dhermes the two new system tests now pass for me. PTAL

@tmatsuo thanks for the help!

[dhermes]

LGTM