Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate novel datastore system test failures #1089

Closed
dhermes opened this issue Aug 25, 2015 · 4 comments
Closed

Investigate novel datastore system test failures #1089

dhermes opened this issue Aug 25, 2015 · 4 comments
Assignees
@dhermes
Labels
api: datastore Issues related to the Datastore API. testing

Comments

@dhermes
Copy link
Contributor

dhermes commented Aug 25, 2015

  1. 502 on RPC

    ======================================================================
ERROR: test_allocate_ids (datastore.TestDatastoreAllocateIDs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/datastore.py", line 47, in test_allocate_ids
    allocated_keys = CLIENT.allocate_ids(CLIENT.key('Kind'), num_ids)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/datastore/client.py", line 418, in allocate_ids
    incomplete_key_pbs)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 372, in allocate_ids
    datastore_pb.AllocateIdsResponse)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 115, in _rpc
    data=request_pb.SerializeToString())
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 92, in _request
    raise make_exception(headers, content, use_json=False)
gcloud.exceptions.GCloudError: 502 <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/errors/logo_sm_2.png) no-repeat}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/errors/logo_sm_2_hr.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:55px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>

    From

    1. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/77234688
    2. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/100902340

  2. Query results eventually consistent (this shouldn't occur because we use entity groups)

From: https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/77772358
3. 503 Backend Error

======================================================================
ERROR: test_ancestor_query (datastore.TestDatastoreQuery)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/datastore.py", line 40, in tearDown
    CLIENT.delete_multi(keys)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/datastore/batch.py", line 228, in __exit__
    self.commit()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/datastore/transaction.py", line 170, in commit
    super(Transaction, self).commit()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/datastore/batch.py", line 203, in commit
    self.dataset_id, self.mutation, self._id)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/datastore/connection.py", line 329, in commit
    datastore_pb.CommitResponse)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/datastore/connection.py", line 113, in _rpc
    data=request_pb.SerializeToString())
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/datastore/connection.py", line 90, in _request
    raise make_exception(headers, content, use_json=False)
gcloud.exceptions.ServiceUnavailable: 503 Backend Error

From: https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/78109143
4. 500 Internal Error

ERROR: test_query_group_by (datastore.TestDatastoreQuery)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/datastore.py", line 322, in test_query_group_by
    entities = list(query.fetch(limit=expected_matches + 1))
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/datastore/query.py", line 444, in __iter__
    self.next_page()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/datastore/query.py", line 408, in next_page
    transaction_id=transaction and transaction.id,
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 266, in run_query
    _datastore_pb2.RunQueryResponse)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 117, in _rpc
    data=request_pb.SerializeToString())
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 94, in _request
    raise make_exception(headers, content, use_json=False)
gcloud.exceptions.InternalServerError: 500 Internal Error

From: https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/98468887

This could be solved by reviving #535
@dhermes dhermes added api: datastore Issues related to the Datastore API. testing labels Aug 25, 2015
@dhermes dhermes self-assigned this Aug 25, 2015
@dhermes dhermes changed the title Investigate novel test failure (missing ACK ID) Investigate novel test failure (502 on allocate IDs) Aug 25, 2015
@dhermes dhermes changed the title Investigate novel test failure (502 on allocate IDs) Investigate novel datastore system test failure (502 on allocate IDs) Aug 27, 2015
@dhermes dhermes changed the title Investigate novel datastore system test failure (502 on allocate IDs) Investigate novel datastore system test failures Aug 27, 2015
@dhermes
Copy link
Contributor Author

dhermes commented Aug 31, 2015

@jgeewax I just added a 503 error (just happened) to accompany our 502 error from last week.

@pcostell
Copy link
Contributor

pcostell commented Nov 8, 2015

In 2. the only failure I see is test_empty_kind. Note that it is possible that your delete from another test fails, which would then result in this test failing. Perhaps it makes more sense to randomly choose a kind here and assert that the result is empty?

For 1. and 3., the next time it happens can you grab the logs from the Datastore service task (the version ah-builtin-datastoreservice in your test project)? Note that you may need to view this using the old App Engine Admin Console and not the Developers Console. Or just shoot me an email and I can look into it.

@dhermes
Copy link
Contributor Author

dhermes commented Nov 8, 2015

Will do! Thanks for taking a look.

ISTM that the failed delete should also fail the previous test, but we see all successes except for test_empty_kind.

From the mismatched output:

- [{'author': 'Silvano',
-   'isDraft': False,
-   'publishedAt': datetime.datetime(2001, 1, 1, 0, 0, tzinfo=<UTC>),
-   'rating': 4.5,
-   'tags': ['pasta', 'homemade'],
-   'title': 'How to make the perfect homemade pasta',
-   'wordCount': 450},
-  {'author': 'Silvano',
-   'isDraft': False,
-   'publishedAt': datetime.datetime(2001, 1, 1, 0, 0, tzinfo=<UTC>),
-   'rating': 5.0,
-   'tags': ['pizza', 'grill'],
-   'title': 'How to make the perfect pizza in your grill',
-   'wordCount': 400}]

the failed delete would have been after test_save_multiple.

Am I missing something here?

@tseaver tseaver mentioned this issue Mar 16, 2016
Closed
@dhermes dhermes mentioned this issue May 17, 2016
Closed
@dhermes dhermes added the flaky label Aug 11, 2016
@dhermes
Copy link
Contributor Author

dhermes commented Aug 11, 2016

@tseaver Closing this until we see more in the wild

@dhermes dhermes closed this as completed Aug 11, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dhermes dhermes

Labels
api: datastore Issues related to the Datastore API. testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dhermes @pcostell