Swapping PyCrypto for pyOpenSSL. #1338

dhermes · 2016-01-02T00:26:10Z

This was done because PyCrypto does not install easily on Windows. pyOpenSSL is managed by PyCA (the Python crypto authority) and has a mature release process.

This change was influenced by discussions about #1009.

dhermes · 2016-01-02T00:28:34Z

To "confirm" this implementation matches our current one I did the following:

>>> import json
>>> from oauth2client.client import GoogleCredentials
>>> from OpenSSL import crypto
>>> from gcloud import credentials
>>> creds = GoogleCredentials.get_application_default()  # Env. var. -> path to JSON key
>>> pkey = crypto.load_privatekey(crypto.FILETYPE_PEM, creds._private_key_pkcs8_text)
>>> data = b'foo'
>>> new_signed_bytes = crypto.sign(pkey, data, 'SHA256')
>>> curr_signed_bytes = credentials._get_signature_bytes(creds, data)
>>> new_signed_bytes == curr_signed_bytes
True

dhermes · 2016-01-02T00:31:20Z

H/T to @tseaver for pointing out that cryptography was managed by the PyCA which led me to discover that pyOpenSSL was too.

gcloud/_testing.py

@@ -20,6 +20,8 @@ class _Monkey(object):

    def __init__(self, module, **kw):
        self.module = module
+        if len(kw) == 0:  # pragma: NO COVER
+            raise ValueError('_Monkey was used with nothing to monkey-patch')


This was done because PyCrypto does not install easily on Windows. pyOpenSSL is managed by PyCA (the Python crypto authority) and has a mature release process. This change was influenced by discussions about googleapis#1009.

dhermes · 2016-01-07T17:25:32Z

@jonparrott Is pyOpenSSL supported on App Engine?

theacodes · 2016-01-07T17:35:05Z

I don't think so; only pycrypto and the core ssl library.

dhermes · 2016-01-07T17:36:10Z

OK. PyCrypto has a pretty crappy install story (latest version isn't on PyPI, pip install fails on Windows).

It may be best to take the oauth2client approach and just implement using both pyOpenSSL and PyCrypto but require neither in setup.py.

theacodes · 2016-01-07T17:36:42Z

Sounds reasonable.

tseaver · 2016-01-07T20:32:36Z

I'd really hate to have our codebase cluttered with both the current PyCrypto-based implementation and the new pyOpenSSL-based one.

dhermes · 2016-01-07T20:37:32Z

The original goal: better Windows experience, is not really possible with PyCrypto since the maintainers don't have great windows support.
pyOpenSSL not working on Google App Engine is something we can avoid if we allow PyCrypto as a fallback since it is supported on GAE
I wouldn't exactly call it cluttered. We already have both implementations and they are both short (and only used for signing bytes). We've mentioned before we wanted to push the signing upsteam into oauth2client anyhow.

Maybe now is the time to bite the bullet and get both implementations into oauth2client?

Then the question becomes

Are we OK having a component that depends on one of pyOpenSSL and PyCrypto being installed if we don't include either one in our setup.py?

tseaver · 2016-01-07T22:42:43Z

Are we OK having a component that depends on one of pyOpenSSL and PyCrypto being installed if we don't include either one in our setup.py?

I really don't want to introduce imperative platform-based variations in the requirements (breaking wheel generation, for instance). Is there a PEP 508-supported way to detect GAE?

theacodes · 2016-01-07T22:44:03Z

Is there a PEP 508-supported way to detect GAE?

No.

dhermes · 2016-01-07T22:44:58Z

@jonparrott Wrote https://github.com/jonparrott/Darth-Vendor which probably makes him the foremost expert on packaging for GAE 😀

dhermes · 2016-01-08T19:41:51Z

@jonparrott After some digging I realized this can be done in pure Python using pyasn1, pyasn1-modules and rsa. Is there any vendor-ing issue for these libraries on GAE?

theacodes · 2016-01-08T19:42:58Z

@dhermes if they have any native components, then yes. Let me verify.

theacodes · 2016-01-08T19:44:13Z

@dhermes those should be fine. Go for it. 👍

dhermes · 2016-01-08T19:51:53Z

Good deal. Thanks for doing it for me (I know I could've RTFM instead of wasting your time).

theacodes · 2016-01-08T20:26:46Z

I know I could've RTFM instead of wasting your time

Considering how much of your time I've monopolized elsewhere, I'd say I owe you.

dhermes · 2016-01-08T21:25:36Z

As I was starting to do the rsa/pyasn1 implementation I realized GAE users are [covered] by using oauth2client.appengine.AppAssertionCredentials so we don't need to worry if they have support for either pyOpenSSL or PyCrypto.

@jonparrott What does darth vendor do if a package can't be installed / imported in GAE?

theacodes · 2016-01-08T21:34:17Z

@dhermes nothing. pip handles the installation, not the vendor tool. Pip will happily stage a binary package into the lib directory. :(

dhermes · 2016-01-08T21:38:04Z

OK. I suppose we could try/except ImportError on the pyOpenSSL import. Though it seems strange to do for something in setup.py.

tseaver · 2016-01-14T20:48:44Z

Would the try:..except: in setup.py be to make the dependency on PyCrypto be "soft" FBO the GAE environment?

dhermes · 2016-01-14T21:00:47Z

What did you have in mind?

i.e. Try and except what import? How would it help GAE?

tseaver · 2016-01-14T21:16:39Z

I'm responding to:

OK. I suppose we could try/except ImportError on the pyOpenSSL import. Though it seems strange to do for something in setup.py.

dhermes · 2016-01-14T21:17:22Z

I meant try/except in our code, not in setup.py.

theacodes · 2016-01-14T21:26:00Z

setup.py doesn't run on app engine, it runs on the user's machine when they vendor the package.

dhermes · 2016-01-14T21:30:23Z

@jonparrott Do you think it's fine to just leave pyopenssl as a dep since on GAE the user has the fancy http://oauth2client.readthedocs.org/en/stable/source/oauth2client.appengine.html module (now moved to contrib)

theacodes · 2016-01-14T21:33:30Z

I think you're in the clear as long as you don't ever try to use the library within GAE code.

dhermes · 2016-01-14T21:37:05Z

@tseaver GAE is "no longer" a blocker for using pyopenssl. Remaining issues?

tseaver · 2016-01-14T21:52:26Z

@dhermes

GAE is "no longer" a blocker for using pyopenssl. Remaining issues?

I thought GAE was the only reason to keep the pyopenssl codepath around at all. Am I mistaken?

dhermes · 2016-01-14T21:54:42Z

Nope its Windows. Check out the description of this PR.

dhermes · 2016-01-14T21:55:16Z

I just scared myself into thinking that it'd be a blocker for GAE but forgot that GAE and GCE had custom credentials types

tseaver · 2016-01-14T21:58:00Z

Oops, I misspoke: I meant I thought the old codepath (PyCrypto) was being kept around FBO GAE. If that isn't so, then lets just punt and make pyOpenSSL the only way we do this.

dhermes · 2016-01-14T21:59:49Z

Got it. All good to merge?

(PS I am planning on pushing this upstream to oauth2client at some point.)

tseaver · 2016-01-14T22:04:06Z

LGTM. I had lost track of the fact that you already dropped the old codepath.

Swapping PyCrypto for pyOpenSSL.

dhermes added packaging auth labels Jan 2, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jan 2, 2016

dhermes reviewed Jan 2, 2016
View reviewed changes

Swapping PyCrypto for pyOpenSSL.

948cccf

This was done because PyCrypto does not install easily on Windows. pyOpenSSL is managed by PyCA (the Python crypto authority) and has a mature release process. This change was influenced by discussions about googleapis#1009.

dhermes force-pushed the swap-crypto-libs branch from ae525ed to 948cccf Compare January 2, 2016 01:18

dhermes mentioned this pull request Jan 4, 2016

AppVeyor test commit. #1220

Closed

dhermes added a commit that referenced this pull request Jan 14, 2016

Merge pull request #1338 from dhermes/swap-crypto-libs

30acf33

Swapping PyCrypto for pyOpenSSL.

dhermes merged commit 30acf33 into googleapis:master Jan 14, 2016

dhermes deleted the swap-crypto-libs branch January 14, 2016 22:06

This was referenced Jan 21, 2016

Move bytes signing code upstream into oauth2client #1408

Closed

Upgrading version to 0.9.0. #1418

Merged

kvdb mentioned this pull request Feb 18, 2016

On GAE: "pyOpenSSL must be installed to load a private key" #1487

Closed

Swapping PyCrypto for pyOpenSSL. #1338

Swapping PyCrypto for pyOpenSSL. #1338

Conversation

dhermes commented Jan 2, 2016

dhermes commented Jan 2, 2016

dhermes commented Jan 2, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Jan 7, 2016

theacodes commented Jan 7, 2016

dhermes commented Jan 7, 2016

theacodes commented Jan 7, 2016

tseaver commented Jan 7, 2016

dhermes commented Jan 7, 2016

tseaver commented Jan 7, 2016

theacodes commented Jan 7, 2016

dhermes commented Jan 7, 2016

dhermes commented Jan 8, 2016

theacodes commented Jan 8, 2016

theacodes commented Jan 8, 2016

dhermes commented Jan 8, 2016

theacodes commented Jan 8, 2016

dhermes commented Jan 8, 2016

theacodes commented Jan 8, 2016

dhermes commented Jan 8, 2016

tseaver commented Jan 14, 2016

dhermes commented Jan 14, 2016

tseaver commented Jan 14, 2016

dhermes commented Jan 14, 2016

theacodes commented Jan 14, 2016

dhermes commented Jan 14, 2016

theacodes commented Jan 14, 2016

dhermes commented Jan 14, 2016

tseaver commented Jan 14, 2016

dhermes commented Jan 14, 2016

dhermes commented Jan 14, 2016

tseaver commented Jan 14, 2016

dhermes commented Jan 14, 2016

tseaver commented Jan 14, 2016