I just spent a long time diagnosing an RSA public key exchange problem. Google was of very little help so hopefully this article will get picked up save someone else the trouble in the future.
The problem is this an RSA public key PEM or DER generated by Ruby’s
OpenSSL::PKey::RSA are unreadable by OpenSSL, Bouncy Castle and probably other crypto tools. For example, if you try to load a public key PEM file generated by
OpenSSL::PKey::RSA with the openssl command you get the following error
$ openssl rsa -text -pubin < my_pub_key.pem unable to load Public Key 16879:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:647:Expecting: PUBLIC KEY
and if you try to load a DER generated by
OpenSSL::PKey::RSA you get this error
$ openssl rsa -text -pubin -inform DER < my_pub_key.cer unable to load Public Key 16880:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1294: 16880:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:380:Type=X509_ALGOR 16880:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:749:Field=algor, Type=X509_PUBKEY
The actual issue is that Ruby’s
#to_der generate a PKCS#1 public key format while the
openssl rsa command only works PKCS#8 formatted public keys. Both of the forms are grammars in the dreaded ASN.1 binary format which completely obscures the differences between them. Both formats encode the exact same information but the PCKS#8 style does it in a more complicated way.
I have not found a good solution to this problem. There does not seem to be any way to make
OpenSSL::PKey::RSA#to_pem generate anything other than a PKCS#1 style key. I suspect that OpenSSL is able to handle PKCS#1 public keys but there does not seem to be any way to get the
openssl command to do so. Similarly, it seems from Bouncy Castle’s API docs that you should be able to coerce it into accepting PKCS#1 public keys does not do so by default.
OpenSSL::PKey::RSA#new works just fine if handed a PKCS#8 PEM. That has allowed me to work around this issue by using the openssl command to generate the PKCS#8 style PEM files (both private and public using
openssl genrsa and
openssl rsa -pubout respectively) and then just reading/serving those files as needed. It is not great but it does work.