I just spent a long time diagnosing an RSA public key exchange problem. Google was of very little help so hopefully this article will get picked up save someone else the trouble in the future.
The problem is this an RSA public key PEM or DER generated by Ruby’s OpenSSL::PKey::RSA are unreadable by OpenSSL, Bouncy Castle and probably other crypto tools. For example, if you try to load a public key PEM file generated by OpenSSL::PKey::RSA with the openssl command you get the following error
$ openssl rsa -text -pubin < my_pub_key.pem
unable to load Public Key
16879:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:647:Expecting: PUBLIC KEY
and if you try to load a DER generated by OpenSSL::PKey::RSA you get this error
$ openssl rsa -text -pubin -inform DER < my_pub_key.cer
unable to load Public Key
16880:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1294:
16880:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:380:Type=X509_ALGOR
16880:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:749:Field=algor, Type=X509_PUBKEY
The actual issue is that Ruby’s OpenSSL::PKey::RSA#to_pem and #to_der generate a PKCS#1 public key format while the openssl rsa command only works PKCS#8 formatted public keys. Both of the forms are grammars in the dreaded ASN.1 binary format which completely obscures the differences between them. Both formats encode the exact same information but the PCKS#8 style does it in a more complicated way.
I have not found a good solution to this problem. There does not seem to be any way to make OpenSSL::PKey::RSA#to_pem generate anything other than a PKCS#1 style key. I suspect that OpenSSL is able to handle PKCS#1 public keys but there does not seem to be any way to get the openssl command to do so. Similarly, it seems from Bouncy Castle’s API docs that you should be able to coerce it into accepting PKCS#1 public keys does not do so by default.
Fortunately, OpenSSL::PKey::RSA#new works just fine if handed a PKCS#8 PEM. That has allowed me to work around this issue by using the openssl command to generate the PKCS#8 style PEM files (both private and public using openssl genrsa and openssl rsa -pubout respectively) and then just reading/serving those files as needed. It is not great but it does work.