HSM RSA OAEP Encryption with Asymmetric hashes fails

Question

I am using the PKCS11Interop library to Encrypt and Decrypt data with the parameters below.

CK_RSA_PKCS_OAEP_PARAMS p = new CK_RSA_PKCS_OAEP_PARAMS();
p.HashAlg = (uint)CKM.CKM_SHA256;
p.Mgf = (uint)CKG.CKG_MGF1_SHA1;
p.Source = (uint)CKZ.CKZ_DATA_SPECIFIED;
p.SourceData = IntPtr.Zero;
p.SourceDataLen = 0;

CK_MECHANISM mech = CkmUtils.CreateMechanism(CKM.CKM_RSA_PKCS_OAEP, p);

The error I get is CKR_MECHANISM_PARAM_INVALID when attempting to Encrypt/Decrypt. But when I use CKG_MGF1_SHA256 for MGF1 then both encryption and decryption works.

Am I missing something or mismatched hashes aren't supported by HSM boxes?

Answer 1

In the PKCS#1 standard, the random oracles (both hash functions) are identical.

If you want SHA-1, use (uint)CKM.CKM_SHA_1 as hash algorithm.