Signing/verifying and encrypting/decrypting using OpenSSL with libp11

OpenSSL can be used with pkcs11 engine provided by the libp11 library, and complemented by p11-kit that helps multiplexing between various tokens and PKCS#11 modules (for example, the system that the following was tested on supports: YubiHSM 2, YubiKey NEO, YubiKey 4, Generic PIV tokens and SoftHSM 2 software-emulated tokens).

Signing and Verifying

RSA-PKCS#1 v1.5

openssl dgst -engine pkcs11 -keyform engine -sign "pkcs11:token=YubiHSM;id=%04%01;type=private" -out t3200.pkcs1.sig -sha384 t3200.dat
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
openssl dgst -engine pkcs11 -keyform engine -verify "pkcs11:token=YubiHSM;id=%04%01;type=public" -signature t3200.pkcs1.sig -sha384 t3200.dat
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
Verified OK
$

RSA-PSS

~/openssl-1.1/bin/openssl dgst -engine pkcs11 -keyform engine -sign "pkcs11:token=YubiHSM;id=%04%01;type=private" -out t6400.txt.sigpss -sigopt rsa_padding_mode:pss -sha384 t6400.txt
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
~/openssl-1.1/bin/openssl dgst -engine pkcs11 -keyform engine -verify "pkcs11:token=YubiHSM;id=%04%01;type=public" -signature t6400.txt.sigpss -sigopt rsa_padding_mode:pss -sha384 t6400.txt
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
Verified OK
$

ECDSA

openssl dgst -engine pkcs11 -keyform engine -sign "pkcs11:token=YubiHSM;id=%02%03;type=private" -sha384 -out t3200.ecdsa.sig t3200.dat
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
openssl dgst -engine pkcs11 -keyform engine -verify "pkcs11:token=YubiHSM;id=%02%03;type=public" -sha384 -signature t3200.ecdsa.sig t3200.dat
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
Verified OK
$

Encrypting and Decrypting

RSA-PKCS

cat t64.txt
4aa58c448f3264c777be1b5ad94cf3e0a68911ed3f18db9e568ff2179e263f76
~/openssl-1.1/bin/openssl pkeyutl -engine pkcs11 -keyform engine -pubin -encrypt -inkey "pkcs11:token=YubiHSM;id=%04%02;type=public" -pkeyopt rsa_padding_mode:pkcs1 -in t64.txt -out t64.txt.pkcs1
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
~/openssl-1.1/bin/openssl pkeyutl -engine pkcs11 -keyform engine -decrypt -inkey "pkcs11:token=YubiHSM;id=%04%02;type=private" -pkeyopt rsa_padding_mode:pkcs1 -in t64.txt.pkcs1
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
4aa58c448f3264c777be1b5ad94cf3e0a68911ed3f18db9e568ff2179e263f76
$

RSA-OAEP

cat t64.txt
4aa58c448f3264c777be1b5ad94cf3e0a68911ed3f18db9e568ff2179e263f76
~/openssl-1.1/bin/openssl pkeyutl -engine pkcs11 -keyform engine -pubin -encrypt -inkey "pkcs11:token=YubiHSM;id=%04%02;type=public" -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha384 -pkeyopt rsa_mgf1_md:sha384 -in t64.txt -out t64.txt.oaep
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
~/openssl-1.1/bin/openssl pkeyutl -engine pkcs11 -keyform engine -decrypt -inkey "pkcs11:token=YubiHSM;id=%04%02;type=private" -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha384 -pkeyopt rsa_mgf1_md:sha384 -in t64.txt.oaep
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
4aa58c448f3264c777be1b5ad94cf3e0a68911ed3f18db9e568ff2179e263f76
$

ECDH

openssl pkeyutl -engine pkcs11 -keyform engine -derive -inkey "pkcs11:token=YubiHSM;id=%02%04;type=private" -peerkey peer_key.der
engine "pkcs11" set.
Enter PKCS#11 token PIN for YubiHSM:
34a03079c38947a679a924f3e20657cd4f69dd36df395b7e759e727524da87dc