KMIP vs PKCS#11: the difference is whose code runs in your process

HashiCorp Vault can be a PKCS#11 token. You download libvault-pkcs11.so from releases.hashicorp.com, point your application at it, and C_Sign works. What the library does with that call is open a TLS connection to Vault and send a KMIP request. The PKCS#11 half of the arrangement ships only as a compiled shared library. The KMIP half is a protocol with an IANA port, 5696, and a binary encoding you can read off the OASIS site. Both halves need Vault Enterprise.

I keep coming back to that product because it is both sides of this argument in one download.

I read the KMIP 1.4 and 2.1 specifications against PKCS#11 3.2 last week for the error and library pages on this site, and then spent a day on the other half: what the vendors actually ship, what open source exists on each side, and what a project that supports both has to say.

What a PKCS#11 library is when it arrives

PKCS#11 is a C header. Everything that makes it useful with a network HSM is the shared object the HSM vendor hands you, and that object is where the vendor’s wire protocol lives.

The minimal Luna client Thales documents for containers is libCryptoki2.so, two more libraries, Chrystoki.conf, an openssl.cnf, a folder of certificates, and three tools, vtl, configurator and lunacm, for exchanging those certificates with the appliance and registering the client. The 10.8.0 release notes say the Linux installer “is compiled as .rpm packages. To install on a Debian-based distribution, such as Ubuntu, alien is used to convert the packages.” You need build-essential for that.

Entrust’s nShield library, libcknfast.so, comes inside the Security World software, which wants Java on the path before it installs, bundles a Tcl runtime for its tools, and routes every call through a hardserver daemon whose configuration is read at start-up. A PCIe card adds a kernel driver build.

AWS builds cloudhsm-pkcs11 separately for Amazon Linux 2 and 2023, RHEL 8, 9 and 10, Ubuntu 22.04, 24.04 and 26.04, and Windows, two architectures each, and serves the packages from an S3 bucket. SDK 5 dropped the client daemon, so the library itself now holds the TLS connection to the cluster. It claims PKCS#11 2.40, the 2015 version.

Three vendors, three private protocols, three installers, and in each case the code that speaks the protocol is loaded into your application’s address space with your application’s permissions. It sees every buffer you hand it and every buffer it hands back.

In 2023 Heiko Schäfer was integrating YubiHSM 2 with OpenPGP and read the source of yubihsm_pkcs11.so while doing it. C_GetAttributeValue for the subject, issuer and serial number of a certificate object declared an 8 KB buffer on the stack, never initialised it, never updated the length, and returned it. The caller got 8 KB of whatever the process had been doing. CVE-2023-39908, fixed in yubihsm-shell 2.4.1. What I take from it is that yubihsm-shell is open source and the bug was found by someone reading it during ordinary integration work. I do not know what the equivalent bug looks like in libCryptoki2.so, and outside Thales nobody can look.

Thales’s own compliance page says where the arrangement sits against FIPS 140: “Luna HSM Client software does not affect FIPS compliance; only the HSM firmware version.” The client is outside the validated boundary. The certificate covers the firmware at the other end of the wire. The closed binary in your process buys the vendor a wire protocol it never had to publish, and buys you nothing the certificate counts.

What KMIP is when it arrives

KMIP is TCP 5696, TLS with a client certificate, and TTLV, a tag-type-length-value encoding in which every standard tag begins with the byte 0x42. There is a client in whatever language you are in. PyKMIP is Apache 2.0 Python and released 0.11.0 on 9 September. Thales publishes its own Go client, kmip-go, on GitHub. NLnet Labs wrote a Rust one for Krill. libkmip is C11 and has not moved since November 2023, which says something about how much a C client needs to move.

Percona’s builds of MySQL and MongoDB ship a KMIP keyring in the open-source edition, tested, per the MySQL docs, against PyKMIP, Vault, CipherTrust Manager and Fortanix. Oracle’s MySQL Enterprise has the same thing as keyring_okv, Enterprise only. MongoDB’s own KMIP is Enterprise only. MariaDB has had the ticket open since May 2016. vSphere’s standard key provider is a KMIP 1.1 client and vCenter is the only thing in the cluster that talks to the server. NetApp ONTAP, tape libraries, storage arrays. The protocol has a lot of clients.

Where it is thin is servers. Thales, Entrust, Utimaco, Fortanix, Oracle, IBM and HashiCorp all sell one. Open source gets you PyKMIP’s server, which its own documentation calls “intended for testing and demonstration purposes only”, keeping keys in SQLite under /tmp, and Cosmian KMS, Rust, KMIP 1.0 through 2.1, with an HSM behind it if it is a Utimaco or a Proteccio, under a Business Source Licence that allows production on two vCPUs and converts to MPL four years after each release. If you want the server end of your system to be open as well, that is one project and a licence to read carefully.

A project that runs both

Krill, NLnet Labs’ RPKI certificate authority, signs with OpenSSL, PKCS#11 or KMIP, and its documentation is the most honest comparison I found. On PKCS#11: Krill “cannot control connection timeout, request throughput, or resource usage” because all of that is inside the vendor library, and the backend “may be local, remote, or clustered invisibly.” On KMIP: “arguably simpler to setup”, and Krill “manages the connection directly”, so it can see and configure it. The one-off keys for signed objects are still generated locally with OpenSSL “because it can be slow” against an HSM, on either path. They tested against CloudHSM, Kryptus, PyKMIP, SoftHSM, Utimaco and YubiHSM 2.

Where KMIP does not reach

If the thing that needs the key is OpenSSL, you get PKCS#11 or nothing. There is a pkcs11 provider for OpenSSL 3 and no KMIP one. The same holds for the JDK, NSS, GnuTLS, OpenSSH and GnuPG.

Most KMIP traffic is not cryptography either. vSphere, ONTAP and the database keyrings call Get: the key comes out of the server and the encryption happens locally. That is key delivery, which is the right design for a disk and the opposite of what an HSM is for. Encrypt, Sign and MAC have been in KMIP since 1.2 as optional profiles. Vault implements them. A KMIP server that does not is a key store you happen to reach over TLS, and replacing PKCS#11 with KMIP only means something if the server implements the cryptographic profiles and you never call Get on a private key.

Every signature over KMIP is a TLS round trip. I have not measured that against a local PKCS#11 call, and Krill’s note above suggests both are slow enough that you design around them.

The seam

The two standards have started reaching for each other. KMIP 2.1 added an operation called PKCS#11 that tunnels a Cryptoki call to the server and returns a CK_RV. The open-source PKCS#11 world built its own RPC, p11-kit server, over a Unix socket you forward with ssh, because the standard never gave it one. opencryptoki ships p11kmip to move an AES key from a token to a KMIP server and back. KMIP 3.0 went through public review in July and August with ML-KEM, ML-DSA and Encapsulate and Decapsulate in it, and is not a standard yet.

What I want from a network HSM is the wire, because the wire is what I can audit, replay and test without the vendor in the room. KMIP gives me one. The vendor’s PKCS#11 library gives me one I cannot read, loaded into my process, outside the FIPS boundary. When the operations I need fit in KMIP’s cryptographic profiles I will take the protocol. When OpenSSL or Java has to hold the key I will take the blob, and give it a process of its own over p11-kit’s socket, so that the closed code and mine at least stop sharing memory.