Intermediate mTLS - How to set up with example in NodeJS?

How to create a server and clients in NodeJS connected by mTLS with certificates signed with Custom Intermediate Certificate Authorities (CA).

Piotr Gawędanodejsintermediate catlsmTlsserver

Intermediate mTLS - How to set up with example in NodeJS?

Sometimes it happens that we have to create multi cluster application where microservices connect each other. So for that we must verify that connection in encrypted and request from client is from our network.

Code and generated files you can find on GitHub repo: https://github.com/gawsoftpl/gawsoft.com/tree/main/blog/mtls-intermediate

Best option is encrypt traffic network by TLS and signed each certificates for our microservices by ROOT CA. But we have multiple clusters so for sign certificate we have to store ROOT CA private key on each cluster. This is a security issue. Preferred option will be introduction of an intermediary.

Following graph demonstrates the Intermediate Certificate Authorities (CA) hierarchy in example deployment:

Cluster1
Cluster2
Sign Intermediate Cert
Sign Intermediate Cert
Sign certifiate
Sign certifiate
Sign certifiate
Sign certifiate
Microservice1
Intermediate CA
Microservice2
Microservice1
Intermediate CA
Microservice2
ROOT CA

Above graph show us that we can store on each server independent Intermediate Certificate Authorities signed by ROOT CA. Main CA will be stored in safe offline storage. When one of Intermediate CA will be exposed, it will not affect on other Intermediate CA certificates.

Below the article you find script to generate ROOT CA. During this process we will generate two files:

  • root.ca.key.pem - Private key used only to sign intermediate CA
  • root.ca.cert.pem - Public certificate to verify intermediate CA and Microservices certificates.

In production private key of ROOT CA is stored offline for example in Hashicorp Vault. Private key of ROOT CA is used only for sign Intermediate CA certificate. ROOT CA public key is used in our network to verify that client certificate is issued by our ROOT CA private key.

Read also our previous blog post: https://gawsoft.com/blog/mtls-how-to-setup-server-and-client-in-nodejs.html with tutorial how to install mTLS in nodeJS.

Real life example in NodeJS

For demo please imagine you have 2 clusters:

  1. On cluster 1:
    • OrderService - This service is waiting to serve order data.
    • InvoiceService - Will send request to OrderService to collect info about order and create invoice.
  2. On cluster 2:
    • WarehouseService - Will send request to OrderService to collect info about order for prepare shipping.

Cluster1
Cluster2
Sign Intermediate Cert
Sign Intermediate Cert
Send request
Sign certifiate
Sign certifiate
Sign certifiate
Send request
InvoiceService
Intermediate CA2
OrderService
WarehouseService
Intermediate CA1
ROOT CA

All communication in our clusters are encrypted by TLS. OrderService have to validate request from WarehouseService and from InvoiceService. On the other hand WarehouseService and InvoiceService have to known that, OrderService is who he claims to be. For that we will use mTLS signed by our Intermediate CA. Let's start write our commands and code.

Requirements

  • Openssl
  • NodeJS

For our demo I use versions:

OpenSSL 1.1.1o  3 May 2022
NodeJS v16.17.0

1. Prepare directory

mkdir RootCa OrderService WarehouseService InvoiceService IntermediateCaCluster1 IntermediateCaCluster2

2. Generate ROOT CA

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout RootCa/root.ca.key.pem -out RootCa/root.ca.cert.pem

3. Generate Intermediate CA1

Create config file for Intermediate CA1

cat <<\EOF > IntermediateCaCluster1/openssl.conf
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
#subjectAltName=@san

[ req_dn ]
CN = Intermediate CA
L = cluster1
EOF

Create private and public keys:

# Generate private key for IntermediateCA1
openssl genrsa -out IntermediateCaCluster1/ca.key.pem 4096

# Generate request file for create IntermediateCa1 csr
openssl req -new -config IntermediateCaCluster1/openssl.conf -sha256 -new -key IntermediateCaCluster1/ca.key.pem -out IntermediateCaCluster1/certreq.csr

# Create new certificate from CSR and sign IntermediateCa1 cert by RootCa private key
openssl x509 -req -days 730 \
	-CA RootCa/root.ca.cert.pem -CAkey RootCa/root.ca.key.pem -CAcreateserial\
	-extensions req_ext -extfile IntermediateCaCluster1/openssl.conf \
	-in IntermediateCaCluster1/certreq.csr -out IntermediateCaCluster1/ca.cert.pem

And now we prepare final certificate ~~chain~~ file. What is certificate chain? This is simple. Certificate chain is file it contains.

  1. Public certificate of RootCA which we signed Intermediate Certificate
  2. Intermediate Certificate

The easiest way to explain it is using the example of a browser like Chrome. When you enter to https://ebay.com, Chrome receive the certificate chain issued by one of the trusted roots. Browser check in order from bottom to top that certificate are issued by parent. If Root CA is correct and belongs to trusted list, accept connection, otherwise show warning or drop connection.

Root Ca certificate
Intermediate Certificate
Client Certificate

Ok let's generate out certificate chain:

# Generate chains, the orders matters
cat IntermediateCaCluster1/ca.cert.pem RootCa/root.ca.cert.pem > IntermediateCaCluster1/cert-chain.pem

Verify that Intermediate certificate that are signed correct.

# Verify intermediate certificate with root ca certificate
openssl verify -CAfile RootCa/root.ca.cert.pem IntermediateCaCluster1/cert-chain.pem
# Output: IntermediateCaCluster1/cert-chain.pem: OK

Now our IntermediateCaCluster1 folder has below files:

# tree IntermediateCaCluster1
.
├── IntermediateCaCluster1
│   ├── ca.cert.pem -> Intermediate public certificate
│   ├── ca.key.pem -> Intermediate Private key
│   ├── cert-chain.pem -> Intermediate public certificate chains
│   ├── certreq.csr -> Intermediate CSR file, this file now can be delete
│   └── openssl.conf -> Openssl config file for create certificate

4. Generate Intermediate CA for second cluster

We will repeat commands that we did in previous step.

cat <<\EOF > IntermediateCaCluster2/openssl.conf
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
#subjectAltName=@san

[ req_dn ]
CN = Intermediate CA
L = cluster2
EOF

# Generate private key for IntermediateCA2
openssl genrsa -out IntermediateCaCluster2/ca.key.pem 4096

# Generate request file for create IntermediateCa2 csr
openssl req -new -config IntermediateCaCluster2/openssl.conf -sha256 -new -key IntermediateCaCluster2/ca.key.pem -out IntermediateCaCluster2/certreq.csr

# Create new certificate from CSR and sign IntermediateCA2 cert by RootCa private key
openssl x509 -req -days 730 \
	-CA RootCa/root.ca.cert.pem -CAkey RootCa/root.ca.key.pem -CAcreateserial\
	-extensions req_ext -extfile IntermediateCaCluster2/openssl.conf \
	-in IntermediateCaCluster2/certreq.csr -out IntermediateCaCluster2/ca.cert.pem
	
# Generate chains, the orders matters
cat IntermediateCaCluster2/ca.cert.pem RootCa/root.ca.cert.pem > IntermediateCaCluster2/cert-chain.pem

Output tree of directory:

├── IntermediateCaCluster1
│   ├── ca.cert.pem
│   ├── ca.key.pem
│   ├── cert-chain.pem
│   ├── certreq.csr
│   └── openssl.conf
├── IntermediateCaCluster2
│   ├── ca.cert.pem
│   ├── ca.key.pem
│   ├── cert-chain.pem
│   ├── certreq.csr
│   └── openssl.conf

5. Create our Services certificates

5.1 Create Private key and csr file for OrderService

For example this microservice will serve request as localhost. In production you will use your organization domain for example orders.abc.com. In this case please replace localhost with your endpoint name.

# Create Private key and CSR for OrderService
openssl req -out OrderService/cert-request.csr -newkey rsa:2048 -nodes -keyout OrderService/key.pem -subj "/CN=localhost/O=OrderService"

Because OrderService will be hosted in Cluster1 so for create and sign certificate for Order microservice we have to use Intermediate CA from Cluster 1.

openssl x509 -req -sha256 -days 365  -CA IntermediateCaCluster1/ca.cert.pem -CAkey IntermediateCaCluster1/ca.key.pem -extensions server_cert -set_serial 1 -in OrderService/cert-request.csr -out OrderService/cert.pem

Now folder looks like that:

├── OrderService
│   ├── cert.pem - Certificate for Order service signed by IntermediateCaCluster1
│   ├── cert-request.csr - Certificac request file, now can delete it
│   └── key.pem - Private key for order service

5.2 Create Private key and csr file for InvoiceService

openssl req -out InvoiceService/cert-request.csr -newkey rsa:2048 -nodes -keyout InvoiceService/key.pem -subj "/CN=InvoiceService/O=client organization"
openssl x509 -req -sha256 -days 365 -CA IntermediateCaCluster1/ca.cert.pem -CAkey IntermediateCaCluster1/ca.key.pem -set_serial 1 -in InvoiceService/cert-request.csr -out InvoiceService/cert.pem

5.3 Create Private key and csr file for WarehouseService

Because WarehouseService will be stored in Cluster2, we must sign our certificate by IntermediateCaCluster2 certificate.

openssl req -out WarehouseService/cert-request.csr -newkey rsa:2048 -nodes -keyout WarehouseService/key.pem -subj "/CN=WarehouseService/O=client organization"
openssl x509 -req -sha256 -days 365 -CA IntermediateCaCluster2/ca.cert.pem -CAkey IntermediateCaCluster2/ca.key.pem -set_serial 1 -in WarehouseService/cert-request.csr -out WarehouseService/cert.pem

5.4 Verify new certificates

# Verify intermediate certificate chain with root ca certificate
openssl verify -CAfile IntermediateCaCluster1/cert-chain.pem OrderService/cert.pem
openssl verify -CAfile IntermediateCaCluster1/cert-chain.pem InvoiceService/cert.pem
openssl verify -CAfile IntermediateCaCluster2/cert-chain.pem WarehouseService/cert.pem
# Output:
# OrderService/cert.pem: OK
# InvoiceService/cert.pem: OK
# WarehouseService/cert.pem: OK

Final structure of our example project looks like this:

# tree .
.
├── IntermediateCaCluster1
│   ├── ca.cert.pem
│   ├── ca.key.pem
│   ├── cert-chain.pem
│   ├── certreq.csr
│   └── openssl.conf
├── IntermediateCaCluster2
│   ├── ca.cert.pem
│   ├── ca.key.pem
│   ├── cert-chain.pem
│   ├── certreq.csr
│   └── openssl.conf
├── InvoiceService
│   ├── cert.pem
│   ├── cert-request.csr
│   └── key.pem
├── OrderService
│   ├── cert.pem
│   ├── cert-request.csr
│   └── key.pem
├── README.md
├── RootCa
│   ├── root.ca.cert.pem
│   ├── root.ca.cert.srl
│   └── root.ca.key.pem
└── WarehouseService
    ├── cert.pem
    ├── cert-request.csr
    └── key.pem

6. NodeJS code

Time for write example code and execute requests.

cat <<\EOF > OrderService/server.js
const https = require('https');
const fs = require('fs');
const path = require('path');

const hostname = '0.0.0.0';
const port = 3010;

const dirn = __dirname;

const options = {
    ca: fs.readFileSync(path.join(dirn, '../IntermediateCaCluster1/cert-chain.pem')),
    cert: fs.readFileSync(path.join(dirn, 'cert.pem')),
    key: fs.readFileSync(path.join(dirn, 'key.pem')),
    requestCert: true, // Client have to send own certificate, otherwise request will be dropped
    rejectUnauthorized: true, // If client send certficate not signed by certificates in chain, drop connection
};

const server = https.createServer(options, (req, res) => {
    res.statusCode = 200;
    res.setHeader('Content-Type', 'application/json');
    res.end(JSON.stringify({
        firtname: 'John',
        lastname: 'Smith',
        order: [
            {
                name: 'Sofa',
                items: 1,
                price: {
                    amount: 100,
                    currency: 'USD'
                }
            }
        ]
    }));
});

server.on('error',(e)=>console.log(e));

server.listen(port, hostname, () => {
    console.log(`Server running at http://${hostname}:${port}/`);
});
EOF

Most important is below options which I use in the code above.

const options = {
    ca: fs.readFileSync(path.join(dirn, '../IntermediateCaCluster1/cert-chain.pem')),
    cert: fs.readFileSync(path.join(dirn, 'cert.pem')),
    key: fs.readFileSync(path.join(dirn, 'key.pem')),
    requestCert: true, // Client have to send own certificate, otherwise request will be dropped
    rejectUnauthorized: true, // If client send certficate not signed by certificates in chain, drop connection
};

Start OrderService, and please dont't close that server Next commands run in new terminal.

node OrderService/server.js
# Output: Server running at http://0.0.0.0:3010/

Now OrderService is listing for request on port 3010

Create Code for InvoiceService

cat <<\EOF > InvoiceService/client.js
var fs = require('fs');
var https = require('https');
const path = require('path');

const dirn = __dirname;
var options = {
    hostname: 'localhost',
    port: 3010,
    path: '/',
    method: 'GET',
    key: fs.readFileSync(path.join(dirn,'key.pem')),
    cert: fs.readFileSync(path.join(dirn,'cert.pem')),
    ca: fs.readFileSync(path.join(dirn, '../IntermediateCaCluster1/cert-chain.pem'))
};

var req = https.request(options, function(res) {
    res.on('data', function(data) {
        process.stdout.write(data);
    });
});
req.end();
req.on('error', function(e) {
    console.error(e);
});

EOF

Run client request from InvoiceService:

node InvoiceService/client.js 
# Output: {"firtname":"John","lastname":"Smith","order":[{"name":"Sofa","items":1,"price":{"amount":100,"currency":"USD"}}]}%

Create Code for WarehouseService

cat <<\EOF > WarehouseService/client.js
var fs = require('fs');
var https = require('https');
const path = require('path');

const dirn = __dirname;
var options = {
    hostname: 'localhost',
    port: 3010,
    path: '/',
    method: 'GET',
    key: fs.readFileSync(path.join(dirn,'key.pem')),
    cert: fs.readFileSync(path.join(dirn,'cert.pem')),
    ca: fs.readFileSync(path.join(dirn, '../IntermediateCaCluster2/cert-chain.pem'))
};

var req = https.request(options, function(res) {
    res.on('data', function(data) {
        process.stdout.write(data);
    });
});
req.end();
req.on('error', function(e) {
    console.error(e);
});

EOF

Run client request from WarehouseService:

node WarehouseService/client.js 
# Output: {"firtname":"John","lastname":"Smith","order":[{"name":"Sofa","items":1,"price":{"amount":100,"currency":"USD"}}]}%

Great, now between all services connection is encrypted and OrderService authenticate WarehouseService and InvoiceService that both have valid certificate signed by authorize Intermediate CA.

7. Make some tests for incorrect CA

We can try test our authenticate and create fake IntermediateCA for validate that OrderService server check requests correct.

Let's try to generate new Intermediate CA(let's call him WrongIntermediateCa) and sign that WrongIntermediateCa a new certificate for WarehouseService. WrongIntermediateCa will not signed by RootCa so OrderService should drop connection from service with that certificate.

So this is our plan:

  1. Generate Fake Root CA(WrongRootCa)
  2. Generate new IntermediateCA(WrongIntermediateCa) and issue it with a Fake RootCA
  3. Issue certificate for WarehouseService with WrongIntermediateCA
  4. Send request to OrderService and check that return error
mkdir WrongRootCa WrongIntermediateCa

Generate WrongRootCA

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=XYC INC' -keyout WrongRootCa/root.ca.key.pem -out WrongRootCa/root.ca.cert.pem

Generate Wrong Intermediate CA

cat <<\EOF > WrongIntermediateCa/openssl.conf
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
#subjectAltName=@san

[ req_dn ]
CN = Wrong Intermediate CA
L = cluster3
EOF

# Generate private key for WrongIntermediateCa
openssl genrsa -out WrongIntermediateCa/ca.key.pem 4096

# Generate request file for create WrongIntermediateCa
openssl req -new -config WrongIntermediateCa/openssl.conf -sha256 -new -key WrongIntermediateCa/ca.key.pem -out WrongIntermediateCa/certreq.csr

# Create new certificate from CSR and sign WrongIntermediateCa cert by WrongRootCa private key
openssl x509 -req -days 730 \
	-CA WrongRootCa/root.ca.cert.pem -CAkey WrongRootCa/root.ca.key.pem -CAcreateserial\
	-extensions req_ext -extfile WrongIntermediateCa/openssl.conf \
	-in WrongIntermediateCa/certreq.csr -out WrongIntermediateCa/ca.cert.pem
	
# Generate chains, the orders matters
cat WrongIntermediateCa/ca.cert.pem WrongRootCa/root.ca.cert.pem > WrongIntermediateCa/cert-chain.pem
Generate new Certificate signed by WrongIntermediateCA
openssl x509 -req -sha256 -days 365 -CA WrongIntermediateCa/ca.cert.pem -CAkey WrongIntermediateCa/ca.key.pem -set_serial 1 -in WarehouseService/cert-request.csr -out WarehouseService/cert-wrong-ca.pem

Prepare new script to run request with wrong certificate

cat <<\EOF > WarehouseService/client-with-wrong-ca.js
var fs = require('fs');
var https = require('https');
const path = require('path');

const dirn = __dirname;
var options = {
    hostname: 'localhost',
    port: 3010,
    path: '/',
    method: 'GET',
    key: fs.readFileSync(path.join(dirn,'key.pem')),
    cert: fs.readFileSync(path.join(dirn,'cert-wrong-ca.pem')),
    ca: fs.readFileSync(path.join(dirn, '../WrongIntermediateCa/cert-chain.pem'))
};

var req = https.request(options, function(res) {
    res.on('data', function(data) {
        process.stdout.write(data);
    });
});
req.end();
req.on('error', function(e) {
    console.error(e);
});

EOF

node WarehouseService/client-with-wrong-ca.js
#Output: 
#Error: self signed certificate in certificate chain
#    at TLSSocket.onConnectSecure (node:_tls_wrap:1535:34)
#    at TLSSocket.emit (node:events:513:28)
#    at TLSSocket._finishInit (node:_tls_wrap:949:8)
#    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:730:12) {
#  code: 'SELF_SIGNED_CERT_IN_CHAIN'
#}

Great server OrderService drop connection because cert-wrong-ca.pem is not signed by correct Intermediate CA.

References

  1. GitHub repo with examples: https://github.com/gawsoftpl/gawsoft.com/tree/main/blog/mtls-intermediate
  2. https://www.thesslstore.com/blog/root-certificates-intermediate/
  3. https://www.youtube.com/watch?v=x_I6Qc35PuQ
How to enable load balancer in kind kubernetes cluster?

How to enable load balancer in kind kubernetes cluster?

Simple enable load balancer in your kind kubernetes in 30 seconds

Piotr GawędaPiotr Gawęda

Piotr Gawęda

mTLS - How to set up server and client in NodeJs with mTLS?

mTLS - How to set up server and client in NodeJs with mTLS?

How to encrypt the connection between two machines and authorize request from client on server? The article has a script to generate certificates and examples of servers in NodeJS.

Piotr GawędaPiotr Gawęda

Piotr Gawęda

Create monitoring cluster with Thanos, Loki and Istio.

Create monitoring cluster with Thanos, Loki and Istio.

How to create monitoring cluster with Thanos and Loki and send all metrics, logs from other clusters to one place. All encrypted and authenticate via mTLS by Istio Service Mesh.

Piotr GawędaPiotr Gawęda

Piotr Gawęda