Per Vices Router Page

This page aims to provide a comprehensive guide for how we installed OpenWRT and the configuration required to have it load balance across multiple redundant Wide Area Networks (WANs) and enable secure, smart card based, VPN access.

Overview

Our goal for this project was to implement a secure and inexpensive router with the following capabilities:

1. Support for multiple redundant upstream WANS
   • The work we do requires us to have internet access. To help mitigate against outages, we have a couple of different ISPs that provide us with Cable, DSL, and fiber. Ideally, we would like to make use of the aggregate bandwidth, while monitoring the status of each connection and failing over appropriately.
2. Secure, Smart-Card Based, VPN traffic
   • We want our employees to be able to securely access internal resources without compromising our security using two factor authentication.
3. 10Gbps LAN network interfaces, 1Gbps WAN
   • Our internal LAN uses 10GBASE-T to provide a fast connection across machines and facilitate the rapid transfer of (very) large files. We build high performance Software Defined Radios, and debugging them sometimes requires us to share large (multi-gigabyte) files of raw IQ samples. Moreover, we would like to ensure that we can fully use the bandwidth provided by our various ISPs, which may exceed 1Gbps. This requires that the aggregate bandwidth between the LAN and Router is greater than the aggregate bandwidth provided by our ISPs.

We spent some time looking at various possible commercial routers, but we didn’t find any reasonably priced products that met all this criteria. Those products we did find generally cost over 6k. After looking around, we decided to try and do it ourselves.

Though this ended up taking us a week to figure out, we’re pretty pleased with the end result. The following aims to document our work and provide others with a starting point to figuring everything out.

Hardware and Firmware

Materials

The following is a list of the materials we ended up using, followed by some notes.

Item	Model	Manufacturer	Price (USD)
Router	LS1088ARDB	NXP	1850
Programmer	CWH-CTP-BASE-HE (CodeWarrior TAP)	NXP	450
SmartCard	Nitrokey HSM 2	Nitrokey	~80

Router

We looked through a number of the routers supported by OpenWRT, but weren’t immediately able to find a reasonably inexpensive router that met all our requirements - especially when it came to the number of 10G ports, and other features. It occurred to us that rather than look for a commercial router, it might be a good idea to look who makes high performance networking ICs, and then purchase an eval board. These boards generally expose all the features supported by the silicon, and include all the files required to easily flash the device.

This was important, because we wanted to ensure that we could maintain this in the future.

We ended up going with the NXP LS1088ARDB. The eval board comes with two 10G ports (which we decided to interface with our LAN), along with (8) 1G ports (which we decided to use for our WANs). It also has internal PCIe 3.0 slots, DDR4 RAM, and a couple of USB3.0 ports, multiple cores, and hardware crypto processing support - a fairly compelling combination of features for the price. This was a wired only configuration, so we didn’t worry about wirelss connectivity. If that’s your thing, you can try the LS2088?

Programmer

We also purchased the code warrior programmer. This device provides access to the serial port, which is very useful when it comes to programming and configuring the device. It’s possible to avoid using it, but the programmer enables us to see how things loads and speeds things up dramatically.

Smart Card

For the VPN, we were looking to implement two factor authentication. We looked around and decided to go with the Nitrokey HSM 2.

Hardware Configuration

This section includes all the developer configuration you’ll need to successfully access the device. The most substantial piece here is ensuring that you properly configure the TAP Controller.

Configuring TAP Controller

1. Connect TAP controller to UART1 of LS1088ARDB
2. Minicom into the TAP controller serial device. On our machines, we use: sudo minicom -D /dev/ttyACM0
3. Configure the network settings: This can be done using the netparam command

4. Make a note of the bootconfig to find the IP Address of the controller. i.e. bootconfig = dhcp:FSL03AE56 (192.168.128.153) Alternatively, set the IP address manually by using the following commands:

   netparam static_ip_address <ip-address>
   netparam bootconfig static
   reset
   

5. Next, configure the TTY parameters with: tgtty data8 stop1 noparity nortscts noxon echo

6. You can now telnet into the TAP controller: telnet 192.168.128.153 1082

Note that the target serial port number of the CodeWarrior TAP probe is 1082

If controller is still not connecting, try restarting it and consulting the manual which can be found here.

Configuring LS1088ARDB

Open the router and check the following switches to make sure it boots off QSPI 0.

Changing the Default Boot Drive

Boot Source	SW1[1:8]	SW2[1:8]	SW3[1:8]	SW4[1:8]	SW5[1:8]
QSPI NOR flash0	0011 0001	X100 0000	1110 0010	1001 0011	0111 0000
QSPI NOR flash1	0011 0001	X100 0001	1110 0010	1001 0011	0111 0000
SD card	0010 0000	0100 0000	1110 0010	1001 0011	0111 0000

LSDK Flash Layout

Firmware Definition	Max Size	Flash Offset	SD Start Block Number
RCW+PBI+BL2	1 MiB	0x00000000	0x00008
TF-A FIP image (BL31 + TEE (BL32) + U-Boot/UEFI (Bl33))	4 MiB	0x00100000	0x00800
Boot firmware environment	1 MiB	0x00500000	0x02800
Secure boot headers	2 MiB	0x00600000	0x03000
Secure header or DDR PHY FW	512 KiB	0x00800000	0x04000
Fuse provisioning header	512 KiB	0x00880000	0x04400
DPAA1 FMAN ucode	256 KiB	0X00900000	0x04800
QE firmware or DP firmware	256 KiB	0X00940000	0x04A00
Ethernet PHY firmware	256 KiB	0X00980000	0x04C00
Script for flashing image	256 KiB	0x009C0000	0x04E00
DPAA2 MC	3 MiB	0x00A00000	0x05000
DPAA2 DPL	1 MiB	0x00D00000	0x06800
DPAA2 DPC	1 MiB	0x00E00000	0x07000
Device tree	1 MiB	0x00F00000	0x07800
Kernel	16 MiB	0x01000000	0x08000
Ramdisk rfs	32 MiB	0x02000000	0x10000

Installing OpenWRT

There are a couple of ways to install OpenWRT: You may use the default images, or you may compile a new image yourself. Using the default image is best if you’re sure you can download and install all the additional images you need through opkg. Compiling your own image is required if you need to build or include custom modules.

Obtaining Firmware Image

Default Image

This option is only useful if you are sure you can download and install all the modules you need from opkg in openWRT.

• Where to find it
• Go to stable release builds and find the version needed.

Compiling Image

OpenWRT Build Dependencies

# Archlinux

pacman -S --needed bash bc bin86 binutils bzip2 cdrkit \
                  core/which diffutils fastjar findutils \
                  flex gawk gcc gettext git intltool libusb \
                  libxslt make ncurses openssl patch perl-extutils-makemaker \
                  pkgconf python3 rsync sharutils time unzip util-linux \
                  wget zlib

# Ubuntu

sudo apt-get install subversion g++ zlib1g-dev build-essential \
                     git python python3 python3-distutils libncurses5-dev \
                     gawk gettext unzip file libssl-dev wget libelf-dev ecj \
                     fastjar java-propose-classpath

• Targets for other distributions can be found here.

Download OpenWRT Repo

To install the build system run the following commands:

git clone https://github.com/openwrt/openwrt.git
git fetch --tags
git tag -l
git checkout v19.07.2 (Select your current version)
make prereq

Now update your packages using the following commands:

./scripts/feeds update -a
./scripts/feeds install -a

Configure the OpenWRT Kernel

Now to choose the packages we will use. Run this command and choose the right targets, settings, etc.

make menuconfig

Choose: Target System (NXP Layerscape) —> Subtarget (ARMv8 64-bit based boards) —> Profile (NXP LS1088A-RDB Default) —>

The list of packages and where to find them is as follows:

1. conntrack
2. conntrackd
   • Under Network -> Firewall
3. kmod-cryptodev
   • Under Kernel Modules -> Cryptographic API modules
4. luci
   • Under LuCI -> Collections
5. luci-app-mwan3
6. luci-app-openvpn
7. luci-app-sqm
8. luci-app-statistics
   • Under LuCI -> Applications
9. mailsend
   • Under Mail -> mailsend
10. mwan3
    • Under Network -> Routing and Redirection
11. nmap
    • Under Network -> NMAP Suite
12. openvpn-easy-rsa
13. openvpn-openssl
    • Under Network -> VPN

Once you are done, save config file so you never have to do this again. Then make the image:

## X is the number of concurrent jobs. Setting this to the number of cores on the machine makes this part faster
## eg. you can use 'make -j7' if the machine has 8 cores
make -jX

Your image will be in the directory : openwrt/bin/targets/layerscape/armv8_64b/

Flashing Firmware Image

There are two methods of getting images onto the router, the first is using a tftp server and the second is using an sd card. Both methods will be outlined below.

Programing qSPI from uboot over tFTP

First we will need to start the tftp server on our host machine

systemctl start tftp

Dump all files you wish to send in the tftp directory which is located at /srv/tftp

Configure Networking from UBOOT

Enter UBOOT and type the following commands:

INIT SCRIPTS TO START MC (We are using cross, who presently has an IP address of: 192.168.128.202)

Set ipaddr to a unique value that can communicate with the tftp server

Plug into ETH0 of LS1088ARDB, which is DPMAC2@xgmii

setenv ethprime DPMAC2@xgmii
setenv ethact DPMAC2@xgmii
setenv ethaddr d6:69:f9:e6:33:ad
setenv ipaddr 192.168.128.240
setenv serverip 192.168.128.202
ping 192.168.128.202

If you don’t run ping, it won’t work.

Flashing through uboot

This can be dangerous. There are two steps here:

1. Copy the binary over the network to device ram using tftp.
2. Copy the binary from device ram to NAND/qSPI

LS1088ARDB uses QSPI 0 as default, so if you upload a faulty image to QSPI 0 it will no longer boot.

As a result, make sure to test your image on QSPI 1 before putting it on QSPI 0

Copy to flash1:

sf probe 0:1
tftp 0xa0000000 filename
print filesize
sf erase 0x0 +$filesize && sf write 0xa0000000 0x0 $filesize

Reboot: Use one of the following commands: qixis_reset and qixis_reset altbank are relative. i.e if you boot from QSPI 0 and use qixis_reset altbank you will boot from QSPI 1 and vice versa

qixis_reset
qixis_reset altbank

Programing qSPI from uboot using SDCard

For this to work, you need to be able to get into U-boot. Load an sd_boot image onto an sd card. This can be found here. Once you have it, copy the desired firmware image onto the sd card

Load the sd_boot image first, and then copy the desired firmware image onto it after, check that the partition type is Linux, LS1088ARDB will not recognize the sd card otherwise

Insert sd card into ls1088ardb and enter u-boot

Find sd card:

mmc rescan
mmc info

To see the contents of the sd card, device will usually be zero but double check with mmc info and choose the partition where the files are loaded

ls mmc device:partition

For example if the device is 0 and our files are on partition 1,

ls mmc 0:1

Once you have found the file you want to load, run the following commands TEST YOUR IMAGE ON QSPI 1 FIRST, IT WILL NOT BOOT IF YOU PUT A FAULTY IMAGE ON QSPI 0

Make sure to test your image on QSPI 1 before putting it on QSPI 0

There’s really no excuse for wasting your time reflashing everything like this… Follow this simple, non-negotiable, and straight forward rule:

Make sure to test your image on QSPI 1 before putting it on QSPI 0

    sf probe 0:1
load mmc device:partition 0xa0000000 filename
print filesize
sf erase 0x0 +$filesize && sf write 0xa0000000 0x0 $filesize
qixis_reset altbank

Software Configuration

This section concerns itself with the software configuration. It assumes that you’ve successfully installed OpenWRT on to the hardware, and should be more broadly applicable.

Interface Configuration

Initial Interface Configuration

Connect the ethernet cable to ETH8 on LS1088ARDB, this is the only one that is specified by default. Set the ip address of br-lan to a free address in your subnet. The ip address is 192.168.1.1 by default. Connect to LUCI by typing the ipaddr set to LS1088ARDB in a browser. Login, and change the following settings.

1. Change password
2. System > Startup > Disable dnsmasq and Disable ohdhcp

3. Network > Interfaces > Edit and set the following values:

   iPv4 : (ipaddr previously specified above)

Save and Apply these changes

Then navigate to System>Startup>Local, and copy these magic lines:

echo dpni.0  > /sys/bus/fsl-mc/drivers/fsl_dpaa2_eth/unbind && restool dpni destroy dpni.0
echo dpcon.1 > /sys/bus/fsl-mc/devices/dpcon.1/driver/unbind && restool dpcon destroy dpcon.1
echo dpbp.1  > /sys/bus/fsl-mc/devices/dpbp.1/driver/unbind && restool dpbp destroy dpbp.1

ls-addni --num-queues=8 --fs-entries=58 dpmac.2
ls-addni --num-queues=8 --fs-entries=58 dpmac.1
ls-addni --num-queues=8 --fs-entries=58 dpmac.7
ls-addni --num-queues=8 --fs-entries=58 dpmac.8
ls-addni --num-queues=8 --fs-entries=58 dpmac.9
ls-addni --num-queues=8 --fs-entries=58 dpmac.10
ls-addni --num-queues=8 --fs-entries=58 dpmac.3
ls-addni --num-queues=8 --fs-entries=58 dpmac.4
ls-addni --num-queues=8 --fs-entries=58 dpmac.5
ls-addni --num-queues=8 --fs-entries=58 dpmac.6

Restart and now br-lan is connected on eth0 so connect the ethernet accordingly.

The magic lines above set all the ports to their corresponding number, but if you are a masochist here are the steps to do it manually

Creating New Network Interfaces

1. Check network interfaces using the following command

   ls-listni
   

For example, we see one interface dprc.1/dpni.0

1. Unbind and destroy existing interfaces

   echo dpni.0 > /sys/bus/fsl-mc/drivers/fsl_dpaa2_eth/unbind
   restool dpni destroy dpni.0
   

2. Add new network interfaces using the following commands, do them in order since they are named in order of creation ls-addni dpmac.X

3. To list all the DPMAC objects present in a system use the following command: ls-listmac

Configuring OpenWRT modules

Here, we configure the OpenWRT modules for the firewall, MWAN3 (which supports load sharing), OpenVPN (for the VPN), and SQM (to minimize buffer bloat).

Configuring OpenWRT for MWAN3

Configuring LAN’s

As we have our own DHCP server, we’ll want to disable dhcp and ipv6 features. To do so:

1. Edit the LAN interface.
2. Under advanced tab, uncheck “Use Built-in ipv6 management”.
3. Under the DHCP interface tab, click on the general tab, and , check “Ignore Interface”
4. Under the DHCP interface tab, click on advanced, and disable:
   • Router advertisement service
   • DHCPv6 service
   • NDP proxy

Configuring WAN’s

Navigate to Network>Interfaces Add a new interface as a dhcp client and name it according to the port it will represent

Change the following settings. in Advanced:

1. Disable iPv6 management
2. Disable Peer DNS
3. Add the following DNS servers: 192.168.128.2 and 1.1.1.1
4. Set metric to 10,20,30, … Make sure the metric for each Wan is unique in Firewall,
5. Add the Wan to the Wan zone

Connect the new WAN to your modem and check connectivity by pinging google and local machines.

Configuring MWAN3

Navigate to System>Software

1. update package list
2. search for ’ mwan3 ‘
3. Install packages:
   • mwan3
   • luci app-mwan3

Restart LS1088ARDB and check connectivity

Setting up Interfaces

1. Navigate to Network>Load Balancing
2. Go to the interfaces tab and delete all the pre-existing interfaces
3. Create interfaces with names corresponding to the wan networks
4. Change the following settings:
   • Enable the interface at the very top
   • Tracking hostname : 1.1.1.1

Setting up Members

1. Go to the members tab and delete all the pre-existing members
2. Create members with names corresponding to the interfaces
3. Change the following settings:
   • metric = 1
   • weight = 1

Setting up Policies

1. Go to the policies tab and delete all the policies other than balanced
2. Edit the balanced policy, get rid off the default members and add in the members you just created.
3. Leave the last resort to reject

Restart LS1088ARDB and check connectivity

Check status, all connected WAN’s should show up, you can also check your routing table and see that each WAN has its own default route.

To monitor your WANs a useful package :

• luci app-statistics

Setting up Notification Script Go to the notifications tab and paste the following script,

##!/bin/sh

####
## Debugging
####

echo -e "\n Action:\t $ACTION \n Interface:\t $INTERFACE \n Device:\t $DEVICE \n" > /dev/console

### Ensure non-zero string values.
if [ -z "$INTERFACE" ]; then
    echo -e "Interface must be nonzero. Exiting program.\n"
    exit
fi

if [ -z "$DEVICE" ]; then
    echo -e "Device must be nonzero. Exiting program.\n"
    exit
fi

### Actions if ifup
if [ "$ACTION" = "ifup" ]; then
    echo -e "ifup triggered by $INTERFACE on $DEVICE \n" > /dev/console
fi

### Actions if ifdown
if [ "$ACTION" = "ifdown" ]; then
    echo -e "ifdown triggered by $INTERFACE on $DEVICE \n" > /dev/console
fi

### Actions if connected
if [ "$ACTION" = "connected" ]; then
    echo -e "connected triggered by $INTERFACE on $DEVICE \n" > /dev/console
fi

### Actions if disconnected
if [ "$ACTION" = "disconnected" ]; then
    echo -e "disconnected triggered by $INTERFACE on $DEVICE \n" > /dev/console
fi

Configuring the Firewall

Go to firewall, custom rules and paste this:

##!/bin/sh
iptables -N Always
iptables -N Allow
iptables -N Bogus
iptables -N Enemies
iptables -N Friends

iptables -A input_rule -j Bogus
iptables -A input_rule -j Always
iptables -A input_rule -j Enemies
iptables -A input_rule -j Allow

iptables -A forwarding_rule -j Bogus
iptables -A forwarding_rule -j Always
iptables -A forwarding_rule -j Enemies
iptables -A forwarding_rule -j Allow

iptables -A Bogus -p tcp -m tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
iptables -A Bogus -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP
iptables -A Bogus -s 169.254.0.0/16 -j DROP
iptables -A Bogus -s 172.16.0.0/12 -j DROP
iptables -A Bogus -s 192.0.2.0/24 -j DROP
iptables -A Bogus -s 127.0.0.0/8 -i ! lo -j DROP

iptables -A Always -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A Always -i lo -j ACCEPT

iptables -A Enemies  -m recent --name psc --update --seconds 60 -j DROP
iptables -A Enemies -i ! lo -m tcp -p tcp --dport 1433  -m recent --name psc --set -j DROP
iptables -A Enemies -i ! lo -m tcp -p tcp --dport 3306  -m recent --name psc --set -j DROP
iptables -A Enemies -i ! lo -m tcp -p tcp --dport 8086  -m recent --name psc --set -j DROP
iptables -A Enemies -i ! lo -m tcp -p tcp --dport 10000 -m recent --name psc --set -j DROP
iptables -A Enemies -s 99.99.99.99 -j DROP

iptables -A Friends -s 10.8.0.10 -j ACCEPT
iptables -A Friends -s 192.168.128.242 -j ACCEPT
iptables -A Friends -s 192.168.128.153 -j ACCEPT
iptables -A Friends -j DROP

iptables -A Allow -p icmp --icmp-type echo-request -j Friends
iptables -A Allow -p icmp --icmp-type any -m limit --limit 1/second -j ACCEPT
iptables -A Allow -p icmp --icmp-type any -j DROP
iptables -A Allow -p tcp -m state --state NEW -m tcp --dport 22 -j Friends
iptables -A Allow -p tcp -m state --state NEW -m tcp --dport 25 -j ACCEPT
iptables -A Allow -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
iptables -A Allow -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT

if a port should always be open and accept all packets use

iptables -A Always -p udp --dport 123 -j ACCEPT

ban list of ip’s from a file

working on it: https://www.cyberciti.biz/faq/iptables-read-and-block-ips-subnets-from-text-file/

Make sure to test firewall before exposing your network to the world.

OpenVPN

The following OpenVPN configuration is specifically designed to support access through Nitrokeys.

Everyone gets a copy of the the self signed certificate from the certifying authority (CA-cert.pem). The server gets its own private key (server-key.pem), CA signed certificate (server-cert.pem), and dh.pem file. Each client gets their own CA signed certificate and private key (ex. client-key.pem & client-cert.pem) .

OpenWRT OpenVPN server configuration

Set up server config file on the router found at /etc/config/openvpn : Make sure to put the correct file path for your keys

config openvpn 'PVLAN'
    option dev 'tun'
    option comp_lzo 'yes'
    option keepalive '10 60'
    option verb '3'
    option dh '/pki/dh.pem'
    option enabled '1'
    option log 'log/log.log'
    option status '/tmp/log/pvlan.status 5'
    option server 'XXX.XXX.XXX.XXX 255.255.255.0'
    option ifconfig_pool_persist '/etc/openvpn/ipp.txt 600'
    option cipher 'AES-256-CBC'
    option persist_key '1'
    option persist_tun '1'
    option port '1194'
    option cert '/pki/server-cert.pem'
    option key '/pki/server-key.pem'
    option ca '/pki/CA-cert.pem'
    list push 'redirect-gateway autolocal def1'
    list push 'dhcp-option DNS XXX.XXX.XXX.XXX'
    option proto 'udp'

OpenWRT OpenVPN client configuration

Set up client config file on the client machine with this config: Make sure to put the correct file path for your keys

client
dev tun
proto udp
resolv-retry infinite
remote XXX.XXX.XXX.XXX
port 1194
float
auth-nocache
comp-lzo
persist-key
ca /openvpn/CA-cert.pem
cipher AES-256-CBC
verb 3
log /openvpn/vpn.log
status /openvpn/vpn.status 5
pkcs11-id 'www\x2ECardContact\x2Ede/PKCS\x2315\x20emulated/DENKXXXXXXX/SmartCard\x2DHSM\x20\x28UserPIN\x29/11'
pkcs11-providers '/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so'
dhcp-option DNS XXX.XXX.XXX.XXX
dhcp-option DOMAIN-ROUTE
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

Running the client

To run the client you need openvpn and run the following command with the name of your config file:

sudo openvpn --config config.conf

Nitrokey

The following describes how to access the VPN using the Nitrokey HSM2 using certificates stored on the hardware dongle.

There are three main steps to getting the keys initialized for use with the VPN.

1. Setup certificate authority Create the Certificate Authority key and self signed certificate, save these onto master USB

   • Create CA key openssl genrsa -des3 -out CA-key.pem 4096

   • Create self signed certificate openssl req -new -key CA-key.pem -x509 -days 3650 -out CA-cert.pem

2. Setup server key Create server key and CA signed certificate, save these as well as the CA-cert.srl file onto master USB

   • Create server key openssl genrsa -out server-key.pem 4096
   • Create certificate signing request for server key openssl req -new -key server-key.pem -out server.csr
   • Sign CSR to create server cert openssl x509 -req -days 365 -in server.csr -CA CA-cert.pem -CAkey CA-key.pem -CAcreateserial -out server-cert.pem

3. Setup user keys

   • Initialize the key: sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 648219
   • Generate key: pkcs11-tool --module /usr/lib64/opensc-pkcs11.so -l --pin 648219 --keypairgen --key-type rsa:4096 --id XX

   • Generate csr:

     • Create hsm.conf file with the following contents, link the libraries and engines properly. Remember to set the correct pin

       openssl_conf = openssl_def
       
       [openssl_def]
       engines = engine_section
       
       [req]
       distinguished_name = req_distinguished_name
       
       [req_distinguished_name]
       
       [engine_section]
       pkcs11 = pkcs11_section
       
       [pkcs11_section]
       engine_id = pkcs11
       dynamic_path = /usr/lib/engines-1.1/libpkcs11.so
       MODULE_PATH = /usr/lib/pkcs11/opensc-pkcs11.so
       PIN = 648219
       init = 0
       

     • Create CSR

       OPENSSL_CONF=./hsm.conf openssl req -engine pkcs11 -keyform engine -new -key 0:XX -sha256 -out "NAME.csr" -subj "/C=CA/ST=Ontario/L=Toronto/O=Pervices/OU=VPN/CN=NAME"
       

4. Sign and Import Certificate:

   • Sign csr

         openssl x509 -req -days 365 -in NAME.csr -CA CA-cert.pem -CAkey CA-key.pem -CAserial CA-cert.srl -out NAME-cert.pem
     

   • Import cert

         pkcs11-tool --module /usr/lib64/opensc-pkcs11.so -l --pin 648219 --write-object NAME-cert.pem --type cert --id XX
     

5. Change user and admin pins:

   • Change SO pin

         pkcs11-tool --module /usr/lib64/opensc-pkcs11.so --login --login-type so --so-pin 3537363231383830 --change-pin --new-pin 0123456789012345
     

   • Change user pin

         pkcs11-tool --login --pin 648219 --change-pin --new-pin 123456
     

Important

Nitrokey

Here are some important facts about the Nitrokeys:

• Nitrokey HSM
• default SO: 3537363231383830
• default user: 648219

Revoking Nitrokey

If for any reason you need to revoke a certificate, here are the steps to do so:

1. Create a copy of your openssl.cnf file and place it in your local directory and edit the paths to point to the correct files and keys.
2. Create an empty index.txt file

3. Create the crlnumber file with the following contents :

   01
   

4. Add certificate to revocation list:

   openssl ca -config openssl.cnf -revoke client-cert.pem -keyfile CA-key.pem -cert CA-cert.pem
   

5. Create new certificate revocation list

   openssl ca -config openssl.cnf -gencrl -keyfile CA-key.pem -cert CA-cert.pem -out crl.pem
   

6. Replace old revocation list in server files

Client Configuration

This section deals with the user configuration requirements to successfully access the VPN on their laptops.

Set up user laptop

1. Install the following dependencies:
   • opensc openvpn openresolv
   • For Archlinux,
     • Install ccid opensc pcsc-tools from the official repositories.
     • If the card reader does not have a PIN pad, append the line(s) and set enable_pinpad = false in the opensc configuration file /etc/opensc.conf.
     • Note: The package ccid provides a generic USB interface driver for smart card reader. If the smart card at hand is not supported by the generic driver or simply it needs a specific one, feel free to install the best for that device.
     • Start and/or enable the pcscd.service.
     • Install openvpn-update-systemd-resolved from AUR
2. Create directory “openvpn” in the home directory
3. Create file vpn.conf and copy the client configuration shown above
4. Locate where the ‘opensc-pkcs11.so’ engine is and replace pkcs11-provider in vpn.conf
   • Note that the default configuration assumes an ubuntu path; you’ll have to change it to the arch engine
   • ie: replace /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so with /usr/lib/opensc-pkcs11.so
5. Find the pkcs11 id, remembering to specify the correct engine, and replace it in vpn.conf.
   • To find the pkcs11 id:
     • For Ubuntu machines: openvpn --show-pkcs11-ids /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
     • For ArchLinux machines: openvpn --show-pkcs11-ids /usr/lib/opensc-pkcs11.so
   • If required, replace the default path with the valid one. Ie; /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so with /usr/lib/opensc-pkcs11.so
6. Create bash script or run command ” sudo openvpn –config vpn.conf “

User commands

To start the vpn, open a terminal indexing your home directory and run the following script sudo ./vpnpv.sh

Known Issues

Here are a few of the specific performance issues and some details surrounding them.

OpenWRT:

• Openvpn on OpenWRT does not give full server configuration access. Some configuration options are not supported, i.e reading off of a Nitrokey

Nitrokey:

• The certificate authority cannot be kept on the Nitrokey. This is the case because OpenSSL does not allow access to the private key stored on the key when signing certificates.

Useful Commands:

Layerscape Commands

Layerscape Task	Command
Create interface	ls-addni dpmac.X
List interfaces	ls-listni
List MAC addresses	ls-listmac

Nitrokey Commands

initialize key: sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 648219

change SO-pin :

    pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so \
                --login --login-type so --so-pin 3537363231383830 \
                --change-pin --new-pin 0123456789012345`

change user pin : pkcs11-tool --login --pin 648219 --change-pin --new-pin 123456

generate key : pkcs11-tool --module /usr/lib64/opensc-pkcs11.so -l --pin 648219 --keypairgen --key-type rsa: 4096 --id XX

store certificates: pkcs11-tool --module /usr/lib64/opensc-pkcs11.so -l --pin 648219 --write-object testcert.der --type cert -id XX delete certificates : pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so -l --pin 648219 --delete-object --type cert --id XX

delete keys: pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so -l --pin 648219 --delete-object --type privkey --id XX

create certificate signing request: OPENSSL_CONF=./hsm.conf openssl req -engine pkcs11 -keyform engine -new -key 1:10 -sha256 -out “raymii.org.csr” -subj “/C=NL/ST=Zuid Holland/L=Rotterdam/O=Sparkling Network/OU=IT Dept/CN=raymii.org”`

create certificate : OPENSSL_CONF=./hsm.conf openssl req -engine pkcs11 -keyform engine -new -key 1:10 -nodes -days 3560 -x509 -sha256 -out “raymii.org.pem” -subj “/C=NL/ST=Zuid Holland/L=Rotterdam/O=Sparkling Network/OU=IT Dept/CN=raymii.org”`

view csr: openssl req -noout -text -in cert.csr

view cert: openssl x509 -noout -text -in cert.pem

Get URL: p11tool --provider /usr/lib64/opensc-pkcs11.so --list-all (For example: pkcs11: model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DENK0104013;token=SmartCard-HSM%20%28UserPIN%29%00%00%00%00%00%00%00%00%00;id=%10;object=Certificate;type=private)

To find private key URI:

p11tool --provider /usr/lib64/opensc-pkcs11.so --list-tokens
p11tool --provider /usr/lib64/opensc-pkcs11.so --list-all \
        --login "pkcs11: model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DENK0104013;token=SmartCard-HSM%20%28UserPIN%29%00%00%00%00%00%00%00%00%00"

Change SO pin:

    pkcs11-tool --module /usr/lib64/opensc-pkcs11.so \
                --login --login-type so --so-pin 3537363231383830 \
                --change-pin --new-pin ###################

Change user pin:

    pkcs11-tool --login --pin 648219 --change-pin --new-pin 123456