RHEL 7 and NFSv4 with Kerberos

by GS
in Corosync, DRBD, gssproxy, idmapd, Linux, NFS, nfs-utils, Pacemaker, rpc-gssd, rpc-rquotad
on February 12, 2019

Over the past year I have been tasked with building out a large Secure NFSv4 Environment using DRBD, Corosync and Pacemaker and ran into a plethora of issues which included gotcha’s with setting up NFSv4 Server and Client Security settings related to gssproxy/rpc-gssd, how to enforce quotas remotely with rpc-rquotad, to setting up idmapd or sssd, and dealing with some known defects that are not patched below RHEL/Centos 7.6

Kerberos Setup for NFSv4

Some settings are needed in /etc/sysconfig/nfs to make kerberized NFS function correctly. Note if using an alternate keytab such as /etc/security/keytabs/nfs.keytab you must update RPCGSSDARGS and /etc/gssproxy/99-nfs-client.conf and /etc/gssproxy/99-nfs-server.conf. They are as follows and after setting these you must run: “systemctl restart nfs-config”

/etc/sysconfig/nfs settings

RPCNFSCOUNT=16
NFSD_V4_GRACE=10
NFSD_V4_LEASE=10
STATDARG=”–no-notify”
RPCGSSDARGS=”-T 60 -t 60 -k /etc/security/keytabs/nfs.keytab”
GSS_USE_PROXY=yes
SECURE_NFS=yes

/etc/gssproxy/24-nfs-server.conf and /etc/gssproxy/99-nfs-client.conf

Note: As stated above this is only used if using an alternate keytab

cred_store = /etc/security/keytabs/nfs.keytab

Enforcing quotas on an NFSv4 Setup

NFS Export Options

Note: Allowing quota to work over NFS requires the following to options “uquota,gqnoenforce”

options="noatime,nodiratime,uquota,gqnoenforce"

Setup the file system to use xfs quotas

xfs_quota -x -c 'limit bsoft=9g bhard=10g -d' /data/drbd/exports/nfs 
xfs_quota -xc 'report -h' /data/drbd/exports/nfs

Verify rpc-rquotad service is set to startup and is running on the NFS Servers

systemctl enable  rpc-rquotad
systemctl start rpc-rquotad

Setting up idmapd to map NFSv4 Kerberos principals to userids using nsswitch

Note: This maps principals from the defined Kerberos Realms both local and remote to usernames that can be looked up using nsswitch.

[General]
Domain = hdp.senia.org
Local-Realms = HDP.SENIA.ORG, HDPSVC.HDPUSR.SENIA.ORG,HDPUSR.SENIA.ORG
Verbosity = 0

[Mapping]
Nobody-Group = nobody
Nobody-User = nobody

[Static]

[Translation]
Method = nsswitch

Update /etc/sysctl.d/90-filesystem.conf for idmap cache timeout

Note: this parameter is required if not your id’s could be incorrectly cached if a problem occurs.

fs.nfs.idmap_cache_timeout = 60

RHEL/Centos defects related to NFSv4 and Kerberos and GSSProxy

When attempting to utilize Non Local Kerberos Principal names we noticed that they would be mapped to nobody and because on the NFS Client the gssproxy/nfs-utils code has some known defects which are explained in the below links and are completely resolved with RHEL 7.6.

Tags: corosync, drbd, gssproxy, idmapd, Linux, nfs, pacemaker, rpc-gssd, rpc-rquotad

RHEL 7 and NFSv4 with Kerberos