Minikube NFS mounts

Minikube is great for having a Kubernetes cluster as local Docker development environment. In a development workflow you probably have source code on your host machine and it would be great if the Docker containers in the Kubernetes cluster could mount this. So changes made to the code can be tested and are visible quickly. The steps on this blog post are tested with Minikube version v0.28.0.

Mounting is possible for example with:

minikube start --mount --mount-string ./sources:/sources

And by using hostPath in the volume. 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
  namespace: default
  labels:
    app: my-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-api
  template:
    metadata:
      labels:
        app: my-api
    spec:
      containers:
      - name: my-api
        imagePullPolicy: Never
        image: my-api:v1
        ports:
        - containerPort: 80
        volumeMounts:
        - name: code
          mountPath: /var/www/app
      volumes:
      - name: code
        hostPath:
          path: /sources/my-api
---
apiVersion: v1
kind: Service
metadata:
  name: my-api
  namespace: default
  labels:
    app: my-api
spec:
  selector:
    app: my-api
  ports:
  - name: app-port
    port: 80
    nodePort: 30101
  type: NodePort

By default the VirtualBox VM driver is used. By using the mount above you will get a VirtualBox shared folder. For a project with quite a lot of files, like a PHP framework as Symfony, your load time will be around 20-30 seconds per page request, which is terribly slow.

However NFS mounts can be used and they will give awesome performance! View the comparison. The first page request will load in about five seconds and after that in milliseconds.

NFS server

Configure your sources directory as export (/etc/exports) in the NFS server that runs on your host machine. This way containers can mount source code. This is a lot faster than a default VirtualBox shared folder mount. You only have to do this once, the NFS service will load /etc/exports at (re)boot.

NOTE: The Minikube IP can be different after a minikube delete and minikube start command. Make sure that your NFS export contains the correct Minikube IP again.

Mac OS X

echo "$(realpath .)/sources -alldirs -mapall="$(id -u)":"$(id -g)" $(minikube ip)" | sudo tee -a /etc/exports && sudo nfsd restart

Check if the entry is active by executing on your host machine:

showmount -e 127.0.0.1

This should output something like:

Exports list on 127.0.0.1:
/Absolute/path/to/sources 192.168.99.100

Linux

Similar to the command above for Mac OS X, but the service name can be different depending on the distribution.

Windows

With https://github.com/winnfsd/winnfsd it even works great on Windows! But make sure you configure the NFS version 3 instead of the default version 4.

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: default-sources-volume
spec:
  capacity:
    storage: 15Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard
  nfs:
    # The address 192.168.99.1 is the Minikube gateway to the host. This way
    # not the container IP will be visible by the NFS server on the host machine,
    # but the IP address of the `minikube ip` command. You will need to
    # grant access to the `minikube ip` IP address.
    server: 192.168.99.1
    path: '/C/Users/myname/minikube/sources/default'
  mountOptions:
    - nfsvers=3
    - udp

UDP protocol because of https://github.com/winnfsd/winnfsd/issues/68. The content of nfs-pathfile.txt. Access is allowed to subdirectories inside this directory.

C:\Users\myname\minikube\sources

The command:

C:\Users\myname\bin\WinNFSd.exe -pathFile "C:\Users\myname\etc\nfs-pathfile.txt"

The Kubernetes and WinNFSd path is case sensitive, make sure it matches!

NFS mounts

The examples in this section are prefixed with “default-“, referring to the default namespace. The examples also work if multiple namespaces are used! Just repeat the steps for each namespace and replace “default” with the proper namespace. Make sure you also change the “namespace” key inside the yaml files.

An important thing to notice is that persistent volumes are global and persistent volume claims live inside a namespace.

Configure a PersistentVolume in a file default-sources-volume.yaml:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: default-sources-volume
spec:
  capacity:
    storage: 15Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard
  nfs:
    # The address 192.168.99.1 is the Minikube gateway to the host. This way
    # not the container IP will be visible by the NFS server on the host machine,
    # but the IP address of the `minikube ip` command. You will need to
    # grant access to the `minikube ip` IP address.
    server: 192.168.99.1
    path: '/Absolute/path/to/sources/default'

Apply it with:

kubectl apply -f ./default-sources-volume.yaml

Configure a PersistentVolumeClaim in a file default-sources-volume-claim.yaml: 

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: sources-volume-claim
  namespace: default
spec:
  storageClassName: standard
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 15Gi

Apply it with:

kubectl apply -f ./default-sources-volume-claim.yaml

Now you can use the NFS persistent volume to your host machine in your container. Create a file my-api.yaml: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
  namespace: default
  labels:
    app: my-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-api
  template:
    metadata:
      labels:
        app: my-api
    spec:
      containers:
      - name: my-api
        imagePullPolicy: Never
        image: my-api:v1
        ports:
        - containerPort: 80
        volumeMounts:
        - name: code
          mountPath: /var/www/app
          subPath: my-api
      volumes:
      - name: code
        persistentVolumeClaim:
          claimName: sources-volume-claim
---
apiVersion: v1
kind: Service
metadata:
  name: my-api
  namespace: default
  labels:
    app: my-api
spec:
  selector:
    app: my-api
  ports:
  - name: app-port
    port: 80
    nodePort: 30101
  type: NodePort

Apply it with:

kubectl apply -f ./my-api.yaml

Now you have awesome fast mounts from your containers to your host machine and a very nice development workflow!

Alternative: NFS mount directly in the VM

Minikube has a sub command “ssh”, this way you can for example build Docker images with the docker client inside the Minikube VM with:

minikube ssh "docker build -t myimage:latest -f /host-sources ."

To create the NFS /host-sources mount:

echo 'Mounting (NFS) /host-sources inside the Minikube VM'
sources_dir=$(realpath .)/sources
# In case of Windows drive C, uppercase the drive letter so it matches with the WinNFSd exports
sources_dir=${sources_dir/\/c\//\/C\/}
# The address 192.168.99.1 is the Minikube gateway to the host. NFS version 3 to be compatible with WinNFSd
minikube ssh "sudo mkdir -p /host-sources && sudo mount -t nfs -o nfsvers=3,tcp 192.168.99.1:${sources_dir} /host-sources"

Use udp instead of tcp in case of Windows to avoid hanging issues. This NFS mount can also be reused for Docker containers with a hostPath volume mount.

Tags: ,,,,