Deploying an NFS Server in K8S and exposing it externally
up vote
2
down vote
favorite
I am in the process of setting up a NFS server on my K8S cluster. I want it to act as a NFS server for external entities i.e. client will be from outside the K8S cluster such as VMs.
The port requirements for the Docker image are :
==================================================================
SERVER STARTUP COMPLETE
==================================================================
----> list of enabled NFS protocol versions: 4.2, 4.1, 4
----> list of container exports:
----> /exports *(rw,no_subtree_check)
----> list of container ports that should be exposed:
----> 111 (TCP and UDP)
----> 2049 (TCP and UDP)
----> 32765 (TCP and UDP)
----> 32767 (TCP and UDP)
So I have created a Debian Stretch docker image. When I run it using docker run, I can successfully expose /exports and mount it from other systems.
docker run -v /data:/exports -v /tmp/exports.txt:/etc/exports:ro
--cap-add SYS_ADMIN -p 2049:2049 -p 111:111 -p 32765:32765
-p 32767:32767 8113b6abeac
The above command spins up my docker container and when I do
mount.nfs4 <DOKCER_HOST_IP>:/exports /mount/
from another VM, I can successfully mount the volume.
So everything up until here is A OK!
Now the task is to deploy this in K8S.
My stateful-set definition is:
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: nfs-provisioner
spec:
selector:
matchLabels:
app: nfs-provisioner
serviceName: "nfs-provisioner"
replicas: 1
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccount: nfs-provisioner
terminationGracePeriodSeconds: 10
imagePullSecrets:
- name: artifactory
containers:
- name: nfs-provisioner
image: repository.hybris.com:5005/test/nfs/nfs-server:1.2
ports:
- name: nfs
containerPort: 2049
- name: mountd
containerPort: 20048
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
- name: filenet
containerPort: 32767
- name: filenet-udp
containerPort: 32767
protocol: UDP
- name: unknown
containerPort: 32765
- name: unknown-udp
containerPort: 32765
protocol: UDP
securityContext:
privileged: true
env:
- name: SERVICE_NAME
value: nfs-provisioner
- name: NFS_EXPORT_0
value: '/exports *(rw,no_subtree_check)'
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: export-volume
mountPath: /exports
volumes:
- name: export-volume
hostPath:
path: /var/tmp
As you can see, I have specified all the ports (both TCP and UDP)
And now to expose this to the outside world and not just inside the cluster, my service.yaml file deceleration is :
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
type: NodePort
ports:
- name: nfs
port: 2049
- name: mountd
port: 20048
- name: rpcbind
port: 111
- name: rpcbind-udp
port: 111
protocol: UDP
- name: filenet
port: 32767
- name: filenet-udp
port: 32767
protocol: UDP
- name: unknown
port: 32765
- name: unknown-udp
port: 32765
protocol: UDP
selector:
app: nfs-provisioner
This results in
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nfs-provisioner NodePort 10.233.43.135 <none> 2049:30382/TCP,20048:31316/TCP,111:32720/TCP,111:32720/UDP,32767:30173/TCP,32767:30173/UDP,32765:31215/TCP,32765:31215/UDP 32m
Now I try to mount /exports from another node/VM that is external to the
K8S cluster.
I've tried
mount.nfs4 <K8S_Node_IP>:/exports /mount/
and I've tried
mount.nfs4 -o port=<NodePort> <K8S_Node_IP>:/exports /mount/
Ive tried each NodePort one at a time. But none of them work. I get the error :
mount.nfs4 -o port=31316 <K8S_Node_IP>:/exports /mount/
mount.nfs4: mount to NFS server '<K8S_Node_IP>:/exports' failed: RPC Error: Unable to receive
I'm unsure as to what might be the issue here. Is it that I need to specify all the nodePorts? If so, how can I do that?
docker
kubernetes nfs nfsclient edited Nov 21 at 20:34
Rico
25.1k94864
asked Nov 21 at 19:56
Jason Stanley
705727
add a comment |
up vote
2
down vote
favorite
I am in the process of setting up a NFS server on my K8S cluster. I want it to act as a NFS server for external entities i.e. client will be from outside the K8S cluster such as VMs.
The port requirements for the Docker image are :
==================================================================
SERVER STARTUP COMPLETE
==================================================================
----> list of enabled NFS protocol versions: 4.2, 4.1, 4
----> list of container exports:
----> /exports *(rw,no_subtree_check)
----> list of container ports that should be exposed:
----> 111 (TCP and UDP)
----> 2049 (TCP and UDP)
----> 32765 (TCP and UDP)
----> 32767 (TCP and UDP)
So I have created a Debian Stretch docker image. When I run it using docker run, I can successfully expose /exports and mount it from other systems.
docker run -v /data:/exports -v /tmp/exports.txt:/etc/exports:ro
--cap-add SYS_ADMIN -p 2049:2049 -p 111:111 -p 32765:32765
-p 32767:32767 8113b6abeac
The above command spins up my docker container and when I do
mount.nfs4 <DOKCER_HOST_IP>:/exports /mount/
from another VM, I can successfully mount the volume.
So everything up until here is A OK!
Now the task is to deploy this in K8S.
My stateful-set definition is:
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: nfs-provisioner
spec:
selector:
matchLabels:
app: nfs-provisioner
serviceName: "nfs-provisioner"
replicas: 1
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccount: nfs-provisioner
terminationGracePeriodSeconds: 10
imagePullSecrets:
- name: artifactory
containers:
- name: nfs-provisioner
image: repository.hybris.com:5005/test/nfs/nfs-server:1.2
ports:
- name: nfs
containerPort: 2049
- name: mountd
containerPort: 20048
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
- name: filenet
containerPort: 32767
- name: filenet-udp
containerPort: 32767
protocol: UDP
- name: unknown
containerPort: 32765
- name: unknown-udp
containerPort: 32765
protocol: UDP
securityContext:
privileged: true
env:
- name: SERVICE_NAME
value: nfs-provisioner
- name: NFS_EXPORT_0
value: '/exports *(rw,no_subtree_check)'
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: export-volume
mountPath: /exports
volumes:
- name: export-volume
hostPath:
path: /var/tmp
As you can see, I have specified all the ports (both TCP and UDP)
And now to expose this to the outside world and not just inside the cluster, my service.yaml file deceleration is :
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
type: NodePort
ports:
- name: nfs
port: 2049
- name: mountd
port: 20048
- name: rpcbind
port: 111
- name: rpcbind-udp
port: 111
protocol: UDP
- name: filenet
port: 32767
- name: filenet-udp
port: 32767
protocol: UDP
- name: unknown
port: 32765
- name: unknown-udp
port: 32765
protocol: UDP
selector:
app: nfs-provisioner
This results in
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nfs-provisioner NodePort 10.233.43.135 <none> 2049:30382/TCP,20048:31316/TCP,111:32720/TCP,111:32720/UDP,32767:30173/TCP,32767:30173/UDP,32765:31215/TCP,32765:31215/UDP 32m
Now I try to mount /exports from another node/VM that is external to the
K8S cluster.
I've tried
mount.nfs4 <K8S_Node_IP>:/exports /mount/
and I've tried
mount.nfs4 -o port=<NodePort> <K8S_Node_IP>:/exports /mount/
Ive tried each NodePort one at a time. But none of them work. I get the error :
mount.nfs4 -o port=31316 <K8S_Node_IP>:/exports /mount/
mount.nfs4: mount to NFS server '<K8S_Node_IP>:/exports' failed: RPC Error: Unable to receive
I'm unsure as to what might be the issue here. Is it that I need to specify all the nodePorts? If so, how can I do that?
docker
kubernetes nfs nfsclient edited Nov 21 at 20:34
Rico
25.1k94864
asked Nov 21 at 19:56
Jason Stanley
705727
Have you considered exposing the ports via external-ip + load balancer?
– William Chong
Nov 21 at 20:13
My K8S cluster is on Baremetal. So I dont have access to loadbalancers.
– Jason Stanley
Nov 21 at 20:46
add a comment |
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I am in the process of setting up a NFS server on my K8S cluster. I want it to act as a NFS server for external entities i.e. client will be from outside the K8S cluster such as VMs.
The port requirements for the Docker image are :
==================================================================
SERVER STARTUP COMPLETE
==================================================================
----> list of enabled NFS protocol versions: 4.2, 4.1, 4
----> list of container exports:
----> /exports *(rw,no_subtree_check)
----> list of container ports that should be exposed:
----> 111 (TCP and UDP)
----> 2049 (TCP and UDP)
----> 32765 (TCP and UDP)
----> 32767 (TCP and UDP)
So I have created a Debian Stretch docker image. When I run it using docker run, I can successfully expose /exports and mount it from other systems.
docker run -v /data:/exports -v /tmp/exports.txt:/etc/exports:ro
--cap-add SYS_ADMIN -p 2049:2049 -p 111:111 -p 32765:32765
-p 32767:32767 8113b6abeac
The above command spins up my docker container and when I do
mount.nfs4 <DOKCER_HOST_IP>:/exports /mount/
from another VM, I can successfully mount the volume.
So everything up until here is A OK!
Now the task is to deploy this in K8S.
My stateful-set definition is:
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: nfs-provisioner
spec:
selector:
matchLabels:
app: nfs-provisioner
serviceName: "nfs-provisioner"
replicas: 1
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccount: nfs-provisioner
terminationGracePeriodSeconds: 10
imagePullSecrets:
- name: artifactory
containers:
- name: nfs-provisioner
image: repository.hybris.com:5005/test/nfs/nfs-server:1.2
ports:
- name: nfs
containerPort: 2049
- name: mountd
containerPort: 20048
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
- name: filenet
containerPort: 32767
- name: filenet-udp
containerPort: 32767
protocol: UDP
- name: unknown
containerPort: 32765
- name: unknown-udp
containerPort: 32765
protocol: UDP
securityContext:
privileged: true
env:
- name: SERVICE_NAME
value: nfs-provisioner
- name: NFS_EXPORT_0
value: '/exports *(rw,no_subtree_check)'
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: export-volume
mountPath: /exports
volumes:
- name: export-volume
hostPath:
path: /var/tmp
As you can see, I have specified all the ports (both TCP and UDP)
And now to expose this to the outside world and not just inside the cluster, my service.yaml file deceleration is :
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
type: NodePort
ports:
- name: nfs
port: 2049
- name: mountd
port: 20048
- name: rpcbind
port: 111
- name: rpcbind-udp
port: 111
protocol: UDP
- name: filenet
port: 32767
- name: filenet-udp
port: 32767
protocol: UDP
- name: unknown
port: 32765
- name: unknown-udp
port: 32765
protocol: UDP
selector:
app: nfs-provisioner
This results in
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nfs-provisioner NodePort 10.233.43.135 <none> 2049:30382/TCP,20048:31316/TCP,111:32720/TCP,111:32720/UDP,32767:30173/TCP,32767:30173/UDP,32765:31215/TCP,32765:31215/UDP 32m
Now I try to mount /exports from another node/VM that is external to the
K8S cluster.
I've tried
mount.nfs4 <K8S_Node_IP>:/exports /mount/
and I've tried
mount.nfs4 -o port=<NodePort> <K8S_Node_IP>:/exports /mount/
Ive tried each NodePort one at a time. But none of them work. I get the error :
mount.nfs4 -o port=31316 <K8S_Node_IP>:/exports /mount/
mount.nfs4: mount to NFS server '<K8S_Node_IP>:/exports' failed: RPC Error: Unable to receive
I'm unsure as to what might be the issue here. Is it that I need to specify all the nodePorts? If so, how can I do that?
docker
kubernetes nfs nfsclient edited Nov 21 at 20:34
Rico
25.1k94864
asked Nov 21 at 19:56
Jason Stanley
705727
I am in the process of setting up a NFS server on my K8S cluster. I want it to act as a NFS server for external entities i.e. client will be from outside the K8S cluster such as VMs.
The port requirements for the Docker image are :
==================================================================
SERVER STARTUP COMPLETE
==================================================================
----> list of enabled NFS protocol versions: 4.2, 4.1, 4
----> list of container exports:
----> /exports *(rw,no_subtree_check)
----> list of container ports that should be exposed:
----> 111 (TCP and UDP)
----> 2049 (TCP and UDP)
----> 32765 (TCP and UDP)
----> 32767 (TCP and UDP)
So I have created a Debian Stretch docker image. When I run it using docker run, I can successfully expose /exports and mount it from other systems.
docker run -v /data:/exports -v /tmp/exports.txt:/etc/exports:ro
--cap-add SYS_ADMIN -p 2049:2049 -p 111:111 -p 32765:32765
-p 32767:32767 8113b6abeac
The above command spins up my docker container and when I do
mount.nfs4 <DOKCER_HOST_IP>:/exports /mount/
from another VM, I can successfully mount the volume.
So everything up until here is A OK!
Now the task is to deploy this in K8S.
My stateful-set definition is:
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: nfs-provisioner
spec:
selector:
matchLabels:
app: nfs-provisioner
serviceName: "nfs-provisioner"
replicas: 1
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccount: nfs-provisioner
terminationGracePeriodSeconds: 10
imagePullSecrets:
- name: artifactory
containers:
- name: nfs-provisioner
image: repository.hybris.com:5005/test/nfs/nfs-server:1.2
ports:
- name: nfs
containerPort: 2049
- name: mountd
containerPort: 20048
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
- name: filenet
containerPort: 32767
- name: filenet-udp
containerPort: 32767
protocol: UDP
- name: unknown
containerPort: 32765
- name: unknown-udp
containerPort: 32765
protocol: UDP
securityContext:
privileged: true
env:
- name: SERVICE_NAME
value: nfs-provisioner
- name: NFS_EXPORT_0
value: '/exports *(rw,no_subtree_check)'
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: export-volume
mountPath: /exports
volumes:
- name: export-volume
hostPath:
path: /var/tmp
As you can see, I have specified all the ports (both TCP and UDP)
And now to expose this to the outside world and not just inside the cluster, my service.yaml file deceleration is :
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
type: NodePort
ports:
- name: nfs
port: 2049
- name: mountd
port: 20048
- name: rpcbind
port: 111
- name: rpcbind-udp
port: 111
protocol: UDP
- name: filenet
port: 32767
- name: filenet-udp
port: 32767
protocol: UDP
- name: unknown
port: 32765
- name: unknown-udp
port: 32765
protocol: UDP
selector:
app: nfs-provisioner
This results in
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nfs-provisioner NodePort 10.233.43.135 <none> 2049:30382/TCP,20048:31316/TCP,111:32720/TCP,111:32720/UDP,32767:30173/TCP,32767:30173/UDP,32765:31215/TCP,32765:31215/UDP 32m
Now I try to mount /exports from another node/VM that is external to the
K8S cluster.
I've tried
mount.nfs4 <K8S_Node_IP>:/exports /mount/
and I've tried
mount.nfs4 -o port=<NodePort> <K8S_Node_IP>:/exports /mount/
Ive tried each NodePort one at a time. But none of them work. I get the error :
mount.nfs4 -o port=31316 <K8S_Node_IP>:/exports /mount/
mount.nfs4: mount to NFS server '<K8S_Node_IP>:/exports' failed: RPC Error: Unable to receive
I'm unsure as to what might be the issue here. Is it that I need to specify all the nodePorts? If so, how can I do that?
docker
kubernetes nfs nfsclient docker
kubernetes nfs nfsclient edited Nov 21 at 20:34
Rico
25.1k94864
asked Nov 21 at 19:56
Jason Stanley
705727
edited Nov 21 at 20:34
Rico
25.1k94864
asked Nov 21 at 19:56
Jason Stanley
705727
edited Nov 21 at 20:34
Rico
25.1k94864
edited Nov 21 at 20:34
Rico
25.1k94864
edited Nov 21 at 20:34
Rico
25.1k94864
25.1k94864
asked Nov 21 at 19:56
Jason Stanley
705727
asked Nov 21 at 19:56
Jason Stanley
705727
asked Nov 21 at 19:56
Jason Stanley
705727
705727
Have you considered exposing the ports via external-ip + load balancer?
– William Chong
Nov 21 at 20:13
My K8S cluster is on Baremetal. So I dont have access to loadbalancers.
– Jason Stanley
Nov 21 at 20:46
add a comment |
Have you considered exposing the ports via external-ip + load balancer?
– William Chong
Nov 21 at 20:13
My K8S cluster is on Baremetal. So I dont have access to loadbalancers.
– Jason Stanley
Nov 21 at 20:46
Have you considered exposing the ports via external-ip + load balancer?
– William Chong
Nov 21 at 20:13
Have you considered exposing the ports via external-ip + load balancer?
– William Chong
Nov 21 at 20:13
My K8S cluster is on Baremetal. So I dont have access to loadbalancers.
– Jason Stanley
Nov 21 at 20:46
My K8S cluster is on Baremetal. So I dont have access to loadbalancers.
– Jason Stanley
Nov 21 at 20:46
add a comment |
1 Answer
1
active
oldest
votes
up vote
2
down vote
The issue here is that all the NodePorts are different as seen externally as from:
----> 111 (TCP and UDP)
----> 2049 (TCP and UDP)
----> 32765 (TCP and UDP)
----> 32767 (TCP and UDP)
You can try an L4 load balancer that exposes exactly those ports on a given IP address (internal or external) and forwards them to the nodePorts (which is what type=LoadBalancer does too).
Another option is to hard code the NodePorts in your services to match exactly the ones of the containers:
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
type: NodePort
ports:
- name: nfs
port: 2049
nodePort: 2049
- name: mountd
port: 20048
nodePort: 20048
- name: rpcbind
port: 111
nodePort: 111
- name: rpcbind-udp
port: 111
nodePort: 111
protocol: UDP
- name: filenet
port: 32767
nodePort: 32767
- name: filenet-udp
port: 32767
nodePort: 32767
protocol: UDP
- name: unknown
port: 32765
nodePort: 32765
- name: unknown-udp
port: 32765
nodePort: 32765
protocol: UDP
selector:
app: nfs-provisioner
You will have to change the nodePort range (--service-node-port-range) on the kubelet though. This is so that you can use 2049 and 111.
You can also change the ports that you NFS server listens on for 2049 (nfs) and 111 (portmapper) for example, that way you don't have to change --service-node-port-range
answered Nov 21 at 20:51
Rico
25.1k94864
Thanks Rico. I do not have a LoadBalancer so cannot try that option since im running K8S on Bare Metal. I did try out playing with the--service-node-port-rangeoption and it works. A simplemount -t nfs 10.27.239.31:/exports /mnt/datafrom my client with this configuration works ! So thanks for that. But this is a hack eventually since I did--service-node-port-range=99-32767. As for changing the nfs and portmapper port numbers, i will eventually run into the same problem from the client side since I cannot use multiple different ports.
– Jason Stanley
Nov 22 at 0:40
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
The issue here is that all the NodePorts are different as seen externally as from:
----> 111 (TCP and UDP)
----> 2049 (TCP and UDP)
----> 32765 (TCP and UDP)
----> 32767 (TCP and UDP)
You can try an L4 load balancer that exposes exactly those ports on a given IP address (internal or external) and forwards them to the nodePorts (which is what type=LoadBalancer does too).
Another option is to hard code the NodePorts in your services to match exactly the ones of the containers:
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
type: NodePort
ports:
- name: nfs
port: 2049
nodePort: 2049
- name: mountd
port: 20048
nodePort: 20048
- name: rpcbind
port: 111
nodePort: 111
- name: rpcbind-udp
port: 111
nodePort: 111
protocol: UDP
- name: filenet
port: 32767
nodePort: 32767
- name: filenet-udp
port: 32767
nodePort: 32767
protocol: UDP
- name: unknown
port: 32765
nodePort: 32765
- name: unknown-udp
port: 32765
nodePort: 32765
protocol: UDP
selector:
app: nfs-provisioner
You will have to change the nodePort range (--service-node-port-range) on the kubelet though. This is so that you can use 2049 and 111.
You can also change the ports that you NFS server listens on for 2049 (nfs) and 111 (portmapper) for example, that way you don't have to change --service-node-port-range
answered Nov 21 at 20:51
Rico
25.1k94864
Thanks Rico. I do not have a LoadBalancer so cannot try that option since im running K8S on Bare Metal. I did try out playing with the--service-node-port-rangeoption and it works. A simplemount -t nfs 10.27.239.31:/exports /mnt/datafrom my client with this configuration works ! So thanks for that. But this is a hack eventually since I did--service-node-port-range=99-32767. As for changing the nfs and portmapper port numbers, i will eventually run into the same problem from the client side since I cannot use multiple different ports.
– Jason Stanley
Nov 22 at 0:40
add a comment |
up vote
2
down vote
The issue here is that all the NodePorts are different as seen externally as from:
----> 111 (TCP and UDP)
----> 2049 (TCP and UDP)
----> 32765 (TCP and UDP)
----> 32767 (TCP and UDP)
You can try an L4 load balancer that exposes exactly those ports on a given IP address (internal or external) and forwards them to the nodePorts (which is what type=LoadBalancer does too).
Another option is to hard code the NodePorts in your services to match exactly the ones of the containers:
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
type: NodePort
ports:
- name: nfs
port: 2049
nodePort: 2049
- name: mountd
port: 20048
nodePort: 20048
- name: rpcbind
port: 111
nodePort: 111
- name: rpcbind-udp
port: 111
nodePort: 111
protocol: UDP
- name: filenet
port: 32767
nodePort: 32767
- name: filenet-udp
port: 32767
nodePort: 32767
protocol: UDP
- name: unknown
port: 32765
nodePort: 32765
- name: unknown-udp
port: 32765
nodePort: 32765
protocol: UDP
selector:
app: nfs-provisioner
You will have to change the nodePort range (--service-node-port-range) on the kubelet though. This is so that you can use 2049 and 111.
You can also change the ports that you NFS server listens on for 2049 (nfs) and 111 (portmapper) for example, that way you don't have to change --service-node-port-range
answered Nov 21 at 20:51
Rico
25.1k94864
Thanks Rico. I do not have a LoadBalancer so cannot try that option since im running K8S on Bare Metal. I did try out playing with the--service-node-port-rangeoption and it works. A simplemount -t nfs 10.27.239.31:/exports /mnt/datafrom my client with this configuration works ! So thanks for that. But this is a hack eventually since I did--service-node-port-range=99-32767. As for changing the nfs and portmapper port numbers, i will eventually run into the same problem from the client side since I cannot use multiple different ports.
– Jason Stanley
Nov 22 at 0:40
add a comment |
up vote
2
down vote
up vote
2
down vote
The issue here is that all the NodePorts are different as seen externally as from:
----> 111 (TCP and UDP)
----> 2049 (TCP and UDP)
----> 32765 (TCP and UDP)
----> 32767 (TCP and UDP)
You can try an L4 load balancer that exposes exactly those ports on a given IP address (internal or external) and forwards them to the nodePorts (which is what type=LoadBalancer does too).
Another option is to hard code the NodePorts in your services to match exactly the ones of the containers:
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
type: NodePort
ports:
- name: nfs
port: 2049
nodePort: 2049
- name: mountd
port: 20048
nodePort: 20048
- name: rpcbind
port: 111
nodePort: 111
- name: rpcbind-udp
port: 111
nodePort: 111
protocol: UDP
- name: filenet
port: 32767
nodePort: 32767
- name: filenet-udp
port: 32767
nodePort: 32767
protocol: UDP
- name: unknown
port: 32765
nodePort: 32765
- name: unknown-udp
port: 32765
nodePort: 32765
protocol: UDP
selector:
app: nfs-provisioner
You will have to change the nodePort range (--service-node-port-range) on the kubelet though. This is so that you can use 2049 and 111.
You can also change the ports that you NFS server listens on for 2049 (nfs) and 111 (portmapper) for example, that way you don't have to change --service-node-port-range
answered Nov 21 at 20:51
Rico
25.1k94864
The issue here is that all the NodePorts are different as seen externally as from:
----> 111 (TCP and UDP)
----> 2049 (TCP and UDP)
----> 32765 (TCP and UDP)
----> 32767 (TCP and UDP)
You can try an L4 load balancer that exposes exactly those ports on a given IP address (internal or external) and forwards them to the nodePorts (which is what type=LoadBalancer does too).
Another option is to hard code the NodePorts in your services to match exactly the ones of the containers:
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
type: NodePort
ports:
- name: nfs
port: 2049
nodePort: 2049
- name: mountd
port: 20048
nodePort: 20048
- name: rpcbind
port: 111
nodePort: 111
- name: rpcbind-udp
port: 111
nodePort: 111
protocol: UDP
- name: filenet
port: 32767
nodePort: 32767
- name: filenet-udp
port: 32767
nodePort: 32767
protocol: UDP
- name: unknown
port: 32765
nodePort: 32765
- name: unknown-udp
port: 32765
nodePort: 32765
protocol: UDP
selector:
app: nfs-provisioner
You will have to change the nodePort range (--service-node-port-range) on the kubelet though. This is so that you can use 2049 and 111.
You can also change the ports that you NFS server listens on for 2049 (nfs) and 111 (portmapper) for example, that way you don't have to change --service-node-port-range
answered Nov 21 at 20:51
Rico
25.1k94864
answered Nov 21 at 20:51
Rico
25.1k94864
answered Nov 21 at 20:51
Rico
25.1k94864
answered Nov 21 at 20:51
Rico
25.1k94864
25.1k94864
Thanks Rico. I do not have a LoadBalancer so cannot try that option since im running K8S on Bare Metal. I did try out playing with the--service-node-port-rangeoption and it works. A simplemount -t nfs 10.27.239.31:/exports /mnt/datafrom my client with this configuration works ! So thanks for that. But this is a hack eventually since I did--service-node-port-range=99-32767. As for changing the nfs and portmapper port numbers, i will eventually run into the same problem from the client side since I cannot use multiple different ports.
– Jason Stanley
Nov 22 at 0:40
add a comment |
Thanks Rico. I do not have a LoadBalancer so cannot try that option since im running K8S on Bare Metal. I did try out playing with the--service-node-port-rangeoption and it works. A simplemount -t nfs 10.27.239.31:/exports /mnt/datafrom my client with this configuration works ! So thanks for that. But this is a hack eventually since I did--service-node-port-range=99-32767. As for changing the nfs and portmapper port numbers, i will eventually run into the same problem from the client side since I cannot use multiple different ports.
– Jason Stanley
Nov 22 at 0:40
Thanks Rico. I do not have a LoadBalancer so cannot try that option since im running K8S on Bare Metal. I did try out playing with the
--service-node-port-range option and it works. A simple mount -t nfs 10.27.239.31:/exports /mnt/data from my client with this configuration works ! So thanks for that. But this is a hack eventually since I did --service-node-port-range=99-32767. As for changing the nfs and portmapper port numbers, i will eventually run into the same problem from the client side since I cannot use multiple different ports.– Jason Stanley
Nov 22 at 0:40
Thanks Rico. I do not have a LoadBalancer so cannot try that option since im running K8S on Bare Metal. I did try out playing with the
--service-node-port-range option and it works. A simple mount -t nfs 10.27.239.31:/exports /mnt/data from my client with this configuration works ! So thanks for that. But this is a hack eventually since I did --service-node-port-range=99-32767. As for changing the nfs and portmapper port numbers, i will eventually run into the same problem from the client side since I cannot use multiple different ports.– Jason Stanley
Nov 22 at 0:40
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53419624%2fdeploying-an-nfs-server-in-k8s-and-exposing-it-externally%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Have you considered exposing the ports via external-ip + load balancer?
– William Chong
Nov 21 at 20:13
My K8S cluster is on Baremetal. So I dont have access to loadbalancers.
– Jason Stanley
Nov 21 at 20:46