Using Local Storage from Kubernetes Nodes: HostPath, Rancher local-path-provisioner, and OpenEBS

In Kubernetes, choosing the right storage solution can be a game changer, especially when dealing with local storage from the nodes. In this blog, I will discuss three such options that I tested in my lab. Each has its strengths and limitations, and based on your needs, you can decide which one fits best.

Introduction

Kubernetes offers many ways to handle storage, but what if you want to use the local disks on your nodes?

• Local storage is useful for workloads that don’t need complex distributed storage solutions but still require persistent volumes.
• Using local storage on Kubernetes worker nodes allows pods to access fast, node local storage directly from the host. This ensures low latency and high performance by keeping data close to the application.

In this post, we will explore three different methods to provision local storage in Kubernetes:

• Kubernetes Native HostPath – Quick and easy, but with some trade-offs.
• Rancher local-path-provisioner – More automated solution for local storage.
• OpenEBS – Flexible with multiple backends.

I will break down each option and it’s pros and cons.

1. Kubernetes Native HostPath

HostPath is the most straightforward way to use local storage in Kubernetes. It allows you to mount a specific directory from a Kubernetes node directly into a pod. No additional setup is required – just define a path on the host, and Kubernetes will make it available inside your container.

Pros:

• Simple and straightforward: Very easy to set up.
• Direct access: Uses the node’s directory directly, so there is no additional overhead.

Cons:

• No dynamic provisioning: You have to manually manage directories on the nodes.
• No storage quotas: Pods can consume as much space as the node allows, regardless of the PVC size.
• No replication: Since it’s just a local path, there is no redundancy in case of node failure.

2. Rancher local-path-provisoner

The Rancher local-path-provisioner takes the concept of HostPath and adds automation. It dynamically provisions local storage directories when PVCs are created. This helps streamline the process and removes the need for manual directory management on the nodes.

Pros:

• Dynamic provisioning: Automatically creates directories for PVCs, reducing manual effort.
• Easy to set up: Quick to deploy and integrate with Kubernetes.

Cons:

• No storage limits: Similar to HostPath, there is no built-in mechanism to limit how much storage a pod can consume.
• No replication: It’s still tied to a single nodes storage, so no built-in redundancy.

3. OpenEBS

OpenEBS is a Kubernetes-native solution that offers a range of storage backends, making it far more flexible than the previous options. Whether you want to stick with directory based storage or use more advanced setups like LVM or replicated storage, OpenEBS has you covered.

Pros:

• Multiple backends: You can choose between directory based storage, LVM, ZFS and replicated storage.
• Storage limits with LVM: Unlike HostPath and Rancher, OpenEBS with LVM allows you to enforce storage limits on PVCs.
• Replication: OpenEBS supports replication across nodes, adding fault tolerance.

Cons:

• More complex setup: OpenEBS offers more flexibility, but it can be slightly more complex to set up and manage.
• Directory option limitations: Similar to HostPath, the directory based option in OpenEBS doesn’t enforce storage limits.

Bonus Mention: CEPH

For environments where you need distributed storage and replication, CEPH is a great option, and I have used it successfully in production. However, this post focuses on solutions that use local storage from the Kubernetes nodes, so we will leave CEPH for another discussion.

Conclusion

Choosing the right local storage solution depends on your needs:

• HostPath is great for its simplicity but lacks dynamic provisioning and storage limits.
• Rancher local-path-provisioner automates provisioning but, like HostPath, doesn’t enforce storage quotas.
• OpenEBS provides the flexibility, allowing you to set storage limits with LVM, and supports replicated storage.

The best choice for your Kubernetes setup will depend on your specific storage needs.

Next Steps

In my next post, I will walk through the installation and configuration of each solution I have tested in the lab. Stay tuned!

References

• https://kubernetes.io/docs/concepts/storage/volumes/#hostpath
• https://github.com/rancher/local-path-provisioner
• https://openebs.io/