MARWAN docs NOC knowledge base
DocsVirtualization

Build a hyperconverged Proxmox VE 9 cluster with Ceph

Plan, install and configure a three-node Proxmox VE 9 cluster with Ceph storage for a university or research centre: hardware sizing, networks, cluster, Ceph pools and CephFS, from the command line.

Published Sami Ait Ali Oulahcen
About this guide

The written companion to the hyperconverged Proxmox cluster session of the RIMER 2025 campus networks workshop (Nouakchott, February 2025), updated for Proxmox VE 9 in November 2026. The slides show the same build in the web interface; this page gives the commands.

What you'll build#

A hyperconverged cluster runs virtual machines and stores their disks on the same servers. Three pieces do the work:

Layer Software Role
Compute Proxmox VE (QEMU/KVM, LXC) Runs VMs and containers, live migration, HA
Storage Ceph Replicates VM disks across the nodes, no SAN needed
Network Linux bridges and VLANs Connects VMs to your campus VLANs

With three nodes and Ceph keeping three copies of every block, any one server can fail or be taken down for maintenance while every VM keeps running.

flowchart LR
  M["Management + cluster<br/>1 GbE"] --- N1 & N2 & N3
  N1["pve1<br/>VMs + Ceph OSDs"] --- C["Ceph network<br/>10/25 GbE"]
  N2["pve2<br/>VMs + Ceph OSDs"] --- C
  N3["pve3<br/>VMs + Ceph OSDs"] --- C
  N1 & N2 & N3 --- V["VM networks<br/>campus VLANs"]

Plan the hardware#

Number of nodes. Use an odd number, 3, 5 or 7, so the cluster always has a majority (quorum) when one node is lost. Three is the minimum for Ceph.

CPU. Count 1 to 4 vCPUs per VM. One physical core gives two vCPUs with hyper-threading. Keep a few cores per node for Proxmox and Ceph, roughly one core per Ceph disk.

RAM. Count 1 to 4 GB per typical VM or container, plus about 1 GB per TB of Ceph storage on the node (in practice at least 4 GB per OSD), plus a few GB for the system.

Disks. SSDs (or NVMe) for Ceph if you can afford them. Ceph keeps three copies, so raw capacity must be about three times the usable space you need, plus room to recover from a failed node: plan to fill no more than 70 to 80%. Install Proxmox itself on a separate small mirrored pair.

No hardware RAID for Ceph

Ceph does the replication. Give it plain disks through an HBA, or switch the RAID controller to HBA/IT mode. A RAID volume hides failing disks from Ceph and hurts performance.

Network. At least three separate networks, ideally on separate ports:

Network Speed Carries
Management and cluster 1 GbE Web interface, SSH, corosync heartbeat
Ceph 10/25 GbE or more Replication between nodes
VM 10 GbE Traffic of the VMs, trunk with your campus VLANs

Corosync, the cluster heartbeat, is sensitive to latency. Keep it off the Ceph links, which can saturate during recovery. A second corosync link on another network is good insurance.

In this guide the nodes are:

Node Management Ceph
pve1 192.0.2.11/24 198.51.100.11/24
pve2 192.0.2.12/24 198.51.100.12/24
pve3 192.0.2.13/24 198.51.100.13/24

Prepare and install#

  1. In the BIOS or UEFI, enable virtualization (Intel VT-x / AMD-V, and VT-d / AMD-Vi).
  2. Put the Ceph disks in HBA mode, or remove them from any RAID volume.
  3. Download the Proxmox VE 9 ISO from proxmox.com, write it to a USB stick, and install every node. Pick ZFS RAID1 on the two system disks, give each node its management address and a fully qualified name (pve1.example.ac.ma).

Proxmox VE 9 is based on Debian 13 "trixie". Without a subscription, switch each node to the no-subscription repository. The repository files use the deb822 format:

/etc/apt/sources.list.d/proxmox.sources
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

Disable the enterprise repository (set Enabled: no in pve-enterprise.sources), then update:

# apt update && apt full-upgrade

Make sure every node resolves the others by name (/etc/hosts or DNS) and keeps the time in sync (chrony is installed by default).

Configure the networks#

Do this on each node, with that node's addresses. vmbr0 carries management, the second port carries Ceph, and vmbr1 is a VLAN-aware bridge on the trunk to your campus switches.

/etc/network/interfaces (pve1)
auto lo
iface lo inet loopback

# Management and cluster
auto vmbr0
iface vmbr0 inet static
    address 192.0.2.11/24
    gateway 192.0.2.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

# Ceph
auto ens1f0
iface ens1f0 inet static
    address 198.51.100.11/24
    mtu 9000

# VM networks: trunk with the campus VLANs, no IP address on the host
auto vmbr1
iface vmbr1 inet manual
    bridge-ports ens1f1
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

Apply with ifreload -a. With a VLAN-aware bridge you don't create one bridge per VLAN: each VM network card gets its VLAN tag (for example tag=120) in its settings.

Jumbo frames only end to end

mtu 9000 on the Ceph network helps, but only if every switch port on that network allows it. Test with ping -M do -s 8972 198.51.100.12 before relying on it.

Create the cluster#

On pve1:

# pvecm create campus-cluster --link0 192.0.2.11

On pve2 and pve3, join it (you'll be asked for pve1's root password):

# pvecm add 192.0.2.11 --link0 192.0.2.12
# pvecm add 192.0.2.11 --link0 192.0.2.13

Check that all three nodes are members and the cluster is quorate:

# pvecm status

Set up Ceph#

On every node, install the Ceph packages from the no-subscription repository:

# pveceph install --repository no-subscription

On pve1 only, write the initial configuration with the Ceph network:

# pveceph init --network 198.51.100.0/24

On every node, create a monitor and a manager. Three monitors keep Ceph's own quorum when one node is down:

# pveceph mon create
# pveceph mgr create

On every node, turn each data disk into an OSD (one disk, one OSD). Check the device names with lsblk first: this wipes the disk.

# pveceph osd create /dev/sdb
# pveceph osd create /dev/sdc

When all OSDs are in, ceph -s should show HEALTH_OK and all OSDs up and in.

Storage for VM disks#

Create a replicated pool (3 copies, still writable with 2) and add it as Proxmox storage in one go:

# pveceph pool create vm-pool --size 3 --min_size 2 --add_storages

CephFS for ISO images and templates#

CephFS needs a metadata server. Create one on each node (one is active, the others stand by), then the file system:

# pveceph mds create
# pveceph fs create --name cephfs --add-storage

The new cephfs storage appears on all nodes. Upload ISO images and container templates there once and every node can use them.

Make VMs highly available#

A VM on Ceph storage can run on any node. Add it to HA so the cluster restarts it on another node if its host fails:

# ha-manager add vm:100 --state started

Before planned maintenance on a node, move its VMs away with live migration, or use the node maintenance mode described in Put a Proxmox node into maintenance mode.

Check and operate#

Check Command
Cluster membership and quorum pvecm status
Ceph health ceph -s, ceph health detail
OSD layout and usage ceph osd df tree
Pool usage ceph df
HA state ha-manager status

A few habits keep the cluster healthy:

  • Keep Ceph usage under about 75%, so it can rebuild the copies of a lost node.
  • Upgrade one node at a time: migrate its VMs away, upgrade, reboot, wait for HEALTH_OK, then the next.
  • Back up VMs with Proxmox Backup Server or vzdump to storage outside the cluster. Ceph replication isn't a backup.
  • Watch the cluster: the Prometheus and Grafana guide covers the monitoring stack.

Further reading#