The written companion to the hyperconverged Proxmox cluster session of the RIMER 2025 campus networks workshop (Nouakchott, February 2025), updated for Proxmox VE 9 in November 2026. The slides show the same build in the web interface; this page gives the commands.
What you'll build#
A hyperconverged cluster runs virtual machines and stores their disks on the same servers. Three pieces do the work:
| Layer | Software | Role |
|---|---|---|
| Compute | Proxmox VE (QEMU/KVM, LXC) | Runs VMs and containers, live migration, HA |
| Storage | Ceph | Replicates VM disks across the nodes, no SAN needed |
| Network | Linux bridges and VLANs | Connects VMs to your campus VLANs |
With three nodes and Ceph keeping three copies of every block, any one server can fail or be taken down for maintenance while every VM keeps running.
flowchart LR M["Management + cluster<br/>1 GbE"] --- N1 & N2 & N3 N1["pve1<br/>VMs + Ceph OSDs"] --- C["Ceph network<br/>10/25 GbE"] N2["pve2<br/>VMs + Ceph OSDs"] --- C N3["pve3<br/>VMs + Ceph OSDs"] --- C N1 & N2 & N3 --- V["VM networks<br/>campus VLANs"]
Plan the hardware#
Number of nodes. Use an odd number, 3, 5 or 7, so the cluster always has a majority (quorum) when one node is lost. Three is the minimum for Ceph.
CPU. Count 1 to 4 vCPUs per VM. One physical core gives two vCPUs with hyper-threading. Keep a few cores per node for Proxmox and Ceph, roughly one core per Ceph disk.
RAM. Count 1 to 4 GB per typical VM or container, plus about 1 GB per TB of Ceph storage on the node (in practice at least 4 GB per OSD), plus a few GB for the system.
Disks. SSDs (or NVMe) for Ceph if you can afford them. Ceph keeps three copies, so raw capacity must be about three times the usable space you need, plus room to recover from a failed node: plan to fill no more than 70 to 80%. Install Proxmox itself on a separate small mirrored pair.
Ceph does the replication. Give it plain disks through an HBA, or switch the RAID controller to HBA/IT mode. A RAID volume hides failing disks from Ceph and hurts performance.
Network. At least three separate networks, ideally on separate ports:
| Network | Speed | Carries |
|---|---|---|
| Management and cluster | 1 GbE | Web interface, SSH, corosync heartbeat |
| Ceph | 10/25 GbE or more | Replication between nodes |
| VM | 10 GbE | Traffic of the VMs, trunk with your campus VLANs |
Corosync, the cluster heartbeat, is sensitive to latency. Keep it off the Ceph links, which can saturate during recovery. A second corosync link on another network is good insurance.
In this guide the nodes are:
| Node | Management | Ceph |
|---|---|---|
| pve1 | 192.0.2.11/24 | 198.51.100.11/24 |
| pve2 | 192.0.2.12/24 | 198.51.100.12/24 |
| pve3 | 192.0.2.13/24 | 198.51.100.13/24 |
Prepare and install#
- In the BIOS or UEFI, enable virtualization (Intel VT-x / AMD-V, and VT-d / AMD-Vi).
- Put the Ceph disks in HBA mode, or remove them from any RAID volume.
- Download the Proxmox VE 9 ISO from proxmox.com, write
it to a USB stick, and install every node. Pick ZFS RAID1 on the two system disks, give each node
its management address and a fully qualified name (
pve1.example.ac.ma).
Proxmox VE 9 is based on Debian 13 "trixie". Without a subscription, switch each node to the no-subscription repository. The repository files use the deb822 format:
/etc/apt/sources.list.d/proxmox.sourcesTypes: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Disable the enterprise repository (set Enabled: no in pve-enterprise.sources), then update:
# apt update && apt full-upgrade
Make sure every node resolves the others by name (/etc/hosts or DNS) and keeps the time in sync
(chrony is installed by default).
Configure the networks#
Do this on each node, with that node's addresses. vmbr0 carries management, the second port
carries Ceph, and vmbr1 is a VLAN-aware bridge on the trunk to your campus switches.
/etc/network/interfaces (pve1)auto lo
iface lo inet loopback
# Management and cluster
auto vmbr0
iface vmbr0 inet static
address 192.0.2.11/24
gateway 192.0.2.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
# Ceph
auto ens1f0
iface ens1f0 inet static
address 198.51.100.11/24
mtu 9000
# VM networks: trunk with the campus VLANs, no IP address on the host
auto vmbr1
iface vmbr1 inet manual
bridge-ports ens1f1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
Apply with ifreload -a. With a VLAN-aware bridge you don't create one bridge per VLAN: each VM
network card gets its VLAN tag (for example tag=120) in its settings.
mtu 9000 on the Ceph network helps, but only if every switch port on that network allows it.
Test with ping -M do -s 8972 198.51.100.12 before relying on it.
Create the cluster#
On pve1:
# pvecm create campus-cluster --link0 192.0.2.11
On pve2 and pve3, join it (you'll be asked for pve1's root password):
# pvecm add 192.0.2.11 --link0 192.0.2.12
# pvecm add 192.0.2.11 --link0 192.0.2.13
Check that all three nodes are members and the cluster is quorate:
# pvecm status
Set up Ceph#
On every node, install the Ceph packages from the no-subscription repository:
# pveceph install --repository no-subscription
On pve1 only, write the initial configuration with the Ceph network:
# pveceph init --network 198.51.100.0/24
On every node, create a monitor and a manager. Three monitors keep Ceph's own quorum when one node is down:
# pveceph mon create
# pveceph mgr create
On every node, turn each data disk into an OSD (one disk, one OSD). Check the device names with
lsblk first: this wipes the disk.
# pveceph osd create /dev/sdb
# pveceph osd create /dev/sdc
When all OSDs are in, ceph -s should show HEALTH_OK and all OSDs up and in.
Storage for VM disks#
Create a replicated pool (3 copies, still writable with 2) and add it as Proxmox storage in one go:
# pveceph pool create vm-pool --size 3 --min_size 2 --add_storages
CephFS for ISO images and templates#
CephFS needs a metadata server. Create one on each node (one is active, the others stand by), then the file system:
# pveceph mds create
# pveceph fs create --name cephfs --add-storage
The new cephfs storage appears on all nodes. Upload ISO images and container templates there
once and every node can use them.
Make VMs highly available#
A VM on Ceph storage can run on any node. Add it to HA so the cluster restarts it on another node if its host fails:
# ha-manager add vm:100 --state started
Before planned maintenance on a node, move its VMs away with live migration, or use the node maintenance mode described in Put a Proxmox node into maintenance mode.
Check and operate#
| Check | Command |
|---|---|
| Cluster membership and quorum | pvecm status |
| Ceph health | ceph -s, ceph health detail |
| OSD layout and usage | ceph osd df tree |
| Pool usage | ceph df |
| HA state | ha-manager status |
A few habits keep the cluster healthy:
- Keep Ceph usage under about 75%, so it can rebuild the copies of a lost node.
- Upgrade one node at a time: migrate its VMs away, upgrade, reboot, wait for
HEALTH_OK, then the next. - Back up VMs with Proxmox Backup Server
or
vzdumpto storage outside the cluster. Ceph replication isn't a backup. - Watch the cluster: the Prometheus and Grafana guide covers the monitoring stack.
Further reading#
- Deploy Hyper-Converged Ceph Cluster, Proxmox wiki
- Proxmox VE administration guide
- Migrating from VMware to Proxmox on this site