By default, Proxmox networking uses the Linux bridge. There is no issue with the Linux bridges and it actually works really well. I have been using my PVE for two years now with Linux bridges. I recently switched to OVS for several reasons.
- OVS can be configured as layer 2 or layer 3 bridges. Linux bridge, as far as I know, is just layer 2
- I want to simplify my bridge setup. With OVS, I only have one bridge and use the VLAN tag. With Linux bridge, I would have one bridge per VLAN
- OVS supports more protocols such as VxLAN, GRE, STT, LLDP, etc. Linux bridge, on the other hand, does support some protocols, but not as much as OVS
- As a network guy and in my personal opinion, I think, OVS has a cleaner logic than the Linux bridge
Figure 1 shows the Network settings when creating a VM.
Somehow the Open vSwitch is not installed by default at least from Proxmox version 6 and below. I kind of hope it would come with it already. To install the Open vSwitch, use the web UI shell or SSH into the Proxmox server. In the web UI, there are a couple of ways to get to the shell. See Figure 2 as a reference.
- Click on the PVE node then click on the Shell button on the upper right corner right below the Create VM button
- navigate to PVE node > Shell
If we don’t install the openvswitch-switch, the system will get an error message Open VSwitch is not installed (need package 'openvswitch-switch') (500).
Therefore, once access to the shell has been established, enter the command below to install Open vSwitch:
apt install -y openvswitch-switch
Don’t forget to make a backup of the current interfaces just in case something goes wrong. While you are in the shell, enter the command below to backup the original config file.
cp /etc/network/interfaces /etc/network/interfaces.bak
After installing the Open vSwitch and making a backup of the /etc/network/interfaces
, we would need to delete the default Linux bridge. To do so, navigate to :
- Data Center > PVE Node > System > Network
- Select the vmbr0
- Click on Remove
Once the Linux bridge vmbr0 has been removed, we need to create OVS bridge
- Data Center > PVE Node > System > Network
- Click on Create
- Select the OVS Bridge
- Enter the physical interface name in the Bridge ports field
- Click Create
After creating a bridge, we would need to create an in-band interface to access the Proxmox VE server. We could have also done this while creating the OVS bridge by filling out the IPv4/CIDR and Gateway (IPv4). However, I wanted to make my config clean for me, so in my case, I am going to create an OVS IntPort for in-band management.
- Data Center > PVE Node > System > Network
- Click on Create
- Select OVS IntPort
- Enter a name for this inband IntPort in the Name field
- Enter the IP address and prefix in the IPv4/CIDR field
- Enter the gateway address in the Gateway field
- Click on Create
For VLAN tags, create another OVS IntPort for each VLAN ID.
- Data Center > PVE Node > System > Network
- Click on Create
- Select OVS IntPort
- Enter a name for this IntPort in the Name field
- Enter the VLAN ID in the VLAN Tag field
- Click on Create
- Repeat steps 1 through 2.3 for each VLAN
In addition, every OVS IntPort we create will be automatically be added to the OVS Bridge. Once done, reboot the Proxmox server for the network changes to become active. Now, that all the VLAN tags are active, assigning the VM or LXC to specific VLAN can be easily done by just specifying the VLAN ID in the Networking tab in VM or LXC during their creation as shown in Figure 1
When creating a VM or LXC, entering a VLAN ID in the VLAN Tag field the OVS will behave like an access port. Leaving the VLAN Tag field empty, OVS will behave like a trunk port; therefore, the guest VM can be configured VLAN tagging within the VM itself. In addition to leaving the VLAN Tag empty, if the guest VM is not tagging its interface, it would use the untagged or native VLAN.
I hope you will this helpful. Cheers!
This was very helpful because I knew that I needed to switch from Linux Bridges to OVS, but what not ready to make the plunge. Using your tutorial, I make the jump and on paper, everything looks good. I am getting ready to spin up pfSense with all of my VLANs and I am wondering how to trunk from OVS to my Cisco router. I reinstalled PM so that the management interface now has a 10.0.0.0/24, along with my switch with a primary 10.0.0.0/24 address. The reason for this is because on my pfSense box, I am using 192.168.0.0/24 for… Read more »
I don’t have a Cisco switch handy but off the top of head; On the Cisco switch, the basic the Cisco trunk link is: ! interface g0/0 switchport switchport mode trunk switchport trunk native vlan 10 switchport trunk allow vlan all exit ! The native vlan is the key here. This is where your Proxmox inband management UI and SSH access will be. On the Proxmox side of things, check Figure 5, for inband access. Check Figure 6, for each VLAN tagged. At this point, your Proxmox is trunking. So when you create a VM/CT, under the Network tab, you… Read more »
I seem to have an issue with making the inband. Should the IP address and gateway be the same as the host, or should I have used another IP address and gateway??? Every time I try to add an IP address on the VLAN, it gives me a fail with an error saying the address conflicts with the inband. Proxmox Server: 192.168.10.250 gw 192.168.10.254 inband: 192.168.10.250 gw 192.168.10.254 I have not be able to configure these, but it should be like this vlan10: 192.168.10.1 gw 192.168.10.254 vlan20: 192.168.20.1 gw 192.168.20.1 vlan30: 192.168.30.1 gw 192.168.20.1 etc. Now should I be using… Read more »
Should the IP address and gateway be the same as the host, or should I have used another IP address and gateway??? The host’s IP address should be unique. This also applies to your VM/CT or any other networks. You cannot have duplicate IP addresses. Proxmox Server: 192.168.10.250 gw 192.168.10.254 inband: 192.168.10.250 gw 192.168.10.254 What is the difference between the Proxmox Server and the inband? I am a little confused about what you are trying to do here. The inband is literally the IP address of your server/host/hypervisor (not the VM/CT). The VM and CT are the guests. I have… Read more »
In one fell swoop, you clarified many things for me. I always assumed that when I installed proxmox, I just set the IP address as a host in my management subnet and never gave it any thought. If I am reading your message clearly, I should have set the host IP address to (for instance) 10.0.0.250/24 and under ovs create the inband to 192.168.10.250 (which would fall into my MGMT VLAN once I spin up pfsense. Furthermore, switch the Cisco back to a L3 and also create the VLAN02, which will be attached to port 10 which will connect the… Read more »
Actually, re-reading your above answer, I got some clarity.
“On PVE Network | Your L3 switch or router
vlan10: no ip addr| gw 192.168.10.254
vlan20: no ip addr| gw 192.168.20.1
vlan30: no ip addr| gw 192.168.30.1”
Now it makes sense… So then I just setup pfsense with the interfaces and vlans and it will pass to ovs which will pass it on to the Cisco switch.
If I am reading your message clearly, I should have set the host IP address to (for instance) 10.0.0.250/24 and under ovs create the inband to 192.168.10.250 (which would fall into my MGMT VLAN once I spin up pfsense. I don’t know where you are getting the 10.0.0.250/24. You are the only one who can answer that since it is your network. I would highly recommend for you to look at this playlist from YouTube https://www.youtube.com/playlist?list=PL6F144CF03CB2381B. It may clarify some networking questions that you may have. Last question. You asked if the laptop could handle the 802.1q? I would have… Read more »
Also, should I have ip routing on or off? switch-01(config)# ip routing Also, I was trying to figure out my interface name to make sure that I used your example above: ” switchport trunk allowed vlan add 20,30,40 switchport trunk native vlan 10 ! interface gigabitethernet10 description WAN switchport mode access switchport access vlan 2 ! exit ” This is my current config: set system mode router file SSD indicator encrypted @ ssd-control-start ssd config ssd file passphrase control unrestricted no ssd file integrity control ssd-control-end cb0a3fdb1f3a1af4e4430033719968c0 ! vlan database vlan 2,10,20,30,40,50,60,70,80 exit voice vlan oui-table add 0001e3 Siemens_AG_phone________ voice… Read more »
Also, should I have ip routing on or off? I am not familiar with the switch you have, but the ip routing enables the layer3 capability on the Cisco switch. Therefore, you would not need the ip default-gateway command, but you would need a default route via ip route command. Anyways, the ip routing command will allow your L3 switch to do routing between networks. So if you have multiple VLANs and the gateway is the Cisco switch then the switch can route between these VLANs by default. However, for simplicity sake, if you want to segregate traffic between VLANs… Read more »
First off, I am using a Cisco SG-300 switch. It would seem that some of the IOS commands have been removed or have a change in syntax. This as been an ordeal in trying to get this to work. I was working with one of the old Netgate pfSense APU unit which had three interfaces and more or less worked. The WAN interface on the pfSense plugged into the DSL gateway; The LAN interface went to the Cisco, but since I really didn’t make use of a LAN, it was no big bust; And the OPT interface also went into… Read more »
What IP do you have on the OPNSense for VLAN10?
It is hard to troubleshoot the problem without some kind of diagram. Can you draw your topology?
The reason you are getting the Cisco web UI via 192.168.10.1 is that IP exist on VLAN10 of your Cisco switch at least according to the config you have provided from your previous reply.
Your Cisco switch config for VLAN2 and gigabitethernet10 seem to be correct. Therefore, your Cisco switch should get a public IP from your modem.
I have a couple of network diagrams in pdf format… How to do I get a copy to you?
Can you screenshot the drawings and put it on something like https://imgur.com/ and link me to it?
Ok, give this a try:
https://imgur.com/FP5gtQU
https://imgur.com/G053eWT
The switch config in the 2nd drawing does not match the switch config you provided from your previous comment. According to your 2nd drawing note, port 7 is trunk; however, port 7 from your previous comment was an access port. Anyways, who is the gateway for your pfsense, is it the Cisco switch or the Arris box? The steps below will make the pfsense your firewall/router and everything will be protected by the pfsense. Do not migrate youra VLANs gateway if you do not want to do this. Make a config backup of the Cisco switch, just in case you… Read more »
I am about to give this a shot. There is one issue that I am not clear on. Wouldn’t I see need to create the VLAN 20, 30, and 40 on the switch because of the need to have those VLANs available for the WAP (which I have setup with VLAN10 up and VLAN20, VLAN30 and VLAN40 as tagged?
If your management is vlan 10 keep this untagged and the rest tagged.
Also, and I am sure that I am kicking a dead horse, but… The host IP address for the proxmox host is 192.168.10.250. Should I set the IntPort for the management port to be 192.168.10.251???
You can always try it. I have my pve node IP on the IntPort as I described in step 11.