Changing Mellanox ConnectX VPI Ports to Ethernet or InfiniBand in Linux

In our recent Mellanox ConnectX-5 VPI 100GbE and EDR IB Review, we showed a unique feature of the Mellanox VPI cards: they can run in InfiniBand or Ethernet modes. Indeed, one can have a single adapter and use either protocol which is handy when you have a server with limited PCIe slots, but a need to access both types of high-speed networks. We have been using Mellanox VPI cards since the ConnectX-3 generation and even had an old guide around how to Change Mellanox ConnectX-3 VPI Cards between Infiniband and Ethernet in Windows. During the process of an upcoming server review, we had the opportunity to do the same using ConnectX-5 and Linux. In this short guide, we are going to show you how to change a Mellanox ConnectX VPI port to either Ethernet or Infiniband in Linux.

Changing Mellanox VPI Ports Prerequisites

There are a few things you will need to get started.

First, you will need a VPI capable cards. Mellanox makes three main types of cards: Ethernet only, Infiniband only, and VPI cards capable of both. You need the VPI versions and you may need to check a model number against a spec sheet to ensure you have a capable part.

Next, you need Mellanox drivers and tools installed. Here we suggest, that even if your OS has native support for Mellanox networking drivers you should download and install current drivers.

This is also a Linux guide. As we showed in the above-linked Windows article, you can do the same in Windows via the GUI.

Now that those prerequisites are settled, the process is very quick and easy.

Changing Mellanox VPI Ports from Ethernet to InfiniBand

Swapping from Infiniband to Ethernet or back on a Mellanox ConnectX-5 VPI card is really simple. There are tools to help you do this, but we have a simple three-step process in the lab. First, we see what devices are installed.

In this case, we have two Mellanox ConnectX-5 VPI cards (CX556A’s) installed in the GPU server.

We can then select the port configuration. Here, 1 is for InfiniBand and 2 is for Ethernet.

The third step is to reboot. That is it. Here is a video with this in action:

We may pull this section out to be its own piece for others to reference later.

Final Words

This is an extremely useful feature if you need to access both 40GbE / FDR 56Gbps InfiniBand (ConnectX-3) or 100GbE / EDR 100Gbps InfiniBand (ConnectX-4 / ConnectX-5.) You can have one port set for each protocol and access both. This is one of the key reasons why we gave the Mellanox ConnectX-5 VPI card a coveted STH Editor’s Choice award in our recent review. Mellanox is the only vendor currently supporting both on the same card.