How to configure ESXi to boot via Software iSCSI?

http://blogs.vmware.com/vsphere/2011/11/how-to-configure-esxi-to-boot-via-software-iscsi.html

Introduction

VMware introduced support for iSCSI back in the ESX 3.x days. However, ESX could only boot from an iSCSI LUN if a hardware iSCSI adapter was used. Hosts could not boot via VMware's iSCSI driver using a NIC with special iSCSI capabilities.

It quickly became clear that there was a need for booting via Software iSCSI. VMware's partners are developing blade chassis containing blade servers, storage and network interconnects in a single rack. The blades are typically disk-less, and in many cases have iSCSI storage. The requirement is to have the blade servers boot off of an iSCSI LUN using NICs with iSCSI capabilities, rather than using dedicated hardware iSCSI initiators.

In ESXi 4.1, VMware introduced support for booting the host from an iSCSI LUN via the Software iSCSI adapter. Note that support was introduced for ESXi only, and not classic ESX.

Check that the NIC is supported for iSCSI Boot

Much of the configuration for booting via Software iSCSI is done via the BIOS settings of the NICs and the host. Ensure that you are using a compatible NIC by checking the VMware HCL. This is important, but be aware. If you select a particular NIC and you see iSCSI as a feature, you might assume that you are good to go with using it to boot. This is not the case.

Too see if a particular NIC is supported for iSCSI boot, you need to set the I/O Device Type to Network (not iSCSI) and then check the foot notes. If the foot notes states that iBFT is supported, then this card may be used for Boot from iSCSI (I'll explain iBFT later). Yes, this is all rather cryptic and difficult to follow in my opinion. I'm going to see if I can get this changed internally to make it a little more intuitive.

Steps to configure BIOS for Software iSCSI Boot

Now that you have verified that your NIC is supported, lets move on to the configuration steps. First step is to go into the BIOS of the NIC and ensure that it is enabled for iSCSI Boot. Here is how one would do it on a HP DL series:

Similarly, here is how you would do this on a DELL PowerEdge R710:

The next step is to get into the NIC configuration. In my testing I used a Broadcom NetXtreme NIC, which comes with a boot agent. Broadcom’s Multi-Boot Agent (MBA) software utility enables a host to execute a boot process using images from remote servers, including iSCSI targets. You access the MBA by typing <Control>S during the boot sequence:

This takes us into the MBA Configuration Menu:

Select iSCSI as the boot protocol. The key sequence CTRL-K will allow you to access the iSCSI  Configuration settings. If iSCSI isn’t available as a boot protocol, it may mean that the iSCSI firmware has not been installed, or that iSCSI has not been enabled on the NIC. There are a number of different parameters to configure. The main menu lists all available parameters.

First select General Parameters. In this example, I am going to use static IP information, so I need to set Disabled for theTCP/IP parameters via DHCP and iSCSI parameters via DHCP parameters.

When doing the initial install, Boot to iSCSI Target must also be left Disabled. You will need to change it to Enabled for subsequent boots. I'll tell you when later on in the post. You should therefore end up with settings similar to the following:

Press <Esc> to exit the General Parameters Configuration Screen and then select Initiator Parameters. At the iSCSI Initiator Parameters Configuration screen, one would enter values for the IP Address, Subnet Mask, Default Gateway, Primary DNS, and Secondary DNS parameters as needed. If authentication is required then enter the CHAP ID (Challenge Handshake Authentication Protocol) and CHAP Secret parameters.

Press <Esc> to return to the Main Menu and then select the 1st Target Parameters. Enter values for the Target IP Address, Target name, and Login information. The iSCSI Name corresponds to the iSCSI initiator name to be used by the client system. If authentication is required then enter the CHAP ID and CHAP Secret parameters. Note also that the Boot LUN ID(which LUN on the target we will use) is also selected here.

Press <Esc> to return to the Main Menu and then press <Esc> again to display the Exit Configuration screen and then select Exit and Save the Configuration. That completes the BIOS configuration. We are now ready to install ESXi onto an iSCSI LUN via the software iSCSI initiator.

Steps to install ESXi onto an iSCSI LUN via Software iSCSI 

After configuring the MBA parameters in the Broadcom NIC, you can now go ahead with the ESXi installation. The install media for ESXi is placed in the CDROM as per normal. The next step is to ensure that Boot Controller/device order is set in the BIOS. For Broadcom cards, the NIC should be before the CDROM in the boot order.

When the host is powered on, the system BIOS loads a NIC's OptionROM code and starts executing. The NIC's OptionROM contains bootcode and iSCSI initiator firmware. The iSCSI initiator firmware establishes an iSCSI session with the target.

On boot, a successful login to the target should be observed before installation starts. In this example, the iSCSI LUN is on a NetApp Filer. If you get a failure at this point, you need to revisit the configuration steps done previously. Note that this screen doesn't appear for very long:

The installation now begins.

As part of the install process, what could best be described as a memory-only VMkernel is loaded. This needs to discover suitable LUNs for installation, one of which is the iSCSI LUN. However, for the VMkernel's iSCSI driver to communicate with the target, it needs the TCP/IP protocol to be setup. This is all done as part of one the start-up init script. The NIC's OptionROM is also responsible for handing-off the initiator and target configuration data to the VMkernel. The hand-off protocol is called iBFT (iSCSI Firmware Boot Table). Once the required networking is setup, an iSCSI session is established to the target configured in the iBFT and LUNs beneath the targets are discovered and registered with VMkernel SCSI stack (PSA).

If everything is successful during the initial install, you will be offered the iSCSI LUN as a destination for the ESXi image, similar to the following:

You can now complete the ESXi installation as normal.

Steps to Boot ESXi from an iSCSI LUN via Software iSCSI 

Once the install has been completed, a single iSCSI Configuration change is required in the iSCSI Configuration General Parameter. The change is to set the 'Boot to iSCSI target' to Enabled.

Now you can reboot the host and it should boot ESXi from the iSCSI LUN via the software iSCSI initiator.

Gotchas

1. Make sure your NIC is on the HCL for iSCSI boot. Remember to check the foot notes of the NIC.
2. Make sure that your device has a firmaware version that supports iSCSI boot.
3. Make sure that the iSCSI configuration settings for initiator and target are valid.
4. Check the login screen to make sure your initiator can login to the target.
5. Multipathing is not supported at boot, so ensure that the 1st target path is working.
6. If you make changes to the physical network, these must be reflected in the iBFT.
7. A new CLI command, esxcfg-swiscsi -b -q, displays the iBFT settings in the VMkernel.