Cisco Catalyst 4500 Series Zero-Downtime IOS Upgrade Process for Supervisor Engine 7-E, 7L-E, 6L-E and V-10GE Redundant Configurations

The Supervisor Engine is the heart of the Cisco Catalyst 4500 and 6500 series switches. It is the equivalent of an engine in a car. Every packet that enters or exits your switch passes through the Supervisor Engine and, naturally, the Cisco IOS is the software that brings this beast to life.

Cisco makes an enormous effort to bring new features to its products through new versions of its popular IOS software, especially for the enterprise-class series switches such as the 4500 and 6500 Catalyst switches. Unlike other vendor operating systems, it is always advisable to keep your Cisco Catalyst operating system up to date by fitting it with the latest IOS image.

To be eligible to download an IOS image from Cisco’s website, one must have a valid contract support with Cisco Systems. This contract not only makes you eligible for all software upgrades for the duration of your contract, but also binds Cisco in delivering Top-Class support for your covered device(s).

Note: Information on installing the Cisco 4507R-E/4507R+E Catalyst Switch, Supervisor Engines, Line cards and power supplies can be found in our following articles which, combined, contain over 30 pictures of all switch components, including the passive backplane, fantray and more:

• Cisco 4507R+E Layer 3 Installation: Redundant WS-X45-SUP7L-E Supervisor Engines & WS-X4648-RJ45V+E Line Card

• Installation of a Cisco Catalyst 4507R-E Layer 3 Switch
  

Are Network Services Affected During Or After The Catalyst Supervisor Engine(s) Upgrade?

This is perhaps one of the most frequently asked question that troubles engineers, administrators and IT Managers who have not dealt with the Supervisor Engine upgrade process before.  Who wouldn’t be? This is, after all, the network backbone switch and when it comes to large campus, Enterprise networks or networks that operate around the clock, downtime is not an option.

At this point, we should note that when upgrading a Supervisor Engine the engine must be reloaded (rebooted) in order for the system to load the new IOS image. For the Cisco Catalyst 4503 and 4506 series, which can only accept a single Supervisor Engine, this means network service interruption is unavoidable as the single Supervisor Engine must reboot. On the other hand, the Cisco Catalyst 4507R and 4510R series are capable of accepting up to two Supervisor Engines (hence the ‘R’ – Redundancy), therefore in case of a 4507R/4510R with two Supervisor Engines installed, IOS upgrades can be performed with guaranteed Zero-Service-Interruption. 

Note: The process we are about to describe was performed on a live 4507R+E with two Supervisor Engines 7L-E, on a network of 120+ users and 11 servers connected to our 4507R+E via dual 10Gbps fiber optic links (one 10Gbps link on each Supervisor Engine).

Quick Overview Of The Supervisor Engine Upgrade Process

Before we dive into the upgrade process, let’s take a quick look at the steps to be followed. This will help understand the process and caveats of each step.

When upgrading a system with redundant Supervisor Engines the upgrade process has to be performed in a specific way as each Supervisor Engine is upgraded in turn.  

Following is a brief overview of the upgrade steps:

• Load the new IOS image on to the Active Supervisor Engine (SE1)
• Copy IOS image to Standby Supervisor Engine (SE2)
• Configure Supervisor Engines to load the new image upon reboot
• Set Configuration-Register variable to ensure newest image is loaded upon bootup
• Force reload of Standby Supervisor Engine (SE2) & Switchover to Standby Supervisor Engine (SE2). This now becomes the new Active Supervisor Engine
• Force reload of previously Active Supervisor Engine (SE1).

Step 1: Loading The New IOS Image On To The Active Supervisor Engine (SE1)

The first step is to copy the new IOS image on to the active Supervisor Engine (SE1). For this, a TFTP server is required for the file transfer. Users can download a selection of Free TFTP Servers from our FTP/TFTP Servers & Clients download section.

Once the TFTP server is ready, we issue the necessary command to initiate the file transfer:

Note that the new IOS image is saved to the bootflash: file system.  Cisco Supervisor Engines use the bootflash: file system rather than the flash: file system which most of us are used to.

If in doubt, simply make use of the show file system command that will reveal the file systems on your Catalyst switch:

Step 2: Copy IOS Image To Standby Supervisor Engine (SE2)

Once the IOS image is loaded on to the active Supervisor Engine (SE1), it must be copied to the standby Supervisor Engine (SE2). For this, we use the copy bootflash: slavebootflash: command. Note that the slavebootflash: file system refers to the bootflash: of the standby Supervisor Engine, regardless of which physical engine is on standby mode.

To verify the image is correctly loaded on both Supervisor Engine bootflash systems, use the show bootflash & show slavebootflash commands:

Note:  Notice that the first IOS image listed is the previous version (03.03.00.SG.151-1.SG). We’ll need to keep this information in mind for our next step.

Step 3: Configure Supervisor Engines To Load The New Image Upon Reboot

In our next step, we configure the active Supervisor Engine to load the new IOS image when it reboots. This is easily done using the boot system flash command as shown below:

It is highly likely the boot system flash bootflash command already exists in the system’s configuration for the previous IOS image, so we’ll need to remove the command from the configuration to ensure the newest IOS image we just uploaded (03.04.00.SG.151-2.SG) is the only one referenced:

Next, we must ensure the configuration is saved to the startup-configuration:

Readers wondering about the % VRF table-id 0 not activeCompressed message when saving the running-configuration should not be alarmed as this is a cosmetic bug and can be safely ignored. If no such message was reported when saving the configuration, it means that the IOS currently running does not have this cosmetic bug.

On another note, every time we save our configuration to startup-config the system will immediately synchronize the configuration changes to the standby supervisor. When this happens, we’ll receive a message, similar to the one show above, confirming the synchronization has been successful.

Step 4: Set Configuration-Register Variable To Ensure Newest Image Is Loaded Upon Bootup

Cisco Supervisor Engines, by factory default, have their configuration register set to 0x2101. While this value is a combination of settings, we will focus on two specific values: 0x2101 & 0x2102.  The value 0x2101 instructs the system to boot the first system image in the onboard flash memory (bootflash). This is usually the oldest image in the flash. The value of 0x2102 instructs the system to use the image specified in the BOOT environment variable, which is essentially whatever was specified in the previous step (No.3) using the boot system flash bootflash: command.

To view the environment variables on both Supervisor Engines, use the show bootvar command:

Notice how the newly uploaded IOS image cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin is set for the BOOT variable, however the Configuration register value is 0x2101, which means it will force the Supervisor Engine to boot the first image it is going to find on the bootflash. Which image is that?  Let’s refresh our memory:

As we can see, despite the boot system flash command being correctly set to load the second IOS image, the configuration register setting of 0x2101 will force the Supervisor Engine to boot the first image found, that is: cat4500e-universal.SPA.03.03.00.SG.151-1.SG.bin.

To overcome this problem, we have two options: 

1) Delete all older IOS images from the Bootflash 

2) Set the configuration register to 0x2102. 

We decided to set the configuration register just to be on the safe side:

Take note that the system is confirming that both configuration register and startup-config have been successfully synchronized with the standby supervisor.

…and there’s that cosmetic bug again :)

Step 5: Force Reload Of Standby Supervisor (SE2) & Switchover To Standby Supervisor Engine (SE2)

At this point we are ready to force the standby Supervisor Engine(SE2) to reload. Once this happens, the Supervisor Engine (SE2) will load the new IOS. Once we confirm the new IOS is loaded, we can then make the standby Supervisor Engine (SE2) the active Supervisor Engine.

To force the reload of the standby Supervisor Engine (SE2), use the redundancy reload peer command:

The second (standby) Supervisor Engine is now restarting. This process will take a couple of minutes and will have no negative impact on the 4507R switch.

Once the IOS has loaded and the restart process is complete we will receive a message similar to the following:

*Mar 21 01:24:54.312: %C4K_REDUNDANCY-6-DUPLEX_MODE: The peer Supervisor has been detected
*Mar 21 01:25:36.066: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 4 (WS-X45-SUP7L-E S/N: CAT1714L4T4 Hw: 1.1) is online
*Mar 21 01:25:36.094: %C4K_REDUNDANCY-6-MODE: ACTIVE supervisor initializing for sso mode
*Mar 21 01:25:36.344: %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been established
*Mar 21 01:25:37.098: %C4K_REDUNDANCY-6-MODE: ACTIVE supervisor initializing for sso mode

The above messages indicate that the standby Supervisor Engine has successfully completed its reboot and is fully synchronized with the active Supervisor Engine. The system shows the product ID (WS-X45-SUP7L-E), serial number and hardware revision of the Supervisor Engine that has just established communication with the active Supervisor Engine.

To verify that the standby Supervisor Engine is running the latest and greatest IOS image we just loaded, use the show module command:

The show module command provides a generous amount of information including all line cards installed, Supervisor Engine models, IOS software versions, serial numbers, operational status and redundancy mode (SSO by default).

We’ve highlighted our standby Supervisor Engine which happens to be in slot No.4. Notice the software image the system is reporting for both Supervisor Engines. Our current active Supervisor Engine is running version 03.03.00.SG, while our standby Supervisor Engine is running version 03.04.00.SG – the new IOS loaded!

We’ve now confirmed that the standby Supervisor Engine has loaded the new IOS and is fully operational, waiting eagerly to manage our 4507R switch and all its data!!

Step 6: Force Reload Of Previously Active Supervisor Engine (SE1)

On our final step, we will need to force the reload of the active Supervisor Engine (SE1) so it too can boot using the new IOS image. Forcing a Supervisor Engine switch over is an experience when you consider what’s really happening inside the switch at that moment. During the process of the switchover we don’t get LEDs lighting up, flashing like crazy. The SUP ACTIVE LED will simply switch off from the current active Supervisor Engine and switch on on our previously standby Supervisor Engine, indicating it is now the new active engine.

To initiate the Supervisor Engine switchover, use the redundancy force-switchover command as shown below:

Once the command is entered and we’ve confirmed by hitting ENTER, we will lose our telnet session to the 4507R. This is normal expected behaviour – do not be alarmed!  Apart from the switch cutting our telnet session, users will not notice any service disruption – the switchover will be completely transparent to them no matter the network load during the switchover.

To reconnect to the switch, simply telnet back into the same IP address. If connected via console cable it will be necessary to connect it to the new active supervisor engine in order to continue controlling the switch.

Engineers who would like to monitor, via telnet or direct console cable connection (to the new active engine), the reload progress of the Supervisor Engine can use the show module command. By typing the command we will see the 4507R identifying a Supervisor Engine in slot 3 (that’s SE1 that is reloading), however, further down we will see that no information about the engine’s MAC address or IOS software is provided due to the fact that it has not fully booted into its new IOS. In addition the redundancy status of the first engine is Disabled – an expected result since the Supervisor Engine (SE1) has not booted yet.

At this point, we can continue to issue the show module command and monitor the output changes. At some point the system will show the Supervisor Engine (SE1) loaded with the new IOS, however, the redundancy status will go through the following phases until it is ready (Standby hot):

Standby Supervisor Redundancy Status Cycle:

1. Disabled
2. In progress to Standby cold
3. Standby cold
4. In progress to Issu negotiation la
5. In progress to Standby config
6. In progress to Standby bulk  
7. Standby hot

When the Supervisor Engine reaches Standby hot status, it is ready to take over in the event the active Supervisor Engine fails.

Following is the expected output when the Supervisor Engine IOS is loaded and fully synced with the active Supervisor Engine:

Summary

This article explained the importance of upgrading the Cisco Supervisor Engine IOS to the latest version, the process that needs to be followed for the upgrade and how the Catalyst switch’s operations are affected during or after the upgrade. We also covered the IOS upgrade process for redundant Supervisor Engine configurations, how to force a Supervisor Engine switchover and monitoring the Supervisor Engine bootup and sync process.

More articles covering Cisco Catalyst switches and LAN switching technologies can be found in our Cisco Knowledgebase Switching category.