Resolving Cisco Switch & Router ‘DHCP Server Pool Exhausted-Empty’ Error – Client IP Assignment Failure

In previous articles, we showed how it is possible to configure a Cisco router or Catalyst switch to provide DHCP server services to network clients. Everything usually works without a problem, however there are times when the Cisco DHCP server stops assigning IP addresses and we need to look into the issue and resolve it as quickly as possible. System messages such as ‘POOL EXHAUSTED’, ‘ASSIGNMENT FAILURE’ & ‘address pool Guest-VLAN is empty’ provide some basic information, however further investigation is required to identify the real cause.

Small-sized networks usually have DHCP services configured on their Cisco router, while large-sized networks (with multiple VLANs) assign DHCP services to their backbone layer-3 switch (Catalyst 6500, 4500, 3750 etc). The good news is that configuration and debugging commands are identical for both Cisco Catalyst switches and Cisco routers.

Debugging DHCP Server On Cisco Catalyst Switch & Cisco Router

The first symptoms of DHCP server issues are users nagging that they cannot connect to the network because they haven’t got an IP address, and that’s where the fun begins.

Assuming no configuration changes have been made to the Cisco DHCP server, the best way to troubleshoot the problem is to enable debugging on the dhcp server.  The debug ip dhcp events & debug ip dhcp server packets are useful debugging commands that will help us identify what is happening:

The key information provided by our debugging is highlighted in bold. This information tells us that our address pool named Guest-WiFi-VLAN is the DHCP Pool where we have a problem because the pool is empty, which means the DHCP server has no more free IP addresses to assign to new clients.

The next step is to understand why there are no more free IP addresses. A common reason is that there are more clients in the specific VLAN requesting IP addresses, than what the DHCP server can assign. Let’s see if this is the case.

First, we take a look at the configured IP address range for that VLAN/Pool. Note that our Guest VLAN is assigned to VLAN7:

Let’s check and see how many clients have been assigned an IP address for VLAN7:

The output surprisingly shows us that we have only 3 clients to which IP addresses have been allocated. So the question now is where did all the rest of the 231 (234-3) IP addresses go?

Another useful command to check the DHCP pool usage is the show ip dhcp pool. It provides the overall usage of the pool alongside with the total addresses, leased and excluded addresses:

Here we can again confirm that from the 254 total IP addresses, 251 are excluded and 3 are leased. Note that the Excluded addresses includes the manually excluded and conflicted IP addresses.

The Current index column shows the next IP address that will be assigned by the DHCP server. Under normal operation, we would expect to see and IP address within the 192.168.7.0 network, however the value of 0.0.0.0 means that there are no more available IP addresses to lease.

The next step is to check the DHCP server for possible conflicts using the show ip dhcp conflict command – we are sure to find something here:

To save space, we had to remove the rest of the command’s output. Surprisingly enough, we found that all 231 IP addresses were listed in the dhcp conflict table.  As shown above, the IP addresses are listed by date of conflict with the older entries shown first.

Understanding The DHCP Conflict Table & Cisco DHCP Server Functionality

Before a Cisco DHCP server hands out an IP address to a client, it always ARPs and then pings the address to be handed out to make sure no one is using it.

When a Cisco DHCP server discovers a conflict, it will place the IP address into the conflict table stating the address was conflicting and how it came to that conclusion, as noted under the Detection method column.

If for any reason the client who is already using the IP address that is about to be handed out by the Cisco DHCP server, does not respond to the ping from the DHCP server, the DHCP server will lease out the IP address since it cannot identify any conflict issues.

The first thing the client will do once the IP address assignment is complete, is to send out a gratuitous ARP message with its new IP address. If not reply is received, then it is safe to assume no one else is using it. However if it does receive a gratuitous ARP reply, then it will indicate that another device on the network is already using that address.

Assuming a gratuitous ARP reply is received, the client will send a DECLINE message to the DHCP server, rejecting the IP address it was just assigned. Since Cisco DHCP server has seen two gratuitous ARP messages and discovered there is a conflict, it will move the IP address into its conflict table and assign the next available IP address to the client.

Clearing The IP DHCP Conflict Table

When the DHCP server detects there is a conflict of an IP address before or right after it is assigned to a client, it will automatically remove the IP address from the DHCP pool and move it to the DHCP conflict table.  The IP address in question will remain there until an administrator sees and clears the DHCP conflict table.

If DHCP conflicts are occurring frequently, it is only a matter of time until all available IP addresses are moved to the DHCP conflict table and the DHCP Pool is left empty with no IP addresses to hand out.

We can clear the DHCP conflict table by using the clear ip dhcp conflict * command. This will instruct the DHCP server to clear the conflict table and return all IP addresses to the DHCP Pool.  In case we have multiple VLANs and Pools, the command will affect them as well:

Issuing the show ip dhcp conflict command confirms that there are no more IP addresses in the table.

The show ip dhcp pool command will now show all previously conflicted IP addresses, available to be handed out to our clients.