CAT 6500 and 7600 Series Routers and Switches TCAM Allocation Adjustment Procedures

Introduction

This document describes how to adjust the routing Ternary Content Addressable Memory (TCAM) allocations on 3BXL modules for the Cisco Catalyst 6500 and 7600 Series routers and switches.

Background Information

On the Catalyst 6500 and 7600 Series platforms, all of the routing information is stored in special high-speed memory called TCAM. More specifically, the Catalyst 6500 and 7600 Series platforms have three different types of TCAM:

• Forwarding Information Base (FIB), or routing TCAM
• Access Control List (ACL) TCAM
• Netflow TCAM

When a route is programmed into the Cisco Express Forwarding (CEF) table in the main memory (RAM), a second copy of that route is stored in the hardware TCAM memory on the Supervisor as well as any Distributed Forwarding Card (DFC) modules on the linecards.

Problem

This document focuses on the FIB TCAM; however, the information in this document can also be used in order to resolve these error messages:

%MLSCEF-SP-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry
 usage is at 95% capacity for IPv4 unicast protocol

%MLSCEF-DFC4-7-FIB_EXCEPTION: FIB TCAM exception, Some
 entries will be software switched 

%MLSCEF-SP-7-FIB_EXCEPTION: FIB TCAM exception, Some
 entries will be software switched

It is important to be aware that the aforementioned errors are reported only once when the TCAM limit is exceeded. Even if the total number of routes withdraws below the configured limit, the switch remains in the exception state until it is cleared:

7600#show mls cef exception status
       Current IPv4 FIB exception state = TRUE
       Current IPv6 FIB exception state = FALSE
       Current MPLS FIB exception state = FALSE

As a result of this TCAM exception condition, connectivity is affected and might result in elevated CPU usage due to software switching.

On August 8th, 2014, the Classless Inter-Domain Routing (CIDR) Report, which provides statistics on the global Internet routing table, reported that the global Internet routing table had passed 512,000 routes.

Most platforms have more than enough TCAM space to support larger routing tables, but the default configurations might require adjustment. As the Internet routing table approaches 512,000 routes, it can cause the Catalyst 6500 and 7600 3BXL-based modules to exceed the default routing TCAM allocations.

Note: The VS-S2T-10G-XL supervisor engine and DFC4XL modules support 1,000,000 routes that are dynamically shared between IPv4 and IPv6 by default.

This table shows the Supervisor modules and DFCs that are affected by the continued growth of the IPv4 Internet routing table:

On the 3BXL Series modules, the default FIB TCAM space for the IPv4 routes is 512k routing entries. You can enter the show mls cef maximum-routes command in order to view this information:

7600#show mls cef max
FIB TCAM maximum routes :
=======================
Current :-
-------
 IPv4 + MPLS         - 512k (default)
 IPv6 + IP Multicast - 256k (default)

Note: Each IPv6 route consumes two TCAM entries. Thus, 256,000 IPv6 routes is equal to 512,000 IPv4 routes.

The FIB TCAM is a single block of memory that is shared between the IPv4 routes/Multiprotocol Label Switching (MPLS) labels and the IPv6 routes/Multicast routes.

The total memory space cannot be changed unless you replace the Supervisor and the DFC module, but you can adjust the amount of memory that is allocated to either the IPv4/MPLS or the IPv6/Multicast.

Solution

You can enter the mls cef maximum-routes ip <number in thousands> command in order to adjust the number of routing entries that are allocated to the IPv4. This does not increase the overall size of the FIB TCAM, but it reduces the number of routing entries that are allocated to the IPv6 in order to increase the amount of TCAM space for the IPv4.

It is very important to check the number of MPLS, IPv6, and Multicast routes present before you increase the allocation for IPv4 routes. Enter the show mls cef summary command in order to verify the total amount of routes per protocol:

7600# show mls cef summary 
Total routes:                     513525
    IPv4 unicast routes:          513507 
        IPv4 non-vrf routes:      513507 
        IPv4 vrf routes:          0 
    IPv4 Multicast routes:        3 
    MPLS routes:                  1
    IPv6 unicast routes:          5 
        IPv6 non-vrf routes:      5 
        IPv6 vrf routes:          0 
    IPv6 multicast routes:        3 
    EoM routes:                   1

7600(config)# mls cef maximum-routes ip 1000 
 Maximum routes set to 1024000. Configuration will be effective on reboot.

Note: This example reduces the total number of available MPLS labels, IPv6 routes, and IPv4 multicast routes to only 8,000. An increase in the size of the IPv4 TCAM space always reduces the amount of TCAM space that is available to other protocols. Consider the current and future MPLS, IPv6, and multicast needs of your network before you perform TCAM adjustments.

After you adjust the maximum-routes, you must save the running configuration and reboot the switch before the changes become active (there is no operational impact before the reboot). After the reboot, you can enter the show mls cef maximum-routes command in order to view the new TCAM allocations:

7600# show mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
 IPv4                - 1000k
 MPLS                - 8k (default)
 IPv6 + IP Multicast - 8k (default)

After the adjustment, the total FIB TCAM size is unchanged. The total number of MPLS labels for the IPv6 routes that can be used is reduced, which now allows for 1,000,000 IPv4 routes.