C7200-IO-2FE: hardware address filter problem leading to routing loops

[spiritusasper]

Hi everyone

I am having a strange problem with local routing loops among 3 or more routers attached to a hub. Apologies for the long-winded post, I've tried to structure it to make it easier to read.

Regards,
Yorgos

Summary:

It seems that the emulated ethernet hardware incorrectly behaves promiscuously. This behavior only shows up when using a hub, as a switch would mask it.

Topology:

R1, R2, R3, all attached via fa0/0 to a hub. Addressing is 192.168.0.{1,2,3} for R{1,2,3} respectively. MAC addresses as set by default in GNS3 (unicast, locally administered). Nothing fancy.

Configuration:

• GNS3: 1.3.13
• Dynamips: version 0.2.12-amd64/Darwin unstable
• IOS: c7200-adventerprisek9-mz.152-4.M9.image
• Emulated router: C7200, NPE-400, VXR, C7200-IO-2FE

Problem:

When I ping R2 from R1 using ping 192.168.0.2 rep 1 (ie only a single packet), I a get a gigantic flood of packets among all three routers. Looking at the packet trace, I see what is happening (where M1 is R1's MAC address, and so on):

• R1 sends an ICMP echo request with source R1,M1 to destination R2,M2 [correct]
• R2 sends an ICMP echo reply with source R2,M2 to destination R1,M1 [correct]
• R3 forwards R1's original request, with source R1,M3, to destination R2,M2 [incorrect]
• R2 replies to this packet with source R2,M2 to destination R1,M1 [correct]
• R3 forwards this reply with source R2,M3 to R1,M1 [incorrect]
• R1 forwards its own original request with source R1,M1 to R2,M2 [incorrect]

And then all hell breaks loose, as the three routers play a three-way ping-pong amongst themselves, until TTLs start getting exceeded, and then another round of three-way ping-pong starts, as they go on forwarding their ICMP TTL-exceeded messages to one another. End result: from a single ICMP echo request, I got a flood of 153000 packets, and that's only where the maximum capture file size was reached.

Note: when running the above, I had ICMP redirect disabled, in order to simplify the problem. With redirects enabled, I get an even worse problem as redirects are added to the mix.

The lines marked [incorrect] show what I consider incorrect behavior: the router forwarding code is receiving and processing (forwarding) a packet with a unicast destination MAC address not its own. The error seems to be with the emulated ethernet hardware (or the interface between it and the software), since otherwise IOS behavior is correct: the packet should have been filtered at the hardware level.

Tests:

1. Problem occurs when there are more than two routers on the LAN; obviously, the more routers the worse the problem is.
2. If I disable IP forwarding (no ip routing) on all three routers, the looping problem no longer occurs, but Dynamips still shows incorrect behavior: debug ip packet with CEF disabled on R3, shows it is still receiving packets with the wrong MAC address.
3. If I disable IP forwarding on R1 and R2, but leave it enabled on R3, the problem is reduced: R1 sends one request, R2 gets two copies (one from R1 and one from R3) and sends two replies, R1 receives 4 replies (two from R2, and two from R3); the loop does not escalate further.
4. Here it gets interesting: if I enable ip pim dense-mode on interface fa0/0 on R3, the problem goes away completely. In fact, it seems that putting the ethernet hardware in multicast mode prevents the problem from occurring.

Here is what show controllers fa0/0 shows:

HW addr filter: 0x68EB3394, ISL disabled, Promiscuous mode disabled
Entry= 0:  Addr=CA03.4B0C.0008
(All other entries are empty)

And with multicast enabled:

HW addr filter: 0x68EB3394, ISL disabled, Promiscuous mode multicast
Entry= 0:  Addr=CA03.4B0C.0008
(All other entries are empty)

In other words, IOS reports the hardware as functioning correctly; when in multicast mode, that is indeed the case; when in unicast mode, IOS claims that promiscuous mode is disabled, but in fact the hardware seems to be in promiscuous mode.

[spiritusasper]

Following up on my own post: I did the same test with three other FastEthernet port types:

• C7200-IO-FE
• PA-FE-TX
• PA-2FE-TX

The problem only seems to occur with the two-port cards (C7200-IO-2FE and PA-2FE-TX). The single-port cards do not have the problem. Looking again at show controllers fastethernet, I see a consistent difference between the single-port and dual-port interfaces:

Single port (C7200-IO-FE, PA-FE-TX):

HW addr filter: 0xE1A3DA0, ISL Disabled
Entry= 0:  Addr=0100.0CCC.CCCC
Entry= 1:  Addr=0180.C200.0002
Entry= 2:  Addr=FFFF.FFFF.FFFF
Entry= 3:  Addr=FFFF.FFFF.FFFF
Entry= 4:  Addr=FFFF.FFFF.FFFF
Entry= 5:  Addr=FFFF.FFFF.FFFF
Entry= 6:  Addr=FFFF.FFFF.FFFF
Entry= 7:  Addr=FFFF.FFFF.FFFF
Entry= 8:  Addr=FFFF.FFFF.FFFF
Entry= 9:  Addr=FFFF.FFFF.FFFF
Entry=10:  Addr=FFFF.FFFF.FFFF
Entry=11:  Addr=FFFF.FFFF.FFFF
Entry=12:  Addr=FFFF.FFFF.FFFF
Entry=13:  Addr=FFFF.FFFF.FFFF
Entry=14:  Addr=FFFF.FFFF.FFFF
Entry=15:  Addr=CA03.5AF3.0000

Dual port (C7200-IO-2FE, PA-2FE-TX):

HW addr filter: 0x68EC7968, ISL disabled, Promiscuous mode disabled
Entry= 0:  Addr=CA03.5AF3.0038
(All other entries are empty)

Perhaps the problem is that the hardware filter entries are empty?

[julien-duponchelle]

Thanks for the report. I don't have the knowledge to check. Perhaps @grossmj can check this.

[grossmj]

Excellent report, maybe it is not much to fix but unfortunately we do not have anyone who has the necessary knowledge to do it.

[flaviojs]

Different ethernet chips are being emulated:

• dev_dec21140_init -> C7200-IO-FE, PA-FE-TX
• dev_i8254x_init -> C7200-IO-2FE, PA-2FE-TX, C7200-IO-GE-E, PA-GE
• dev_am79c971_init -> PA-4E, PA-8E
• dev_mv64460_init -> NPE-G2
• ...

The dual port ones use dev_i8254x, so the bug should be there.

[flaviojs]

I failed to reproduce this issue with gns 2.2.46 and a C7200 124-11.T1 image.

Maybe the ethernet hub that gns3 1.3.13 used was different from the current one?
Or maybe the current master (951f1f4) already has the root cause fixed?