ARP for a remote subnet machine, eh !

Hi peeps,

Just wanted to check something by everyone to make sure i am not going mad. The way i understand ARP to work is that its to map an IP Address to a Layer 2 MAC address. This is only done for hosts on the same network and if packets need to be routed then an ARP for the next hop MAC address is done and then the packet is sent there for routing.


Picture for Example Purposes

So here is my problem.

I have just swapped a layer 3 switch for the pix firewall in the above example. Everything works fine with the layer 3 switch and all the routes and the ip addresses of the interface on the pix are the same as when the layer 3 switch is in place. The issue is that two of the subnets will not talk to each other.

172.16.0.1 talks to 172.16.1.1
172.16.0.1 talks to 172.16.2.1
172.16.1.1 talks to 172.16.2.1
172.16.2.1 talks to 172.16.3.1
172.16.1.1 will not talk to 172.16.3.1 - and it should be doing

(172.16.0.1 to 172.16.3.1 cannot talk anyway due to security)

So, we reboot some stuff and still nothing happens. Wait over night to hope that some MAC/ARP cache is going to time out, nothing. Try changing ip address on the Pix and Router 2 and changing default gateways accordingly and again still nothing.

Because R2 and beyond is managed by a 3rd party company and is totally out of our control, i need to prove that the fault is with them and not me installing the pix (a capture on the pix interface shows ICMP from 172.16.1.1 leaving the pix but no return ICMP traffic coming back, i am convinced its not our problem but still need to give more proof). So i decide to do a full capture of the segment between the Pix and Router 2.

What i see is that for some reason, Router 2 (between the pix and the router interface) is doing ARP requests directly for 172.16.1.1. i.e. you see a ARP from 10.10.2.1 for 172.16.1.1. I'm sorry but this is well confusing me ? This shouldn't be happening right ? It should be just forwarding to the Pix interface to be routed ?

Anyone any ideas what can be causing this ? The only thing that is left to try is a reboot in the wireless kit for that link but i am grasping at straws a bit.

Cheers

ARP is used in four cases of two hosts communicating:

1)When two hosts are on the same network and one desires to send a packet to the other

2)When two hosts are on different networks and must use a gateway/router to reach the other host

3)When a router needs to forward a packet for one host through another router

4)When a router needs to forward a packet from one host to the destination host on the same network

i am not sure what exactly is happening, but just to be sure again, check the default gateway of every node on the 172.16.1.1 /24 network. it's evident that within the same network, it's working fine, and the pix is able to route between the other networks. trouble starts when this 172.16.1.1/24 network starts to talk to 172.16.3.1/24 on the 10.10.2.0/24 network. check the interface table for routing data when it needs to reach the 172.16.3.1/24 under the 10.10.2.0/24 network.
it just might be that there's no entry of a wrong entry on the routing table for any packets which needs to reach the 172.16.3.1/24.
once you have made sure that outgoing interface is well withing the range of 10.10.2.0/24 for any packet which needs to reach 172.16.3.1/24, you can very well go ahead and talk with that 3rd party. but again on that issue, do bear in mind that networks 172.16.2.1/24 is talking with 172.16.3.1/24. so it still revolves around the issue of which outgoing interface should a packet originating from 172.16.1.1/24, take in order to reach 172.16.3.1/24 via 10.10.2.0/24?
hope this helps.
cheers

Hi Arani,

Thanks very much for the suggestion

I have already checked everything in the post, i have tried 4 times now to get the pix in place so pretty much exauhsted everthing. Just wanted to make sure i wasn't cracking up when i saw the arp traffic on the 10.10.2.0/24 for the remote network......

Another test i tried was to plug in a laptop on the 10.10.2.0/24 subnet, lets say 10.10.2.10. Ping from 192.168.3.1 to 10.10.2.10 works fine (still on our leg of that network). The 10.10.2.1 router 2(well its an ISA 2004 Server just routing, default gateway set to 10.10.2.2) just arps for hosts on 192.168.3.1. Its also at the other end of a wireless connection and everything from the wireless device in our building, onwards is controlled and managed by the 3rd party.

Its all very odd

Thanks for the post matey

hi,
analysed the situaton again. and in that case, yes there seems to be a problem on their end. looks like from interface 10.10.2.1 it cannot find the gateway. the interface 10.10.2.2 should have been the default gateway for any packet destined for the subnet 172.16.3.1/24. outbound from this subnet seems to work fine as you said you were able to ping that laptop. looks like a link down or loss of table info on outbound interface at 10.10.2.2 end. hence it cannot find the interface it needs to forward packets which have 172.16.3.1/24 as destination. no wonder it keeps on sending out ARP packets to try and find out where it can locate that subnet.
i think you should find out more about the setup of that wireless network. 3rd party never worked out well for maintenance
cheers

Hi Arani,

I would have to agree, lol The very strange thing though is that packets from 172.16.3.1 to 172.16.2.1 do work fine in both directions. R2 (which is an ISA Server) has its default gateway set to 10.10.2.2, its very very strange and i am looking forward to finding out whats happening.

I suspect that the fix will be a reboot of the wireless devices, i have heard of some strange things happening with the wireless kit before

yes, maybe if you reboot the wireless router, then it will learn the network again, and rebuild it's routing tables. this might do away with this loss of link information for the 172.16.3.1/24 network. let us know how it goes. i have just completed a mini project for my company to link up two remote sites. hence my absence from the forums. i would love to learn from your experience, and this can help us both add to our repertoire of knowledge