Problem w/ Dual NICs on server both on disparate Networks

Scenario:

Domino Server (HP G5 Windows Server 2003) with two NICs addressed 10.75.x.x and 10.59.59.x. The only Layer one connectivity between the two is the server and the SAN. 10.75 NIC in prodution email network, 10.59.59 NIC on backup network to SAN.

Issue:

We began seeing traffic (port 137 SMB queries or NetbiosIP) from the 10.59.59 network bleeding [(?) no routing evident, NICs NOT teamed] onto our production network. Since there is no evident (yet found) routing in place, this traffic gets directed to our default gateways. At times this traffic hits our gateway segments (internet and WAN) so hard it causes ICMP to be dropped producing floods of false positive 'down' conditions on our NMS boxes and subsequently alarms are generated. This condition occurs on a regular incremented schedule corresponding with back-up intervals. Remote sites across the WAN and VPN (internet) encounter poor performance on APPs served by my location during the events.

Commentary:

I don't administer the server only the routers and edge network appliances and it took me a long time to convince the administrators that it was happening. When I first pinned it down after 4 hours of monitoring and forensics (via affected VPN on Thanksgiving morning) they told me it was impossible. When I finally showed them .pcaps containing the rogue traffic and Edge appliance logs IDing the traffic flows, they still denied it (idiots) LOL! Now they don't want to own it nor do I think they know how to other than sitting on hold with HP support.

Questions:

Has anyone else encountered this issue and what was the solution?
If not, any suggestions?

No, I've never encountered this.

You're already on the right path of isolating the issue to being a specific NIC on that specific server, now you have to further isolate the specific service/process/software on that server that's bound to that NIC to identify how that service/process/software is possibly malfunctioning and producing this traffic.

I would look-up applications that map TCP/UDP connections to processes on the computer, such as TCPview (or simply netstat -b) and then further isolate the issue some more to figure out how it's malfunctioning.

As you already noticed, obscure and odd issues will require more research and analysis than usual issues and that's what discourages some admins from even beginning the work.

Sorry for not having a better answer/suggestion...I've learned that with some issues, you have to go back to the basics to understand the complex side of things.

Windows usually sends broadcast netbios-ns queries (UDP port 137) regularly. Specially when DNS queries fail to resolve a domain name (say www.nonexistant-4456c33453467-blablabla.com ) or when a DNS server is down. I think I'd consider that a normal behavior.

The odd thing is how did this broadcast traffic get passed your gateway router (Or didn't it ?). netbios-ns queries are typically sent to a broadcast IP address (say 192.168.0.255/24, MAC: FF:FF:FF:FF:FF:FF). As you know by default routers don't forward those. Unless you have a WINS server in your network and you have configured it on the Domino Server, in this case the netbios-ns queries will be headed to the WINS servers IP (unicast).

P.S. You can find the WINS settings on the same TCP/IP settings window when you click on the [Advanced] button.

I would be interested in seeing the type of traffic that is occuring ? i.e. a few of the packet fragments from the Wireshark capture would be handy to see source/destination IPs/Ports ? These could be directed NetBIOS traffic going to Public Routable addresses which i have seen before, these could be generated from the Nic on the SAN IP Address and then being routed through the Production Network since that would be the only NIC with a Default Gateway configured (or it should be, make sure you have not set the default gateway up on both NICs which is a common mistake when dual homing).