College of mine reminded me today, that one does not want tcpdump with the NIC in promiscuous mode (the default for tcpdump, turn it off with -p), when debugging problems. And really, there is no other use for tcpdump, than debugging problems. (You don't sniff for fun, do you? And I'd want to use Wireshark in that case anyway.)

So, why is promiscuous mode a bad idea?
Because tcpdump will show you a very different truth - it will show you what's on the wire, but not what ethernet packets your machine really accepts under normal conditions - it will only accept packets which destination address is set to the machines ethernet address (plus some multicast stuff, but I'm usually not interested in those). This will especially get you in trouble, if you rely on the IP adresses in tcpdumps output to determine if "this packet is for me". You will fool yourself into thinking, "oh these packets all arrive here", and all your further conclusions from this point forward are wrong.

Typical situation for this:

• Machine M is connected to Router R
• Machine M has got more than one IP address, but only the first one is directly bound to the interface
• You swap the ethernet card in machine M (or migrate the whole machine to new hardware, probably more common these days)
• IP connectivity works, but the secondary IPs are not reachable
• because router R caches the ethernet address for all IPs
• only the primary/first IP got updated in the routers ARP cache (= IP address to ethernet address cache)
  
• tcpdump will show you different truths:
• without ethernet address display turned on, plus NIC in promiscuous mode:
• shows that everything is fine
• NIC not in promiscous mode:
• would have shown you that the packets don't arrive ...
  

I failed to recognize this problem two times by now, first time this got us quite some outage time, and Yesterday I saw the symptoms, got the feeling that again the ARP cache is acting up, finally resolved the issue, but I hadn't proof that this was the problem. And I could have had... if I'd used non-promisc mode or turned on ethernet address display in tcpdump.