Here we go. I found the problem.
First the firewall on my side had a Source NAT rule that said any smtp traffic on port 25 was to be forwarded out the public IP. This was taking what was suppose to be internal traffic and blowing it out onto the net. We all know that its not going very far from there.
So I removed the NAT rule which didnt seem to be serving much of a purpose anyways and I was then able to telnet to the server.
Now step 2, the smptdiag utility was resolving the mail server to the wrong IP, not much mail going to go through in that setup. So I went to the remote server's DNS configuration and realized the MX records were incorrect. I basically made 2 MX records. One was mail.domain.com to internal server IP priority 10. The second MX record was the internal server name: server1.domain.com with a priority 10.
Now that the MX records were in place I followed through and made sure that they both had Host A records in DNS. The host A record for mail and server1 both point at the internal IP address of the mail server.
Also I verified that the MX record on the outside world was pointing at the correct public IP, I like
MXToolbox to verify that. Then I made sure the firewalls directed the traffic correctly.
Now I have DNS in place and SMTP (port 25) and POP (port 110) access between both servers. I re-ran smtpdiag on both sides and it still failed on DNS. I went through the servers and ran ipconfig/flushdns, nbtstat -R, and nbtstat -RR.
After 5 minutes everything in DNS propagated correctly and smtpdiag worked perfectly with no issues.
But email was still not being received. I restared the SMTP Service and the Exchange System Attendant service and within a minute mail started flooding in.
So I put a solid 20+ hours into this entire fiasco. I hope this information reaches someone else and helps them out. Another tool I used to help troubleshooting was
VisualRoute. They have a 15 day trial but Im thinking its worth the investment to have full time. And of course the builtin tools always help, ping, tracert, telnet, nslookup, netstat and route print. Two more tools I use that are free downloads from MS are SMTPDiag and PortQry. Also used Wireshark on both Windows and Linux.
There it is, any questions on the process or if you are having the same issues, post and I'll try and help out anyone.
Edited by jaxisland, 15 July 2009 - 11:55 AM.