Having just come through a harrowing ‘network issue’ ordeal, I thought I’d best document the steps back to sanity from out of my naivety. A rough description of the scenario follows.
We have a client using Sage Line 50 who wanted to perform queries on their web members database in conjunction with their in-house accounting information. This set me down the path of setting up an on demand connection and synch, database to database. The solution was a VPN connection between the client’s LAN and their web server.
Having never played seriously with VPNs before, we secured the help of a good colleague of ours with mad IT skillz.. Tino at Forza-IT (site coming soon).
A day on site for Tino and a few days of fiddling on my part got the solution in place. This ran fine for a couple of months, but then a rare kernel upgrade forced a reboot of the web server a week ago.
The first we heard of issues was that none of the client’s machines could access their web member’s database a few days later. Tonight’s conclusion has seen me grow from a ‘poke it and see’ position to a ‘ahhh I see how that’s working’ position. Basically I’d left a failed/partial lt2p/ipsec vpn setup in place as well as the working pptp vpn, but it took a lot of investigating to finally see the light!
The ninja tools I have acquired along the way are as follow.
Network routing and traffic investigation
netstat -rn ... netstat -an | grep LISTEN ... route -n ... telnet example.com 1723 ... tshark -i eth0 proto 47 ... tshark -i eth0 port 500 ... tail -f /var/log/messages
tshark is a traffic watching tool that seems to have taken over the mantle of ethereal, tcpdump, nc (netcat) and installs on CentOS with the wireshark package. It is synonymous with tethereal.
yum install wireshark
Remote syslog harvesting
vi /etc/sysconfig/syslog
Add the -r switch to the options therein
... #SYSLOGD_OPTIONS="-m 0" SYSLOGD_OPTIONS="-r -m 0" ...
Ensure ‘syslog 514/udp’ is listed somewhere in /etc/services
… and finally restart syslogd
service syslog restart
Once you’ve set your router to forward event logs to the syslog server IP, /var/log/messages will harvest the router logs as well as the local events. Have a look at syslog-ng if you want to get more clever with syslog.
The combination of these tools and several hours of reading got me through in the end. The failed VPN connection had setup a network route on reboot that sent any outgoing traffic from the web server to the client router into a black hole. This made investigation really difficult, as pinging/telnet-ing the web server from the lan would send, be seen on the server, but no reply returned. I faffed with the firewall an awful lot turning lots of traffic logging on and off tracking the cause down. In the end, the thing that made the penny drop was seeing traffic over port 500 (IKE) using tshark coming from the lan when I wasn’t trying to initiate a VPN connection from the web server as far as I was aware. This was the IPSEC connection that was set to start ONBOOT, had sprung to life, failed to successfully create a VPN and killed traffic between the two sites for good measure.
Well glad that’s all over, and I’m sure the client will be on Monday … bleurgh!
