Juniper Applicaion Layet Gateway (ALG) Feature Considered Harmful With Microsoft 2008 AD Replication

I'm currently at an Active Directory (AD) shop. I hate AD, but here it seems to be a necessary evil. (I'd much rather run regular LDAP, BIND, etc.)

We ran in to a tricky firewall corner case today that I couldn't find anyone blogging about, so I'm jotting my notes down here for posterity.

It seems that AD domains that were set up post Windows Server 2008 (no 2003 backward compatibility) are now using dynamic ports for RPC.
MSFT KB


These high/dynamic ports DO NOT play nicely with Juniper's Application Layer Gateway features. We're using Juniper SRX boxes for site-to-site IPSEC VPN termination. I wasn't expecting any NAT or gateway features to be active on the Juniper between my tunnel interfaces and trusted interfaces, but ALG appears to have been blocking connectivity.

The symptom was lack of TCP connectivity between domain controllers.

telnet domain1 49152

would fail to connect. (in seemingly very non deterministic ways, but then would block solid for days)

Disabling ALG on the Juniper did the trick.

I doubt disabling ALD will be an issue since we're not using the SRXes as internet gateways. (atleast not these nodes)

Netflow DOS Detector

Set up netflow at your network borders.

Send to a collector and dump 30s windows in raw v5 netflow to a file.
Dump from raw v5 to a text parsable format.

Every 2 minutes or so, use a cron to merge all flow data.

account for:
netflor variations
origin AS
watch pps rates
watch DST port to detect DDOS attacks
graph pps&bps by proto and pps by port

Offensive ssh Keys!

So you've gone an reimaged a system, and that host has new ssh keys, and now you can't just ssh in to it because you've got an "Offending key" in your ~/.ssh/known_hosts file.

You know there's no mitm attack going on here. It's just a common issue when
dealing with lots of machines.

Now you've got to go in and remove one line from that file.
Annoying.

Maybe I shouldn't be reimaging systems so often.

Maybe I should try to save my old keys and then push them out.

Maybe I need a script to make it very easy to fix this problem..
DING!

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef.
Please contact your system administrator.
Add correct host key in /home/jkrauska/.ssh/known_hosts to get rid of this message.
Offending key in /home/jkrauska/.ssh/known_hosts:376
RSA host key for otherbox has changed and you have requested strict checking.
Host key verification failed

So here's a fixer script.
I spent maybe 2 minutes on it, and I used perl for only one line (split), but with some proper bash-fu it should really be done completely in bash. I leave that as an exercise for the reader.

This is a prime example of my tools philosophy. If I need to write something that gets used maybe once a month, there's no need to make it super optimized or perfect. Just "usable".

I called the script "Offending" and put it in my ~/bin dir which is in my PATH.

So I can just copy and past the line that ssh spits, and be on my way.

#!/usr/bin/perl
# Simple bash script to clean known_hosts when you've reinstalled an OS

# USAGE: (paste the line from the ssh output)
# Offending key in /home/jkrauska/.ssh/known_hosts:376
# To remove that line from your known_hosts

($file,$line) = split "\:", $ARGV[2];
if ($file eq "" || $line eq "") {
  print "Error in parsing, unable to grok input\n";
  exit 0;
}
#print "DEBUG FILE:$file\tLINENUMBER:$line\n";

$h=$line-1;
$t=$line+1;

system "head --lines $h $file > $file.new";
system "tail --lines +$t $file >> $file.new";
system "rm $file";
system "mv $file.new $file";
system "chmod 644 $file";