Network Security on the Cheap, part 2: Graylog

So, I've spent the last couple of weeks setting up GrayLog in a cluster.

Giant pain in the butt.

My build was Ubuntu 16.04.  And that works fine.

Lots of stuff that is not mentioned, or even thought about.

A few things I had to solve...

1) Elasticsearch needs to be on the same version
2) Mongodb needs to be at least version 3 or higher.
3) The web URI information is badly documented.
4) A cluster won't form until you are using the same database for all servers.

I set up a router in front of my setup, resulting in a NAT'd public IP address and the cluster behind.  Gives plenty of room for growth without ever readdressing anything externally.

Based on recommendations, I changed the listening port to 12900.   Not sure the implications or reasoning of that.  But it seems to work for me.

So, private IP is IP of the server.
Public IP is the NAT'd IP of the server. 

rest_listen_uri =  private ip/api/

web_listen_uri = private ip

web_endpoint_uri = public ip/api/

rest_transport_uri = private ip/api

forgetting the /api/ on the web_endpoint_uri causes a specific issue

It's the cannot POST issue.

Don't forget the /api/ setting.

Once that is done

I also load balanced the web interface using HAProxy.

Final thing about Graylog, and one I wondered about...

Once I've got the cluster up and working, do I need to load balance my UDP syslog traffic?

Short answer: probably not.

Though it doesn't look like the other servers in the cluster are doing anything, they are.  Easy way to tell is look at the number of messages waiting and processing.  With 2 servers in my cluster, my waiting and processing at 4,000 messages per minute sits in the dozens.  There's never a queue.

So, with two low power computers and a whopping total of 14 GB of RAM, I process 4,000 messages a minute with zero wait.  That includes bursts of up to 6,000 messages per minute.

I'm not saying you won't need to load balance the UDP traffic.  I'm saying you need to base it off your scenario.  And figure out how many messages per minute you are getting.

Generally, UDP is low overhead network traffic anyways.  The general problem is not that you can't receive enough data.  It's that the data takes too long to process, resulting in a queue.  That's the purpose of building a cluster.

Total cost at this point?  3 cast off computers that aren't good for much else.  One of my computers runs 6 GB of RAM and runs 100% CPU usage.   Yet it still handles enough to reduce the load of the primary computer.

Network and Security on the cheap

Networking security and monitoring on the cheap

Building a complex network scheme requires a lot of knowledge and experience.  It also requires a bit of insight and guesswork into what can and should be used.  Trying to do all this without budget is even worse.  You basically don't have any chance to do it correctly.  

Or do you....   

See, part of the problem is not knowing the tools are there in the first place.  And because the Internet is big into open source, you've got plenty of options once you become aware of what they are.  So that's what I'm going to try to deal with here.  Trying to build a defensible architecture based off spending the least amount of money as possible.  Saving the money on these tools allows you to spend that money else where.  Like in good firewall. 

Love them or hate them, here's my recommendations.  

Network monitoring - Paessler PRTG

Log Management - GrayLog

SIEM - OSSIM

Firewall -  Palo Alto

Patch Management - WSUS

Now, before I get yelled at...  

1) I don't deal with much of a web presence, so I don't have to fight with Web Application Firewalls.   I mostly deal with brick and mortar.  If you've got a web front end to deal with, I can't really help you.  

2) Palo Alto is expensive.  But it is awesome.  Money spent there is well worth the spend.   

3) PRTG is only free for 100 sensors.  100 is better than 0, which is probably what you are monitoring now.   And running a ping here and there is not monitoring.   Don't even pretend.   

Sure, many of these platforms are free.  But you will spend a lot of time in sweat equity.  Don't expect an easy time with setup and tuning.  Expect to spend a lot of time beating your head on the wall, trying to figure something out. 

The other part of this is do what everyone says to do, but no one does: document.  

Documentation is the untold glue that holds all these train wrecks together.  Spend a lot more time documenting than you think you should.  Because that first time you have to rewrite a script to get your Graylog server to accept traffic on the correct port, you'll wish you had.  Because you'll forget about that script, and reboot.  And because you didn't write it down, you'll have to figure it out again.  No fun.

Or when your Graylog setup script gets overwritten due to an upgrade.  Fun times.  Fun times.

Leg Work

I always wanted to write a program that took forever to run just due to the amount of processing it did.  Now, I think I’ve got one. 

Unless I remember to run the thing frequently, it takes about 85 minutes to process through all the files I’ve collected to watch the status of my network.  That’s not pleasant, but it’s working on getting better.  It shouldn’t take as long the next time, as I’ve written in a few checks and balances that prevent it from re-checking every single file over again.  I end up running a “select everything and see if you return nothing” comparison.  Afterwards, I move the folder off to a different directory to archive and eventually clean up that directory.

Which, all of that seems to be working properly.  For the most part.  There are still some issues I need to debug, and some error conditions that need to be solved.  But it’s working better than it did.  The second part is that I’ve finally got it to where it will take multiple input units in the same file, so you don’t have to have multiple scripts logging in every five minutes.  The next part is to start working on NAT table translating and interpretation.  I’m not sure how I want to do that as of yet.  Show ip nat translations gives all the translations, but is that enough information?   What else would I want?

I think that’s the basic problem of all network security people.  You want to gather information, but it begs the question of what the correct amount of information is, and what really needs stored.  And there is always the question of when enough is enough.  

Flipping back through my Cisco book, it looks like I’m going to need show ip nat translations verbose for several different reasons.  The main reason is that is tells when that connection was last used, and that’s vital to proper identification of what has happened.  It provides the “when” and “how long” needed to trace information back to the source.

Show ip nat translations just gives a static view of an event that happened. The connection could have lasted for seconds or it could have lasted for hours.  There is no way to tell.  So the need is to translate the verbose method of the call.  

Where do I go from there?  I don’t know.  At some point I’ve got to figure out how make Apache work the way I want it to.  But that?  That’s another day.

Note: bold notations are specific Cisco commands.  

The next steps

Now that the Routing and Switching class is over, it's time to get ready for the CCENT.  I'm aiming for that in the next month.  I was aiming for two weeks, but I can get a voucher for half-priced testing, so I'm going to go for the voucher.

That being said, let's go back to network security.

I had a AHA! moment last week on network security, and it leads me to believe a very large vendor does not really provide security updates.  They also have some serious problems with their code.  It's all Apache web server, but it's an unpatched version of Apache that still has some serious vulnerabilities.

About a year ago, or maybe a little less, I realized when we replaced a router, all of our internal vulnerability scans started passing.  It was weird.  But I didn't know what it could have been.  Over the past couple of weeks, I've been working on replacing their router with my own.  After breaking our internal vulnerability scanner for a couple of weeks, our vendor finally fixed the device.

About the same time, we ran into a situation where a location suddenly started showing up as failing when it had been passing.  Thinking back, during the intervening period we had upgraded the site and swapped out the equipment.  At that point, we had swapped out a router.  And suddenly the site started failing scans.

Not suddenly, but it seems like it.  So I started thinking about what could have gone wrong during the upgrade.  Everything is built from standard equipment.  The results are pretty predictable and cookie cutter.  So how did this cookie fall out of the cutter mess up?  We'd tried a different configuration with the router, and the site ended up failing internal scans.

So, why did this router fail and other pass?  It's pretty simple: access control lists.  There was an implicit deny on the VLAN with the internal scanner.

Now I know why my scans are all passing.  The door to the scanner is closed.  And the equipment we're dealing with is no way near as secure as we thought.  All because of Apache.

It's also the realization that I can fake a passing scan in less than 30 minutes by simply throwing an ACL in every single router we have.  It'd be easy.  I already know the syntax.  Come to think of it, I could make it highly precise, so it wouldn't be something I could automate.

Anyways.

It just pisses me off that a multi-national company can't patch Apache.  Or, that I have to find the holes in their system.

Now, off to figure out Apache on my own so I can ramp up my network base lining and turn all the data I've collected into something usable.

I should probably write an article on that.   It's basically a combination of Java, mySQL, and some Ubuntu cron jobs.

Network Baselines

Like I said, I’ve been working on network baseline analysis.  Beginning problem is that I don’t have a baseline to begin with, nor do I have any way to examine the current baseline of the network.  So, I’m at a loss of where to start. 

I read one book where a basic baseline can be created by pinging all available hosts.  It’s not the greatest baseline, but it is the beginning, and it’s better than nothing.  What I’ve got is nothing.  So what I did is wrote a batch file using a FOR loop to ping all devices and print the output to a file.  After that, I ran an arp –a and appended that to the end of the file. 

So it’s not the greatest baseline.  But it does give me an idea of what standard network performance should be, at least as far as PING goes.  I guess the next part is trying to dump the information into a webpage or a database so the information can be examined later and compared to what it has been at various points. 

I guess I should probably add the ITILv3 documentation to my reading list.  The only problem is I’m not definite the ITIL information actually provides information on how to baseline a network.  I understand the basics and the conceptual theory.  It’s a matter of going out and doing the work.  And sorry, SNMP is not the way to baseline.  Everyone has it turned off due to the insecurities in the system. 

Just a quick look at Cisco, and the only encrypted version they have only supports DES.  So the options are send the data as plaintext, or send it as an algorithm that has already been replaced due to inherent weakness.   15 years ago, DES was cracked in 22 hours.  15 years ago, I was happy with 400 MHz processor running 128 Mb of RAM. 

In comparison, I’m writing this on a laptop with an Intel Core i5 running at 2.5 GHz with 4 GB of RAM.  Shot in the dark, but I think a couple of these suckers could crack DES in a day.  And if someone breaches your network and doesn’t get caught, then what is a day?  What is 10 days? 

The end of one thing, the beginning of another.

I finished the Security+ book, and I think it left me with more questions than answers.  At the moment, I’m questioning how to do a lot of things.  Network baseline analysis is the primary one of those.  At the moment, I’m doing some preliminary reading.  Sure, there are a lot of books out there that say “this should be done”.  None of them discuss how to do network baseline analysis.  I think the best answer I’ve seen so far “there isn’t a standard”.  Which sounds pretty normal with network security.  And that’s why network security is, as a general rule, very splotchy.

With Security+ being finished, it’s off to learning physics VIA a collection of books written by Benjamin Crowell.  Part of me wants to write a long, drawn out blog post describing in detail how I can believe in both science and God at the same time.  But I’m not.  The answer is pretty simple: most of life is not an either/or selection.  Despite simplistic arguments against, it is entirely possible to believe in both at the same time.  The two are not mutually exclusive.  Sorry folks, I can believe in both at the same time.

I’ve touched on the false idea of mutual exclusivity before.   I can’t remember the post, but it’s the argument of people who want to lead you down paths that are only valid if the two items discussed in the beginning really are mutually exclusive.  For the most part, there are very few mutually exclusive items in the world.  I guess in the end, you have to question the assumptions people push at you, and assume everyone has an agenda.  Despite the best arguments, the truth is not the real agenda.

I think Andy Andrews put it best:  People often think logically to the wrong conclusions. 

Really, there was an entire book about that subject.  It was pretty interesting.  

A bit busy

Can you say busy?  Yup.  It’s been.  It’s been the good kind of busy, though.  I spent a few days building a new Active Directory Group Policy.  Default templates work well, but they still need tweaked.  Not everything works out as well as you’d think.  Or at least not always.

I’ve been contemplating security in depth, and I’m pretty sure my old beliefs about security are still true: security is an illusion we like to tell ourselves to ignore reality.  It really is.  Several companies I’ve dealt with are completely clueless as to what they do and how they operate.  They are no more aware of their failings than the man in the moon.  It’s moderately hilarious, but only in the sick, sad sort of way.  I guess there is always a need to upgrade and move forward in a technology related company.  The inertia of doing nothing often gets in the way of real accomplishment. 

I think that will be it for today.  I spent a few minutes working on the Java StopWatch, but not enough to do anything of interest.   Yay.   I have large letters and I set up a font.  But they aren’t centered in the boxes so they are kind of ugly.   But at least the numbers are large and readable.

Security

Something I’m questioning quite a bit is network security.  Network security is a nebulous thing that is never really clearly defined.  I’ve read through most of a Security+ book, but that doesn’t really cover the broad scope of what needs to be done to cover the subject.

I’ve looked at the SANS website, and they might be great but SANS is too busy selling things.  Looking at Cisco, it’s one part of the subject, but it doesn’t cover the entire subject. 

Here’s the real problem: network security is so spread out, ambiguous, and poorly documented that truly learning network security has to be a personal topic.  The individual has to wander through the various resources, hoping to find some piece of information that will secure the network.  So many different possibilities of areas to cover without any real dedication.  It is no wonder that networks are broken into all the time.

I guess that’s my rant for the day.  Lots of work today, so little time to spend at home and do…  anything else.

And I’m pissed at Evernote right now.  It lost about 2-3 hours of my note taking.  There was a lot of interesting information.   And somehow, I’m back to where I was before all my Saturday/Sunday studying.