Upgrades break your stuff.... python 3.4 vs python 3.7

So...  The new version of PRTG rolled out and Python 3.4 is now deprecated and Python 3.7 is a thing. 

What does that mean for me? 

Everything I wrote before is now broke....

so...  that big chunk about paepy...  you can forget all that.

Luckily, I spent the time figuring out how to make all this work.


starting point...

result = []

result is going to hold our collection of strings that we'll add together prior to dumping into PRTG.


Next bit.. 

We need this bit to create the proper JSON format before we start dumping stuff in. 

 result.append(" { \"prtg\" :{ \"text\": \"OK\", \"result\":[")


Second bit... 
result.append("{\"channel\": \"name of the thingy\", \"unit\": \"Custom\", \"value\": " + value_of_importance + ", \"is_float\": \"True\"  , \"primary_channel\": \"True\", \"warning\":\"0\", \"is_limit_mode\": \"True\", \"limit_min_error\":\"0.5\", \"limit_max_error\":\"1.5\", \"limit_error_msg\":\"the message\"}")

So... you're going to have to escape a lot of quotes.  General stuff....  escaping quotes is \".   So if you want a quote (")  you need to write \" to get a quote and not finish the line.

It's a bit pain to say the least.

The not so much pain is that the end result can be dumped into a JSON validator and you can find what needs to be fixed and what is wrong.

I like this one.  https://jsonformatter.curiousconcept.com/

Other web pages I found valuable:

  https://kb.paessler.com/en/topic/80232-how-can-i-monitor-sigfox-callbacks-with-prtg#reply-276903

https://www.paessler.com/manuals/prtg/custom_sensors

Please use the previous article I wrote on troubleshooting to find the actual JSON output and dump that into a validator to make sure your JSON is good before pulling your hair out.  The big thing that is going to be a problem is adding the correct amount of commas after the channels.  

One comma at the end of each channel, except for the last one.

Use the len() command to solve that problem.

To be more specific.....   len() = blah.

Create another variable...

currentRun = 0

Increment variable at the end of the for statement.

if len() > currentRun:

    result.append(",")    #add a a comma

That should be sufficient...

Now....   One everything is done...  

Need to add this bit to the result so we can the end of the JSON.

result.append("]}}")

Ok... got that...  

Now everything is is stuffed into result, but we need to get rid of the commas that normally show up when you just add a bunch of strings together.....

So...

finalResult = ""

         for items in result:

             finalResult += items

    

print(finalResult)

And with that print statement, we're done.

We've got rid of commas in the wrong place, got rid of paepy, and upgraded from Python 3.4 to Python 3.7.....

Looks like we're good.

PRTG: solving your problems

So you wrote a nice custom script to monitor you Next Cool Thing, and it doesn't work.  Not only does it not work, you can't figure out what is wrong with it.

So how do you solve it?

First, turn on sensor debugging. 

Go to the sensor in question, then Settings.  Scroll down until you find this bit. 

Turn that on.

Got that?   Good.

Now the interesting part.  Where are my readings? 

C:\program data\Paessler\PRTG Network Monitor\Logs (Sensors)\

So there you have it.

Kind of.

You get two files for every sensor.   One file indicates all the stuff that was sent to the sensor via PRTG.  That's the file with the name "Result of Sensor XXXX.Data.txt"

The other file is the output after the script has been run.  That's "Result of Sensor XXXX.log"

From there, it's time to go read that second log file and try to figure out what the problem might be.  The big problem I have with troubleshooting Python scripts is the lack of an Idle interpreter in the default installation of PRTG. 

That's survivable.  Notepad is there.  It's just more annoying. 

The second part of reference is that the interesting bits about what failed is generally at the very end of the script output. 

Is this the perfect debugging scenario?  No.  But it does provide the information you need to figure out why your script isn't working.

You did write the script on a machine with better debugging utilities to try and make a good proof of concept, right? 

So best practice in my eyes: write the entire script except the output bits on a separate machine before transferring it to the remote probe and/or primary server.

So...   Have fun, and go squash some bugs.

PRTG: Making Custom Sensors to monitor strange things with Python

#verion 1.0

#last modified 12/2/19

#

#v 1.0   initial revision / prtg integration

#

import sys

import json

import urllib.request

from paepy.ChannelDefinition import CustomSensorResult

if __name__ == "__main__":

    location = json.loads(sys.argv[1])

    

    parsed = "http://" + str(location['host']) + "whatever else in the url"

    

    page = urllib.request.urlopen(parsed).read()

    

    #data comes out as binary type.

    #convert from binary to normal string

    np = page.decode('utf-8')

    scrip = np.split('

    for lines in scrip:

    result = CustomSensorResult("OK")

         result.add_channel(channel_name="channel1", unit="Custom", value=status_value1, is_float=True, primary_channel=True, warning=0, is_limit_mode=True, limit_min_error=0.5, limit_max_error=1.5, limit_error_msg="channel1 failed") 

         result.add_channel(channel_name="channel2", unit="Custom", value=status_value2, is_float=True, is_limit_mode=True, warning=0, limit_min_error=0.5, limit_max_error=1.5, limit_error_msg="channel2 failed")

    print(result.get_json_result())

Base code for a python script sensor for PRTG.   

Using this method, you write a script to scrape a web page, and then present the information to PRTG as a JSON file.

The limits are used to define up/down/warning status.

In this case, I have outputs of a binary 1/0 for working not working.

So how does that work?   

Notice this bit.

result.add_channel(channel_name="channel1", unit="Custom", value=status_value1, is_float=True, primary_channel=True, warning=0, is_limit_mode=True, limit_min_error=0.5, limit_max_error=1.5, limit_error_msg="primary connection 

And breaking that apart, the section here

is_limit_mode=True, limit_min_error=0.5, limit_max_error=1.5, limit_error_msg="channel1 offline"

Let's break these down.

Note: these names aren't the same those expected or presented for EXE/Advanced sensors on the PRTG custom sensor page.

is_limit_mode = Sets PRTG to know that the output has acceptable ranges of input.   

limit_min_error  = This sets the lower limit that defines an error.  Depending on your output, there may not be one.  I'm using outputs of binary 0/1 in this case, so I set it to .5.   Therefore, a 0 output is defined as error state.

limit_max_error  = This set the upper limit that defines an error.   Depending on your output, there may not be one.  In my case, there is never an upper maximum error.   So I set it to 1.5.   

limit_erro_msg = The message you want on PRTG for any device that may be not working.

So, with these setting set correctly, PRTG will report an individual sensor is down based on the values you assign to status_value1 and status_value2.  So now, you can alert based on those settings using normal PRTG alerting.

What this base script doesn't currently do:

1. Parse anything.  Parsing the web page is based entirely on what you are looking for.  The page I was looking at was all table based, so splitting the data into tables made sense.  You will have to handle that portion.
2. Deal gracefully with urlib.urlrequest.open() errors.   You will get a JSON error in PRTG when you try to pull a web page you can't get.  That's a simple try/except statement.  Use this message in your except portion to report failure gracefully.

            result.add_error("Your Error Message Here")

Secondary important thing...   probably the most important.  

See this block? 

 location = json.loads(sys.argv[1])

    

    parsed = "http://" + str(location['host']) + "whatever else in the url"

This block accepts json data as input to the script.  

The second part location['host'] pulls the IP address setup on the sensor to feed that data into the script.   So this script can be written once and run on multiple devices.  That's what makes this script extendable.

Now...   

So, you've got the initial script working.

How in the world do I troubleshoot this thing when I suddenly get a bunch JSON errors when I deploy it?   

That's the subject of another discussion.

monitoring applications

Part of a decent monitoring scheme is making sure business critical things are running all the time.  PRTG and powershell are a great way to do that.

$errorFound = $false


try
{
    if( -not (get-process -ProcessName "blah" -ErrorAction 'silentlycontinue' ))
    {
        throw 
    }
    $textVal = "OK"
    $errorFound = $false
}
catch
{
    $errorFound  = $true
    $textVal = "blah Not Running"
}



write-output ""
write-output ""

write-output "blahProcessRunning"


"Status"
   

"1"
"1"
"
"     



if($errorFound)
{
    write-output "0"
    write-output "1"
    $textVal = "blah Process not running." 

}
else
{
        "0"
        $textVal = "OK"
        "1"
    
}

"" + $textVal + ""
write-output ""
write-output ""
write-output ""

write-output "BlahProcessRunning"


"Status"
   

"1"
"1"
"
"     



if($errorFound)
{
    write-output "0"
    write-output "1"
    $textVal = "blah Process not running." 

}
else
{
        "0"
        $textVal = "OK"
        "1"
    
}

"" + $textVal + ""
write-output "
"

Note: Blogger as far as I know is doesn't have a good way to post full code.  So there's a bit more to it than this, but it's all opening and closing brackets. 

Ok, so there's all the code.  Most of it is pretty dull.

The heavy lifting is done by the get-process ProcessName "blah" section.  Which is pretty self explanatory.  Works great when the process happens to be running.  The problem is what happens when the process doesn't run.

When it doesn't run, you get red powershell text.  Which is what we don't want. Adding -erroraction "silentlycontinue" gets rid of the error text.

But now it fails and doesn't tell us anything.  Also not what we want.

So try-catch.

Which doesn't quite properly work.  It doesn't give any output at all.

The answer was the if(-not ((process)) ) bit followed by the {throw}

So, if you get some sort of failure, silently continue and throw an error.
Which the catch picks up.

And now we've got some nicely formatted text that changes based on whether the process is or is not running.  And with business critical processes, that's what you need.

Now, get-process dumps a lot of interesting data that you could feed back to PRTG.  Frankly, I wasn't that concerned.  I just wanted to know whether the process was running.

As for install procedure, let's not skip that.

Install a PRTG remote probe.
Drop this file in the CustomSensors\EXEXML directory.

Change the policy for the x86 version of powershell to unrestricted.
Or sign the code with your own code signing cert.  I don't know how to do that.  So  I go with the first.

That's in c:\windows\syswow64\windowspowershell\1.0\powershell.exe

Not the same one as opening a command prompt or searching for powershell.exe.

So there you go.  Alert as your company dictates and or want.

Network and Security on the cheap

Networking security and monitoring on the cheap

Building a complex network scheme requires a lot of knowledge and experience.  It also requires a bit of insight and guesswork into what can and should be used.  Trying to do all this without budget is even worse.  You basically don't have any chance to do it correctly.  

Or do you....   

See, part of the problem is not knowing the tools are there in the first place.  And because the Internet is big into open source, you've got plenty of options once you become aware of what they are.  So that's what I'm going to try to deal with here.  Trying to build a defensible architecture based off spending the least amount of money as possible.  Saving the money on these tools allows you to spend that money else where.  Like in good firewall. 

Love them or hate them, here's my recommendations.  

Network monitoring - Paessler PRTG

Log Management - GrayLog

SIEM - OSSIM

Firewall -  Palo Alto

Patch Management - WSUS

Now, before I get yelled at...  

1) I don't deal with much of a web presence, so I don't have to fight with Web Application Firewalls.   I mostly deal with brick and mortar.  If you've got a web front end to deal with, I can't really help you.  

2) Palo Alto is expensive.  But it is awesome.  Money spent there is well worth the spend.   

3) PRTG is only free for 100 sensors.  100 is better than 0, which is probably what you are monitoring now.   And running a ping here and there is not monitoring.   Don't even pretend.   

Sure, many of these platforms are free.  But you will spend a lot of time in sweat equity.  Don't expect an easy time with setup and tuning.  Expect to spend a lot of time beating your head on the wall, trying to figure something out. 

The other part of this is do what everyone says to do, but no one does: document.  

Documentation is the untold glue that holds all these train wrecks together.  Spend a lot more time documenting than you think you should.  Because that first time you have to rewrite a script to get your Graylog server to accept traffic on the correct port, you'll wish you had.  Because you'll forget about that script, and reboot.  And because you didn't write it down, you'll have to figure it out again.  No fun.

Or when your Graylog setup script gets overwritten due to an upgrade.  Fun times.  Fun times.