Author Archives: ingmar.koecher

Managing Windows Services & Service Credentials

Every Windows server runs a seemingly ever increasing number of services which range from built-in services providing core Windows functionality (e.g. Print Spooler, Bitlocker, WMI) to 3rd party services added when installing 3rd party software (e.g. various software update services, MySQL) – all of which run in the context of a specific user account.

For example, Windows Server 2012 includes more than 300 services, about half of which are automatically running (this particular server has SQL Server installed as well):

Services on Windows Server 2012 grouped by user

Services on Windows Server 2012 grouped by user

That user account is either a built-in security principal of Windows (e.g. NetworkService), a user account specifically created for that service, or another user account from the server or domain.

Common Practices
Services should always run under a user account which has the least amount of privileges necessary to do its job. It’s common, and often tempting, to run a service an administrative account like “Administrator”. While this often the easiest way to “get it working”, it’s also the least secure.

When a service runs under the “Administrator” account – especially if it’s the domain Administrator account – the service has almost unrestricted access to all resources on the host or, in case of a domain admin, on the domain. This is not something a service usually needs nor you want. It also means that the service will stop working whenever the password of the Administrator account is changed (the service will continue to work until it is restarted).

Less is Better
Whenever possible, try to use one of the built-in security principals available in Windows to run a service under, or create a specific user account for the service. For example, if you have a file synchronization app which runs as a service, create a “ServiceFileSync” or similar account and configure the service to run under that account. Carefully examine the rights the service requires, and only assign those privileges to the user account which the service actually needs.

When creating the user account, give it a very strong & complex password. Users won’t have to log on with that user account, so the password can be complex and long. You can optionally check the “password does not expire” option if you feel that the password is sufficiently secure and you have a short password expiration policy on your domain which could interfere with the service starting after the password expired.

In domain environments I also recommend giving those user accounts (since you will most likely end up with more than one) either a common prefix or suffix (e.g. svc_mysql) and/or moving the accounts into a specific OU. This makes managing and distinguishing these accounts easier – especially in teams with more than one SysAdmin.

The quick way: Local Services grouped by User Account

Sample output from srvsec

Sample output from srvsec

To view all locally installed services grouped by the user account they are running under, download the EventSentry SysAdmin Tools and just run srvsec.exe. This will show you all locally running services, and group the output by the user account they are running under. Srvsec can also be pointed at a remote host, and can also change the passwords stored in services. Click here for more information on srvsec.

Srvsec is a great tool to quickly see what’s going on a single host, but to manage services on an entire domain effectively a more scalable solution is available: EventSentry + AutoAdministrator – the dynamic duo!

The right way: Making sense of ALL installed services
Even when passwords for service accounts are sufficiently strong, they should still be changed on a regular basis. But which services are installed where and are using which service account?

If this is your first time examining service accounts on your network, you should first identify which services run under which user accounts. EventSentry’s service monitoring feature combined with the web-based reporting really makes this a breeze. Assuming that you have a service monitoring system health package assigned to all of your servers, you can simply open the web reports and navigate to Status – Services and get a birds-eye view of all installed services.

In the Overview view, all installed services are grouped by common attributes, including startup type (automatic startup services vs manual startup services), current status, service name and, most importantly for this post, the service user account.

Service overview of all services installed in a domain / forest.

Overview of all installed services in a domain.

Click the “Show All” link to see all user accounts, or click on a specific user account (e.g. “LocalSystem”) to filter the list and only show services running under this specific user account. In most cases you will want to click on “Detailed” to see a list of all services with more detail.

In addition to filtering and viewing details, you can also click on the header of the

All user accounts used by services

All service user accounts

username (or any other) column to see a chart depicting all user accounts used by services from all monitored servers and workstations.

Any report viewed in the web reports can also be scheduled with a job, e.g. a list of all user accounts used by services could be emailed daily/weekly. Simply click the “Save as Report” link to create a report and setup a job.

Managing Services
The standard way to configure the user account and password used by a service is through the “Services” application in Windows. This works well for one or two servers, but not when you need to update the password for a service on multiple hosts.

Managing services with AutoAdministrator

Managing services with AutoAdministrator

This is where AutoAdministrator comes in: A free graphical tool which lets you do just that (and quite a bit more): Update the username and/or password of a service on multiple servers in a domain or work-group. Since AutoAdministrator is multi-threaded, even tasks affecting a large amount of hosts usually only take a few seconds.

To update the stored password of a service, open AutoAdministrator and select “Services” from the drop-down list on the top left.

Service Key Name

Service Key Name

Next, select the service you wish to update from the “Service key / display name” drop-down. If the service is not listed, simply specify the service key name in the service field. The key name is the internal name used by the service and can be obtained by double-clicking a service name in the “Services” MMC application in Windows.

Updating service credentials

Updating service credentials

Next, click on the “Set logon” tab and specify the new username and/or password. Of course you can also specify other service actions, such as restarting the service or changing the start-up type.

As the next step, select the hosts you wish to apply the selected changes to. You can select hosts from Active Directory, EventSentry, custom groups or work groups (Microsoft Windows Network).

Once the correct hosts are selected, click the “Start” button. The number of hosts which will be affected by any action is always shown on the bottom right of the application.

The Essential Security Tools of the EventSentry SysAdmin Tools

toolsPart of our larger EventSentry network-management solution, the freeware EventSentry SysAdmin Tools offer a set of graphical and command-line utilities designed to help you with your daily administrative tasks. These tools are constantly under development, always being honed to provide simple yet powerful functionality. Three of these tools are vital security utilities: Password Assistant, Service Secure, and Task Secure. Let’s take a look at what they offer.

Password Assistant
Password Assistant is a simple yet powerful tool that lets you update the passwords of user accounts on multiple Windows machines. You simply enter the username, the old password, and the new password (with confirmation); after doing so, you can select the computer name(s) from a network neighborhood list (with a filter option) or choose the computer(s) from a text file. The update process can also be logged to a text file.
All the parampassword_assistanteters of the command-line utility are provided in clear terms: The /u (or /username) parameter specifies the username whose password needs to be changes, the /pwold parameter takes that account’s old password, and the /pwnew parameter accepts the new password; there are also /n (or /network), /f (or /filepath), and /filter parameters for specifying computers, as mentioned above. The /ignore_rest parameter ignores the rest of the labeled arguments following this flag, and the /version parameter displays version information and exits.
A great sample use of Password Assistant is when you need to update the administrator passwords on all of the workstations in your environment. Password Assistant provides a one-stop interface—or a simple command-line utility—for making the task hassle-free.

ServiceSecure
Service Secure provides a simple command interface that displays all of your system’s services, grouped by service account. The tool also lets you easily reset service passwords by specifying username and password rather than having to manually configure those services through the Microsoft Management Console (MMC).srvsec_1
All the parameters of the command-line utility are provided in clear terms: The /p (or /password) parameter sets a password; the /c (or /changepwd) parameter changes a password; the /r (or /restart) parameter restarts the service after the password has been changed; the /u (or /username) parameter lists only those services running under a certain username; the /ignore_rest parameter ignores the rest of the labeled arguments following this flag; the /version parameter displays version information and exits.
Suppose service security has been compromised, and you need to quickly change the passwords of a user account used by a number of services. That task is no longer a logistical nightmare: You can now simply use ServiceSecure in a batch file and update all affected services in your entire network in a matter of moments.

TaskSecure
Task Secure provides a simple command interface that displays all of your system’s scheduled tasks, grouped by task account. The tool also lets you easily manage the passwords stored in scheduled tasks on your network. Simply specify the username and password, and Task Secure will reset the password stored in all scheduled tasks (using the specified username) on the specified computer (local or remote).

TaskSecureAll the parameters of the command-line utility are provided in clear terms: The /r (or /remote_host) parameter lists all the scheduled tasks on a given host; the /u (or /username) parameter lists only those scheduled tasks running under a certain username; the /p (or /password) parameter sets a password for every scheduled task configured for a certain user account; the /ignore_rest parameter ignores the rest of the labeled arguments following this flag; the /version parameter displays version information and exits.
Suppose you need to quickly change the passwords of all the scheduled tasks used by one user. You can now simply use Task Secure in a batch file and update all scheduled tasks in your entire network in a matter of moments.

More to Come!
This is just a taste of the free, constantly evolving tools available in EventSentry SysAdmin Tools. Give them a try—you won’t be able to stop with just one.

Automatically Restarting a Failed Windows Process

Whether it’s a critical process running on a server or an application on your desktop – sometimes processes terminate and need to be restarted – immediately.

With EventSentry & EventSentry Light you can do just that: Automatically restart processes immediately after they terminate.

In the past, one drawback of EventSentry launching a process was the side effect that any process started by the EventSentry agent would run under the same account as the EventSentry agent itself (usually a privileged domain account or LocalSystem).

In this post I’ll discuss how you can work around that limitation in a secure manner using a scheduled task. When the critical process fails, instead of launching the process directly through a process action, EventSentry will trigger a scheduled task instead. Why? Because scheduled tasks allow you to configure under which user a task will run – and the user’s password is securely stored in Windows.

The recipe for accomplishing this feat is as follows:

  • Process Monitoring monitors the process
  • An event log filter looks for the “failed process” event and triggers a process action
  • The process action starts a scheduled task

Let’s look at this in detail. First, on the host where the critical but unstable task is running, you create a schedule task in the Windows “Task Scheduler”. Under General, give the task a descriptive name (“Start Super Important App”) and change the user under which the program should be running under. In most cases you will also want to make sure that you configure the task to run whether a user is logged on or not. Then, under “Actions”, add a new action “Start a program” which points to the executable that should be launched. After you click “OK” you will be prompted for the password for the user.

Scheduled Task

Creating a scheduled task

The next step is to setup process monitoring in EventSentry. Right-click “System Health” and create a new package and assign it to the computer(s) in question. Right-click the newly added package and select “Add – Processes”. Click the newly added object and add the name of the process which should be monitored. You can configure how many instances of the processes are required, and with which severity the event will be logged when the process is inactive.

process monitoring

Configuring process monitoring

Now we create a new “Process” action. Right-click the “Actions” container, select “Add” and enter a descriptive name (e.g. “Trigger Super Important App”). In the Filename field specify:

%SYSTEMROOT%\system32\schtasks.exe

And for the Command Line Arguments enter:

/Run /TN “Start Super Important App”

This uses the built-in Microsoft utility schtasks.exe to run the task we created in our first step. At this point EventSentry will monitor the specified process and log an event if the process is inactive. And while we do have an action to trigger the scheduled task, we still need to tell EventSentry when to launch that action.

EventSentry Process Action

Configuring a process action to start a scheduled task

For the next step, right-click the “Event Logs” container, select “Add Package” and give that package a descriptive name. Then assign the package to the same host. Right-click the newly added package and add a filter by clicking “Add Filter”. In the filter dialog, add the “ Trigger Super Important App” action to the action list and configure the following fields:

Event Log Include Filter Rule

Setting up a rule to trigger the process action

Event Severity: Information
Event Log: Application
Event Source: EventSentry
Category: Process Monitoring
Event ID: 10401
Content Filter (wildcard): *critical_app.exe*

Important Notes: The event severity will need to match whichever severity you selected when adding the process monitoring object in the system health package. The content filter can also be configured to match insertion string #1, in which case the wildcards are not necessary.

And that’s all there’s to it, simply save the configuration when you are done. If the process is running on a remote host then don’t forget to push the configuration to that host.

EventSentry Mobile v1.3 for iOS Available!

We’re excited to announce EventSentry Mobile v1.3 for the iOS platform. The new version remains free, is optimized for iOS 7, sports a refreshed interface and also includes a brand-new feature for pinging hosts in real-time.

Since the new version of EventSentry Mobile was optimized for the new web reports in EventSentry v3.0, it will not work if you are running EventSentry v2.93 on your network.

The interface of the app remains largely the same, but integrates more seamlessly with the iOS 7 flat look. Icons on the network status dialog are now only shown when a host is at least in a warning state, and the icons on the heartbeat dialog have been changed to new flat icons.

EventSentry Mobile Network Status

Network status dialog

Viewing computer details has been tweaked for hosts which are monitored via SNMP since there is currently less information available when compared with Windows hosts monitored by the EventSentry agent.

But since we didn’t feel that visual and internal changes were enough, we also added a new “Tools” section to the app which can:

  • Ping a host
  • Perform a DNS lookup
  • Provide GEO lookup data (when available)

The tools dialog has a single input field with one button where you can enter either an IP address or host name. After you enter the host name or IP address and click the target button, the web reports will ping the host, perform a DNS (reverse) lookup and try to obtain GEO information.

EventSentry Mobile Tools

Ping, DNS & GEO lookup tools in EventSentry Mobile app

The ping and DNS lookup are performed from the web reports rather than from the phone directly, GEO data is pulled by the iOS device from http://freegeoip.net/. If the IP address is not private (e.g. 192.168.1.x) and coordinates are available then the location can be viewed on a map directly on the iOS device as shown below.

Location of IP address after GEO lookup of www.unvienna.org

Location of IP after GEO lookup of www.unvienna.org

Reminder: If you are using EventSentry v2.93, iOS 7 and have automatic updates enabled on your iOS device, then EventSentry Mobile will stop working.

If you have an iPhone or iPad and haven’t already installed EventSentry Mobile then simply search for “EventSentry” in the App store or click here. Enjoy!

CryptoLocker Defense for Sysadmins

It seems as if CryptoLocker has been making the rounds lately, much to the dismay of users who don’t have working backups of their precious office documents.

While I admire Cryptolocker’s simplicity and effectiveness from a purely technical and entrepreneurial standpoint, what the software is doing does appears to be illegal in most countries and so I’d like to offer some advise on how to tame the beast. If you’re looking for a 5-minute fix then I have bad news: implementing the CryptoLocker defense I have outlined below, while completely free, will take a little more than 5 minutes to implement. But knowing that you have an effective defense against CryptoLocker may very well be worth it. After all, CryptoLocker seems to find its way into a lot private networks these days.

CryptoLocker Screenshot

The ideas set forth in this blog post apply mostly to Windows-networks with file servers, but could be adapted for individual computers as well (though this is not covered here – let me know if you’d like me to include this scenario).

About CryptoLocker
For those who have not heard of CryptoLocker yet, it is a piece of software which encrypts pretty much all common office-type documents, including Microsoft Office, AutoCAD, PDFs, images and more. This blog article from MalwareBytes has a complete list of extensions. Once encrypted, CryptoLocker charges you to decrypt (your own files) again. It’s public key cryptography gone wrong; I wonder if Diffie & Hellman saw this one coming. And to make the whole spiel even more interesting, you only get a limited amount of time to pay before your files will remain encrypted. Forever. Oh – and the longer the wait, the more you have to pay. And with recent bit coin exchange rates in excess of USD 1000, the amount that needs to be paid can be uncomfortably high.

It is pretty difficult to defend against something like CryptoLocker other than through usual means of AntiSpyware software, user eduction and strict policies against opening and downloading files from the Internet, email attachments and such. In most cases CryptoLocker comes in form of a ZIP attachment disguised with a PDF icon.

One reason CryptoLocker is so effective – yet difficult to block – is because it exhibits the same behavior as users would: It “simply” accesses and modifies files like a user would. And infecting a machine isn’t all that difficult since CryptoLocker doesn’t require any elevated permissions to run. On the contrary, it wants to run in the same context the user does, so that it can access and see the same files a user does. As such, security features like UAC are utterly useless against ransomware like CryptoLocker – it’s a whole new type of software.

Backups
The most effective defense against CryptoLocker is to have a working, tested backup. Let me repeat this: A WORKING and TESTED backup. Users have lost all their data because they thought that they had a backup in place when their backup was broken in some way.

We’ve seen posts of users who deleted all the files CryptoLocker encrypted, thinking they had a working backup. They had a backup, but it was apparently not recently tested and as a result the user lost all of their data.

Naturally, CryptoLocker does not like backups. It dislikes them so much that when CryptoLocker runs, it even tries to delete any Windows Shadow Copy backups. Cloud backup services (including Dropbox, Skydrive and Google Drive etc.) which keep versions of your files offer some protection, but restoring older versions of your files may be a tedious process.

The Defense
The most obvious defense against CryptoLocker is AntiSpyware software, e.g. MalwareBytes. Most AntiSpyware & AntiVirus software still uses signatures however, so new versions of the ransom ware often remain undetected at least for a few days.

So instead of detecting CryptoLocker itself, we can sniff its tracks so to speak. CryptoLocker’s predictable behavior can be used against it. CryptoLocker’s objective is of course to encrypt and hold hostage as many files as possible, so to increase the likelihood of the user purchasing the decryption key from the thugs.

And it is that very pattern that we will try to exploit and use as a trigger to detect and take corrective measures. The approach consists of measuring how many files are being changed in a certain time interval, and if a certain threshold is being exceeded (say more than 10 files modified in 1 minute) we assume that CryptoLocker found its way into our castle. Even though users modify their documents on a regular basis, users can usually make only so many changes at a time and most likely at a much slower rate than any sort of script / software would.

Another approach would be to create one or more honeypot or canary files, which we know (or hope) a user would not modify. If a checksum change in one of those files were detected, we could (more or less) safely assume that CryptoLocker was on one of his rampages again and take corrective measures. The honeypot file would have to be modifiable by users (otherwise CryptoLocker would also not be able to modify it), which makes accidental modifications by users possible (although somewhat unlikely).

This 2nd approach isn’t quite as solid in my opinion, since CryptoLocker is most certainly adapting to changes, and may skip files that it may suspect are a trap. For example, it could skip small files or skip directories with a very small number of files and so forth.

A more sophisticated approach, where we detect an unusually large number of files changes in a small time period, is going to be harder to circumvent by CryptoLocker. The good news is that we have a free (it’s really free, not a trial) software tool available which can do just that. It can:

  • detect file changes
  • measure the rate of file changes (through event log alerts)
  • stop/start services or launch processes
  • send out alerts

EventSentry (Light) to the rescue
EventSentry Light is the free version of our full-spectrum monitoring & compliance solution EventSentry. The features we can utilize to come up with a defense are:

  • File Checksum Changes (part of System Health Monitoring)
  • Filter Thresholds (part of Event Log Monitoring)
  • Action (control services, send out emails)

File Checksum Monitoring
Monitors any folder and detects file size changes, checksum changes and file additions and deletions. EventSentry Light will log file checksum changes to the event log (it’s big brother can also log them to a database), which in turn is monitored by the real-time event log monitoring component.

Event Log Monitoring & Thresholds
This component supports a variety of sophisticated features, one of which are thresholds. The thresholds feature lets you essentially detect event log entries that occur at a certain pace. For example, if 10 specific events occur in 1 minute then let me know and/or take corrective action.

Service Action, Email Action
EventSentry supports a variety of action types to be triggered when an event occurs, with email usually being the most commonly used one. You can also control services, use REST APIs, launch processes and much more. We’ll use the former to stop the file sharing services (LanmanServer) when we have determined that CryptoLocker is on the loose.

I will go into step-by-step instructions on how to configure EventSentry at the end of the post.

The Baseline
The most difficult thing to determine is the maximum rate of file changes we deem normal, as we need to have a baseline in order to configure the threshold slightly above that. This number will vary from network to network, with file servers serving lots of users obviously requiring a larger threshold. I’d like to repeat that determining the right threshold is very important. If it is too low, then normal user activity will trigger an alarm; if it is too high then the alarm may never be triggered and CryptoLocker won’t be caught in time.

The best approach is to setup file monitoring and let it do its job for 1-2 days to determine a baseline. Once the baseline is established, we can increase it by a certain factor (say 1.5) and use that as the threshold.

Setting up the trap requires 3 steps. In this case we assume that EventSentry is either installed directly on the file server, or an agent is deployed on the file server (in which case you will need to make sure that configuration updates are pushed to the file server(s) in question).

Step 1: Monitoring the directory/ies
In EventSentry, right-click the system health packages and add a new package. Right-click the package, select “assign” and assign it to all file servers. Right-click the package again and add a “File Monitoring” package. Click the new object. Directories are monitored in real time by default, but EventSentry requires a recurring scan as well – in case Windows doesn’t send real time notifications. This is usually a good thing, but when you are monitoring large directories it’s best to set the interval very high (future versions will allow for this to be unchecked).

File Checksum Monitoring Settings

In the package, add all the folders which should be monitored and only check the “checksum change” check box. Do not check any of the other check boxes in the bottom left section at this time. Since we haven’t established a baseline yet, we’ll set the severity of the event log alerts to “Information”. If the monitored folders contain a lot of non-Office files then it may be a good idea to adjust monitoring so that only office files (e.g. .doc, .xls, etc.) are monitored. If you prefer to monitor all files, simply change the setting to the green PLUS icon and make sure the list of exclusions is empty (or specifies files that should be excluded, e.g. *.tmp). Below is a screenshot of how this can be configured.

File Checksum Monitoring Settings

When you save the configuration, EventSentry will enumerate all files in the folder and create an initial checksum for every file. The agent will log event 12215 when the scan starts, and event 12216 when the scan is complete. When that happens, EventSentry is essentially “armed” and will detect, and log, all checksum changes to any of the files in the monitored directories.

At this point we’ll want to let this run for at least 24 hours during a “normal” work day, as to determine how many file changes occur on average. You are going to be at a bit of an advantage if you are running the full or the trial version with database support, as it will be a lot easier to determine the number of file changes occurring through the web-based reporting.

Step 2: Setting up the trap
Now that we have established a baseline, we’re ready to setup a threshold. This time we’ll create a new event log monitoring package. Right-click “Event Log Packages” and add a new package and call it “CryptoLocker Rules”. Like before, assign it to the file servers we are monitoring. Right-click the package again and add a new event log filter. Configure the filter as shown in the screenshot below. Note that we are triggering an email action for now. The content filter can be used to restrict the filter further, e.g. to only match certain directories if you are monitoring several directories with EventSentry.

Event Log Filter Setup

Now things are getting interesting. The goal is to create an error event in the event log when X amount of file checksum changes occur in a given time period. To get there, we’ll start with the “General” tab where we tell the filter what type of event we are interested in (see below). Once that event is defined, we’ll move on to the “Threshold” tab which is where we specify the threshold parameters. For the purpose of an example, let’s assume that we have established a baseline of 100 file checksum changes per day, with a work day starting at 8am and ending at 7pm. Assuming that activity is somewhat spread throughout the day, this amounts to about 9 file changes per hour. Naturally we’ll have to assume that file changes aren’t always evenly spread out throughout the day, but setting up a “if 20 checksum changes occur in 1 minute shut file sharing down” is probably a reasonable threshold. Configure the threshold as shown in the screenshot below, with whichever threshold you came up with.

Event Log Filter Threshold Setup

Step 3: Triggering corrective action
When our threshold is reached, EventSentry will log an error to the event log with event id 10601 and trigger the specified action(s) from the “General” tab (Default Email) one time per threshold interval.

At this point we would merely receive an alert when we suspect that CryptoLocker is at it again. If you are cautious then you can retain this setup for a little while (e.g. a day or two) to ensure that you are not getting any alerts about the threshold being met (assuming that CryptoLocker is not active on your network in which case you should get the emails).

To go all in and trigger a server service shutdown, we’ll need to create a service action now. On Windows, file sharing services are provided by the “Server” service, which uses the internal name of “LanmanServer”. The service action allows you to control any service (start/stop/restart), and in this case we’ll obviously want to stop the server service, so that clients cannot access the file shares on your server anymore. We’ll trigger an email action at the same time of course, so that the sysadmin in charge is aware of what is going on. While shutting down all file services seems a bit extreme, it’s unfortunately the most effective way to prevent more files from becoming encrypted.

So for the next step, right-click the “Actions” container and select “Add Action”. At the selection dialog choose the “Service” action, enter a descriptive name (e.g. “Stop File Sharing”) and hit enter.

Selecting an EventSentry Notification

Then, configure the settings of the service as shown in the screenshot below.

Action to stop the LanmanServer service

The last step of our setup (congratulations if you’ve made it that far) is to assign the service action to the filter we previously created. After all, a service action which isn’t referenced anywhere doesn’t do much good. So head back to the Event Log Packages, find the “CryptoLocker Rules” package and edit the filter in the package. In the action list on top, click the “Add” button and add the action you just created.

Testing
If at all possible I’d recommend testing the EventSentry setup at a time when your users are not interrupted. Adding a few template files to one of the monitored folders and changing them in short succession (a script may be necessary depending on how short your threshold interval is) should trigger the file services shutdown procedure. Once verified, you can just start the “Server” service again.

Conclusion
Just like in the real world, network viruses come in all shapes and sizes – only limited by technology and the imagination of the cyber-evildoers.

I hope that this article gave you some insight into CryptoLocker and a good way to guard against it. As always, make sure that your company has the following in place:

  • Email Attachment scanning
  • Working, tested backups
  • User education
  • AntiSpyware software

With those in place, one should be able to keep future infections to a minimum.

Stay safe & decrypted.

Mobile Alerts: Pushing EventSentry alerts directly to your mobile devices with Prowl and NMA

When it comes to mobile alerts, email seems to still be the prevalent method of choice for many IT pros. There are many good reasons why network alerts delivered via email are convenient:

  • easy to configure
  • uses existing infrastructure
  • every smartphone and tablet supports email
  • supports attachments (e.g. performance charts from EventSentry’s performance alerts)
  • integrates into your existing environment – everybody already uses email!

What’s not to like? Well, of course it turns out that some of these advantages can also be a disadvantage:

  • emails are not real time
  • problems on the email server often don’t surface immediately
  • important alerts can be overlooked in the inbox jungle
  • you cannot be alerted about email problems via email (duh!)

Viable Email Alternatives
Thankfully, there are a number of alternatives that can be used as an email substitute or addition for mobile alerts. In this article I’ll focus on two affordable services: Prowl (for the iOS platform) and Notify My Android (you may have guessed it – for the Android platform) – subsequently referred to as “NMA”. Both of these services consist of apps for their respective platform and a web-based back end which will push the notifications to your device(s) in near real time.

Mobile Alert on iPhone 5 with Prowl

EventSentry event sent to Prowl on an iPhone

Both services offer an HTTP API which we can connect to with EventSentry’s HTTP action. If you have never used the HTTP action in EventSentry, then here is some background: the HTTP action allows you to POST any event (whether it be an event from the security event log or a heartbeat alert for example) to either Prowl’s or NMA’s web service. These notifications are then pushed to one or more mobile devices.

Neither service currently requires a monthly subscription, but both require a purchase (Prowl costs USD 2.99 whereas NMA costs USD 4.99) if you want to send unlimited notifications to your mobile device(s). NMA is a little more welcoming to strangers – it supports up to 5 notifications per day at no charge.

Prowl: Getting Started
The
iOS app costs USD 2.99, and supports up to 1000 API calls (=notifications) per hour. To get started:

  1. Purchase the Prowl: Growl Client from the Apple App Store and install it
  2. Register for free at prowlapp.com
  3. Login & create an API key

Notify My Android (NMA): Getting Started
The Notify My Android app is free in the Google Play store and supports up to 5 API calls (=notifications) per day  for free. Upgrading to a premium account will allow up to 800 API calls per hour. To get started:

  1. Download the Notify My Android app from the Google Play store
  2. Register for free at www.notifymyandroid.com
  3. Login & create an API key
  4. Optional: Upgrade to a Premium account to allow for unlimited notifications

Once you have the mobile app installed and the API key in hand, you can start setting up a HTTP action in EventSentry. Notifications essentially consist of three fields: The application name, a subject as well as a message field, all of which can be customized. As such, it’s up to you configure which part of an event log alert you will put in the subject, and which part you will put into the actual message. In our example, we generate a dynamic notification subject with the host name, event id, event source and event category. The notification body will simply consist of the event message text, though this can be customized as well.

Setting up a HTTP action
Right-click the “Actions” container (or, in v3.0, use the “Add” button in the Ribbon) and create a new HTTP action. The HTTP action requires a URL at minimum, optional credentials and the actual data fields to submit in the HTTP POST. Conveniently, both Prowl & NMA use the same field names. I suspect that they are adhering to some sort of standard, though I couldn’t find any references.

The first parameter to configure is the URL, which depends on the service you use:

Prowl: https://api.prowlapp.com/publicapi/add
NMA: https://www.notifymyandroid.com/publicapi/notify

Configuring authentication credentials is not necessary since you are essentially authenticating with the API key you generated. The last step is configuring the form fields. The bold field on the left is the name of the form element and the text to the right the value. A description follows in italic on the next line:

apikey: abdef123123abababafefefe
The API key you received

application: EventSentry
This will displayed as part of the notification. I use “EventSentry” here, but this can really be anything, so it could be the host name as well for example ($EVENTCOMPUTER)
Max Length: 256

event: $EVENTCOMPUTER-$EVENTID-$EVENTSOURCE-$EVENTCATEGORY
This is the subject of the notification. You can use any variables, including insertion strings $STR1, $STR2 etc.
Max Length: 1024 [Prowl], 1000 [NMA]

description: $EVENTMESSAGE
This is the main message body of the notification, the $EVENTMESSAGE seems like a pretty good candidate for this field
Max Length: 10000

priority: 0 (possible values are -2, -1, 0, 1 and 2)
This field is optional and doesn’t do anything with NMA. With Prowl however, a priority can be set to “2″ (indicating an “emergency”) which may then override quiet hours on the mobile app (if quiet hours are configured in the Prowl mobile app).

Take a look at the respective API documentation for Prowl and NMA as well; each show additional fields which can be configured, as well as additional information that you may or may not find useful.

The screenshot below shows a fully configured HTTP action for Prowl:

EventSentry HTTP Action for Prowl

EventSentry HTTP Action for Prowl

The HTTP action for NMA would look identical, with the exception of the URL which is different. You can click the “Test” button, which will submit the configured data to the specified URL and should, when configured correctly, immediately generate a notification on the mobile device. Please note that event variables will not be resolved when testing.

Threshold
To make sure that your mobile device doesn’t get flooded with alerts (some applications have the tendency to generate not one but hundreds of events in a short period of time), I highly recommend that you setup a threshold on the action or the event log filter referencing the action (I personally prefer the former). You can also setup a schedule so that notifications are only sent on certain days and/or during certain hours.

The last step would be to configure an event log filter to forward select events to the mobile device, something that is beyond the scope of this article. See the tutorials below for more information:

Reliability
It seems only natural to wonder whether alerts sent through these services can be used for mission critical systems. I’ve been using mostly Prowl as I’m an iPhone user, and have been very happy with it’s fast response times (which are almost instant), the service reliability and the stability of the iOS app. Nevertheless, both prowlapp.com and NMA state that you should not solely rely on their system for critical alerts and instead setup multiple channels for mission critical alerts. This sounds nice in theory, but suspect that most sysadmins will not want to dismiss alerts on more than one device – something that can get old if you get a fair number of alerts. Switching to a commercial system like PagerDuty with guaranteed up-time may be preferable in that case. I will talk more about PagerDuty in an upcoming post soon.

My experience with the NMA Android app wasn’t as good during the limited testing I performed. While it worked great when it worked, the app did crash on me a couple of times.

Conclusion
If you’re looking for a way to push alerts to your mobile device from EventSentry without using email and without spending big bucks, Prowl and NMA are worth looking into. They’re affordable, responsive and easy to configure.

EventSentry Light Supercharged

The latest EventSentry update brings significant and very exciting changes to the free light edition, EventSentry Light.

We have always seen EventSentry Light as a successor to the original EventwatchNT – a monitoring tool for small networks to alert sysadmins by sending real-time alerts about event log activity. As its big brother EventSentry continued to mature, most features from EventSentry made it into the light edition as well: Service Monitoring, Disk Space Monitoring, Performance Monitoring and many more. Since EventSentry Light was, and continues to be, free, it needed to distinguish itself from the commercial edition. As such, most feature were somewhat restricted in the light edition which only allowed a limited number of packages, event log filters, performance counters and such.

es_light.png

Over the last year we’ve been getting feedback that EventSentry Light was being constrained too much. Since our goal is, and always has been, to empower sysadmins and not constrain them, we decided to provide our users with more functionality in the free edition. The result of these efforts is build 2.93.1.75, and with this release you can:

  • Monitor event logs and log files in real time, setting up as many filter rules as you’d like, without restrictions.
  • Utilize all advanced event log filter capabilities like thresholds, timers, schedules and more!
  • Create as many event log, log file and system health packages as you like.
  • Utilize all system health monitoring features, such as file checksum monitoring, performance monitoring, service monitoring and more.
  • Create a variety of alerts using mail, HTTP, SNMP traps, Syslog messages and more.
  • Receive SNMP traps from SNMP v3 enabled devices.
  • Monitor up to 2 full hosts and 2 network devices.

Pretty impressive, no? So literally overnight, EventSentry Light has matured into a full-fledged monitoring solution which will alert IT professionals like sysadmins of critical (event) log events, performance issues and much more. What differences with EventSentry remain? A few, but the line is much more clear now:

  1. Reporting. With EventSentry, you get log consolidation, software/hardware inventory, performance trend reports, network dashboards, jobs, JSON/XML/CSV/… APIs and more.
  2. Compliance. EventSentry includes a variety of compliance functionality such as process tracking, logon tracking, account management tracking and more.
  3. Monitor multiple hosts. Monitor as many hosts as you are licensed for, and also utilize command line utilities to automate remote host management.
  4. Support. EventSentry includes quality email and phone support, something we pride ourselves on. EventSentry Light offers forum support.
  5. Mobile iPhone & Android apps are only available in the full edition since they require reporting.

So if you’re not already using EventSentry Light, or using an older version, then you should give it a try. It’s as free as it gets with no registration required, no advertisements and no nagging pop-ups. We hope you like it as much as we do. And did I mention that you can seamlessly upgrade from EventSentry Light to EventSentry? :-)

Don’t forget to check out our other free tools and Facebook / Google+ pages!
Best,
Ingmar.

EventSentry Mobile Updates for iOS & Android

I’m excited to announce that we released updates to both the iPhone as well as Android version of our mobile app, and both updates are now available in the Apple App Store and Google Play Store respectively.

The iOS version now supports the larger iPhone 5 display as well as a variety of improvements in regards to load speed, rotation and so forth.

screenshot_iphone5_1.png

The Android version catches up with the iOS version, adding the heartbeat page and fixing an issue where the app would not launch on devices running Jelly Bean. We’ve also beautified the app a bit and donated some color icons to the menu. Yes, the Android app now has officially more color than the iOS version. How crazy is that?

We also had to change how EventSentry Mobile is distributed through the Google Play Store, where our mobile app is now called “EventSentry Mobile” (instead of just “EventSentry”). Existing users of the mobile app will, unfortunately, need to uninstall and re-install the app to get the latest version. This was necessary due to Google’s policies and a mishap on our part. Future releases of the Android app will support updating however, and we apologize for the inconvenience.

screenshot_android_1.jpg

To get the free update, either search for “EventSentry” on your mobile device, or visit the links below.

Google Play:
https://play.google.com/store/apps/details?id=net.netikus.eventsentry.mobile

iTunes:
https://itunes.apple.com/us/app/eventsentry/id440535744

Now, if you use or like the app – why not rate it and write a short review?

Thanks,
your NETIKUS.NET team.

Monitoring Windows Updates

Automatic Windows Updates are a wonderful thing when they are working as expected, and many organizations employ WSUS or patch management software to keep their infrastructure up to date with the latest Microsoft hot fixes and service packs.

While this works for many, not everybody can afford patch management software, and, while free, managing the disk-hungry WSUS can be a daunting task as well. This leaves some sysadmins to use the old-fashioned Windows Updates to install all the regular and out-of-band patches Microsoft releases.

If you don’t feel comfortable installing patches automatically however, configuring Windows Update to “download updates for later manual installation” is often safer and more predictable. But, if you’re not logging on to the server(s), you won’t know whether one or more updates are ready for installation or not. Even if you’re just managing one server, checking in on a regular basis can be a waste of time.

updates_are_ready.pngThis is where EventSentry and its log file monitoring feature comes in. It turns out that Windows, like a diligent ship captain, logs all activity to a log file. And with all, I really do mean ALL. The file I’m talking about is windowsupdate.log, and it tells you just about everything that’s going on with Windows Update. In 3-4 steps that don’t take longer than 5 minutes, you can setup real-time monitoring of the WindowsUpdate.log file, and be notified when updates are about to be downloaded to a monitored computer.

The screenshot below shows what such an email from EventSentry would look like:

email_approved_updates.png

From then on, you can either get email notifications when patches are downloaded, or use the web-based reporting to view a report from all of your monitored hosts. On a high level, the configuration works like this:

  • 1. Setup a log file (%systemroot%\windowsupdate.log in this case)
  • 2. Create & assign a new log file package
  • 3. Define a log file filter (this tells EventSentry what to look for in the file, and where to send it)
  • 4. Setup an email action (this is usually already setup)
  • 5. Optionally setup an event log filter to forward alerts to email (the default filter setup should automatically forward warnings)

The WindowsUpdate.log is useful for troubleshooting as well, and you can consolidate the content of this file from all of your servers in the central EventSentry database. This makes searching for text and/or comparing the log from multiple servers a breeze. Having the log file accessible through the web reports is also useful when a patch caused problems and the server is offline. You can view the most recent activity from the log file through the web-based reporting even when the server is unavailable.

So how do you set this up? Assuming you have EventSentry v2.93 installed (any edition will do, including the free “light” version), you can follow the steps outlined below. Note that all steps will need to be performed in the EventSentry management console.


1. Setup a file definition.
This tells EventSentry which file you want to monitor, and sets up a logical representation of that file in the EventSentry configuration. In the “Tools” menu, click “Log Files and File Types” and then click the “Add” button.

menu_tools.png

add_file_definition.png

tree_logfile_package.png

2. Create a package and add a filter. Right-click the “Log File Packages” container, select  “Add Package” and choose a descriptive name. Since new packages are unassigned by default, right-click the newly created package, select “Assign” and assign the package to host(s) on which you want to monitor the Windowsupdate log file.

3. Setup a log file filter. This tells EventSentry which content of the monitored file you are interested in. In every log file filter you can configure a database as well as an event log filter.

Right-click the previously created package and select “Add File”. From the list, select the log file definition created in step 1, “WindowsUpdateLog”. Select the new log file.

The database tab determines which content goes to the database (in most cases you will write all file contents to the database), while the event log tab determines which log file contents are written to the event log. For this project, we are interested in the following wildcard matches:

*AU*# Approved Updates =*

*DnldMgr*Updates to download =*

The first wildcard match will tell you the total number of updates which have been approved and will be downloaded, whereas the 2nd line will fire for every individual update which will be downloaded. In most cases the first line is sufficient and the 2nd line can be skipped.

file_filter_eventlog.png

That’s it! With this setup, you will immediately get notified when patches are ready to be installed. The only thing I didn’t mention here is how to setup an email action and corresponding event log filter, since both of these are usually already setup by default. If you need help with this, please check out our documentation and/or
tutorials.

Please note that the full and evaluation version of EventSentry can inventory installed software and patches. This enables you to use the web interface for viewing/searching installed patches, and get (email) alerts when a patch has been successfully installed.

As always, happy monitoring!

Announcing EventSentry Light v2.93.1

We’re excited to announce a new version of EventSentry Light, our free server and log monitoring solution.

EventSentry Light is:

  • completely free
  • does not show ads
  • does not require a registration
  • does not expire

To see all the new features which were added to full release EventSentry v2.93.1, see “EventSentry v2.93.1 – Part 1” and “EventSentry v2.93.1 – Part 2“.

eventsentry_performance_alert.png

In addition to the new features and bug fixes of the 2.93.1 release, we also decided to make the latest version of EventSentry Light even more useful for sysadmins by enabling several features which were previously only available in the full version:

  • Process Action is now available, so you can now launch scripts and/or processes as a response to event log entries
  • Custom event logs as well as custom event log channels (Windows 2008 and later) can now be monitored
  • Services can now be controlled in addition to just being monitored
  • All event logs can now be backed up
  • Event Log backups can be compressed
  • NTP (Network Time Protocol) feature can now adjust the local time
  • Limits can now be applied to actions
  • Email actions: All features are now available
  • Import/Export feature in management console is now available
  • Variables support is now available

In addition to the above new functionality, we also increased many of the existing limitations:

  • # of event log filters: Increased to 5 (from 4)
  • # of monitored services: Increased to 6 (from 4)
  • # of event log backup schedules: Increased to 3 (from 2)
  • # of actions: Increased to 3 (from 2)

EventSentry Light v2.93.1 is a significant upgrade from v2.92, with many new features now available to light users. Remember, with EventSentry Light you can:

So if you’re running EventSentry Light v2.92 or older then the time to upgrade is now! If you’re not using EventSentry at all, then the time to install it is now – you have nothing to loose.


Get EventSentry Light