Author Archives: ingmar.koecher

How to identify long-running processes

I always enjoy visiting customer sites for training or consulting since I learn about their unique challenges and requirements, and how EventSentry can meet them.

During a recent visit an interesting question came up: How can I identify (certain) processes which run longer than a certain time period? It may sound like an odd requirement, but some software suites spawn worker processes which perform certain tasks which take a predictable amount of time, such as processing a document for example. If something goes wrong and one of the worker processes hangs, you’d want to know about it.

EventSentry does include a process monitoring feature which can ensure that a certain number of instances of processes are running, even taking their command line arguments into consideration; however it doesn’t evaluate the duration of process.

Even though you cannot do this out of the box (and given that most users don’t require this sort of thing we’re probably not going to add it), there is a pretty easy solution with a (VB)script and the application scheduler. As a reminder, the application scheduler is the standard way of extending EventSentry’s functionality.

Even though VB(Script) is not the most popular scripting language these days, we like to utilize it for a number of reasons:

* The interpreter (cscript.exe) is pre-installed on all versions of Windows
* It was developed on and for Windows, and can handle easy to moderate scripting pretty well
* It’s easy to read and customize, even by people who don’t write code on a regular basis

Of course you can utilize any scripting language with the application scheduler as long as the interpreter is installed. Now let’s see what this VBScript would look like (if you have ever used the Scriptomatic then the structure of this script may look familiar to you):

On Error Resume Next

Const wbemFlagReturnImmediately = &h10
Const wbemFlagForwardOnly = &h20

' Customize start
Const processName   = "parser.exe"
Const maxAgeSeconds = 120
' Customize end

Dim returnCode
returnCode = 0

Set objWMIService = GetObject("winmgmts:\\localhost\root\CIMV2")
Set colItems = objWMIService.ExecQuery("SELECT * FROM Win32_Process WHERE Caption='" & processName & "'", "WQL", _
                                      wbemFlagReturnImmediately + wbemFlagForwardOnly)

For Each objItem In colItems
    Dim secAge
    secAge = DateDiff("s", WMIDateStringToDate(objItem.CreationDate), Now())
    If secAge > maxAgeSeconds Then
        WScript.Echo "Process " & objItem.Caption & " (" & objItem.ProcessId & ") has been running for " & secAge & " seconds, since " & WMIDateStringToDate(objItem.CreationDate)
        returnCode = 1
    End If

Function WMIDateStringToDate(dtmDate)
     WMIDateStringToDate = CDate(Mid(dtmDate, 5, 2) & "/" & _
     Mid(dtmDate, 7, 2) & "/" & Left(dtmDate, 4) _
     & " " & Mid (dtmDate, 9, 2) & ":" & Mid(dtmDate, 11, 2) & ":" & Mid(dtmDate,13, 2))
End Function

In a nutshell, the script uses WMI to retrieve all running processes and then subtracts the current timestamp from the process start time to determine the runtime (duration) of the process. If it exceeds the pre-configured threshold, the script will return 1 and subsequently log an error to event log.

To get started, first configure the process name and maximum duration in lines 7 & 8. Then, added the script as an embedded script (Tools -> Embedded Scripts) with a descriptive name. Remember to give the file the correct (.vbs) extension here.

Once the file is setup as an embedded script, you can reference it from the application scheduler or an action (although it wouldn’t make much sense to use this script as an action). Create a new system health package, or add the “Application Scheduler” object to an existing system health package. Make sure the package is assigned to the correct computer or group!

To finish, add a schedule to the newly created application scheduler object; in most cases you will want to use a “Recurring Schedule” which will run in regular intervals. On the main application scheduler dialog you will want to make sure that the “Log application return code > 0 to the event log as “Error” is checked. These types of events can then be forwarded to a recipient via email for example.

This script is a pure monitoring script, it won’t take any corrective action by itself. But the script could easily be modified to automatically terminate the process if it has been running for too long. For example, you could either terminate the process with the Terminate() method via WMI, or execute pskill (Sysinternals suite) from within the VBScript. The latter may be more reliable but will require that pskill is installed on all the machines running this script. A modified version of the script is shown below:

' using "Terminate()"
    If secAge > maxAgeSeconds Then
        WScript.Echo "Process " & objItem.Caption & " (" & objItem.ProcessId & ") has been running for " & secAge & " seconds, since " & WMIDateStringToDate(objItem.CreationDate) & ", and will be terminated"

        returnCode = 1
    End If

' using pskill
    If secAge > maxAgeSeconds Then
        WScript.Echo "Process " & objItem.Caption & " (" & objItem.ProcessId & ") has been running for " & secAge & " seconds, since " & WMIDateStringToDate(objItem.CreationDate) & ", and will be terminated"

          WshShell.Exec "PSKill " & objProcess.ProcessId  
        returnCode = 1
    End If

So there you have it, how to keep long-running processes in check. Since embedded scripts are integrated into the EventSentry configuration, there is no need to manage the script on the remote host.

A nice feature of EventSentry is that any email alert you will get will automatically include the output of the script – delivered straight into your inbox.

An alternative to email alerts. Part 1: Using Trello to manage EventSentry’s alerts

Trello is a simple yet powerful and innovative task management / collaboration platform for teams. With Trello, the developers have basically taken the familiar concept of traditional white boards where you add and remove tasks (by writing on them), and moved it to an easy-to-use online tool.

While Trello doesn’t attempt to replace the more complex project management and collaboration tools available (including its own FogBugz platform), it makes keeping track of small ToDo lists and tasks surprisingly simple, while still supporting advanced features such as due dates, attachments, assignments and more. Of course, Trello also includes a very capable mobile app for iOS and Android (I only tested the iOS version).

Trello Overview
And best of all, it’s completely free if you stick with the basic (and for most people completely sufficient) functionality. But what does Trello have to do with EventSentry and cutting down on emails?

We’re always looking for innovative ways to make managing alerts easier and more productive, especially in larger teams. While email alerts certainly serve a purpose and can be quite useful, alerts dispatched via email suffer from a few disadvantages:

  1. Emails sent to multiple recipients make it difficult for the recipient to know whether the alert has been acted upon or not
  2. Alerts which have already been resolved by a team member still remain in your inbox
  3. Emails often get lost amidst other emails and potentially critical alerts may get overlooked

How Trello Works
Trello is organized into boards, each of which can have one or more lists, each of which have multiple cards. Since Trello offers an API, you can use EventSentry’s HTTP action to submit events (alerts) directly to one (or more) Trello lists.

And this is where the fun starts. Once in Trello, alerts (now cards, or “alert cards”) can be acted upon in a variety of creative and useful ways. You can:

  • Receive alerts in your browser when a card is created
  • Move a card to a different list (e.g. “Resolved”, “Under Investigation”, …)
  • Assign one or more people to a card
  • Add comments to a card
  • Assign a due date to a card
  • Mark a card as important (you can even define your own color codes)
  • Receive periodic summary emails if you don’t visit the board

All of these features make managing alerts in teams with multiple SysAdmins much easier. When an alert comes in, anybody can act on it (e.g. add themselves) or assign it another team member. Any changes are immediately visible to all other team members in real-time (and we at NETIKUS love anything real-time).

Integrating EventSentry with Trello is a 3-step process:

  1. Sign up for Trello, create a board and customize the associated lists
  2. Get an API & access key & determine ID of your list
  3. Setup HTTP action in EventSentry and create/modify rules

Signing up for Trello
To get started, navigate to and sign up with an email address. After you log in for the first time, you will automatically get the “Welcome Board” which will show you all the things you can do with Trello. Since we don’t want to use the default board, we click the big PLUS icon on the top right instead and select “New Board”.

Trello Signup
Give the board a descriptive name, e.g. “EventSentry Alerts”. Once created, the board will contain three default lists. You can either leave the list names as they are, or customize them as shown in the screen shot below. I chose “Active”, “Working on” and “Resolved”.

Template board for EventSentry alerts

Template board for EventSentry alerts

Getting an API and access key
Now that you’ve signed up, the next logical step is to get the API key so that EventSentry can start submitting events to Trello. So while you are logged in, navigate to and note down (aka copy & paste) the first value “Key”, a 32 character-long hexadecimal value. This is the “main” key for your user account, and will be used whenever you (or EventSentry) make an API request.

The API key doesn’t actually let us access data from the boards, for which we’ll need an access key. There are different types of access keys with customizable expiration dates available, but in this case we’ll just get a read/write key without an expiration date. Navigate to the following URL to get a universal read/write access key and substitute APIKEY with the key you obtained just before:,write

You will end up with a dialog similar to the one shown above, where you need to click the green “Allow” button. This will issue another hexadecimal key, this time 64 characters in length. Note this key down as well. Of course you can be less generous and issue keys which expire automatically, e.g. after 30 days. See the Trello docs for more details on the different “expiration” options available.

Getting the list ID
Our end goal is to submit cards to the “Active” list on our “EventSentry Alerts” board. In order to add a new card to this list however, we’ll need the list’s ID. Equipped with our main key and access key, we’re almost there. First, navigate to your “EventSentry Alerts” board in Trello (or whichever board you want to submit cards to) and note down the URL. For example, if the URL is, then you’ll want to extract the text between the /b/ and the board name, gePT9Wax in this case. Now, navigate to the URL below, and replace APIKEY with the API key, and ACCESSKEY with the access key:

This will return detailed results in JSON format similar to this:

 "name":"EventSentry Alerts”,
 "prefs”:  { ……… }
 {"id":"561e92617481e9a123aef3b01","name":"Working on","closed":false,"idBoard":"561e92617481e9a123aef3aff","pos":32768,"subscribed":false}, {"id":"561e92617481e9a123aef3b02","name":"Resolved","closed":false,"idBoard":"561e92617481e9a123aef3aff","pos":49152,"subscribed":false}

What we are interested in is the list id of our “Active” list, 561e92617481e9a123aef3b00 in the example above. With the last missing piece of the puzzle in our hands, we’re now ready to setup a HTTP action in EventSentry.

Configuring EventSentry
Right-click the actions container or utilize the ribbon to create a new HTTP action. In the action dialog, specify the following URL, replacing LISTID with the list id we just obtained:

In addition to the URL, we’ll need to specify at least 4 form fields:


The key and token fields need to be replaced with your API key and access key, whereas the name and desc fields can be customized to suit your needs: what I have shown above is just an example which should work reasonably well in most cases. You can add or remove other event variables as you wish. The upcoming v3.1 will include Trello in the template list to make this a bit easier.

Screenshot EventSentry HTTP Action Trello

Configuring an EventSentry HTTP action for Trello

Once the action is configured, click the Test button to ensure that all IDs have been specified correctly. If the test succeeds, then you should see a new card in the “Active Alerts” list in the EventSentry Alerts board.

Of course an action alone will not forward any alerts to Trello, so you will need to make some changes to your filters and packages. You can either modify existing filters / event log packages and replace the email action with the new Trello HTTP action, or add the Trello action to existing event log packages / filters. Remember that actions can be defined on a package-level through the package properties as well which can help save time.

Managing Alert Cards
Once your first alert card arrives in the “Active” lists and is analyzed by a team member, a few actions can be taken:

  • You can add a team member to the card, essentially assigning the alert to them. You can add multiple team members as well
  • If the event is a false alert, it can be moved to a “False Alert” list, which would indicate that an exclusion filter should be setup in EventSentry
  • You can assign a due date, if the alert requires a resolution by a specific date
  • You can add a comment to the card
  • You can label the card (e.g. “Important”)
  • You can archive & delete the card
EventSentry alerts shown on a trello board

EventSentry alerts shown on a trello board

As you can see, despite its simplicity, Trello offers quite a few features to manage and collaborate. This ensures that alerts don’t disappear in an email inbox somewhere and instead are acted upon, while also allowing collaboration with comments, due dates and such.

Additional Tips & Tricks for Trello
In order to get alerted when a new alert card is created in the EventSentry Alert boards, you’ll need to subscribe to the board. This ensures that you will get a notification on your mobile phone, browser (when enabled or email every time there is activity on a board. Activities include new cards being created, cards being moved to a different list, users being added to cards and so forth.

Note: You will not get a notification if the EventSentry Agent is submitting new cards while using your access key (only other users will see the alerts). This is because Trello assumes that you are creating the cards, and subsequently not notifying you about them.

One way to circumvent this restriction is to create a “service” account (e.g. and issue the access token under this user. Then, everybody will see the alerts.

But don’t stop there!
Of course you can use Trello for what it was originally designed to do as well – manage tasks. We’ve found it to be a great and easy way to handle ToDo lists for teams, resulting in more transparency and efficiency. Assigning a task is quick and easy, and team members can easily track progress with projects – without pesky emails floating around between team members.

Now you just have to get all your To-Do items actually done too. But at least I can now move my “Create Trello Blog Post” card into the “Done” list. And that feels good.

Managing Windows Services & Service Credentials

Every Windows server runs a seemingly ever increasing number of services which range from built-in services providing core Windows functionality (e.g. Print Spooler, Bitlocker, WMI) to 3rd party services added when installing 3rd party software (e.g. various software update services, MySQL) – all of which run in the context of a specific user account.

For example, Windows Server 2012 includes more than 300 services, about half of which are automatically running (this particular server has SQL Server installed as well):

Services on Windows Server 2012 grouped by user

Services on Windows Server 2012 grouped by user

That user account is either a built-in security principal of Windows (e.g. NetworkService), a user account specifically created for that service, or another user account from the server or domain.

Common Practices
Services should always run under a user account which has the least amount of privileges necessary to do its job. It’s common, and often tempting, to run a service an administrative account like “Administrator”. While this often the easiest way to “get it working”, it’s also the least secure.

When a service runs under the “Administrator” account – especially if it’s the domain Administrator account – the service has almost unrestricted access to all resources on the host or, in case of a domain admin, on the domain. This is not something a service usually needs nor you want. It also means that the service will stop working whenever the password of the Administrator account is changed (the service will continue to work until it is restarted).

Less is Better
Whenever possible, try to use one of the built-in security principals available in Windows to run a service under, or create a specific user account for the service. For example, if you have a file synchronization app which runs as a service, create a “ServiceFileSync” or similar account and configure the service to run under that account. Carefully examine the rights the service requires, and only assign those privileges to the user account which the service actually needs.

When creating the user account, give it a very strong & complex password. Users won’t have to log on with that user account, so the password can be complex and long. You can optionally check the “password does not expire” option if you feel that the password is sufficiently secure and you have a short password expiration policy on your domain which could interfere with the service starting after the password expired.

In domain environments I also recommend giving those user accounts (since you will most likely end up with more than one) either a common prefix or suffix (e.g. svc_mysql) and/or moving the accounts into a specific OU. This makes managing and distinguishing these accounts easier – especially in teams with more than one SysAdmin.

The quick way: Local Services grouped by User Account

Sample output from srvsec

Sample output from srvsec

To view all locally installed services grouped by the user account they are running under, download the EventSentry SysAdmin Tools and just run srvsec.exe. This will show you all locally running services, and group the output by the user account they are running under. Srvsec can also be pointed at a remote host, and can also change the passwords stored in services. Click here for more information on srvsec.

Srvsec is a great tool to quickly see what’s going on a single host, but to manage services on an entire domain effectively a more scalable solution is available: EventSentry + AutoAdministrator – the dynamic duo!

The right way: Making sense of ALL installed services
Even when passwords for service accounts are sufficiently strong, they should still be changed on a regular basis. But which services are installed where and are using which service account?

If this is your first time examining service accounts on your network, you should first identify which services run under which user accounts. EventSentry’s service monitoring feature combined with the web-based reporting really makes this a breeze. Assuming that you have a service monitoring system health package assigned to all of your servers, you can simply open the web reports and navigate to Status – Services and get a birds-eye view of all installed services.

In the Overview view, all installed services are grouped by common attributes, including startup type (automatic startup services vs manual startup services), current status, service name and, most importantly for this post, the service user account.

Service overview of all services installed in a domain / forest.

Overview of all installed services in a domain.

Click the “Show All” link to see all user accounts, or click on a specific user account (e.g. “LocalSystem”) to filter the list and only show services running under this specific user account. In most cases you will want to click on “Detailed” to see a list of all services with more detail.

In addition to filtering and viewing details, you can also click on the header of the

All user accounts used by services

All service user accounts

username (or any other) column to see a chart depicting all user accounts used by services from all monitored servers and workstations.

Any report viewed in the web reports can also be scheduled with a job, e.g. a list of all user accounts used by services could be emailed daily/weekly. Simply click the “Save as Report” link to create a report and setup a job.

Managing Services
The standard way to configure the user account and password used by a service is through the “Services” application in Windows. This works well for one or two servers, but not when you need to update the password for a service on multiple hosts.

Managing services with AutoAdministrator

Managing services with AutoAdministrator

This is where AutoAdministrator comes in: A free graphical tool which lets you do just that (and quite a bit more): Update the username and/or password of a service on multiple servers in a domain or work-group. Since AutoAdministrator is multi-threaded, even tasks affecting a large amount of hosts usually only take a few seconds.

To update the stored password of a service, open AutoAdministrator and select “Services” from the drop-down list on the top left.

Service Key Name

Service Key Name

Next, select the service you wish to update from the “Service key / display name” drop-down. If the service is not listed, simply specify the service key name in the service field. The key name is the internal name used by the service and can be obtained by double-clicking a service name in the “Services” MMC application in Windows.

Updating service credentials

Updating service credentials

Next, click on the “Set logon” tab and specify the new username and/or password. Of course you can also specify other service actions, such as restarting the service or changing the start-up type.

As the next step, select the hosts you wish to apply the selected changes to. You can select hosts from Active Directory, EventSentry, custom groups or work groups (Microsoft Windows Network).

Once the correct hosts are selected, click the “Start” button. The number of hosts which will be affected by any action is always shown on the bottom right of the application.

The Essential Security Tools of the EventSentry SysAdmin Tools

toolsPart of our larger EventSentry network-management solution, the freeware EventSentry SysAdmin Tools offer a set of graphical and command-line utilities designed to help you with your daily administrative tasks. These tools are constantly under development, always being honed to provide simple yet powerful functionality. Three of these tools are vital security utilities: Password Assistant, Service Secure, and Task Secure. Let’s take a look at what they offer.

Password Assistant
Password Assistant is a simple yet powerful tool that lets you update the passwords of user accounts on multiple Windows machines. You simply enter the username, the old password, and the new password (with confirmation); after doing so, you can select the computer name(s) from a network neighborhood list (with a filter option) or choose the computer(s) from a text file. The update process can also be logged to a text file.
All the parampassword_assistanteters of the command-line utility are provided in clear terms: The /u (or /username) parameter specifies the username whose password needs to be changes, the /pwold parameter takes that account’s old password, and the /pwnew parameter accepts the new password; there are also /n (or /network), /f (or /filepath), and /filter parameters for specifying computers, as mentioned above. The /ignore_rest parameter ignores the rest of the labeled arguments following this flag, and the /version parameter displays version information and exits.
A great sample use of Password Assistant is when you need to update the administrator passwords on all of the workstations in your environment. Password Assistant provides a one-stop interface—or a simple command-line utility—for making the task hassle-free.

Service Secure provides a simple command interface that displays all of your system’s services, grouped by service account. The tool also lets you easily reset service passwords by specifying username and password rather than having to manually configure those services through the Microsoft Management Console (MMC).srvsec_1
All the parameters of the command-line utility are provided in clear terms: The /p (or /password) parameter sets a password; the /c (or /changepwd) parameter changes a password; the /r (or /restart) parameter restarts the service after the password has been changed; the /u (or /username) parameter lists only those services running under a certain username; the /ignore_rest parameter ignores the rest of the labeled arguments following this flag; the /version parameter displays version information and exits.
Suppose service security has been compromised, and you need to quickly change the passwords of a user account used by a number of services. That task is no longer a logistical nightmare: You can now simply use ServiceSecure in a batch file and update all affected services in your entire network in a matter of moments.

Task Secure provides a simple command interface that displays all of your system’s scheduled tasks, grouped by task account. The tool also lets you easily manage the passwords stored in scheduled tasks on your network. Simply specify the username and password, and Task Secure will reset the password stored in all scheduled tasks (using the specified username) on the specified computer (local or remote).

TaskSecureAll the parameters of the command-line utility are provided in clear terms: The /r (or /remote_host) parameter lists all the scheduled tasks on a given host; the /u (or /username) parameter lists only those scheduled tasks running under a certain username; the /p (or /password) parameter sets a password for every scheduled task configured for a certain user account; the /ignore_rest parameter ignores the rest of the labeled arguments following this flag; the /version parameter displays version information and exits.
Suppose you need to quickly change the passwords of all the scheduled tasks used by one user. You can now simply use Task Secure in a batch file and update all scheduled tasks in your entire network in a matter of moments.

More to Come!
This is just a taste of the free, constantly evolving tools available in EventSentry SysAdmin Tools. Give them a try—you won’t be able to stop with just one.

Automatically Restarting a Failed Windows Process

Whether it’s a critical process running on a server or an application on your desktop – sometimes processes terminate and need to be restarted – immediately.

With EventSentry & EventSentry Light you can do just that: Automatically restart processes immediately after they terminate.

In the past, one drawback of EventSentry launching a process was the side effect that any process started by the EventSentry agent would run under the same account as the EventSentry agent itself (usually a privileged domain account or LocalSystem).

In this post I’ll discuss how you can work around that limitation in a secure manner using a scheduled task. When the critical process fails, instead of launching the process directly through a process action, EventSentry will trigger a scheduled task instead. Why? Because scheduled tasks allow you to configure under which user a task will run – and the user’s password is securely stored in Windows.

The recipe for accomplishing this feat is as follows:

  • Process Monitoring monitors the process
  • An event log filter looks for the “failed process” event and triggers a process action
  • The process action starts a scheduled task

Let’s look at this in detail. First, on the host where the critical but unstable task is running, you create a schedule task in the Windows “Task Scheduler”. Under General, give the task a descriptive name (“Start Super Important App”) and change the user under which the program should be running under. In most cases you will also want to make sure that you configure the task to run whether a user is logged on or not. Then, under “Actions”, add a new action “Start a program” which points to the executable that should be launched. After you click “OK” you will be prompted for the password for the user.

Scheduled Task

Creating a scheduled task

The next step is to setup process monitoring in EventSentry. Right-click “System Health” and create a new package and assign it to the computer(s) in question. Right-click the newly added package and select “Add – Processes”. Click the newly added object and add the name of the process which should be monitored. You can configure how many instances of the processes are required, and with which severity the event will be logged when the process is inactive.

process monitoring

Configuring process monitoring

Now we create a new “Process” action. Right-click the “Actions” container, select “Add” and enter a descriptive name (e.g. “Trigger Super Important App”). In the Filename field specify:


And for the Command Line Arguments enter:

/Run /TN “Start Super Important App”

This uses the built-in Microsoft utility schtasks.exe to run the task we created in our first step. At this point EventSentry will monitor the specified process and log an event if the process is inactive. And while we do have an action to trigger the scheduled task, we still need to tell EventSentry when to launch that action.

EventSentry Process Action

Configuring a process action to start a scheduled task

For the next step, right-click the “Event Logs” container, select “Add Package” and give that package a descriptive name. Then assign the package to the same host. Right-click the newly added package and add a filter by clicking “Add Filter”. In the filter dialog, add the “ Trigger Super Important App” action to the action list and configure the following fields:

Event Log Include Filter Rule

Setting up a rule to trigger the process action

Event Severity: Information
Event Log: Application
Event Source: EventSentry
Category: Process Monitoring
Event ID: 10401
Content Filter (wildcard): *critical_app.exe*

Important Notes: The event severity will need to match whichever severity you selected when adding the process monitoring object in the system health package. The content filter can also be configured to match insertion string #1, in which case the wildcards are not necessary.

And that’s all there’s to it, simply save the configuration when you are done. If the process is running on a remote host then don’t forget to push the configuration to that host.

EventSentry Mobile v1.3 for iOS Available!

We’re excited to announce EventSentry Mobile v1.3 for the iOS platform. The new version remains free, is optimized for iOS 7, sports a refreshed interface and also includes a brand-new feature for pinging hosts in real-time.

Since the new version of EventSentry Mobile was optimized for the new web reports in EventSentry v3.0, it will not work if you are running EventSentry v2.93 on your network.

The interface of the app remains largely the same, but integrates more seamlessly with the iOS 7 flat look. Icons on the network status dialog are now only shown when a host is at least in a warning state, and the icons on the heartbeat dialog have been changed to new flat icons.

EventSentry Mobile Network Status

Network status dialog

Viewing computer details has been tweaked for hosts which are monitored via SNMP since there is currently less information available when compared with Windows hosts monitored by the EventSentry agent.

But since we didn’t feel that visual and internal changes were enough, we also added a new “Tools” section to the app which can:

  • Ping a host
  • Perform a DNS lookup
  • Provide GEO lookup data (when available)

The tools dialog has a single input field with one button where you can enter either an IP address or host name. After you enter the host name or IP address and click the target button, the web reports will ping the host, perform a DNS (reverse) lookup and try to obtain GEO information.

EventSentry Mobile Tools

Ping, DNS & GEO lookup tools in EventSentry Mobile app

The ping and DNS lookup are performed from the web reports rather than from the phone directly, GEO data is pulled by the iOS device from If the IP address is not private (e.g. 192.168.1.x) and coordinates are available then the location can be viewed on a map directly on the iOS device as shown below.

Location of IP address after GEO lookup of

Location of IP after GEO lookup of

Reminder: If you are using EventSentry v2.93, iOS 7 and have automatic updates enabled on your iOS device, then EventSentry Mobile will stop working.

If you have an iPhone or iPad and haven’t already installed EventSentry Mobile then simply search for “EventSentry” in the App store or click here. Enjoy!

CryptoLocker Defense for Sysadmins

It seems as if CryptoLocker has been making the rounds lately, much to the dismay of users who don’t have working backups of their precious office documents.

While I admire Cryptolocker’s simplicity and effectiveness from a purely technical and entrepreneurial standpoint, what the software is doing does appears to be illegal in most countries and so I’d like to offer some advise on how to tame the beast. If you’re looking for a 5-minute fix then I have bad news: implementing the CryptoLocker defense I have outlined below, while completely free, will take a little more than 5 minutes to implement. But knowing that you have an effective defense against CryptoLocker may very well be worth it. After all, CryptoLocker seems to find its way into a lot private networks these days.

CryptoLocker Screenshot

The ideas set forth in this blog post apply mostly to Windows-networks with file servers, but could be adapted for individual computers as well (though this is not covered here – let me know if you’d like me to include this scenario).

About CryptoLocker
For those who have not heard of CryptoLocker yet, it is a piece of software which encrypts pretty much all common office-type documents, including Microsoft Office, AutoCAD, PDFs, images and more. This blog article from MalwareBytes has a complete list of extensions. Once encrypted, CryptoLocker charges you to decrypt (your own files) again. It’s public key cryptography gone wrong; I wonder if Diffie & Hellman saw this one coming. And to make the whole spiel even more interesting, you only get a limited amount of time to pay before your files will remain encrypted. Forever. Oh – and the longer the wait, the more you have to pay. And with recent bit coin exchange rates in excess of USD 1000, the amount that needs to be paid can be uncomfortably high.

It is pretty difficult to defend against something like CryptoLocker other than through usual means of AntiSpyware software, user eduction and strict policies against opening and downloading files from the Internet, email attachments and such. In most cases CryptoLocker comes in form of a ZIP attachment disguised with a PDF icon.

One reason CryptoLocker is so effective – yet difficult to block – is because it exhibits the same behavior as users would: It “simply” accesses and modifies files like a user would. And infecting a machine isn’t all that difficult since CryptoLocker doesn’t require any elevated permissions to run. On the contrary, it wants to run in the same context the user does, so that it can access and see the same files a user does. As such, security features like UAC are utterly useless against ransomware like CryptoLocker – it’s a whole new type of software.

The most effective defense against CryptoLocker is to have a working, tested backup. Let me repeat this: A WORKING and TESTED backup. Users have lost all their data because they thought that they had a backup in place when their backup was broken in some way.

We’ve seen posts of users who deleted all the files CryptoLocker encrypted, thinking they had a working backup. They had a backup, but it was apparently not recently tested and as a result the user lost all of their data.

Naturally, CryptoLocker does not like backups. It dislikes them so much that when CryptoLocker runs, it even tries to delete any Windows Shadow Copy backups. Cloud backup services (including Dropbox, Skydrive and Google Drive etc.) which keep versions of your files offer some protection, but restoring older versions of your files may be a tedious process.

The Defense
The most obvious defense against CryptoLocker is AntiSpyware software, e.g. MalwareBytes. Most AntiSpyware & AntiVirus software still uses signatures however, so new versions of the ransom ware often remain undetected at least for a few days.

So instead of detecting CryptoLocker itself, we can sniff its tracks so to speak. CryptoLocker’s predictable behavior can be used against it. CryptoLocker’s objective is of course to encrypt and hold hostage as many files as possible, so to increase the likelihood of the user purchasing the decryption key from the thugs.

And it is that very pattern that we will try to exploit and use as a trigger to detect and take corrective measures. The approach consists of measuring how many files are being changed in a certain time interval, and if a certain threshold is being exceeded (say more than 10 files modified in 1 minute) we assume that CryptoLocker found its way into our castle. Even though users modify their documents on a regular basis, users can usually make only so many changes at a time and most likely at a much slower rate than any sort of script / software would.

Another approach would be to create one or more honeypot or canary files, which we know (or hope) a user would not modify. If a checksum change in one of those files were detected, we could (more or less) safely assume that CryptoLocker was on one of his rampages again and take corrective measures. The honeypot file would have to be modifiable by users (otherwise CryptoLocker would also not be able to modify it), which makes accidental modifications by users possible (although somewhat unlikely).

This 2nd approach isn’t quite as solid in my opinion, since CryptoLocker is most certainly adapting to changes, and may skip files that it may suspect are a trap. For example, it could skip small files or skip directories with a very small number of files and so forth.

A more sophisticated approach, where we detect an unusually large number of files changes in a small time period, is going to be harder to circumvent by CryptoLocker. The good news is that we have a free (it’s really free, not a trial) software tool available which can do just that. It can:

  • detect file changes
  • measure the rate of file changes (through event log alerts)
  • stop/start services or launch processes
  • send out alerts

EventSentry (Light) to the rescue
EventSentry Light is the free version of our full-spectrum monitoring & compliance solution EventSentry. The features we can utilize to come up with a defense are:

  • File Checksum Changes (part of System Health Monitoring)
  • Filter Thresholds (part of Event Log Monitoring)
  • Action (control services, send out emails)

File Checksum Monitoring
Monitors any folder and detects file size changes, checksum changes and file additions and deletions. EventSentry Light will log file checksum changes to the event log (it’s big brother can also log them to a database), which in turn is monitored by the real-time event log monitoring component.

Event Log Monitoring & Thresholds
This component supports a variety of sophisticated features, one of which are thresholds. The thresholds feature lets you essentially detect event log entries that occur at a certain pace. For example, if 10 specific events occur in 1 minute then let me know and/or take corrective action.

Service Action, Email Action
EventSentry supports a variety of action types to be triggered when an event occurs, with email usually being the most commonly used one. You can also control services, use REST APIs, launch processes and much more. We’ll use the former to stop the file sharing services (LanmanServer) when we have determined that CryptoLocker is on the loose.

I will go into step-by-step instructions on how to configure EventSentry at the end of the post.

The Baseline
The most difficult thing to determine is the maximum rate of file changes we deem normal, as we need to have a baseline in order to configure the threshold slightly above that. This number will vary from network to network, with file servers serving lots of users obviously requiring a larger threshold. I’d like to repeat that determining the right threshold is very important. If it is too low, then normal user activity will trigger an alarm; if it is too high then the alarm may never be triggered and CryptoLocker won’t be caught in time.

The best approach is to setup file monitoring and let it do its job for 1-2 days to determine a baseline. Once the baseline is established, we can increase it by a certain factor (say 1.5) and use that as the threshold.

Setting up the trap requires 3 steps. In this case we assume that EventSentry is either installed directly on the file server, or an agent is deployed on the file server (in which case you will need to make sure that configuration updates are pushed to the file server(s) in question).

Step 1: Monitoring the directory/ies
In EventSentry, right-click the system health packages and add a new package. Right-click the package, select “assign” and assign it to all file servers. Right-click the package again and add a “File Monitoring” package. Click the new object. Directories are monitored in real time by default, but EventSentry requires a recurring scan as well – in case Windows doesn’t send real time notifications. This is usually a good thing, but when you are monitoring large directories it’s best to set the interval very high (future versions will allow for this to be unchecked).

File Checksum Monitoring Settings

In the package, add all the folders which should be monitored and only check the “checksum change” check box. Do not check any of the other check boxes in the bottom left section at this time. Since we haven’t established a baseline yet, we’ll set the severity of the event log alerts to “Information”. If the monitored folders contain a lot of non-Office files then it may be a good idea to adjust monitoring so that only office files (e.g. .doc, .xls, etc.) are monitored. If you prefer to monitor all files, simply change the setting to the green PLUS icon and make sure the list of exclusions is empty (or specifies files that should be excluded, e.g. *.tmp). Below is a screenshot of how this can be configured.

File Checksum Monitoring Settings

When you save the configuration, EventSentry will enumerate all files in the folder and create an initial checksum for every file. The agent will log event 12215 when the scan starts, and event 12216 when the scan is complete. When that happens, EventSentry is essentially “armed” and will detect, and log, all checksum changes to any of the files in the monitored directories.

At this point we’ll want to let this run for at least 24 hours during a “normal” work day, as to determine how many file changes occur on average. You are going to be at a bit of an advantage if you are running the full or the trial version with database support, as it will be a lot easier to determine the number of file changes occurring through the web-based reporting.

Step 2: Setting up the trap
Now that we have established a baseline, we’re ready to setup a threshold. This time we’ll create a new event log monitoring package. Right-click “Event Log Packages” and add a new package and call it “CryptoLocker Rules”. Like before, assign it to the file servers we are monitoring. Right-click the package again and add a new event log filter. Configure the filter as shown in the screenshot below. Note that we are triggering an email action for now. The content filter can be used to restrict the filter further, e.g. to only match certain directories if you are monitoring several directories with EventSentry.

Event Log Filter Setup

Now things are getting interesting. The goal is to create an error event in the event log when X amount of file checksum changes occur in a given time period. To get there, we’ll start with the “General” tab where we tell the filter what type of event we are interested in (see below). Once that event is defined, we’ll move on to the “Threshold” tab which is where we specify the threshold parameters. For the purpose of an example, let’s assume that we have established a baseline of 100 file checksum changes per day, with a work day starting at 8am and ending at 7pm. Assuming that activity is somewhat spread throughout the day, this amounts to about 9 file changes per hour. Naturally we’ll have to assume that file changes aren’t always evenly spread out throughout the day, but setting up a “if 20 checksum changes occur in 1 minute shut file sharing down” is probably a reasonable threshold. Configure the threshold as shown in the screenshot below, with whichever threshold you came up with.

Event Log Filter Threshold Setup

Step 3: Triggering corrective action
When our threshold is reached, EventSentry will log an error to the event log with event id 10601 and trigger the specified action(s) from the “General” tab (Default Email) one time per threshold interval.

At this point we would merely receive an alert when we suspect that CryptoLocker is at it again. If you are cautious then you can retain this setup for a little while (e.g. a day or two) to ensure that you are not getting any alerts about the threshold being met (assuming that CryptoLocker is not active on your network in which case you should get the emails).

To go all in and trigger a server service shutdown, we’ll need to create a service action now. On Windows, file sharing services are provided by the “Server” service, which uses the internal name of “LanmanServer”. The service action allows you to control any service (start/stop/restart), and in this case we’ll obviously want to stop the server service, so that clients cannot access the file shares on your server anymore. We’ll trigger an email action at the same time of course, so that the sysadmin in charge is aware of what is going on. While shutting down all file services seems a bit extreme, it’s unfortunately the most effective way to prevent more files from becoming encrypted.

So for the next step, right-click the “Actions” container and select “Add Action”. At the selection dialog choose the “Service” action, enter a descriptive name (e.g. “Stop File Sharing”) and hit enter.

Selecting an EventSentry Notification

Then, configure the settings of the service as shown in the screenshot below.

Action to stop the LanmanServer service

The last step of our setup (congratulations if you’ve made it that far) is to assign the service action to the filter we previously created. After all, a service action which isn’t referenced anywhere doesn’t do much good. So head back to the Event Log Packages, find the “CryptoLocker Rules” package and edit the filter in the package. In the action list on top, click the “Add” button and add the action you just created.

If at all possible I’d recommend testing the EventSentry setup at a time when your users are not interrupted. Adding a few template files to one of the monitored folders and changing them in short succession (a script may be necessary depending on how short your threshold interval is) should trigger the file services shutdown procedure. Once verified, you can just start the “Server” service again.

Just like in the real world, network viruses come in all shapes and sizes – only limited by technology and the imagination of the cyber-evildoers.

I hope that this article gave you some insight into CryptoLocker and a good way to guard against it. As always, make sure that your company has the following in place:

  • Email Attachment scanning
  • Working, tested backups
  • User education
  • AntiSpyware software

With those in place, one should be able to keep future infections to a minimum.

Stay safe & decrypted.

Mobile Alerts: Pushing EventSentry alerts directly to your mobile devices with Prowl and NMA

When it comes to mobile alerts, email seems to still be the prevalent method of choice for many IT pros. There are many good reasons why network alerts delivered via email are convenient:

  • easy to configure
  • uses existing infrastructure
  • every smartphone and tablet supports email
  • supports attachments (e.g. performance charts from EventSentry’s performance alerts)
  • integrates into your existing environment – everybody already uses email!

What’s not to like? Well, of course it turns out that some of these advantages can also be a disadvantage:

  • emails are not real time
  • problems on the email server often don’t surface immediately
  • important alerts can be overlooked in the inbox jungle
  • you cannot be alerted about email problems via email (duh!)

Viable Email Alternatives
Thankfully, there are a number of alternatives that can be used as an email substitute or addition for mobile alerts. In this article I’ll focus on two affordable services: Prowl (for the iOS platform) and Notify My Android (you may have guessed it – for the Android platform) – subsequently referred to as “NMA”. Both of these services consist of apps for their respective platform and a web-based back end which will push the notifications to your device(s) in near real time.

Mobile Alert on iPhone 5 with Prowl

EventSentry event sent to Prowl on an iPhone

Both services offer an HTTP API which we can connect to with EventSentry’s HTTP action. If you have never used the HTTP action in EventSentry, then here is some background: the HTTP action allows you to POST any event (whether it be an event from the security event log or a heartbeat alert for example) to either Prowl’s or NMA’s web service. These notifications are then pushed to one or more mobile devices.

Neither service currently requires a monthly subscription, but both require a purchase (Prowl costs USD 2.99 whereas NMA costs USD 4.99) if you want to send unlimited notifications to your mobile device(s). NMA is a little more welcoming to strangers – it supports up to 5 notifications per day at no charge.

Prowl: Getting Started
iOS app costs USD 2.99, and supports up to 1000 API calls (=notifications) per hour. To get started:

  1. Purchase the Prowl: Growl Client from the Apple App Store and install it
  2. Register for free at
  3. Login & create an API key

Notify My Android (NMA): Getting Started
The Notify My Android app is free in the Google Play store and supports up to 5 API calls (=notifications) per day  for free. Upgrading to a premium account will allow up to 800 API calls per hour. To get started:

  1. Download the Notify My Android app from the Google Play store
  2. Register for free at
  3. Login & create an API key
  4. Optional: Upgrade to a Premium account to allow for unlimited notifications

Once you have the mobile app installed and the API key in hand, you can start setting up a HTTP action in EventSentry. Notifications essentially consist of three fields: The application name, a subject as well as a message field, all of which can be customized. As such, it’s up to you configure which part of an event log alert you will put in the subject, and which part you will put into the actual message. In our example, we generate a dynamic notification subject with the host name, event id, event source and event category. The notification body will simply consist of the event message text, though this can be customized as well.

Setting up a HTTP action
Right-click the “Actions” container (or, in v3.0, use the “Add” button in the Ribbon) and create a new HTTP action. The HTTP action requires a URL at minimum, optional credentials and the actual data fields to submit in the HTTP POST. Conveniently, both Prowl & NMA use the same field names. I suspect that they are adhering to some sort of standard, though I couldn’t find any references.

The first parameter to configure is the URL, which depends on the service you use:


Configuring authentication credentials is not necessary since you are essentially authenticating with the API key you generated. The last step is configuring the form fields. The bold field on the left is the name of the form element and the text to the right the value. A description follows in italic on the next line:

apikey: abdef123123abababafefefe
The API key you received

application: EventSentry
This will displayed as part of the notification. I use “EventSentry” here, but this can really be anything, so it could be the host name as well for example ($EVENTCOMPUTER)
Max Length: 256

This is the subject of the notification. You can use any variables, including insertion strings $STR1, $STR2 etc.
Max Length: 1024 [Prowl], 1000 [NMA]

description: $EVENTMESSAGE
This is the main message body of the notification, the $EVENTMESSAGE seems like a pretty good candidate for this field
Max Length: 10000

priority: 0 (possible values are -2, -1, 0, 1 and 2)
This field is optional and doesn’t do anything with NMA. With Prowl however, a priority can be set to “2” (indicating an “emergency”) which may then override quiet hours on the mobile app (if quiet hours are configured in the Prowl mobile app).

Take a look at the respective API documentation for Prowl and NMA as well; each show additional fields which can be configured, as well as additional information that you may or may not find useful.

The screenshot below shows a fully configured HTTP action for Prowl:

EventSentry HTTP Action for Prowl

EventSentry HTTP Action for Prowl

The HTTP action for NMA would look identical, with the exception of the URL which is different. You can click the “Test” button, which will submit the configured data to the specified URL and should, when configured correctly, immediately generate a notification on the mobile device. Please note that event variables will not be resolved when testing.

To make sure that your mobile device doesn’t get flooded with alerts (some applications have the tendency to generate not one but hundreds of events in a short period of time), I highly recommend that you setup a threshold on the action or the event log filter referencing the action (I personally prefer the former). You can also setup a schedule so that notifications are only sent on certain days and/or during certain hours.

The last step would be to configure an event log filter to forward select events to the mobile device, something that is beyond the scope of this article. See the tutorials below for more information:

It seems only natural to wonder whether alerts sent through these services can be used for mission critical systems. I’ve been using mostly Prowl as I’m an iPhone user, and have been very happy with it’s fast response times (which are almost instant), the service reliability and the stability of the iOS app. Nevertheless, both and NMA state that you should not solely rely on their system for critical alerts and instead setup multiple channels for mission critical alerts. This sounds nice in theory, but suspect that most sysadmins will not want to dismiss alerts on more than one device – something that can get old if you get a fair number of alerts. Switching to a commercial system like PagerDuty with guaranteed up-time may be preferable in that case. I will talk more about PagerDuty in an upcoming post soon.

My experience with the NMA Android app wasn’t as good during the limited testing I performed. While it worked great when it worked, the app did crash on me a couple of times.

If you’re looking for a way to push alerts to your mobile device from EventSentry without using email and without spending big bucks, Prowl and NMA are worth looking into. They’re affordable, responsive and easy to configure.

EventSentry Light Supercharged

The latest EventSentry update brings significant and very exciting changes to the free light edition, EventSentry Light.

We have always seen EventSentry Light as a successor to the original EventwatchNT – a monitoring tool for small networks to alert sysadmins by sending real-time alerts about event log activity. As its big brother EventSentry continued to mature, most features from EventSentry made it into the light edition as well: Service Monitoring, Disk Space Monitoring, Performance Monitoring and many more. Since EventSentry Light was, and continues to be, free, it needed to distinguish itself from the commercial edition. As such, most feature were somewhat restricted in the light edition which only allowed a limited number of packages, event log filters, performance counters and such.


Over the last year we’ve been getting feedback that EventSentry Light was being constrained too much. Since our goal is, and always has been, to empower sysadmins and not constrain them, we decided to provide our users with more functionality in the free edition. The result of these efforts is build, and with this release you can:

  • Monitor event logs and log files in real time, setting up as many filter rules as you’d like, without restrictions.
  • Utilize all advanced event log filter capabilities like thresholds, timers, schedules and more!
  • Create as many event log, log file and system health packages as you like.
  • Utilize all system health monitoring features, such as file checksum monitoring, performance monitoring, service monitoring and more.
  • Create a variety of alerts using mail, HTTP, SNMP traps, Syslog messages and more.
  • Receive SNMP traps from SNMP v3 enabled devices.
  • Monitor up to 2 full hosts and 2 network devices.

Pretty impressive, no? So literally overnight, EventSentry Light has matured into a full-fledged monitoring solution which will alert IT professionals like sysadmins of critical (event) log events, performance issues and much more. What differences with EventSentry remain? A few, but the line is much more clear now:

  1. Reporting. With EventSentry, you get log consolidation, software/hardware inventory, performance trend reports, network dashboards, jobs, JSON/XML/CSV/… APIs and more.
  2. Compliance. EventSentry includes a variety of compliance functionality such as process tracking, logon tracking, account management tracking and more.
  3. Monitor multiple hosts. Monitor as many hosts as you are licensed for, and also utilize command line utilities to automate remote host management.
  4. Support. EventSentry includes quality email and phone support, something we pride ourselves on. EventSentry Light offers forum support.
  5. Mobile iPhone & Android apps are only available in the full edition since they require reporting.

So if you’re not already using EventSentry Light, or using an older version, then you should give it a try. It’s as free as it gets with no registration required, no advertisements and no nagging pop-ups. We hope you like it as much as we do. And did I mention that you can seamlessly upgrade from EventSentry Light to EventSentry? :-)

Don’t forget to check out our other free tools and Facebook / Google+ pages!

EventSentry Mobile Updates for iOS & Android

I’m excited to announce that we released updates to both the iPhone as well as Android version of our mobile app, and both updates are now available in the Apple App Store and Google Play Store respectively.

The iOS version now supports the larger iPhone 5 display as well as a variety of improvements in regards to load speed, rotation and so forth.


The Android version catches up with the iOS version, adding the heartbeat page and fixing an issue where the app would not launch on devices running Jelly Bean. We’ve also beautified the app a bit and donated some color icons to the menu. Yes, the Android app now has officially more color than the iOS version. How crazy is that?

We also had to change how EventSentry Mobile is distributed through the Google Play Store, where our mobile app is now called “EventSentry Mobile” (instead of just “EventSentry”). Existing users of the mobile app will, unfortunately, need to uninstall and re-install the app to get the latest version. This was necessary due to Google’s policies and a mishap on our part. Future releases of the Android app will support updating however, and we apologize for the inconvenience.


To get the free update, either search for “EventSentry” on your mobile device, or visit the links below.

Google Play:


Now, if you use or like the app – why not rate it and write a short review?

your NETIKUS.NET team.