Defeating Ransomware with EventSentry – Remediation

Since Ransomware is still all the rage – literally – I decided to write a 4th article with a potentially better method to stop an ongoing infection. In part 1, part 2 and part 3 we focused mostly on detecting an ongoing Ransomware infection and utilized the “nuclear” option to prevent it from spreading: stopping the “server” service which would prevent any client from accessing files on the affected server.

While these methods are certainly effective, there are other more targeted steps you can take instead of or in addition to shutting down the server service, provided that all hosts susceptible to a Ransomware infection are monitored by EventSentry.

When EventSentry detects an ongoing Ransomware infection, it can usually determine the infected user by extracting the domain user name from the 4663 event. Simply disabling the user is insufficient however, since a disabled user can continue to access the network (and wreak havoc) as long as he or she doesn’t log off. Any subsequent log on attempt would of course fail, but that provides little comfort when the user’s computer continues to plow through hundreds or thousands of documents, relentlessly encrypting everything in its path.

As such, the only reliable way to stop the ongoing infection, given only the user name, is to log off the user. While logging a user off remotely is possible using the query session and logoff.exe commands, I prefer to completely shut down the offending computer in order to reduce the risk of any future malicious activity. Logging the user off remotely may still be preferable in a terminal server environment (let me know if you want me to cover this in a future article).

Knowing the user name is of course great, but how do we find out which computer he or she is logged on to? If you have EventSentry deployed across your entire network – including workstations – then you can get this info by querying the console logon reports in the EventSentry web reports. If you are not so lucky to have EventSentry deployed in your entire environment (we offer significant discounts for large quantities of workstation licenses – you can request a quote here) then we can still obtain this information from the “net session” command in Windows.

Net Session Output
Net Session Output

We’ve created a little script named antiransom_shutdown.vbs which, given a user name, will report back from which remote IP this user most recently accessed the local server and optionally shut it down. Here are some usage examples:

Find out from which computer boris.johnson most recently accessed this server:
cscript.exe C:\Scripts\antiransom_shutdown.vbs boris.johnson

Find out from which computer boris.johnson most recently accessed this server AND shut the remote host down (if found):
cscript.exe C:\Scripts\antiransom_shutdown.vbs boris.johnson shutdown

The script uses only built-in Windows commands, as such there is no need to install anything else on the server where it’s run.

When executed with the “shutdown” parameter, the script will issue a shutdown command to the remote host, which will display a (customizable) warning message to the user indicating that the computer is being shutdown because of a potential infection. The timeout is 5 seconds by default but can be customized in the script. It’s recommended to keep the timeout short (5-10 seconds) in order to neutralize the threat as quickly as possible while still giving the user a few moments to know what is happening.

The overall setup of the Ransomware detection is still the same, we’re setting up a threshold filter to detect a higher than usual frequency of certain 4663 events and trigger an action in response. Only this time we don’t shut down the server service, but instead trigger this script. To properly execute the action, configure it as shown in the screenshot below. The executable is cscript.exe (the interpreter for .vbs files) and the command line parameters are the name of the script, $STR2 and “shutdown”.

Remote workstation shut down
Remote workstation shut down

So what’s the better and safer approach to freeze an ongoing Ransomware infection? Shutting down the server service is the most reliable approach – since it doesn’t require the workstation to be reachable and will almost certainly succeed. Remotely shutting down a workstation has minimal impact on operations but may not always succeed. See below for the pros and cons of each approach:

File Sharing Shutdown
Pros: 100% effective
Cons: Potentially larger disruption than necessary, false positive unnecessarily disrupts business

Remote Workstation Shutdown
Pros: Only disables infected user/workstation, even if false positive
Cons: Requires workstation to be reachable

This ends up being one of those “it depends” situations where you will have to decide what’s the best approach based on your environment. I would personally go with the remote workstation shutdown option in large networks where the vast majority of workstations are desktops reachable (and not firewalled) from the file server. In smaller, more distributed networks with a lot of laptops, I would go with the file service shutdown “nuclear” option.

A hybrid approach may also be an option for those opting for the remote workstation shutdown method: trigger a remote workstation shutdown during business hours when IT staff is available on short notice, but configure the file service shutdown after business hours when it’s safer and affects fewer people. All this can be configured in EventSentry by creating two filters which are identical except for the action and the day/time settings.

It’s important to point out that the EventSentry agent by default runs under the LocalSystem account, a built-in user account which does not have sufficient privileges on a remote host to issue the shutdown command. You can elevate the permissions of the EventSentry agent and work-around this limitation in 2 ways:

  1. Change the service account (fast): Changing the service account the EventSentry service uses to a domain account with administrative permissions will allow the agent to remotely shut down a remote host. This will have to be done on every file server which may issue shut down commands (you can use AutoAdministrator to update multiple file servers if necessary).
  2. Give the “Force shutdown from a remote system” user right: It’s not necessary to issue domain-wide admin rights to the EventSentry agent, the key right the agent needs is just the “Force shutdown from a remote system” user right. The quickest way to deploy this setting is of course through group policy:

    a) Open the “Group Policy Management Editor”
    b) Edit an existing policy (e.g. “Default Domain Policy”) or create a new group policy
    c) Navigate to “Computer Configuration\Policies\Windows Settings\Security Settings\Local Policies\User Rights Assignment”
    d) Double-click the “Force shutdown from a remote system” user right and add both “Administrators” and the computer accounts of the file servers to the list. Alternatively you can also create a group, add the file servers to the group, and add that group to the policy (keep in mind that you will need to restart the file servers if you go with the group method).

    Once the group policy setting has propagated to the workstations, the remote shut down initiated from the file server(s) should succeed.

    Change the "Force shutdown from a remote system" user right
    Change the “Force shutdown from a remote system” user right


Good luck protecting your network against Ransomware infections, also remember to verify your backups – no protection is 100% effective.

Perfect hardware for a TV-based dashboard

Dashboards are a great way to visualize large amounts of information in a concise matter. In IT we usually display various types of network data from a monitoring software, but dashboards are used in all sorts of environments. You can visualize stock data or just show a map of all trucks in a fleet with their current position.

If you work for a large company with a dedicated NOC then you’ll likely have an integrated setup with 4 or more TVs, connected to hardware specialized for dashboards or, at the very least, a powerful PC with multiple PCI cards.

But not everybody has the budget or the need for a NOC like AT&Ts, and one or two TVs can be sufficient for most networks – provided the dashboard is well-designed and customizable of course.


Most dashboards require a fairly recent web browser (if you are unlucky even Adobe Flash), making some sort of a PC or Mac the preferred hardware to power that dashboard. Most IT departments have a plethora of old PCs sitting around, and it can be tempting to resurrect one of those boxes and give them a new life as a dashboard PC. After all, you’re “just” displaying a web page.

In reality, older hardware can a have hard time keeping up with modern browsers and the frenzy of Javascript operations that come with a busy dashboard. The dashboards often run well for some time (hours or days – depending on the hardware and the dashboard), but ultimately buckle under the load. The result is a dashboard that skips updates or breaks down altogether. Even if you do have a decent PC sitting around, it’s hardly a perfect solution since even small PCs take up a considerable amount of space, and cables can quickly get in the way. And I think we can all agree that the last thing we need more of are cables.

Low-cost integrated devices like the Raspberry Pi are tempting, but not perfect either. They’re not usually designed to be used with graphical interfaces, much less with memory and CPU hungry applications like web browsers displaying dashboards.

After trying everything from Raspberry Pi, old Mac Mini hardware and more, we finally found a solution for under $100 – which has now worked quite well for several months: The 1st generation Intel Compute Stick which you can get from online retailers like Amazon, NewEgg and others.

Intel Compute Stick
Intel Compute Stick

Even in its 1st generation (the one we tested) the Intel Compute Stick running Windows 10 Home performed surprisingly well. We’ve been running an EventSentry dashboard (which of course we’re hoping you are running as well) on it since February on Microsoft’s new Edge browser, and we’ve never had an issue.

The Intel Compute Stick features 2 Gb of RAM, is powered by a quad-core intel Atom processor and has 30 Gb of storage, of which more than half are available. This is of course not a machine you’ll want to render videos or play video games on, but plenty sufficient for a web browser from our experience. We were actually pleasantly surprised by how responsive the little device felt overall. Even though you cannot join a domain, you can still install the EventSentry agent on the machine to keep an eye on performance and other system metrics for example.

But there are of course some caveats, as is to be expected from a computer that costs less than $100 and is not much bigger than a USB memory stick. If you’re using Bluetooth and Wifi then you’ll only need to connect the power cord and the setup is clean. Since the stick also sports a single USB 2.0 port, we used a USB hub along with a USB-based Ethernet adapter to connect it to our LAN as well as connect a keyboard/mouse. USB 2.0 didn’t negatively affect performance in our limited use case scenario.

If you need more hardware, maybe because your dashboards are particularly taxing, then you can purchase a newer and faster model as well. The 2nd generation Intel Computer Sticks start around $149 and the high-end models include as much as 64Gb of disk space and 4Gb of RAM.

My first computer was a 80286 with 1Mb of RAM and a 20Mb hard drive, and it was about as big as two shoe boxes. It’s impressive to see a device this small perform that well. If you have the need to turn a TV into a full-blown desktop, then I’d definitely recommend the Intel Computer Stick(s)!

Additional Notes on EventSentry Update v3.2.1.30

Our latest patch for EventSentry v3.2 (v3.2.1.30) requires some additional information in addition to the release notes.

Heartbeat Monitoring (Agent Status)
By default, the EventSentry Heartbeat Monitor ensures that all remote agents are running by querying the status of the remote “EventSentry” service. While is an accurate way to ensure the remote agent is running, the Microsoft RPC mechanism isn’t very efficient when connecting to remote hosts across a slow (WAN) link, and concurrently checking the service status of 100+ hosts at the same time can on occasion also cause issues. In these situations, the heartbeat agent may not be able to monitor all hosts in the configured monitoring interval. Furthermore, querying the remote status of a service requires that the EventSentry Heartbeat Agent run under a domain account, otherwise the dreaded “Access Denied” error appears on the heartbeat status page in the web reports.

To address these issues for larger EventSentry deployments (500+ hosts) and deployments where the remote agents are connected through a slower WAN link, we have added the ability to query the remote agent status through the EventSentry database where the remote agents periodically check in. This check is enabled by default for new installations, but existing installations will need to make a database permission change in order to give the heartbeat agent permission to query the agent status. More information can be found here.

In the next release of EventSentry (v3.3), this functionality will be configurable, and the heartbeat agent will also be able to determine the current agent status by communicating directly with the collector service (when enabled) for even better accuracy. The Heartbeat Monitor will always attempt to revert back to the legacy method of checking the service status directly if it cannot obtain the status through other means.

Service Monitoring: Configuration Changes
EventSentry distinguishes between three types of service changes: Status changes (e.g. Running to Stopped), service configuration changes (e.g. changes to the startup type) and services being added or removed. Up until release, all status changes and service configuration changes were logged with the same event severity, which we didn’t think was very fitting since the status change of a service is very different to a change of the service itself. As such, starting with, only service status changes will be logged under the severity configured under “Monitor Service Status Changes” category. All other service changes will be logged under the severity configured under “Monitor Service Addition / Removal” category.

Management Console: Quicktools
The EventSentry QuickTools allow you to run an application/script against a server or workstation in your EventSentry configuration. EventSentry includes a few default QuickTools entries, such as “Reboot”, “Remote Desktop” and others. Starting with the latest release we added a new “Hide” option, which will not show the executed application on the desktop. This will be useful for integrating our upcoming VNC wrapper scripts (Blog article coming soon), which will allow you to install & launch a (Tiger)VNC client directly from the EventSentry management console.

EventSentry Light 3.2
Starting with this release, EventSentry Light v3.2 will also be available. We have good news for all EventSentry Light users: We have increased the number of full hosts you can remotely manage to 5, and also increased the number of network devices you can monitor to 5. As such you can now monitor up to 10 hosts with EventSentry Light completely for free.

Defeating Ransomware with EventSentry & Auditing

There seems to be a new variant of ransomware popping up somewhere every few months (Locky being the most recent one), with every new variation targeting more users / computers / networks and circumventing protections put in place by the defenders for their previous counterparts. The whole thing has turned into a cat and mouse game, with an increasing number of software companies and SysAdmins attempting to come up with effective countermeasures.

I’ve already proposed two ways to counteract ransomware on file servers with EventSentry in part 1 and part 2, both of which take a little bit of time to implement (although I’d argue less than it would take to restore all of your files from backups). In this post I’m proposing a third, and better, method with the following improvements:

In the first article we configured file integrity monitoring on a volume, and if the number of file modifications occurring during a certain time interval exceeded a preset threshold, the ransomware would be stopped in its tracks. In the seconds article we used bait (canary) files to accomplish the same thing.

In this third installment we’ll keep track of the number of file modifications made by a user to detect if an infection is underway. To effectively defeat ransomware, we have to be able to distinguish between legitimate user activity and an infection. To date we know this:

  • Users add/change/remove files, but the number of changes made by a user in a short amount of time (say 15 min) is generally small
  • Ransomware always runs in the context of a user, and as such an infection will usually come from one user (unless things go really awry and multiple users are infected). The approach here will work equally well, regardless of the number of infections.

Thus, to detect an infection, EventSentry will be counting the number of file modifications (event 4663) with its advanced threshold capabilities. If the threshold is exceeded, EventSentry will trigger an action of your choice (e.g. disable the user, remove a file share, stop the server service, …) to limit the damage of the ransomware.

Here is what you need:

  • Object Access / File System Auditing enabled
  • Auditing enabled on the files which are to be protected
  • EventSentry installed on the server which needs to be protected

This  KB article explains how to configure EventSentry and enable auditing (preferably through group policy) on one or more directories. I recommend referencing the KB article when you’re ready to configure everything. Pretty much everything in the KB article applies here, although we will make a small change to the threshold settings of the filter (last paragraph of section (4)).

Windows Folder Auditing
Windows Folder Auditing

Once auditing is setup, Windows will log event 4663 for every write access which is performed by a user. An example event looks like this:

Windows Event 4663
Windows Event 4663

The default behavior of a filter threshold in EventSentry is to simply count every filter match towards the threshold. In our case, every 4663 event encountered would count towards the threshold. You can think of there being one bucket for all 4663 events, with the bucket being emptied whenever the threshold period expires, say every 5 minutes. If the bucket fills up we can trigger an alert.

This doesn’t work so well on a file server, where potentially hundreds of users are constantly modifying files. It would take some time to come up with a good baseline (how many file modifications are considered “normal”) that we could use as a threshold, and there would still be a chance for a false positive. For example, a lot of 4663 events could be generated during a busy day at the office, thus causing the threshold to reach its limit.

A better way is to assign each user their own “personal” threshold which we can then monitor. Think of it like each user having their own bucket. If a user writes to a file, EventSentry adds the 4663 event only to that user’s bucket. Subsequently, an alert is only triggered when a user’s bucket is full. Any insertion string of an event can be used to create a new bucket.

We can do this by utilizing the insertion string capabilities of the filter threshold feature. Setting this up is surprisingly easy – all we have to do is change the Threshold Options to “Event”, click the “Insertion Strings” button and select the correct insertion string. What is the correct insertion string? The short answer is #1.

The long answer lies in the “Event Message Browser”, which you can either find through the Tools – Utilities menu in the EventSentry Management Console or in the EventSentry SysAdmin Tools. Once in there, click on “Security”, then “Microsoft-Windows-Security-Auditing”, then 4663. You will see that the number next to the field identifying the calling user (“Security ID”) is %1.

Event 4663 Definition
Event 4663 Definition

Enough with the theory, here is what you need to implement it (assuming EventSentry is already installed on the servers hosting the file share(s)):

  1. Enable global auditing globally and audit the file share(s). See section 2 & 3 of KB 279.
  2. Determine what action you want to take when a ransomware infection has been detected. See either section 1 of KB 279 or “Dive! Stopping the Server Service” from the previous blog post.
  3. Create a package & filter looking for 4663 events. See section 4 of KB 279 and review the additional threshold settings below.

Customizing the threshold
Once you have the package & threshold filter for 4663 events in place, we need to modify the threshold settings as explained above. Edit the filter, click the threshold tab and make sure your filter looks like the one shown below:

Threshold Settings
Threshold Settings

The only variable setting is the actual threshold, since it depends on how fast the particular variant of ransomware would be modifying files. A couple of things to keep in mind:

  • The interval shouldn’t be too long, otherwise it will take too long before the infection is detected.
  • Make sure the actual event log filter is only looking at 4663 events, no other event ids.

With the above example, any user modifying any file (on a given server) more than 30 times in 3 minutes will trigger any action associated with the filter, e.g. shutting down the server service. Note that the action listed in the General tab will be triggered as soon as the threshold is met. If 30 4663 events for a single user are generated within 45 seconds, the action will be triggered after 45 seconds, it won’t wait 3 minutes.

Bonus – Disabling a user
One advantage of intercepting 4663 events is that we can extract information from them and pass them to commands. While shutting down the Server service is pretty much essential, there are a few other things you can do once you have data from the events, e.g. the username, available. You can now do things like:

  • Disabling the user
  • Removing the user from the share permissions
  • Revoking access to select folders for the user

There are a couple of caveats when (trying to) disable a user however:

  1. The user account (usually the computer account) under which the EventSentry service runs under (usually LocalSystem) needs to be part of the Account Operators group so that it has permission to disable a user
  2. Disabling a user is usually not enough though, since Windows won’t automatically disconnect the user or revoke access. As such, any ransom/crypto process already running will continue to run – even if the user has been disabled.

Disabling a user account from the command line is surprisingly simple (leave Powershell in the drawer). To disable the user john.doe, simply run this command:

net user john.doe /domain /active:no

Note that since “net user” doesn’t support a domain prefix (MYDOMAIN\john.doe won’t work), we need to make sure that we pass only the username (which is insertion string %2) and the /domain switch to ensure the user is disabled on the domain controller. Of course you would need to omit the /domain switch if the users connecting to the share are local users. The action itself would look like the screenshot below, where $STR2 will be substituted by EventSentry with the actual user listed in the event 4663:

Action to disable a user
Action to disable a user


That’s it, now just push the configuration and you should be much better prepared to take any ransomware attacks heading your users way.

Oh, and check those backups, would you?



3-2-1-Go! EventSentry 3.2.1 is out!

I am e-x-c-i-t-e-d to announce the availability of EventSentry v3.2 and tell you more about the new features and improvements. So, if you’re looking for a little bit more than the release notes then read on!

The biggest new feature in 3.2 is the collector, a new central component which enables a 3-tier architecture in EventSentry. Traditionally, the EventSentry agents have been communicating directly with email servers, databases and other services. While this usually worked well – and is still desirable in many setups – it does impose a limitation in some scenarios:

  • The SMTP server cannot be configured to allow relaying and/or accepting SMTP connections from remote clients
  • The central database cannot be configured to allow connections from remote clients
  • Agents need to communicate over an insecure medium like the Internet
  • Installing ODBC drivers is not an option
  • Remote agents communicate over unreliable network connections (e.g. satellite, laptops, …)

The collector addresses the above limitations by acting similar to a proxy between the remote agents and the service (e.g. database). In a nutshell, it provides the following benefits:

  • Agents only communicate with the collector over a single port
  • All traffic can be encrypted and compressed
  • Database connection details do not need to be stored on the agents anymore
  • All collected data is cached on the agents if and while the collector is unreachable

Whether you will need the collector or not will largely depend on your network setup. If all of your hosts are in the same data center and/or the same LAN, the collector may provide little benefit. If you are a MSP and monitoring remote sites and laptops however, then the collector is probably what you have been waiting for!

When upgrading (or installing from scratch), the post-installer configuration assistant will ask you whether you are interested in enabling the collector.

Collector Status
Collector Status in maintenance menu

If you are installing from scratch, then enabling the collector during the installation is all you need to do. When upgrading, an additional step is required – an action needs to be configured to use the collector. While the collector service is installed & started during the upgrade when selected, it will not enable any of the existing actions to use the collector. As such, if you want to route data for a specific action through the collector, that needs to be configured. Simply edit the action and click the “Use Collector” check box on the bottom left and push the configuration.

In version 3.2.1, the following actions can be routed through the collector:

  • Database
  • Email (SMTP)
  • Syslog
  • Text File

Since the collector, when enabled, is a critical component, we recommend monitoring the collector stats either through the collector status page (Maintenance -> Collector Status) or by adding the collector status tile to one of your dashboards.

There is one other advantage the collector can bring when routing emails through it:

  • Emails from multiple hosts can be grouped together (if the action polling interval is sufficiently high)
  • Action thresholds can now be applied centrally

Both features can help reduce the number of emails you receive from EventSentry, usually a popular thing to do!

Compliance Modules
EventSentry has always included the compliance tracking components which monitor and interpret Windows security events. Compliance tracking provides process, console, account management and other tracking reports. While popular and extremely useful, the compliance reports themselves don’t tell the user which particular compliance requirement they address.

Say Hello to the new compliance modules, which provide detailed, out-of-the-box reports for:

  • GLBA
  • Sarbanes Oxley

Once a compliance module is enabled, it will install a number of reports that pertain to the specific compliance requirement that was enabled. Every report will be associated with a specific control (e.g. PCI 10.2.2) and allow you to setup a required review, job and more.

PCI Compliance Reports
Example of PCI compliance reports

(Network) Switch Mapping
Finding the port on a switch to which a server, workstation or network device is connected is often a time-consuming and annoying process for most SysAdmins. Starting with version 3.2, EventSentry tries to ease that pain by showing exactly to which switch – and port – a host is connected to. All you need to do is add the switch to the EventSentry configuration, make sure that it can be monitored via SNMP and that it provides the MAC to port mappings via SNMP (OID – iso.identified-organization.dod.internet.mgmt.mib-2.bridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbPort). This feature should work well with all mainstream managed switched, and we haven’t run into a switch yet where this feature wasn’t provided or did not work.

Server Room Cables
Server Room Cables

Once EventSentry pulls the MAC to Port mappings, you will be able to retrieve the collected information in two ways:

  • Through the Inventory – Switch page, which will show all monitored switches and connected devices
  • Through the Inventory – Host page. If the switch port can be detected, it will be displayed next to the IP address of the network card

Since switches only provide MAC addresses, EventSentry attempts to map MAC addresses to host names and IP addresses by analyzing the hardware inventory details as well as the ARP status table when available. As such, it is recommended to enable the ARP component of the network services if the results on your switch inventory page are incomplete.

EventSentry Switch Port Indicator
EventSentry Switch Port Indicator

Web Reports Improvements
Starting with a visual overhaul of the interface, the web reports also received an internal overhaul to improve overall performance, especially when using multiple profiles. The performance trends page can now display multiple charts on a single page, and the host inventory page now shows the highest supported USB version on that host.

Managing multiple reports is easier now through the ability to bulk-edit reporting settings. Reports can now also be saved to a folder instead of being emailed.

Finally, the web reports are now also officially available in 6 additional languages: French, Spanish, Polish, Portuguese, Dutch and Italian. This brings the total number of supported languages in the web reports to 9!

Management Console Improvements
Improvements in the management console pertain mostly to remote update and computer management. Hosts can now be imported from a network scan, which is particularly useful when managing network devices which often don’t show up in Active Directory. The network scan is multi-threaded and can scan a class C subnet in a few seconds and even supports checking TCP ports for hosts which have ICMP disabled.

Remote update can now store the result of every activity in CSV file(s), and output from remote update can be toggled with the context menu to apply remote update actions to a sub-set of hosts easily.

Also new is the ability to export all event log filters to a CSV file allowing you to analyze the results in your favorite spreadsheet application to identify issues, duplicates etc.

That’s all folks. Time to get cracking on 3.3!

EventSentry SysAdmin Tools: New SNMP query utility “snmptool”

I’m excited to announce a new version of our free EventSentry SysAdmin Tools which, in addition to bug fixes and minor improvements, also includes a new command-line tool: snmptool. This brings the total number of utilities in the toolkit to thirty (30)!

Free SNMP tools for Windows® are not easy to find and often require you to memorize the various OIDs in order to test a remote host’s SNMP functionality, or to get useful information back.

Our free snmptool utility solves that problem by giving you a simple utility which downloads a variety of stats, depending on what the remote host provides via SNMP, and displays it to the user. For example, if you are querying a VMWare® ESXi™ host with the snmptool, it will – among other stats – enumerate all VMs configure on the host, whereas it will display switch port mappings when querying a switch.

The snmptool currently retrieves the following:

  • System Description string
  • Operating System
  • Uptime
  • Current CPU usage
  • Network interfaces (name, MAC address, IP if available)
  • Mounted disks
  • Running processes
  • Virtual Machines (ESXi™ only)
  • Switch port mappings

Running the utility is incredibly easy, simply specify the SNMP credentials and the remote host, and the utility will do the rest on its own:

C:\>snmptool /u public linuxserver
System Description: Linux openvas.netikus.local 4.32.22-573.7.1.el6.x86_128 #1 SMP Tue Sep 22 22:00:00 UTC 2019 x86_128
OS Info:            Linux 4.32.22-573.7.1.el6.x86_128 #1
Current Uptime:     3 years, 321 days, 3 hours and 52 minutes
CPU Usage:          0%
01: eth0 00-80-73-C3-57-BF (
01: DISK / (13892 Mb, 67% free)
02: DISK /dev/shm (938 Mb, 100% free)
03: DISK /boot (476 Mb, 75% free)
01: init, PID=1
02: watchdog/1, PID=10
03: ext4-dio-unwrit, PID=1000
04: kauditd, PID=1035
05: migration/2, PID=11
06: flush-253:0, PID=1129
07: stopper/2, PID=12
08: kdmremove, PID=129
09: ksoftirqd/2, PID=13
10: kstriped, PID=130
125: kthrotld/3, PID=91
126: pciehpd, PID=92
127: kpsmoused, PID=94
128: usbhid_resumer, PID=95
129: deferwq, PID=96
130: jbd2/sda1-8, PID=999

The output is completely dynamic, if no processes are found (e.g. you are querying a switch) then that section will simply be omitted.

In addition to the brand new snmptool, version of the EventSentry SysAdmin Tools includes the following other improvements:

Added the new /a parameter which checks the target folder against a pattern for additional safety

Added support for authenticating against a login page, including login pages which redirect

I hope the new utility and other improvements will help make your job easier. Oh, and you can download the EventSentry SysAdmin Tools here.


Reliable SMS Alerting with the SMSEagle

Emails are still the alert type of choice for most network administrators, but they come with a major pitfall: They either rely on your local email server, the Internet connection, or worse: both.

So if you need to get notified when your email server or Internet connection are down – and most likely you will want to get notified – text messages (aka as SMS) are a convenient way out of this monitoring conundrum.

Email to SMS, or web-based SMS services can help in some situations, but you need a different kind of beast when the Internet connection is down.

This is where hardware solutions like the SMSEagle come in. The SMSEagle is essentially a cell phone with an Ethernet jack, a web server + API, allowing you to send text messages either through its web interface or API (preferred).


The SMSEagle and similar devices usually only require three things:

  • A valid SIM card
  • A physical location with coverage from your provider
  • An Ethernet connection to your network

With the SMSEagle in place, you no longer have to fear loss of Internet connectivity or mail server downtime – a text alert is just a few seconds away when combined with a monitoring solution like EventSentry. In fact, the SMSEagle is somewhat unique in that it actually offers its own basic network monitoring capabilities which can be used in addition to EventSentry – or to monitor the host running the EventSentry Heartbeat Monitor.

Since the SMSEagle features a web-based API, EventSentry can submit certain alerts, e.g. pertaining to Internet connectivity or the availability of a specific host and/or service, directly to the SMSEagle through its HTTP action, which will then send a SMS alert directly to your mobile device of choice.

The SMSEagle accepts mini SIM cards and works with the GSM/GPRS/EGPRS 900/1800/1900 MHz wave bands. For users in the U.S. this unfortunately means that the Verizon & Sprint customers will not be able to utilize this device. The setup of the device is straightforward: You first insert the SIM card, connect it to the Ethernet and power on the device. The device automatically assigns itself IP address but can also retrieve its IP address via DHCP. More instructions & details are available in the manual.

Once you’ve setup the SMSEagle and verified that SMS messages can be sent and received, you can create an EventSentry HTTP action and point it to your SMSEagle device. The HTTP action includes a number of templates to quickly load all required fields for a specific HTTP API; simply select the SMSEagle from the drop-down and specify all the required fields such as host name and so forth. Use the test button to ensure the action is setup correctly.

EventSentry HTTP Action
EventSentry HTTP Action configured for the SMSEagle

Once the action is setup it can be referenced by one or more filters so that it is triggered under the right circumstances. Assuming that you are already monitoring a host outside your network (e.g. your ISPs DNS server, a public web site) to determine whether your Internet connection is available or not, you would want to look for EventSentry event 11000, which indicates when a host changes its availability status, such as:

EventSentry Heartbeat Alert
EventSentry Heartbeat Alert

Host ispdns (Internet) changed its PING status from OK to ERROR. The reason for the status change was: “100% packets lost”.

A filter setup for this event would look like something like this:

EventSentry Heartbeat Filter
Filter for EventSentry heartbeat alert

Here, we are looking for an application error generated by the Heartbeat Monitoring category of the EventSentry source. We’re also further restricting the filter to only alert us on status changes of the “ispdns” host when it goes off-line. Since the SMSEagle is listed as the action, this particular event (alert) will be sent to the SMSEagle action.

The insertion strings can be determined by either clicking on the “Lookup” button on the filter dialog, or by clicking on the “Preview” button when adding a content filter.

If you’re located in the Europe then the SMSEagle is a non-brainer, but it’s also a good option for US-based customers who work with a GSM-based cell provider like T-Mobile or AT&T. Otherwise, there are a few other devices out there who work similarly , and as long as they offer a HTTP-based API, integrating EventSentry with them should be easy.

A final note for users in the U.S.: The device comes with a European power connector (“Europlug”) which won’t work in the U.S., you will need to purchase a simple adapter either at your local electronics store (if such a thing exists) or online retailer. A voltage transformer is not necessary since the device operates from 100V to 240V.

An alternative to email alerts. Part 1: Using Trello to manage EventSentry’s alerts

Trello is a simple yet powerful and innovative task management / collaboration platform for teams. With Trello, the developers have basically taken the familiar concept of traditional white boards where you add and remove tasks (by writing on them), and moved it to an easy-to-use online tool.

While Trello doesn’t attempt to replace the more complex project management and collaboration tools available (including its own FogBugz platform), it makes keeping track of small ToDo lists and tasks surprisingly simple, while still supporting advanced features such as due dates, attachments, assignments and more. Of course, Trello also includes a very capable mobile app for iOS and Android (I only tested the iOS version).

Trello Overview
And best of all, it’s completely free if you stick with the basic (and for most people completely sufficient) functionality. But what does Trello have to do with EventSentry and cutting down on emails?

We’re always looking for innovative ways to make managing alerts easier and more productive, especially in larger teams. While email alerts certainly serve a purpose and can be quite useful, alerts dispatched via email suffer from a few disadvantages:

  1. Emails sent to multiple recipients make it difficult for the recipient to know whether the alert has been acted upon or not
  2. Alerts which have already been resolved by a team member still remain in your inbox
  3. Emails often get lost amidst other emails and potentially critical alerts may get overlooked

How Trello Works
Trello is organized into boards, each of which can have one or more lists, each of which have multiple cards. Since Trello offers an API, you can use EventSentry’s HTTP action to submit events (alerts) directly to one (or more) Trello lists.

And this is where the fun starts. Once in Trello, alerts (now cards, or “alert cards”) can be acted upon in a variety of creative and useful ways. You can:

  • Receive alerts in your browser when a card is created
  • Move a card to a different list (e.g. “Resolved”, “Under Investigation”, …)
  • Assign one or more people to a card
  • Add comments to a card
  • Assign a due date to a card
  • Mark a card as important (you can even define your own color codes)
  • Receive periodic summary emails if you don’t visit the board

All of these features make managing alerts in teams with multiple SysAdmins much easier. When an alert comes in, anybody can act on it (e.g. add themselves) or assign it another team member. Any changes are immediately visible to all other team members in real-time (and we at NETIKUS love anything real-time).

Integrating EventSentry with Trello is a 3-step process:

  1. Sign up for Trello, create a board and customize the associated lists
  2. Get an API & access key & determine ID of your list
  3. Setup HTTP action in EventSentry and create/modify rules

Signing up for Trello
To get started, navigate to and sign up with an email address. After you log in for the first time, you will automatically get the “Welcome Board” which will show you all the things you can do with Trello. Since we don’t want to use the default board, we click the big PLUS icon on the top right instead and select “New Board”.

Trello Signup
Give the board a descriptive name, e.g. “EventSentry Alerts”. Once created, the board will contain three default lists. You can either leave the list names as they are, or customize them as shown in the screen shot below. I chose “Active”, “Working on” and “Resolved”.

Template board for EventSentry alerts
Template board for EventSentry alerts

Getting an API and access key
Now that you’ve signed up, the next logical step is to get the API key so that EventSentry can start submitting events to Trello. So while you are logged in, navigate to and note down (aka copy & paste) the first value “Key”, a 32 character-long hexadecimal value. This is the “main” key for your user account, and will be used whenever you (or EventSentry) make an API request.

The API key doesn’t actually let us access data from the boards, for which we’ll need an access key. There are different types of access keys with customizable expiration dates available, but in this case we’ll just get a read/write key without an expiration date. Navigate to the following URL to get a universal read/write access key and substitute APIKEY with the key you obtained just before:,write

You will end up with a dialog similar to the one shown above, where you need to click the green “Allow” button. This will issue another hexadecimal key, this time 64 characters in length. Note this key down as well. Of course you can be less generous and issue keys which expire automatically, e.g. after 30 days. See the Trello docs for more details on the different “expiration” options available.

Getting the list ID
Our end goal is to submit cards to the “Active” list on our “EventSentry Alerts” board. In order to add a new card to this list however, we’ll need the list’s ID. Equipped with our main key and access key, we’re almost there. First, navigate to your “EventSentry Alerts” board in Trello (or whichever board you want to submit cards to) and note down the URL. For example, if the URL is, then you’ll want to extract the text between the /b/ and the board name, gePT9Wax in this case. Now, navigate to the URL below, and replace APIKEY with the API key, and ACCESSKEY with the access key:

This will return detailed results in JSON format similar to this:

 "name":"EventSentry Alerts”,
 "prefs”:  { ……… }
 {"id":"561e92617481e9a123aef3b01","name":"Working on","closed":false,"idBoard":"561e92617481e9a123aef3aff","pos":32768,"subscribed":false}, {"id":"561e92617481e9a123aef3b02","name":"Resolved","closed":false,"idBoard":"561e92617481e9a123aef3aff","pos":49152,"subscribed":false}

What we are interested in is the list id of our “Active” list, 561e92617481e9a123aef3b00 in the example above. With the last missing piece of the puzzle in our hands, we’re now ready to setup a HTTP action in EventSentry.

Configuring EventSentry
Right-click the actions container or utilize the ribbon to create a new HTTP action. In the action dialog, specify the following URL, replacing LISTID with the list id we just obtained:

In addition to the URL, we’ll need to specify at least 4 form fields:


The key and token fields need to be replaced with your API key and access key, whereas the name and desc fields can be customized to suit your needs: what I have shown above is just an example which should work reasonably well in most cases. You can add or remove other event variables as you wish. The upcoming v3.1 will include Trello in the template list to make this a bit easier.

Screenshot EventSentry HTTP Action Trello
Configuring an EventSentry HTTP action for Trello

Once the action is configured, click the Test button to ensure that all IDs have been specified correctly. If the test succeeds, then you should see a new card in the “Active Alerts” list in the EventSentry Alerts board.

Of course an action alone will not forward any alerts to Trello, so you will need to make some changes to your filters and packages. You can either modify existing filters / event log packages and replace the email action with the new Trello HTTP action, or add the Trello action to existing event log packages / filters. Remember that actions can be defined on a package-level through the package properties as well which can help save time.

Managing Alert Cards
Once your first alert card arrives in the “Active” lists and is analyzed by a team member, a few actions can be taken:

  • You can add a team member to the card, essentially assigning the alert to them. You can add multiple team members as well
  • If the event is a false alert, it can be moved to a “False Alert” list, which would indicate that an exclusion filter should be setup in EventSentry
  • You can assign a due date, if the alert requires a resolution by a specific date
  • You can add a comment to the card
  • You can label the card (e.g. “Important”)
  • You can archive & delete the card
EventSentry alerts shown on a trello board
EventSentry alerts shown on a trello board

As you can see, despite its simplicity, Trello offers quite a few features to manage and collaborate. This ensures that alerts don’t disappear in an email inbox somewhere and instead are acted upon, while also allowing collaboration with comments, due dates and such.

Additional Tips & Tricks for Trello
In order to get alerted when a new alert card is created in the EventSentry Alert boards, you’ll need to subscribe to the board. This ensures that you will get a notification on your mobile phone, browser (when enabled or email every time there is activity on a board. Activities include new cards being created, cards being moved to a different list, users being added to cards and so forth.

Note: You will not get a notification if the EventSentry Agent is submitting new cards while using your access key (only other users will see the alerts). This is because Trello assumes that you are creating the cards, and subsequently not notifying you about them.

One way to circumvent this restriction is to create a “service” account (e.g. and issue the access token under this user. Then, everybody will see the alerts.

But don’t stop there!
Of course you can use Trello for what it was originally designed to do as well – manage tasks. We’ve found it to be a great and easy way to handle ToDo lists for teams, resulting in more transparency and efficiency. Assigning a task is quick and easy, and team members can easily track progress with projects – without pesky emails floating around between team members.

Now you just have to get all your To-Do items actually done too. But at least I can now move my “Create Trello Blog Post” card into the “Done” list. And that feels good.

Managing Windows Services & Service Credentials

Every Windows server runs a seemingly ever increasing number of services which range from built-in services providing core Windows functionality (e.g. Print Spooler, Bitlocker, WMI) to 3rd party services added when installing 3rd party software (e.g. various software update services, MySQL) – all of which run in the context of a specific user account.

For example, Windows Server 2012 includes more than 300 services, about half of which are automatically running (this particular server has SQL Server installed as well):

Services on Windows Server 2012 grouped by user
Services on Windows Server 2012 grouped by user

That user account is either a built-in security principal of Windows (e.g. NetworkService), a user account specifically created for that service, or another user account from the server or domain.

Common Practices
Services should always run under a user account which has the least amount of privileges necessary to do its job. It’s common, and often tempting, to run a service an administrative account like “Administrator”. While this often the easiest way to “get it working”, it’s also the least secure.

When a service runs under the “Administrator” account – especially if it’s the domain Administrator account – the service has almost unrestricted access to all resources on the host or, in case of a domain admin, on the domain. This is not something a service usually needs nor you want. It also means that the service will stop working whenever the password of the Administrator account is changed (the service will continue to work until it is restarted).

Less is Better
Whenever possible, try to use one of the built-in security principals available in Windows to run a service under, or create a specific user account for the service. For example, if you have a file synchronization app which runs as a service, create a “ServiceFileSync” or similar account and configure the service to run under that account. Carefully examine the rights the service requires, and only assign those privileges to the user account which the service actually needs.

When creating the user account, give it a very strong & complex password. Users won’t have to log on with that user account, so the password can be complex and long. You can optionally check the “password does not expire” option if you feel that the password is sufficiently secure and you have a short password expiration policy on your domain which could interfere with the service starting after the password expired.

In domain environments I also recommend giving those user accounts (since you will most likely end up with more than one) either a common prefix or suffix (e.g. svc_mysql) and/or moving the accounts into a specific OU. This makes managing and distinguishing these accounts easier – especially in teams with more than one SysAdmin.

The quick way: Local Services grouped by User Account

Sample output from srvsec
Sample output from srvsec

To view all locally installed services grouped by the user account they are running under, download the EventSentry SysAdmin Tools and just run srvsec.exe. This will show you all locally running services, and group the output by the user account they are running under. Srvsec can also be pointed at a remote host, and can also change the passwords stored in services. Click here for more information on srvsec.

Srvsec is a great tool to quickly see what’s going on a single host, but to manage services on an entire domain effectively a more scalable solution is available: EventSentry + AutoAdministrator – the dynamic duo!

The right way: Making sense of ALL installed services
Even when passwords for service accounts are sufficiently strong, they should still be changed on a regular basis. But which services are installed where and are using which service account?

If this is your first time examining service accounts on your network, you should first identify which services run under which user accounts. EventSentry’s service monitoring feature combined with the web-based reporting really makes this a breeze. Assuming that you have a service monitoring system health package assigned to all of your servers, you can simply open the web reports and navigate to Status – Services and get a birds-eye view of all installed services.

In the Overview view, all installed services are grouped by common attributes, including startup type (automatic startup services vs manual startup services), current status, service name and, most importantly for this post, the service user account.

Service overview of all services installed in a domain / forest.
Overview of all installed services in a domain.

Click the “Show All” link to see all user accounts, or click on a specific user account (e.g. “LocalSystem”) to filter the list and only show services running under this specific user account. In most cases you will want to click on “Detailed” to see a list of all services with more detail.

In addition to filtering and viewing details, you can also click on the header of the

All user accounts used by services
All service user accounts

username (or any other) column to see a chart depicting all user accounts used by services from all monitored servers and workstations.

Any report viewed in the web reports can also be scheduled with a job, e.g. a list of all user accounts used by services could be emailed daily/weekly. Simply click the “Save as Report” link to create a report and setup a job.

Managing Services
The standard way to configure the user account and password used by a service is through the “Services” application in Windows. This works well for one or two servers, but not when you need to update the password for a service on multiple hosts.

Managing services with AutoAdministrator
Managing services with AutoAdministrator

This is where AutoAdministrator comes in: A free graphical tool which lets you do just that (and quite a bit more): Update the username and/or password of a service on multiple servers in a domain or work-group. Since AutoAdministrator is multi-threaded, even tasks affecting a large amount of hosts usually only take a few seconds.

To update the stored password of a service, open AutoAdministrator and select “Services” from the drop-down list on the top left.

Service Key Name
Service Key Name

Next, select the service you wish to update from the “Service key / display name” drop-down. If the service is not listed, simply specify the service key name in the service field. The key name is the internal name used by the service and can be obtained by double-clicking a service name in the “Services” MMC application in Windows.

Updating service credentials
Updating service credentials

Next, click on the “Set logon” tab and specify the new username and/or password. Of course you can also specify other service actions, such as restarting the service or changing the start-up type.

As the next step, select the hosts you wish to apply the selected changes to. You can select hosts from Active Directory, EventSentry, custom groups or work groups (Microsoft Windows Network).

Once the correct hosts are selected, click the “Start” button. The number of hosts which will be affected by any action is always shown on the bottom right of the application.

How the EventSentry SysAdmin Tools Focus on File System Maintenance

EventSentry SysAdmin ToolsOver the past couple months, we’ve taken time to go through the various EventSentry SysAdmin Tools, one by one, and show you how they can benefit your environment in powerful ways. We’ve talked about the security tools, the networking tools, and the “check” monitoring utilities. As you know, the SysAdmin Tools offer a set of graphical and command-line utilities designed to help you with your daily administrative tasks. These tools are always being honed to provide simple yet powerful functionality.

This month, let’s take a look at the extremely beneficial file-system utilities: ADSList, CheckSum, DirMon, DirectorySize, FileReplace, PurgeTemp, and SuperDel. Here’s what they can do.

ADSList scans a folder structure to find any alternate data streams (aka “hidden” data streams). Alternate date streams are a feature of the NTFS file system in which you can hide payload (additional files) inside existing files. The jury is still out about whether malware uses these streams, but it’s always a good idea to make sure nobody has hidden something malicious in alternate data streams, because the Windows Explorer and directory listings don’t show them.

ADSList lists any alternate data streams that are associated with a file. When the tool finds an alternate data stream, it displays the name of the stream along with the regular file the stream is associated with. The output will also show a summary that lists the number of files analyzed, the number of files that have an alternate data stream associated with them, the number of alternate data streams that have been found, and the elapsed time.

The main purpose of ADSList is to give you a command-line utility that can be run/scheduled on a regular basis to reveal any hidden streams on a server or workstation. The /s option lets you include subdirectories.

CheckSum generates a one-way checksum (error detection scheme) of a file with a configurable algorithm and displays it onscreen. This capability is useful for ensuring the integrity of a file and making sure that it hasn’t been modified. CheckSum not only supports the SHA set of cryptographic hash functions (e.g., SHA256, SHA512), but also less secure hash functions (e.g., MD5).

To display and create a file’s checksum, simply supply the filename as the first argument. Keep in mind that generating checksums of large files (e.g., greater than 100Mb) can take a significant amount of time and CPU time.

The CheckSum utility is also included in EventSentry as an add-on to the File Monitoring feature, which can automatically generate SHA checksums and detect file modifications based on checksum changes.

Directory Monitor (DirMon) is a useful troubleshooting tool that monitors a directory (and optionally subdirectories) and displays all file changes in real-time. You simply run it on the command line, and it displays any file activity occurring on a given folder (or subfolder).

DirMon will show you when files are added, deleted, or modified. DirMon also lets you specifically include or exclude filters, so you can skip files that you aren’t interested in or show only files that you are interested in. The /I (/includefiles) option includes only files that match a wildcard filter, and the /e (/exclude) option does the opposite. The /s (/subdirectories) option includes subdirectories.

The DirectorySize (dirsize.exe) utility calculates the current size of a directory, including subdirectories, and displays it onscreen. The output shows the number of files and directories searched, and the total size in physical (actual size taken up on the disk) and logical (actual file size) bytes.

DirectorySize will process the current directory if you pass no command-line arguments.

PurgeTemp is a new and exciting tool that lets you purge files that are older than a certain number of days. The tool traverses the %TEMP% directory (or a manually specified directory) and deletes files that have not been modified in 120 days (by default). Because it scans the temp folder by default, you can incorporate PurgeTemp into a login script or run it with Task Scheduler to clean up temp files, for example. It’s a great way to keep users’ temp folders small.

You can customize and configure all of PurgeTemp’s parameters, including /t (time in days) and /p (path). When called without arguments, PurgeTemp simply shows the configured temp directory, the number of files in the directory, and their cumulative size.

SuperDelete (superdel.exe) essentially deletes all instances of a specific file. It parses a directory (including subdirectories) and deletes multiple occurrences of one file.

Suppose you have a thumbs.db file that Windows Explorer creates in every folder containing images, and you want to remove that from every folder on a drive. You can use SuperDelete for that purpose, using the <directory> variable to specify the directory to search (subdirectories are included), and the <fileToDelete> variable to find all occurrences of a file in the directory (wildcards are supported).

FileReplace is a command-line utility that parses a directory (including subdirectories) and replaces multiple occurrences of one template file with a template file of the same name.

Suppose you have 50 instances of various myfile.txt files scattered on your computer. You can quickly replace them all with a new myfile.txt file.

Another useful example is this: You have file C:\WebSite\Default\index.html and want to replace all other index.html files in the directory D:\WWW (including subdirectories) with C:\WebSite\Default\index.html. FileReplace lets you accomplish that with one command.

Streamline Your File System!

This is just another taste of the free, constantly evolving tools available in EventSentry SysAdmin Tools. Give them a try—they’re all free and will help you manage your IT infrastructure more effectively.