Agent vs Agentless: Why you should monitor (event) logs with an agent-based log monitoring solution

The debate as to whether agent-based or agent-less monitoring is “better” has been answered many times over the years in magazine / online articles, blog posts, vendor white papers and others. Unfortunately, most of these articles are often incomplete, inaccurate, biased, or a combination thereof.

To make things slightly more confusing, different ISV use different methods for monitoring servers and workstations. Some use agents, some don’t, and a small few offer both both methods. But what is ultimately the best method?

What are you monitoring?

First it’s important determine what is being monitored to determine whether an agent-based or agent-less approach is better. For example, collecting system metrics like performance data usually creates fewer challenges then transmitting large amounts of (event) log data. Furthermore, agent-based monitoring is not an option for devices which run a proprietary embedded OS (think switch, printer, …) where you can’t install an agent in the first place.

Consequently I’ll be focusing on monitoring (event) logs with an emphasis on Microsoft Windows in this post. Having developed both agent-based as well as agent-less components in C++ over the years I feel that I am in a good position to objectively compare agent-based with agent-less approaches.

The Myths

Monitoring software is of course not the only type of software that uses agents, a lot of other enterprise software (backup, deployment, A/V …) uses agents as well. Below are some of the myths as to what (monitoring) using agents entails:

  • Agents may use up too many resources on the monitored hosts and slow down the monitored machines
  • Agents can become unstable and negatively affect the host OS
  • Deploying and managing agents is tedious and time-consuming
  • Installing agents may require the installation & deployment of dependencies the agents need (.NET, Java, …)
  • Installing third-party software will decrease the security of the monitored host

The Reality

It’s understandable that software which is installed on potentially every server and workstation in a network undergoes some level of scrutiny, but would you be surprised to learn that agents excel in the following areas:

1. Security: Better security since agents push data to a central component, instead of the monitored server being configured to allow remote collection.
2. Reliability: Agents can temporarily store and cache monitored logs if connectivity to the central monitoring server is lost, even if local logs are no longer available. Agents can also take corrective actions more quickly because they can work in isolation (offline). Mobile devices cannot be monitored with agent-less solutions since they cannot be reached by the central monitoring component.
3. Performance: Agents can apply local filtering rules and only transmit data which is valuable, thus increasing throughput while decreasing network utilization.
4. Functionality: They offer more capabilities since there are essentially no limits as to what type of information can be gathered by an agent since it has full access to the monitored system.

The Easy Way Out

Developing agents along with an easy-to-use deployment mechanism requires a lot of time and resources, so it doesn’t come as a surprise to learn that many vendors prefer to monitor hosts without agents. To compensate for the short-fall, ISV which solely have to rely on an agent-less approach will do their best to:

  • Emphasize that they do not use agents
  • Persuade you that agent-less monitoring is preferable

The irony, when promoting a solution as agent-less, is that even so-called agent-less solutions do in fact utilize and agent – the only difference being that the agent is (usually) integrated into Windows. Windows doesn’t just magically service remote clients asking for a boatload of WMI data – it processes these requests through the WMI service, which, for all intents and purposes, is an agent. For example, accessing the Windows event logs via WMI traverses significantly more layers than accessing the event logs directly.

Conclusion

With the exception of network devices where an agent cannot be installed, agent-based solutions will provide a more thorough monitoring experience 9 out of 10 times – assuming that the agent meets all the checklist requirements below.

Some event log monitoring vendors will try to convince you that agent-less monitoring is better & easier (easier for whom?) – but don’t fall for it. We’ve been tweaking and improving the EventSentry agent for more than 10 years, and as a result EventSentry offers one of the most advanced and efficient Windows agents for log monitoring on the market. Developing a rock-solid, secure and fast agent is hard, but it’s the only sensible approach which doesn’t cut corners.

There are situations when deploying a full-scale monitoring solution with agents is not possible, for example when you are tasked with monitoring a third-party network where installing any software is not an option. While unfortunate, an agent-less monitoring solution can fill the gap in this case.

EventSentry also utilizes SNMP (agent-less) to gather inventory, performance metrics as well as other system data from non-Windows devices, including Linux hosts. This collection method does suffer from the above limitations, but since log data is pushed from Non-Windows devices via the Syslog protocol, it’s an acceptable compromise.

Don’t compromise when it comes to monitoring the (event) logs of your Windows infrastructure and select an architecture which scales and offers security & performance.

Technical Comparison

The table below examines the difference between agent-based and agent-less solutions in greater detail.

Resource Utilization & Performance
Agent-Based
Agent-Less
Verdict
  • Usually higher throughput since agents can analyze, filter and evaluate log entries before sending them across the network.
  • Local resource utilization depends on the implementation of the agent.
  • Agent can access (event) logs directly via efficient API access.
  • Network utilization is likely much higher since more logs have to be transmitted across the network before being evaluated. Local filtering capabilities are limited and depend on the protocol (usually WMI).
  • Network latency and utilization affect performance of monitoring solution. Network utilization cannot be controlled.
  • Accessing (event) log data remotely through WMI is much less efficient.
  • Over-saturation of central monitoring component can negatively affect monitoring of entire infrastructure.
Higher network utilization combined with the fact that remote log collection will still utilize CPU cycles on the remote host (e.g. through WMI provider) favors agent-based solutions.
Agent-less solutions have a single point of failure, while agents can filter & evaluate data locally before transmitting them to a central database.
EventSentry Agents are designed to be essentially invisible under normal operations and do not impact the host system negatively in any way.
Stability & Reliability
Agent-Based
Agent-Less
Verdict
  • Failure of an agent does not affect monitoring of other hosts.
  • Locally collected data can be cached if central monitoring component is temporarily unavailable.
  • Failure of a central component may negatively affect deployed agents if they rely on the central component and cannot cache data.
  • Failure of central monitoring system will affect and potentially disable monitoring of all hosts.
  • Hosts which lose network connectivity (e.g. laptops) cannot be monitored while unreachable.
Agent-based solutions have an advantage since local data can be cached and corrective actions can be executed even when the central monitoring component is unavailable. Cache data & logs are re-transmitted – even if the local logs have been cleared or overwritten.
Agent-based solutions can monitor hosts even when disconnected from LAN.
The EventSentry Agent auto-recovers if the process aborts unexpectedly, and by default alerts the user when this occurs. When using the collector (default), the agent caches all data locally and retransmits when the network connection becomes available again.
Deployment
Agent-Based
Agent-Less
Verdict
  • Has to be deployed either with vendor management software and/or with third-party deployment software if vendor provides installation package (e.g. MSI).
  • Larger deployments will require multiple central monitoring components, potentially distributed over several LANs.
  • Only hosts in local LAN can be monitored.
Depends on deployment tools made available by vendor as well as management tools in place for configuring Windows settings. A poorly developed deployment tool would favor an agent-less solution.
EventSentry Agents can be deployed (multi-threaded) with the management console or through 3rd party deployment software by creating a MSI installer on the fly. When using the collector (default), agent updates (patches) can be deployed automatically.
Dependencies
Agent-Based
Agent-Less
Verdict
  • Agent may have dependencies on third-party frameworks
  • Depends on whether the mechanism utilized by the monitoring software requires a Windows component to be added and/or configured.
Depends on whether agent has dependencies and whether configuration changes need to be made on the monitored hosts.
The EventSentry Agent does not depend on any 3rd party frameworks or libraries.
Security
Agent-Based
Agent-Less
Verdict
  • Potential security issues if installed agent exposes itself to the network (if not firewalled) and/or suffers from local vulnerabilities which can be exploited.
  • Remote log collection has to be enabled and at least the central monitoring component needs to have remote access.
  • Secure data transmission relies on protocols and settings from Windows.
  • Enabling multiple methods for gathering data remotely (e.g. WMI) provides additional attack vectors.
  • Credentials (usually Windows user/password) for remote systems must be stored in a central location so that the remote hosts can be queried. If the central system gets compromised, critical credentials can be exploited.
Since agent-based solutions do not require permanent remote access and monitored hosts can therefore be hardened more, they are inherently more secure IF the agent doesn’t suffer from an insecure design and/or vulnerabilities.
Agent-based solutions also have more control over how data is transmitted from the remote hosts.
If there is general concern against third-party software then the product in question should be researched in a vulnerability database like http://www.cvedetails.com.
The EventSentry Agent does not open any ports on a monitored host and resides in a secured location on disk. The agent transmits compressed data securely via TLS to the collector. No major security vulnerabilities have been discovered in the EventSentry agent since its first release in 2002.
Scope & Functionality
Agent-Based
Agent-Less
Verdict
  • Agents have full access to the monitored system and can choose which technology to utilize to get the required data (API, WMI, registry, …)
  • Easily execute local corrective action like launching a script or process
  • Agent-less solutions are limited to remote APIs provided by the monitored host, most commonly WMI. While WMI does offer a lot of functionality, there are limitations.
  • Executing scripts on remote host is more involved and only possible when host is reachable.
Agent-based solutions have an advantage since they can utilize multiple technologies to obtain data, including highly efficient direct API access. Agents can also trigger (corrective) actions locally even while the agent is unreachable.
Agent-less solutions can only monitor data which is made available by the remote protocol.
The EventSentry Agent accesses log files, event logs and other system health data almost exclusively via direct API calls. The more resource-intensive WMI interface is only used minimally, for very specific purposes. Corrective action can be taken directly on the monitored host, often only in milliseconds after an error condition (event) has occurred.

 

Appendix: Checklist

When evaluating software that offers agents then you can utilize the check list below for evaluation purposes.

Resource Utilization
An agent needs to consume as little resources as possible under normal operations. With the exception of short (and unusual) peak periods, a user should never know that an agent is running on their server or workstation – period.
The thought of having a resource-hogging agent running on a server sends shivers down the backs of many SysAdmins, and the agents used by certain AntiVirus vendors that rhyme with Taffy didn’t set a good precedent.
Stability & Reliability
The agent needs to run at all times without crashing – the SysAdmin needs to be able to go to sleep knowing that his agents will reliably monitor all servers and workstations. Unstable agents are just no fun, especially when they negatively impact the host OS.
If an agent that encounters an issue, it needs to at least auto-recover and communicate the issue to the admin.
Deployment
Agent deployment and management needs to be streamlined and easy – it shouldn’t be a burden on the end user. And while agent deployment is important, agent management, keeping the remote agent up-to-date, is equally important and should – ideally – be handled automatically.
Most SysAdmins have enough work the way it is, the last thing they need is baby-sitting agents of their monitoring solution.
Dependencies
The more dependencies an agent has, the more difficult it is to deploy the agent. Agents that rely on complex frameworks like .NET, Java or specific Visual Studio runtimes are difficult and time-consuming to deploy.
Furthermore, any third-party software that is installed as a dependency creates an additional attack vector and needs to then be kept up-to-date.
Security
An agent needs to be 100% secure and cannot expose the monitored host to any additional security risks. I will explain below why using agents is actually more secure than not using an agent – even though this seems counter-intuitive at first glance.

Detecting Web Server Scans in Real-Time

Any web site exposed to the Internet is constantly being probed by bots, malicious hackers and other evildoers in an attempt to take over the machine, gain access to unauthorized data, install back doors and so forth. Detecting probing attempts as early as possible and taking corrective action as soon possible is key to maintaining a secure network.

Manual probing usually involves investigating the HTTP headers to determine the type of web server (e.g. IIS, Apache, Nginx), viewing HTML sources and possibly attempt to access well-known pages in order to determine whether any well-known web-based software (WordPress, CRM, OWA, …) is installed.

IIS Email Alert
Email alert from an IIS web site scan

If the attacker prefers the sledgehammer approach then he or she may also point a vulnerability scanner such as OpenVAS at the web server, which will reveal vulnerabilities with a minimum amount of work. Automated systems aren’t as surgical and will usually just look for specific vulnerabilities by checking for the existence of various URLs on the web server.

But whether it’s a manual probe, a vulnerability scan or a bot, all methods usually result in a non-existent page (URL) being attempted to be accessed, resulting in a “Page Not Found”, 404 error at some point. As such, a larger than usual amount of 404 errors can be a good indicator that suspicious activity is occurring on your web server. If you are a little paranoid like me then you could even look for every single 404 error that occurs on your web server. The same technique can be applied to other errors as well, such as “Access Denied” errors for example if the web site is secured by ACLs.

EventSentry’s log file monitoring feature can monitor Windows-based log files in real time and trigger alerts and/or corrective actions by applying sophisticated rulesets to all parsed text.

Log File Flow
Log File Flow (Icon made by Freepik from www.flaticon.com)

I’ll explain how this can be setup based on an IIS web server, but the same generic steps would apply to other web servers as well.

  1. Define the log file
    The first step is to tell EventSentry which log file you’d like to monitor in the management console. Using the ribbon click on “Packages”, “Log Files” and “Define Files”. In the “Log Files” section on top, click on the plus icon (+) and define the log file. Give the file a descriptive name, specify the path to the log file and select “Non-Delimited” as the file type. Make sure to utilize wild cards or variables for the log file path if the name of the log file is dynamic, as shown in the screenshot below:

    IIS Log File Setup
    IIS Log File Setup

    If you plan on storing contents in the EventSentry database as well then you can also select a matching log file definition (such as IIS 7) as the log file type. More information on log file types can be found in our IIS Log File Monitoring with EventSentry screen cast.

  2. Setup a log file filter
    A log file filter defines where content from the log file is routed to. In this example we’ll route 404 errors to the Application event log. Using the ribbon again, and while still in the log file context, click on “Add Package” on the top left to create a new package – give the package a descriptive name. Then, click the “Assign” button to assign this package globally, to a group or individual host. (remember that you can also assign packages dynamically). Now click “Add” in the “Log File” section to add the previously configured log file to the package.

    Log File Filter
    Specify how log file content is routed

    In the resulting dialog we can configure the log file filter to send log file contents to the database, the event log or both. For the purpose of this example we will only log certain lines of the log file to the event log – those matching the wildcard filter * 404* (note the space between the first * and 404) as shown in the screenshot above. You can also use a regex expression for a more sophisticated match type.

  3. Setup Event Log Filter
    At this point EventSentry will log an informational event every time the text ” 404″ is logged to the specified log file. In order to dispatch (e.g. to an email recipient) this event however, an include filter needs to be setup which should look similar to the screenshot below:

    Event Log Filter for Log File Alert
    Event Log Filter

That’s all that is required to trigger an email or process every time a 404 error is triggered on your web server. Read on to refine this setup and only get alerted when the same remote IP address triggers a certain number of 404 errors within a certain time period – fun!

Additional Resources
Owasp.org is a great resource for web developers which provides a plethora of information to help keep web sites secure. The Owasp Top 10 document illustrates what the most critical web application security flaws are.

Tutorial: Delimited Log File Monitoring
Screen Cast: Log File Monitoring with EventSentry

Bonus for Advanced Users (requires EventSentry v3.3 or later)
Getting alerts whenever specific text – like a 404 error – are logged is quite useful, but utilizing EventSentry’s advanced event log filter & thresholds features can reduce noise and make monitoring log file contents even more actionable.

EventSentry supported utilizing insertion strings from events for quite some time, allowing you to use those insertion strings either in actions (e.g. an email subject, a parameter for a script) or thresholds. Since events don’t always utilize insertion strings properly, or custom content in events needs to be parsed separately, EventSentry v3.3 and later let you define insertion strings based on regular expressions. The screenshot below shows insertion strings before and after a regular expression fitted for IIS 7.5 is applied to EventSentry’s log file monitoring alert:

Apply RegEx to Event
Overriding insertion strings by applying a regular expression

You can learn more about insertion strings here, and view insertion strings either with the Event Message Browser or the EventSentry Management Console (Tools -> Utilities). The regular expression for a default IIS 7.5 setup is as follows:

([0-9]{4}-[0-9]{2}-[0-9]{2}) ([0-9]{2}:[0-9]{2}:[0-9]{2}) ([0-9\\.]*) ([A-Z]*) (.*?) (.*?) ([0-9]*) (.*?) ([0-9\\.]*) (.*?) ([0-9]*) ([0-9]*) ([0-9]*) ([0-9]*)

Since insertion strings can be used in variables (e.g. $STR1 … $STR14) and thresholds, overriding insertion strings in an event has two main benefits:

  • Use any field from the log file in the email subject and other action fields
  • Create thresholds based on log file content – e.g. create dynamic run-time thresholds for each IP address

Regular expressions are set using the “Advanced” button on the “Generic” tab of an event log filter. In the advanced dialog, simply click “Edit” in the “Insertion String Override” section.

Using insertion strings in emails
The generic EventSentry email subject is nice, but a customized subject reflecting the type of alert would certainly be better:

Red Alert: IIS scan detected from IP $STR9

This is possible with the redefined insertion strings, since #9 (=$STR9) is the remote host’s IP address. To set a custom subject, click the “Advanced” button on the “Generic” tab of an event log filter.

Using insertion strings in thresholds
By default, threshold counters are increased every time an event matches the corresponding filter. To stick with our example here, we could configure EventSentry to let us know if more than three 404 errors occur within 5 minutes. But we’d essentially be throwing all events into the same bucket. If you look at it in detail however, you realize that it makes a difference whether three 404 errors are a result of activity from the same remote host, or three different remote hosts.

Since events are often a result of specific activity by something or somebody, it’s important that we can correlate multiple events. In our example, the “something” is the remote host, which is represented by insertion string $STR9. As such, we can configure our threshold to use $STR9 as the common identifier, and create unique thresholds based on the run-time value of $STR9. By doing that, we will trip the threshold only if the same remote host accesses a non-existing URL three times, but not if three different remote hosts only access one non-existent URL each.

Event Log Filter Threshold
Event Log Filter Threshold

The same technique can be applied to thresholds for failed logon events. It’s usually acceptable if a user types the wrong password a few times, but a large number of failed logons from the same user are not. Just applying a threshold to all 4625 events is usually not practicable since many users occasionally type a wrong password. But by tying the threshold to the insertion string representing the user name (they are 6 & 7 in case you are curious), we can create a separate threshold for every user and avoid false positives.

Defeating Ransomware with EventSentry – Remediation

Since Ransomware is still all the rage – literally – I decided to write a 4th article with a potentially better method to stop an ongoing infection. In part 1, part 2 and part 3 we focused mostly on detecting an ongoing Ransomware infection and utilized the “nuclear” option to prevent it from spreading: stopping the “server” service which would prevent any client from accessing files on the affected server.

While these methods are certainly effective, there are other more targeted steps you can take instead of or in addition to shutting down the server service, provided that all hosts susceptible to a Ransomware infection are monitored by EventSentry.

When EventSentry detects an ongoing Ransomware infection, it can usually determine the infected user by extracting the domain user name from the 4663 event. Simply disabling the user is insufficient however, since a disabled user can continue to access the network (and wreak havoc) as long as he or she doesn’t log off. Any subsequent log on attempt would of course fail, but that provides little comfort when the user’s computer continues to plow through hundreds or thousands of documents, relentlessly encrypting everything in its path.

As such, the only reliable way to stop the ongoing infection, given only the user name, is to log off the user. While logging a user off remotely is possible using the query session and logoff.exe commands, I prefer to completely shut down the offending computer in order to reduce the risk of any future malicious activity. Logging the user off remotely may still be preferable in a terminal server environment (let me know if you want me to cover this in a future article).

Knowing the user name is of course great, but how do we find out which computer he or she is logged on to? If you have EventSentry deployed across your entire network – including workstations – then you can get this info by querying the console logon reports in the EventSentry web reports. If you are not so lucky to have EventSentry deployed in your entire environment (we offer significant discounts for large quantities of workstation licenses – you can request a quote here) then we can still obtain this information from the “net session” command in Windows.

Net Session Output
Net Session Output

We’ve created a little script named antiransom_shutdown.vbs which, given a user name, will report back from which remote IP this user most recently accessed the local server and optionally shut it down. Here are some usage examples:

Find out from which computer boris.johnson most recently accessed this server:
cscript.exe C:\Scripts\antiransom_shutdown.vbs boris.johnson

Find out from which computer boris.johnson most recently accessed this server AND shut the remote host down (if found):
cscript.exe C:\Scripts\antiransom_shutdown.vbs boris.johnson shutdown

The script uses only built-in Windows commands, as such there is no need to install anything else on the server where it’s run.

When executed with the “shutdown” parameter, the script will issue a shutdown command to the remote host, which will display a (customizable) warning message to the user indicating that the computer is being shutdown because of a potential infection. The timeout is 5 seconds by default but can be customized in the script. It’s recommended to keep the timeout short (5-10 seconds) in order to neutralize the threat as quickly as possible while still giving the user a few moments to know what is happening.

The overall setup of the Ransomware detection is still the same, we’re setting up a threshold filter to detect a higher than usual frequency of certain 4663 events and trigger an action in response. Only this time we don’t shut down the server service, but instead trigger this script. To properly execute the action, configure it as shown in the screenshot below. The executable is cscript.exe (the interpreter for .vbs files) and the command line parameters are the name of the script, $STR2 and “shutdown”.

Remote workstation shut down
Remote workstation shut down

So what’s the better and safer approach to freeze an ongoing Ransomware infection? Shutting down the server service is the most reliable approach – since it doesn’t require the workstation to be reachable and will almost certainly succeed. Remotely shutting down a workstation has minimal impact on operations but may not always succeed. See below for the pros and cons of each approach:

File Sharing Shutdown
Pros: 100% effective
Cons: Potentially larger disruption than necessary, false positive unnecessarily disrupts business

Remote Workstation Shutdown
Pros: Only disables infected user/workstation, even if false positive
Cons: Requires workstation to be reachable

This ends up being one of those “it depends” situations where you will have to decide what’s the best approach based on your environment. I would personally go with the remote workstation shutdown option in large networks where the vast majority of workstations are desktops reachable (and not firewalled) from the file server. In smaller, more distributed networks with a lot of laptops, I would go with the file service shutdown “nuclear” option.

A hybrid approach may also be an option for those opting for the remote workstation shutdown method: trigger a remote workstation shutdown during business hours when IT staff is available on short notice, but configure the file service shutdown after business hours when it’s safer and affects fewer people. All this can be configured in EventSentry by creating two filters which are identical except for the action and the day/time settings.

Prerequisites
It’s important to point out that the EventSentry agent by default runs under the LocalSystem account, a built-in user account which does not have sufficient privileges on a remote host to issue the shutdown command. You can elevate the permissions of the EventSentry agent and work-around this limitation in 2 ways:

  1. Change the service account (fast): Changing the service account the EventSentry service uses to a domain account with administrative permissions will allow the agent to remotely shut down a remote host. This will have to be done on every file server which may issue shut down commands (you can use AutoAdministrator to update multiple file servers if necessary).
  2. Give the “Force shutdown from a remote system” user right: It’s not necessary to issue domain-wide admin rights to the EventSentry agent, the key right the agent needs is just the “Force shutdown from a remote system” user right. The quickest way to deploy this setting is of course through group policy:

    a) Open the “Group Policy Management Editor”
    b) Edit an existing policy (e.g. “Default Domain Policy”) or create a new group policy
    c) Navigate to “Computer Configuration\Policies\Windows Settings\Security Settings\Local Policies\User Rights Assignment”
    d) Double-click the “Force shutdown from a remote system” user right and add both “Administrators” and the computer accounts of the file servers to the list. Alternatively you can also create a group, add the file servers to the group, and add that group to the policy (keep in mind that you will need to restart the file servers if you go with the group method).

    Once the group policy setting has propagated to the workstations, the remote shut down initiated from the file server(s) should succeed.

    Change the "Force shutdown from a remote system" user right
    Change the “Force shutdown from a remote system” user right

 

Good luck protecting your network against Ransomware infections, also remember to verify your backups – no protection is 100% effective.

Defeating Ransomware with EventSentry & Auditing

There seems to be a new variant of ransomware popping up somewhere every few months (Locky being the most recent one), with every new variation targeting more users / computers / networks and circumventing protections put in place by the defenders for their previous counterparts. The whole thing has turned into a cat and mouse game, with an increasing number of software companies and SysAdmins attempting to come up with effective countermeasures.

I’ve already proposed two ways to counteract ransomware on file servers with EventSentry in part 1 and part 2, both of which take a little bit of time to implement (although I’d argue less than it would take to restore all of your files from backups). In this post I’m proposing a third, and better, method with the following improvements:

In the first article we configured file integrity monitoring on a volume, and if the number of file modifications occurring during a certain time interval exceeded a preset threshold, the ransomware would be stopped in its tracks. In the seconds article we used bait (canary) files to accomplish the same thing.

In this third installment we’ll keep track of the number of file modifications made by a user to detect if an infection is underway. To effectively defeat ransomware, we have to be able to distinguish between legitimate user activity and an infection. To date we know this:

  • Users add/change/remove files, but the number of changes made by a user in a short amount of time (say 15 min) is generally small
  • Ransomware always runs in the context of a user, and as such an infection will usually come from one user (unless things go really awry and multiple users are infected). The approach here will work equally well, regardless of the number of infections.

Thus, to detect an infection, EventSentry will be counting the number of file modifications (event 4663) with its advanced threshold capabilities. If the threshold is exceeded, EventSentry will trigger an action of your choice (e.g. disable the user, remove a file share, stop the server service, …) to limit the damage of the ransomware.

Here is what you need:

  • Object Access / File System Auditing enabled
  • Auditing enabled on the files which are to be protected
  • EventSentry installed on the server which needs to be protected

This  KB article explains how to configure EventSentry and enable auditing (preferably through group policy) on one or more directories. I recommend referencing the KB article when you’re ready to configure everything. Pretty much everything in the KB article applies here, although we will make a small change to the threshold settings of the filter (last paragraph of section (4)).

Windows Folder Auditing
Windows Folder Auditing

Once auditing is setup, Windows will log event 4663 for every write access which is performed by a user. An example event looks like this:

Windows Event 4663
Windows Event 4663

The default behavior of a filter threshold in EventSentry is to simply count every filter match towards the threshold. In our case, every 4663 event encountered would count towards the threshold. You can think of there being one bucket for all 4663 events, with the bucket being emptied whenever the threshold period expires, say every 5 minutes. If the bucket fills up we can trigger an alert.

This doesn’t work so well on a file server, where potentially hundreds of users are constantly modifying files. It would take some time to come up with a good baseline (how many file modifications are considered “normal”) that we could use as a threshold, and there would still be a chance for a false positive. For example, a lot of 4663 events could be generated during a busy day at the office, thus causing the threshold to reach its limit.

A better way is to assign each user their own “personal” threshold which we can then monitor. Think of it like each user having their own bucket. If a user writes to a file, EventSentry adds the 4663 event only to that user’s bucket. Subsequently, an alert is only triggered when a user’s bucket is full. Any insertion string of an event can be used to create a new bucket.

We can do this by utilizing the insertion string capabilities of the filter threshold feature. Setting this up is surprisingly easy – all we have to do is change the Threshold Options to “Event”, click the “Insertion Strings” button and select the correct insertion string. What is the correct insertion string? The short answer is #1.

The long answer lies in the “Event Message Browser”, which you can either find through the Tools – Utilities menu in the EventSentry Management Console or in the EventSentry SysAdmin Tools. Once in there, click on “Security”, then “Microsoft-Windows-Security-Auditing”, then 4663. You will see that the number next to the field identifying the calling user (“Security ID”) is %1.

Event 4663 Definition
Event 4663 Definition

Enough with the theory, here is what you need to implement it (assuming EventSentry is already installed on the servers hosting the file share(s)):

  1. Enable global auditing globally and audit the file share(s). See section 2 & 3 of KB 279.
  2. Determine what action you want to take when a ransomware infection has been detected. See either section 1 of KB 279 or “Dive! Stopping the Server Service” from the previous blog post.
  3. Create a package & filter looking for 4663 events. See section 4 of KB 279 and review the additional threshold settings below.

Customizing the threshold
Once you have the package & threshold filter for 4663 events in place, we need to modify the threshold settings as explained above. Edit the filter, click the threshold tab and make sure your filter looks like the one shown below:

Threshold Settings
Threshold Settings

The only variable setting is the actual threshold, since it depends on how fast the particular variant of ransomware would be modifying files. A couple of things to keep in mind:

  • The interval shouldn’t be too long, otherwise it will take too long before the infection is detected.
  • Make sure the actual event log filter is only looking at 4663 events, no other event ids.

With the above example, any user modifying any file (on a given server) more than 30 times in 3 minutes will trigger any action associated with the filter, e.g. shutting down the server service. Note that the action listed in the General tab will be triggered as soon as the threshold is met. If 30 4663 events for a single user are generated within 45 seconds, the action will be triggered after 45 seconds, it won’t wait 3 minutes.

Bonus – Disabling a user
One advantage of intercepting 4663 events is that we can extract information from them and pass them to commands. While shutting down the Server service is pretty much essential, there are a few other things you can do once you have data from the events, e.g. the username, available. You can now do things like:

  • Disabling the user
  • Removing the user from the share permissions
  • Revoking access to select folders for the user

There are a couple of caveats when (trying to) disable a user however:

  1. The user account (usually the computer account) under which the EventSentry service runs under (usually LocalSystem) needs to be part of the Account Operators group so that it has permission to disable a user
  2. Disabling a user is usually not enough though, since Windows won’t automatically disconnect the user or revoke access. As such, any ransom/crypto process already running will continue to run – even if the user has been disabled.

Disabling a user account from the command line is surprisingly simple (leave Powershell in the drawer). To disable the user john.doe, simply run this command:

net user john.doe /domain /active:no

Note that since “net user” doesn’t support a domain prefix (MYDOMAIN\john.doe won’t work), we need to make sure that we pass only the username (which is insertion string %2) and the /domain switch to ensure the user is disabled on the domain controller. Of course you would need to omit the /domain switch if the users connecting to the share are local users. The action itself would look like the screenshot below, where $STR2 will be substituted by EventSentry with the actual user listed in the event 4663:

Action to disable a user
Action to disable a user

 

That’s it, now just push the configuration and you should be much better prepared to take any ransomware attacks heading your users way.

Oh, and check those backups, would you?

 

 

Automatically restarting services or processes based on resource usage

In the ideal world, every software we install on our servers and workstations uses as few resources as possible, doesn’t have memory or handle leaks and never crashes.

But in reality, Sysadmins often have to deal with temperamental business-critical third-party applications (or in-house developed) which exhibit a number of issues, including:

  • Memory Leak: The application keeps eating away at the available memory like a chubby caterpillar chewing on a leaf
  • Handle Leak: The application continuously increases its handle count, which takes away from kernel memory over time
  • CPU Spike: The application uses all CPU time of one or more cores

When one of these issues is encountered, a manual application (or service) restart, along with a potential bug report, is usually the only solution. Consequently, keeping a close eye on both Windows and third-party software – especially on servers – is considered good practice. But even better than looking is being proactive of course, for example by automatically restarting a service which uses too much memory or CPU.

Frozen Leak

This is where EventSentry comes in. EventSentry doesn’t just analyze metrics available through Windows performance counters (e.g. CPU usage, handle or memory count of a process.), it also allows you to take corrective action based on granular rule sets. This ensures that all active applications are behaving nicely by staying within pre-defined performance boundaries.

To get there, we utilize 3 features in EventSentry:

1. Performance Monitoring
2. Event Log monitoring
3. Service restart or process action

Since examples usually work best, I will outline the steps required to restart the printer spooler service if it uses more than 100 Mb of RAM. This is for illustration purposes only, I’m not suggesting that the printer spooler service should not use more than 100 Mb of RAM.

Performance Monitoring
Application performance monitoring is already setup out-of-the-box via the “Performance Applications” System Health package. This package, by default, is assigned to all hosts and collects key application metrics in the EventSentry database. Since this package is generic and captures all processes (without generating alerts), we’ll create a separate package that will only monitor the spooler service.

Unless you resort to scripting, it is unfortunately not easily possible to automatically link process names (as they are reported by the Windows performance monitoring subsystem) to a service name. As such, we will need to first find out the process of the service we are monitoring and then monitor only that instance of the performance counter. To determine the process for a given service, simply view the properties of the services in the “Services” or “View local services” application and look for the “Path to executable” field. New versions of Windows also show a list of all services in task manager and let you jump to the process by clicking on “Go to details”. The name of the instance is the process name without the .exe extension, spoolsv in this case.

The next step is to create a new System Health package and add a performance object. Select the System Health packages container, click “Add package” from the ribbon and enter a suitable name. Select the newly created package and add the performance object to the package. Now select the “Performance” object and click the “+” icon to add a new performance object to monitor. Every performance object in EventSentry requires at least a name (to describe the counter) as well as the actual Windows performance counter. The respective performance counter for monitoring the memory usage of a process is Process(*)\Working Set, and since we are only interested in the spooler process we will use of the Process(spoolsv)\Working Set performance counter. When you are done, the dialog should look similar to what is shown below:

Performance Counter Setup
Specifying the performance counter to monitor the memory usage of the spooler process

The default frequency is 10 seconds which works well for most counters, but you can increase this frequency for counters which change only minimally over the short term (as is usually the case for memory usage and handle count), so we will use 30 seconds in this case.

Now that we are successfully tracking the memory usage of the spooler service, we need to setup a hard limit in order to get an event when that limit is exceeded. Click on the “Alert” tab and configure the dialog as shown below:

Specifying the alert limit for the performance counter
Specifying the alert limit for the performance counter

We are only concerned with the top section of the dialog, please see the documentation for more details on the “Notify at most …” and below options.

The last step in this section is to assign the package: Select the package, click “Assign” in the ribbon and assign the package to a computer or group. EventSentry is now tracking the memory usage of the spoolsv process and will log a warning event if the memory usage exceeds 100 Mb.

Action
EventSentry uses actions to send emails, toggle services or start processes. Since we want to restart the spooler service, we’ll create a Service action. Select the “Actions” container and click the “Add” button. Select the “Service” action type and assign it a descriptive name, e.g. “Restart Print Spooler Service”.

The configuration of this action is probably the most simple in this tutorial – just specify the service name and the desired action as shown below:

Specifying the service to be restarted
Specifying the service to be restarted

Connecting the dots: Event Log Filter
We’re monitoring the memory usage of the spooler now and have an action which can restart the spooler service, but how do we connect the two? You probably guessed it – with an event log filter. Event Log filters allow you to connect an event (e.g. memory usage is too high) with an action (e.g. restart spooler service).

We’ll create an event log filter which will look for the exact event that is being logged when the memory usage of our performance counter exceeds 100 Mb, and trigger the service restart action.

Similar to what we did with the system health package, right-click the “Event Log Packages” container (or use the ribbon) to create a new event log package and assign it to the computer(s) and or group(s) in question.

Then, add a new INCLUDE filter to the package. Alternatively you can also click the “Alerts” button while the performance object is selected to go through a wizard. Either way, the filter should look like the screenshot below:

Specifying the event properties which will trigger the service restart action
Specifying the event properties which will trigger the service restart action

Now, when the performance monitor writes event id 12104 with the above properties, EventSentry will trigger the “Restart Print Spooler Service” action which should reset the memory usage of the process. As an added bonus, an email is also fired off so that the operator knows that EventSentry took the corrective action.

Note: Don’t forget to push the configuration to any remote hosts if necessary.

Now sit back and relax knowing that another thing is taken care of for you.

Trapping CryptoLocker/CryptoWall with Honey (Part 2/3)


! Updates !
There has been a follow-up post to this article with even better approaches to defeating ransomware. I highly recommend that you jump directly to the most recent article which offers the best & easiest approach for protecting against Ransomware:

Defeating Ransomware with EventSentry & Auditing (Part 3)


When I wrote my first, original post about CryptoLocker (“CryptoLocker Defense for Sysadmins”), I didn’t intend there to be a part 2 or even a part 3. But alas, due to the “popularity” of CryptoLocker and the recent release of CryptoWall 4.0 I decided to write a much-needed sequel to my first blog post. Part #2 differs from the first part with a different (and more simple) detection “algorithm” combined with a more reliable way to stop the “Server” service when CryptoLocker is indeed detected.

The capitol of São Tomé and PríncipeSurprisingly (or not surprisingly), almost 2 years after I wrote my first article, CryptoLocker and its descendants like CryptoWall are still around, thriving, and keeping Sysadmins around the world busy. A recent report stated that CryptoWall 3.0 cost victims a combined $325 million, although it fails to mention whether this is an annual or lifetime figure. This is the same as the GDP of the small African country of São Tomé and Príncipe (population of about 200,000) in 2014.

Now there is something to think about – the criminals behind the various ransomware software collected as much money as a country with 200,000 people. Alright, this is all very interesting but doesn’t help us protect ourselves from ransomware so let’s focus.

In part #1 we used EventSentry’s file monitoring feature to index and inventory all files on a susceptible file share, a very accurate and resilient way to detect any sort of software which would modify large numbers of files in a short time period. While this approach works well, it does require more time to setup and may not work in real-time when monitoring extremely large directories. Consequently we’ll be using a different approach here, and we will look at yet another approach in part #3.

EventSentry’s file checksum monitoring feature was originally intended to monitor only key Operating System folders such as the System32 directory, but increased customer demand prompted us to tweak the feature over time to allow real-time monitoring of even very large folders (as is the case for file servers) as well. But enough of the past, let’s tackle Crypto*.

What’s New?
New versions of software (especially free) are usually exciting, but I’m guessing that the latest “improvements” rolled into the various types of ransomware, including CryptoWall, are only exciting for security researches and the people behind the ransomware. There are three major new features included (for free) in the latest version of CryptoWall:

  • Files are not only encrypted, but file names are now also mangled, making it almost impossible to link the encrypted file(s) with their originals.
  • Shadow copies are being deleted if possible, so that past versions of files are no longer accessible
  • The encryption process seems to be less linear and less complete, resulting in some folders being left alone and thus making detection more difficult.

HoneyThe Theory
If you’ve been working in the IT (security) field for a while then you’ll have probably heard of honeypots before. Honeypots are usually systems emulating a production server with the purpose of detecting an attacker and potentially triggering counter-measures or alerts.

We’ll apply the concept of honeypots to detect CryptoLocker, but instead of emulating entire systems we’ll plant on or more fake files throughout on one or more file shares with the assumption that any CryptoLocker infection will attempt to change and encrypt those files. Once detected, we can trigger a counter measure such as stopping the server service. The three biggest risk factors with this approach are:

  • Accidental modification by a user
  • CryptoLocker detects (and skips) the honeypot files
  • The bait files get modified too late

But not to worry – we can mitigate all of the risks.

Accidental Modification
Since any unsuspecting user with write access may accidentally modify or delete our bait file, it’s possible that some users curiosity may result in some sort of a accidental DoS attack. Making the file read-only defeats the purpose of detecting CryptoLocker of course, since CryptoLocker itself won’t be able to modify it. I was able to come up with two possible solutions for this problem:

1. Give the file a boring name which discourages users from opening it (e.g. meeting_notes_cl1.docx)
2. Put clear instructions into the file in large font, instructing users not to modify or delete the file. I’d recommend against mentioning any words like CryptoLocker, Virus, etc since CryptoLocker may be parsing the contents of the file.

Example Bait File

Honeypot is not sweet enough
Since we don’t have access to CryptoLocker and its constantly evolving code, we don’t know whether it has any honeypot detection capabilities, and if it does, how it attempts to detect them. Since I’m rather safe than sorry, I’m assuming that it has some basic capabilities. It could be as simple as skipping files which are smaller than a pre-defined threshold or looking for specific file names. E.g., based on this article CryptoLocker could now skip any file named meeting_notes_cl1.docx. To maximize our chances for success:

1. Make sure the file is not too small and exhibits properties of other office documents (e.g. 1Mb in size, multiple pages)
2. Give the file a unique, meaningful name, see previous paragraph.

Once you have created the file, place it strategically on your file server among other office documents. I recommend deploying multiple honeypot files if you have multiple file shares. It may be advisable to give the files unique names (e.g. meeting_notes_cl1.docx, meeting_notes_cl2.docx, …) as well.

Too little, too late
The bait file getting modified too late is the biggest risk unfortunately. If you have a directory with 50,000 files but only one bait file, then it won’t help us to detect CryptoLocker. Since we don’t know how CryptoLocker enumerates files (alphabetical, sorted by size, …), it’s probably best to sprinkle them throughout the various vulnerable file shares, using file names which start various letters of the alphabet, e.g.:

  • a_meeting_notes_cl.docx
  • m_meeting_notes_cl.docx
  • s_meeting_notes_cl.docx
  • z_meeting_notes_cl.docx

A name pattern is not required but helpful when configuring EventSentry later, since it allows you to just specify a wildcard (e.g. *_meeting_notes_cl.docx) instead of specifying dozens of files manually.

Creating multiple bait files is particularly important for newer versions of CryptoLocker which doesn’t always parse/encrypt all directories. So it’s best to create multiple bait files and distribute them across multiple directories, e.g.:

  • marketing\m_meeting_notes_cl.docx
  • sales\a_meeting_notes_cl.docx
  • accounts_payable\z_meeting_notes_cl.docx

This way we’ll have a higher chance of detecting malicious behavior. CryptoWall is fast (of course depending on the speed of the infected host) and can often encrypt tens of thousands of files in an hour.

Implementation
We will use EventSentry’s File Checksum Monitoring feature to monitor the bait files and trigger events when one or more of these files are changed or deleted (=renamed). When they are, we will trigger a script which will stop the server service on the file server in order to avoid more damage being done. Click here to learn more about EventSentry’s architecture.

Monitoring files only for (checksum) changes is no longer sufficient since newer variants of CryptoWall not only modify but also rename (and subsequently delete) documents.

In EventSentry, create a new System Health package with the name “CryptoLocker Detection” and assign it to any server on your network that is monitoring with EventSentry and is serving files through a file share. Now, add the “File Checksum Monitoring” object to the package and ensure the following:

  • “Monitor folder(s) in real time” is checked
  • Disable (uncheck) both “Only verify checksum when ..” optimizations

Then, click the “plus” icon to add the (first) folder where a bait file exists to the list of monitored folders. There are a few things to consider when setting this up

  • The folder/directory name should be specified as it exists on the file server, UNC paths are not recommended.
  • Check the “Include Sub Directories” check box you are monitoring files in sub folders
  • Check “Detect File Deletions” and “Detect File Checksum Changes”. File size increases and decreases may also be checked but is not required.
  • Configure the “Files” section to “Only monitor files that are included below” and specify the file name either with a full (relative) path or with a wild card.
  • Select a severity of “Error” under “Log to Event Log as”

File Monitoring Configuration

Example
Your file server has a directory called C:\FileShares\Marketing with two sub directories, Ads and Images. If we were to add a bait file to both subdirectories (say specs.docx and meeting1.docx) then we would specify C:\FileShares\Marketing as the folder, and then add

  • Ads\specs.docx and
  • Images\meeting1.docx

as the files to be monitored. This is because we always specify the path relative to the main folder being monitored when specifying the file names.

Splendid, EventSentry will now log an event to the event log when any of these files change. Try it out – open the file in word, make a change & save – you should get an alert in the event log almost instantly.

Process action to stop the server serviceDive! Stopping the Server Service
Stopping the server service may seem like a drastic step, but it’s unfortunately the most efficient way to prevent an impending CryptoLocker infection from spreading. Sure, blowing up the bridge might seem crazy at first, but if it prevents an army of Zombies (who obviously can’t swim) from entering your town, then we can probably live with the collateral damage.

You can stop a service in 2 ways with EventSentry; with the “Service / Process Control” action as well as with a custom script. Creating a “Service / Process Control” action is easier, but only works for stopping services which have no dependencies. You can probably guess where I’m going with this – the server service depends on other services (e.g. when the “File Sharing Role” is enabled) and thus cannot be stopped with the EventSentry action. Consequently we will go a different route and create a process action instead, which essentially allows you to trigger any process, script etc. Better safe than sorry.

Right-click the Actions container and click “Add” to create a new action called “Stop Server Service”, and select “Process” as the action type. Specify “net.exe” as the Filename, and “stop lanmanserver /yes” as the command line arguments. The “/yes” switch ensures that any service which depends on the “Server” service also gets stopped.

Connecting the dots
Since we now assume that a modification of one or more of our bait files only happens when a CryptoLocker outbreak is under way, the only thing missing now is to have the file change event trigger the process action and shut down the service.

EventSentry uses the concept of “Event Log Filters” to link events to actions, such as sending an email and/or triggering a process. Filters need to be part of an “Event Log Package”, and we can now either create a new package or add our filter to an existing package. For documentation purposes and to keep things orderly we will create a new event log package called “CryptoLocker Prevention”.

We do this by selecting the “Packages – Event Logs” container and clicking “Add” from the ribbon, you can also right-click that container. Give it a descriptive name and select the package, which we now need to assign to one or more hosts and/or groups. Click “Assign” in the ribbon to assign the package, you can also make the package global by clicking the respective button.

With the package all ready to go, we now need to add the filter. With the package still selected, on the ribbon click the “Add” button under “Event Log” and select “Include”. This event log filter, as is, would not apply to any event, since no event log and no severity is selected.

Event Log Filter

Anything detected by the EventSentry agent (e.g. a file checksum change, service status change, low disk space) is logged to the Application event log with the source “EventSentry”, a matching category (e.g. “File Monitoring”) and usually with a configurable or dynamic severity. In our case the file checksum change events will be logged as Errors, as configured earlier.

So let’s first configure the event properties as shown in the screenshot:

Log: Application
Event Severity: Error
Source: EventSentry
Category: File Monitoring

We also add the “Stop Server Service to the list of actions to be triggered. Since we may have other system health packages which log File Monitoring events, we want to make sure that this filter only applies to those, which we do by restricting the filter further with an event id as well as with a Content Filter.

For CryptoLocker we want to get notified about every change that happens to our bait file. Whether it’s deleted, a checksum change or a file size change. As such, we leave the event id field empty and specify the “File Monitoring” category instead.

Important Note: If you are running a German version of Windows, the category will need to be specified in German (“Dateiüberwachung”) since EventSentry is localized for German.

Our filter could still apply to unrelated file checksum changes (e.g. OS files were changed by a Windows Update), but since any file checksum change event includes the package name which triggered the event, we can filter based on that name (we called the package “CryptoLocker Detection”) to ensure that we only match file changes from CryptoLocker. In the “Content Filter” section click the “+” button to add a new content filter.

The quickest way to specify the content filter is to leave the “Wildcard match” setting in place and simply specify *CryptoLocker Detection* as the content filter. A more elegant way is to use an Insertion String match and selecting insertion string 5, which represents the package name (click “Preview” to see the insertion string numbers).

Event Log Content Filter

The setup is now complete, and you can now push the configuration to the remote host(s) which has the bait files and should be protected. If you have multiple file servers with a different directory structure, then you can easily create multiple system health packages which contain a file monitoring object, and assign them accordingly. For example, you could create packages named:

  • CryptoLocker Detection Server1
  • CryptoLocker Detection Server5

The process action doesn’t have to be duplicated, since the stopping the service is the same process for all hosts. The event log filter may need to be adjusted depending on how it was setup. A wild card like *CryptoLocker Detection* would match “CryptoLocker Detection Server5” as well, but an insertion string filter would need to be modified to something like CryptoLocker Detection* in order to match multiple more than one package.

An alternative to email alerts. Part 2: Integrating EventSentry with Slack

Slack is a flexible, web-based messaging app for teams which supports multiple (mobile) platforms with the goal of streamlining communication and collaboration. In some ways, Slack feels like a combination of Hipchat, IRC and Dropbox (you can also send alerts to Hipchat from EventSentry).

Event submitted to slack by EventSentry

By creating channels (e.g. chat rooms) and also allowing for direct 1:1 communication, interacting with your team is straightforward and easy. What’s more, you can send 3rd party feeds such as Twitter notifications directly into a channel, giving you the ability to see data from multiple sources in one central location.

Slack naturally lends itself to receiving EventSentry notifications with its web-based API. And, since you can create multiple channels, it’s easy to divert EventSentry alerts into different channels, e.g. #alerts-windows and #alerts-development.

Since Slack offers native clients (e.g. Windows, MacOSX) in addition to their web-based site, you can choose to opt in to receive a visual notification every time an alert is received by Slack.

If you’re not already using Slack, then you can sign up for free at http://slack.com and see for yourself whether this is a communication platform you will want to use. If you already use Slack, then you will just need to make sure that the “slackbot remote integration” is setup. You can find information on how to enable this integration here: https://api.slack.com/slackbot.

Enabling slackbot

Once the Slackbot Remote Control integration is configured in Slack, follow these steps within the EventSentry Management Console using the slackbot URL provided by Slack to set up an HTTP action. The HTTP action will be triggered by one or more filters in an event log package to submit events/alerts to Slack.

1. Add a new action
In the left pane right click “Actions” and select “Add Action”

Adding an EventSentry action

2. Selecting the correct action type
On the “Action Selection” screen type your name for the new action (Ex: Slack), select “HTTP” and then press OK

EventSentry Action Dialog

3. Configuring the HTTP action
In the right pane you should now see the settings for your new Slack action, update the URL field with the slackbot URL provided by Slack. If you did not specify a channel in your url after the token add “&channel=%23” followed by the name of your channel.

Example: https://team.slack.com/services/hooks/slackbot?token=zk2jR22I34AK24IpEd7tdyroGt&channel=%23es-alerts

Configuring the HTTP action for Slack

4. Configuring the data to submit

In the right pane configure the Data you’d like slackbot to post to the channel, the following is suggested for basic information:

$EVENTCOMPUTER: $EVENTID:$EVENTSOURCE:$EVENTCATEGORY by $PACKAGE - $FILTER
 $EVENTMESSAGE

Configuring data to submit

Additional variables can be found in our documentation.

Once the action is configured click “Test” and you should see the test message in Slack.

5. Finishing up

You can now apply the action to any existing or new packages and filters. To learn more about filters review the documentation or walk through this tutorial.

Save the EventSentry configuration and push the configuration to the remote hosts.

Done!

CryptoLocker Defense for Sysadmins (Part 1/3)


! Updates !
There have been 2 (!) follow-up posts to this article (part 2 and part 3), with even better approaches to defeating ransomware. I highly recommend that you jump directly to the most recent article which offers the best & easiest approach for protecting against Ransomware:

Defeating Ransomware with EventSentry & Auditing (part 3)


It seems as if CryptoLocker has been making the rounds lately, much to the dismay of users who don’t have working backups of their precious office documents.

While I admire Cryptolocker’s simplicity and effectiveness from a purely technical and entrepreneurial standpoint, what the software is doing does appears to be illegal in most countries and so I’d like to offer some advise on how to tame the beast. If you’re looking for a 5-minute fix then I have bad news: implementing the CryptoLocker defense I have outlined below, while completely free, will take a little more than 5 minutes to implement. But knowing that you have an effective defense against CryptoLocker may very well be worth it. After all, CryptoLocker seems to find its way into a lot private networks these days.

CryptoLocker Screenshot

The ideas set forth in this blog post apply mostly to Windows-networks with file servers, but could be adapted for individual computers as well (though this is not covered here – let me know if you’d like me to include this scenario).

About CryptoLocker
For those who have not heard of CryptoLocker yet, it is a piece of software which encrypts pretty much all common office-type documents, including Microsoft Office, AutoCAD, PDFs, images and more. This blog article from MalwareBytes has a complete list of extensions. Once encrypted, CryptoLocker charges you to decrypt (your own files) again. It’s public key cryptography gone wrong; I wonder if Diffie & Hellman saw this one coming. And to make the whole spiel even more interesting, you only get a limited amount of time to pay before your files will remain encrypted. Forever. Oh – and the longer the wait, the more you have to pay. And with recent bit coin exchange rates in excess of USD 1000, the amount that needs to be paid can be uncomfortably high.

It is pretty difficult to defend against something like CryptoLocker other than through usual means of AntiSpyware software, user eduction and strict policies against opening and downloading files from the Internet, email attachments and such. In most cases CryptoLocker comes in form of a ZIP attachment disguised with a PDF icon.

One reason CryptoLocker is so effective – yet difficult to block – is because it exhibits the same behavior as users would: It “simply” accesses and modifies files like a user would. And infecting a machine isn’t all that difficult since CryptoLocker doesn’t require any elevated permissions to run. On the contrary, it wants to run in the same context the user does, so that it can access and see the same files a user does. As such, security features like UAC are utterly useless against ransomware like CryptoLocker – it’s a whole new type of software.

Backups
The most effective defense against CryptoLocker is to have a working, tested backup. Let me repeat this: A WORKING and TESTED backup. Users have lost all their data because they thought that they had a backup in place when their backup was broken in some way.

We’ve seen posts of users who deleted all the files CryptoLocker encrypted, thinking they had a working backup. They had a backup, but it was apparently not recently tested and as a result the user lost all of their data.

Naturally, CryptoLocker does not like backups. It dislikes them so much that when CryptoLocker runs, it even tries to delete any Windows Shadow Copy backups. Cloud backup services (including Dropbox, Skydrive and Google Drive etc.) which keep versions of your files offer some protection, but restoring older versions of your files may be a tedious process.

The Defense
The most obvious defense against CryptoLocker is AntiSpyware software, e.g. MalwareBytes. Most AntiSpyware & AntiVirus software still uses signatures however, so new versions of the ransom ware often remain undetected at least for a few days.

So instead of detecting CryptoLocker itself, we can sniff its tracks so to speak. CryptoLocker’s predictable behavior can be used against it. CryptoLocker’s objective is of course to encrypt and hold hostage as many files as possible, so to increase the likelihood of the user purchasing the decryption key from the thugs.

And it is that very pattern that we will try to exploit and use as a trigger to detect and take corrective measures. The approach consists of measuring how many files are being changed in a certain time interval, and if a certain threshold is being exceeded (say more than 10 files modified in 1 minute) we assume that CryptoLocker found its way into our castle. Even though users modify their documents on a regular basis, users can usually make only so many changes at a time and most likely at a much slower rate than any sort of script / software would.

Another approach would be to create one or more honeypot or canary files, which we know (or hope) a user would not modify. If a checksum change in one of those files were detected, we could (more or less) safely assume that CryptoLocker was on one of his rampages again and take corrective measures. The honeypot file would have to be modifiable by users (otherwise CryptoLocker would also not be able to modify it), which makes accidental modifications by users possible (although somewhat unlikely).

This 2nd approach isn’t quite as solid in my opinion, since CryptoLocker is most certainly adapting to changes, and may skip files that it may suspect are a trap. For example, it could skip small files or skip directories with a very small number of files and so forth.

A more sophisticated approach, where we detect an unusually large number of files changes in a small time period, is going to be harder to circumvent by CryptoLocker. The good news is that we have a free (it’s really free, not a trial) software tool available which can do just that. It can:

  • detect file changes
  • measure the rate of file changes (through event log alerts)
  • stop/start services or launch processes
  • send out alerts

EventSentry (Light) to the rescue
EventSentry Light is the free version of our full-spectrum monitoring & compliance solution EventSentry. The features we can utilize to come up with a defense are:

  • File Checksum Changes (part of System Health Monitoring)
  • Filter Thresholds (part of Event Log Monitoring)
  • Action (control services, send out emails)

File Checksum Monitoring
Monitors any folder and detects file size changes, checksum changes and file additions and deletions. EventSentry Light will log file checksum changes to the event log (it’s big brother can also log them to a database), which in turn is monitored by the real-time event log monitoring component.

Event Log Monitoring & Thresholds
This component supports a variety of sophisticated features, one of which are thresholds. The thresholds feature lets you essentially detect event log entries that occur at a certain pace. For example, if 10 specific events occur in 1 minute then let me know and/or take corrective action.

Service Action, Email Action
EventSentry supports a variety of action types to be triggered when an event occurs, with email usually being the most commonly used one. You can also control services, use REST APIs, launch processes and much more. We’ll use the former to stop the file sharing services (LanmanServer) when we have determined that CryptoLocker is on the loose.

I will go into step-by-step instructions on how to configure EventSentry at the end of the post.

The Baseline
The most difficult thing to determine is the maximum rate of file changes we deem normal, as we need to have a baseline in order to configure the threshold slightly above that. This number will vary from network to network, with file servers serving lots of users obviously requiring a larger threshold. I’d like to repeat that determining the right threshold is very important. If it is too low, then normal user activity will trigger an alarm; if it is too high then the alarm may never be triggered and CryptoLocker won’t be caught in time.

The best approach is to setup file monitoring and let it do its job for 1-2 days to determine a baseline. Once the baseline is established, we can increase it by a certain factor (say 1.5) and use that as the threshold.

Setting up the trap requires 3 steps. In this case we assume that EventSentry is either installed directly on the file server, or an agent is deployed on the file server (in which case you will need to make sure that configuration updates are pushed to the file server(s) in question).

Step 1: Monitoring the directory/ies
In EventSentry, right-click the system health packages and add a new package. Right-click the package, select “assign” and assign it to all file servers. Right-click the package again and add a “File Monitoring” package. Click the new object. Directories are monitored in real time by default, but EventSentry requires a recurring scan as well – in case Windows doesn’t send real time notifications. This is usually a good thing, but when you are monitoring large directories it’s best to set the interval very high (future versions will allow for this to be unchecked).

File Checksum Monitoring Settings

In the package, add all the folders which should be monitored and only check the “checksum change” check box. Do not check any of the other check boxes in the bottom left section at this time. Since we haven’t established a baseline yet, we’ll set the severity of the event log alerts to “Information”. If the monitored folders contain a lot of non-Office files then it may be a good idea to adjust monitoring so that only office files (e.g. .doc, .xls, etc.) are monitored. If you prefer to monitor all files, simply change the setting to the green PLUS icon and make sure the list of exclusions is empty (or specifies files that should be excluded, e.g. *.tmp). Below is a screenshot of how this can be configured.

File Checksum Monitoring Settings

When you save the configuration, EventSentry will enumerate all files in the folder and create an initial checksum for every file. The agent will log event 12215 when the scan starts, and event 12216 when the scan is complete. When that happens, EventSentry is essentially “armed” and will detect, and log, all checksum changes to any of the files in the monitored directories.

At this point we’ll want to let this run for at least 24 hours during a “normal” work day, as to determine how many file changes occur on average. You are going to be at a bit of an advantage if you are running the full or the trial version with database support, as it will be a lot easier to determine the number of file changes occurring through the web-based reporting.

Step 2: Setting up the trap
Now that we have established a baseline, we’re ready to setup a threshold. This time we’ll create a new event log monitoring package. Right-click “Event Log Packages” and add a new package and call it “CryptoLocker Rules”. Like before, assign it to the file servers we are monitoring. Right-click the package again and add a new event log filter. Configure the filter as shown in the screenshot below. Note that we are triggering an email action for now. The content filter can be used to restrict the filter further, e.g. to only match certain directories if you are monitoring several directories with EventSentry.

Event Log Filter Setup

Now things are getting interesting. The goal is to create an error event in the event log when X amount of file checksum changes occur in a given time period. To get there, we’ll start with the “General” tab where we tell the filter what type of event we are interested in (see below). Once that event is defined, we’ll move on to the “Threshold” tab which is where we specify the threshold parameters. For the purpose of an example, let’s assume that we have established a baseline of 100 file checksum changes per day, with a work day starting at 8am and ending at 7pm. Assuming that activity is somewhat spread throughout the day, this amounts to about 9 file changes per hour. Naturally we’ll have to assume that file changes aren’t always evenly spread out throughout the day, but setting up a “if 20 checksum changes occur in 1 minute shut file sharing down” is probably a reasonable threshold. Configure the threshold as shown in the screenshot below, with whichever threshold you came up with.

Event Log Filter Threshold Setup

Step 3: Triggering corrective action
When our threshold is reached, EventSentry will log an error to the event log with event id 10601 and trigger the specified action(s) from the “General” tab (Default Email) one time per threshold interval.

At this point we would merely receive an alert when we suspect that CryptoLocker is at it again. If you are cautious then you can retain this setup for a little while (e.g. a day or two) to ensure that you are not getting any alerts about the threshold being met (assuming that CryptoLocker is not active on your network in which case you should get the emails).

To go all in and trigger a server service shutdown, we’ll need to create a service action now. On Windows, file sharing services are provided by the “Server” service, which uses the internal name of “LanmanServer”. The service action allows you to control any service (start/stop/restart), and in this case we’ll obviously want to stop the server service, so that clients cannot access the file shares on your server anymore. We’ll trigger an email action at the same time of course, so that the sysadmin in charge is aware of what is going on. While shutting down all file services seems a bit extreme, it’s unfortunately the most effective way to prevent more files from becoming encrypted.

So for the next step, right-click the “Actions” container and select “Add Action”. At the selection dialog choose the “Service” action, enter a descriptive name (e.g. “Stop File Sharing”) and hit enter.

Selecting an EventSentry Notification

Then, configure the settings of the service as shown in the screenshot below.

Action to stop the LanmanServer service

The last step of our setup (congratulations if you’ve made it that far) is to assign the service action to the filter we previously created. After all, a service action which isn’t referenced anywhere doesn’t do much good. So head back to the Event Log Packages, find the “CryptoLocker Rules” package and edit the filter in the package. In the action list on top, click the “Add” button and add the action you just created.

Testing
If at all possible I’d recommend testing the EventSentry setup at a time when your users are not interrupted. Adding a few template files to one of the monitored folders and changing them in short succession (a script may be necessary depending on how short your threshold interval is) should trigger the file services shutdown procedure. Once verified, you can just start the “Server” service again.

Conclusion
Just like in the real world, network viruses come in all shapes and sizes – only limited by technology and the imagination of the cyber-evildoers.

I hope that this article gave you some insight into CryptoLocker and a good way to guard against it. As always, make sure that your company has the following in place:

  • Email Attachment scanning
  • Working, tested backups
  • User education
  • AntiSpyware software

With those in place, one should be able to keep future infections to a minimum.

Stay safe & decrypted.

Mobile Alerts: Pushing EventSentry alerts directly to your mobile devices with Prowl and NMA

When it comes to mobile alerts, email seems to still be the prevalent method of choice for many IT pros. There are many good reasons why network alerts delivered via email are convenient:

  • easy to configure
  • uses existing infrastructure
  • every smartphone and tablet supports email
  • supports attachments (e.g. performance charts from EventSentry’s performance alerts)
  • integrates into your existing environment – everybody already uses email!

What’s not to like? Well, of course it turns out that some of these advantages can also be a disadvantage:

  • emails are not real time
  • problems on the email server often don’t surface immediately
  • important alerts can be overlooked in the inbox jungle
  • you cannot be alerted about email problems via email (duh!)

Viable Email Alternatives
Thankfully, there are a number of alternatives that can be used as an email substitute or addition for mobile alerts. In this article I’ll focus on two affordable services: Prowl (for the iOS platform) and Notify My Android (you may have guessed it – for the Android platform) – subsequently referred to as “NMA”. Both of these services consist of apps for their respective platform and a web-based back end which will push the notifications to your device(s) in near real time.

Mobile Alert on iPhone 5 with Prowl
EventSentry event sent to Prowl on an iPhone

Both services offer an HTTP API which we can connect to with EventSentry’s HTTP action. If you have never used the HTTP action in EventSentry, then here is some background: the HTTP action allows you to POST any event (whether it be an event from the security event log or a heartbeat alert for example) to either Prowl’s or NMA’s web service. These notifications are then pushed to one or more mobile devices.

Neither service currently requires a monthly subscription, but both require a purchase (Prowl costs USD 2.99 whereas NMA costs USD 4.99) if you want to send unlimited notifications to your mobile device(s). NMA is a little more welcoming to strangers – it supports up to 5 notifications per day at no charge.

Prowl: Getting Started
The
iOS app costs USD 2.99, and supports up to 1000 API calls (=notifications) per hour. To get started:

  1. Purchase the Prowl: Growl Client from the Apple App Store and install it
  2. Register for free at prowlapp.com
  3. Login & create an API key

Notify My Android (NMA): Getting Started
The Notify My Android app is free in the Google Play store and supports up to 5 API calls (=notifications) per day  for free. Upgrading to a premium account will allow up to 800 API calls per hour. To get started:

  1. Download the Notify My Android app from the Google Play store
  2. Register for free at www.notifymyandroid.com
  3. Login & create an API key
  4. Optional: Upgrade to a Premium account to allow for unlimited notifications

Once you have the mobile app installed and the API key in hand, you can start setting up a HTTP action in EventSentry. Notifications essentially consist of three fields: The application name, a subject as well as a message field, all of which can be customized. As such, it’s up to you configure which part of an event log alert you will put in the subject, and which part you will put into the actual message. In our example, we generate a dynamic notification subject with the host name, event id, event source and event category. The notification body will simply consist of the event message text, though this can be customized as well.

Setting up a HTTP action
Right-click the “Actions” container (or, in v3.0, use the “Add” button in the Ribbon) and create a new HTTP action. The HTTP action requires a URL at minimum, optional credentials and the actual data fields to submit in the HTTP POST. Conveniently, both Prowl & NMA use the same field names. I suspect that they are adhering to some sort of standard, though I couldn’t find any references.

The first parameter to configure is the URL, which depends on the service you use:

Prowl: https://api.prowlapp.com/publicapi/add
NMA: https://www.notifymyandroid.com/publicapi/notify

Configuring authentication credentials is not necessary since you are essentially authenticating with the API key you generated. The last step is configuring the form fields. The bold field on the left is the name of the form element and the text to the right the value. A description follows in italic on the next line:

apikey: abdef123123abababafefefe
The API key you received

application: EventSentry
This will displayed as part of the notification. I use “EventSentry” here, but this can really be anything, so it could be the host name as well for example ($EVENTCOMPUTER)
Max Length: 256

event: $EVENTCOMPUTER-$EVENTID-$EVENTSOURCE-$EVENTCATEGORY
This is the subject of the notification. You can use any variables, including insertion strings $STR1, $STR2 etc.
Max Length: 1024 [Prowl], 1000 [NMA]

description: $EVENTMESSAGE
This is the main message body of the notification, the $EVENTMESSAGE seems like a pretty good candidate for this field
Max Length: 10000

priority: 0 (possible values are -2, -1, 0, 1 and 2)
This field is optional and doesn’t do anything with NMA. With Prowl however, a priority can be set to “2” (indicating an “emergency”) which may then override quiet hours on the mobile app (if quiet hours are configured in the Prowl mobile app).

Take a look at the respective API documentation for Prowl and NMA as well; each show additional fields which can be configured, as well as additional information that you may or may not find useful.

The screenshot below shows a fully configured HTTP action for Prowl:

EventSentry HTTP Action for Prowl
EventSentry HTTP Action for Prowl

The HTTP action for NMA would look identical, with the exception of the URL which is different. You can click the “Test” button, which will submit the configured data to the specified URL and should, when configured correctly, immediately generate a notification on the mobile device. Please note that event variables will not be resolved when testing.

Threshold
To make sure that your mobile device doesn’t get flooded with alerts (some applications have the tendency to generate not one but hundreds of events in a short period of time), I highly recommend that you setup a threshold on the action or the event log filter referencing the action (I personally prefer the former). You can also setup a schedule so that notifications are only sent on certain days and/or during certain hours.

The last step would be to configure an event log filter to forward select events to the mobile device, something that is beyond the scope of this article. See the tutorials below for more information:

Reliability
It seems only natural to wonder whether alerts sent through these services can be used for mission critical systems. I’ve been using mostly Prowl as I’m an iPhone user, and have been very happy with it’s fast response times (which are almost instant), the service reliability and the stability of the iOS app. Nevertheless, both prowlapp.com and NMA state that you should not solely rely on their system for critical alerts and instead setup multiple channels for mission critical alerts. This sounds nice in theory, but suspect that most sysadmins will not want to dismiss alerts on more than one device – something that can get old if you get a fair number of alerts. Switching to a commercial system like PagerDuty with guaranteed up-time may be preferable in that case. I will talk more about PagerDuty in an upcoming post soon.

My experience with the NMA Android app wasn’t as good during the limited testing I performed. While it worked great when it worked, the app did crash on me a couple of times.

Conclusion
If you’re looking for a way to push alerts to your mobile device from EventSentry without using email and without spending big bucks, Prowl and NMA are worth looking into. They’re affordable, responsive and easy to configure.

EventSentry Light Supercharged

The latest EventSentry update brings significant and very exciting changes to the free light edition, EventSentry Light.

We have always seen EventSentry Light as a successor to the original EventwatchNT – a monitoring tool for small networks to alert sysadmins by sending real-time alerts about event log activity. As its big brother EventSentry continued to mature, most features from EventSentry made it into the light edition as well: Service Monitoring, Disk Space Monitoring, Performance Monitoring and many more. Since EventSentry Light was, and continues to be, free, it needed to distinguish itself from the commercial edition. As such, most feature were somewhat restricted in the light edition which only allowed a limited number of packages, event log filters, performance counters and such.

es_light.png

Over the last year we’ve been getting feedback that EventSentry Light was being constrained too much. Since our goal is, and always has been, to empower sysadmins and not constrain them, we decided to provide our users with more functionality in the free edition. The result of these efforts is build 2.93.1.75, and with this release you can:

  • Monitor event logs and log files in real time, setting up as many filter rules as you’d like, without restrictions.
  • Utilize all advanced event log filter capabilities like thresholds, timers, schedules and more!
  • Create as many event log, log file and system health packages as you like.
  • Utilize all system health monitoring features, such as file checksum monitoring, performance monitoring, service monitoring and more.
  • Create a variety of alerts using mail, HTTP, SNMP traps, Syslog messages and more.
  • Receive SNMP traps from SNMP v3 enabled devices.
  • Monitor up to 2 full hosts and 2 network devices.

Pretty impressive, no? So literally overnight, EventSentry Light has matured into a full-fledged monitoring solution which will alert IT professionals like sysadmins of critical (event) log events, performance issues and much more. What differences with EventSentry remain? A few, but the line is much more clear now:

  1. Reporting. With EventSentry, you get log consolidation, software/hardware inventory, performance trend reports, network dashboards, jobs, JSON/XML/CSV/… APIs and more.
  2. Compliance. EventSentry includes a variety of compliance functionality such as process tracking, logon tracking, account management tracking and more.
  3. Monitor multiple hosts. Monitor as many hosts as you are licensed for, and also utilize command line utilities to automate remote host management.
  4. Support. EventSentry includes quality email and phone support, something we pride ourselves on. EventSentry Light offers forum support.
  5. Mobile iPhone & Android apps are only available in the full edition since they require reporting.

So if you’re not already using EventSentry Light, or using an older version, then you should give it a try. It’s as free as it gets with no registration required, no advertisements and no nagging pop-ups. We hope you like it as much as we do. And did I mention that you can seamlessly upgrade from EventSentry Light to EventSentry? 🙂

Don’t forget to check out our other free tools and Facebook / Google+ pages!
Best,
Ingmar.