Windows supports a code-signing feature called Authenticode, which allows a software publisher to digitally sign executable files (e.g. .exe, .msi, …) so that users can verify their autenticity. The digital signature of a file can be viewed in the file properties in Windows explorer on the “Digital Signature” tab.
Digital signature verification has been added to the checksum utility, which already calculates the checksum and entropy of a file. When using the new /s switch, checksum.exe will tell you whether:
the file is digitally signed
a counter signature exists
the digital signature is valid
the algorithm used (e.g. SHA 256)
who signed the file
who issued the certificate
when the file was signed
The utility also sets the ERRORLEVEL variable accordingly; if a signature check is requested with the /s switch but the file is unsigned, then checksum.exe will return %ERRORLEVEL% 2. Below is a sample output of the utility in action:
Digital signature verification will be added to EventSentry’s FIM monitoring component (“File Checksum Monitoring”) in the upcoming v3.4.3 release, which will give you the option to only get notified when unsigned files are changed, thus reducing overall noise.
You can download the latest version from here – enjoy!
In part one I provided a high level overview of PowerShell and the potential risk it poses to networks. Of course we can only mitigate some PowerShell attacks if we have a trace, so going forward I am assuming that you followed part 1 of this series and enabled
Script Block Logging
Security Process Tracking (4688/4689)
I am dividing this blog post into 3 distinct sections:
We start by attempting to prevent PowerShell attacks in the first place, decreasing the attack surface. Next we want to detect malicious PowerShell activity by monitoring a variety of events produced by PowerShell and Windows (with EventSentry). Finally, we will mitigate and stop attacks in their tracks. EventSentry’s architecture involving agents that monitor logs in real time makes the last part possible.
But before we dive in … the
PowerShell Downgrade Attack
In the previous blog post I explained that PowerShell v2 should be avoided as much as possible since it offers zero logging, and that PowerShell v5.x or higher should ideally be deployed since it provides much better logging. As such, you would probably assume that basic script activity would end up in of the PowerShell event logs if you enabled Module & ScriptBlock logging and have at least PS v4 installed. Well, about that.
So let’s say a particular Windows host looks like this:
PowerShell v5.1 installed
Module Logging enabled
ScriptBlock Logging enabled
Perfect? Possibly, but not necessarily. There is one version of PowerShell that, unfortunately for us, doesn’t log anything useful whatsoever: PowerShell v2. Also unfortunately for us, PowerShell v2 is installed on pretty much every Windows host out there, although only activated (usable) on those hosts where it either shipped with Windows or where the required .NET Framework is installed. Unfortunately for us #3, forcing PowerShell to use version 2 is as easy as adding -version 2 to the command line. So for example, the following line will download some payload and save it as calc.exe without leaving a trace in any of the PowerShell event logs:
does the exact same thing. So when doing pattern matching we need to use something like -v* 2 to ensure we can catch this parameter.
Microsoft seems to have recognized that PowerShell is being exploited for malicious purposes, resulting in some of the advanced logging options like ScriptBlockLogging being supported in newer versions of PowerShell / Windows. At the same time, Microsoft also pads itself on the back by stating that PowerShell is – by far –the most securable and security-transparent shell, scripting language, or programming language available. This isn’t necessarily untrue – any scripting language (Perl, Python, …) can be exploited by an attacker just the same and would leave no trace whatsoever. And most interpreters don’t have the type of logging available that PowerShell does. The difference with PowerShell is simply that it’s installed by default on every modern version of Windows. This is any attackers dream – they have a complete toolkit at their fingertips.
So which Operating Systems are at risk?
PowerShell Version 2 Risk
Active By Default
Windows 2008 R2
Windows 8 & later
Potentially Vulnerable – depends on .NET Framework v2.0
Windows 2012 & later
Potentially Vulnerable – depends on .NET Framework v2.0
Versions of Windows susceptible to Downgrade Attack
OK, so that’s the bad news. The good news is that unless PowerShell v2 was installed by default, it isn’t “activated” unless the .NET Framework 2.0 is installed. And on many systems that is not the case. The bad news is that .NET 2.0 probably will likely be installed on some systems, making this downgrade attack feasible. But another good news is that we can detect & terminate PowerShell v2 instances with EventSentry (especially when 4688 events are enabled) – because PowerShell v2 can’t always be uninstalled (see table above). And since we’re on a roll here – more bad news is that you can install the required .NET Framework with a single command:
Of course one would need administrative privileges to run this command, something that makes this somewhat more difficult. But attacks that bypass UAC exist, so it’s feasible that an attacker accomplishes this if the victim is a local administrator.
According to a detailed (and very informative) report by Symantec, PS v2 downgrade attacks haven’t been observed in the wild (of course that doesn’t necessarily mean that they don’t exist), which I attribute to the fact that most organizations aren’t auditing PowerShell sufficiently, making this extra step for an attacker unnecessary. I do believe that we will start seeing this more, especially with targeted attacks, as organizations become more aware and take steps to secure and audit PowerShell.
Well, I think you get the hint: PowerShell v2 is bad news, and you’ll want to do one or all of the following:
Uninstall PowerShell v2 whenever possible
Prevent PowerShell v2 from running (e.g. via AppLocker)
Detect and terminate any instances of PowerShell v2
If you so wish, then you can read more about the PowerShell downgrade attack and detailed information on how to configure AppLocker here.
Uninstall PowerShell v2
Even if the .NET Framework 2.0 isn’t installed, there is usually no reason to have PowerShell v2 installed. I say usually because some Microsoft products like Exchange Server 2010 do require it and force all scripts to run against version 2. PowerShell version 2 can manually be uninstalled (Windows 8 & higher, Windows Server 2012 & higher) from Control Panel’s Program & Features or with a single PowerShell command: (why of course – we’re using PowerShell to remove PowerShell!):
While running this script is slightly better than clicking around in Windows, it doesn’t help much when you want to remove PowerShell v2.0 from dozens or even hundreds of hosts. Since you can run PowerShell remotely as well (something in my gut already tells me this won’t always be used for honorable purposes) we can use Invoke-Command cmdlet to run this statement on a remote host:
Just replace WKS1 with the host name from which you want to remove PowerShell v2 and you’re good to go. You can even specify multiple host names separated by a comma if you want to run this command simultaneously against multiple hosts, for example
Well congratulations, at this point you’ve hopefully accomplished the following:
Enabled ModuleLogging and ScriptBlockLogging enterprise-wide
Identified all hosts running PowerShell v2 (you can use EventSentry’s inventory feature to see which PowerShell versions are running on which hosts in a few seconds)
Uninstalled PowerShell v2 from all hosts where supported and where it doesn’t break critical software
Terminate PowerShell v2
Surgical Termination using 4688 events
If you cannot uninstall PowerShell v2.0, don’t have access to AppLocker or want to find an easier way than AppLocker then you can also use EventSentry to terminate any powershell.exe process if we detect that PowerShell v2.0 was invoked with the -version 2 command line argument. We do this by creating a filter that looks for 4688 powershell.exe events that include the -version 2 argument and then link that filter to an action that terminates that PID.
If an attacker tries to launch his malicious PowerShell payload using the PS v2.0 engine, then EventSentry will almost immediately terminate that powershell.exe process. There will be a small lag between the time the 4688 event is logged and when EventSentry sees & analyzes the event, so it’s theoretically possible that part of a script will begin executing. In all of the tests I have performed however, even a simple “Write-Host Test” PowerShell command wasn’t able to execute properly because the powershell.exe process was terminated before it could run. This is likely because the PowerShell engine does need a few milliseconds to initialize (after the 4688 event is logged), enough time for EventSentry to terminate the process. As such, any malicious script that downloads content from the Internet will almost certainly terminated in time before it can do any harm.
The above approach won’t prevent all instances of PowerShell v2.0 from running however, for example when the PowerShell v2.0 prompt is invoked through a shortcut. In order to prevent those instances of PowerShell from running we’ll need to watch out for Windows PowerShell event id 400, which is logged anytime PowerShell is launched. This event tells us which version of PowerShell was just launch via the EngineVersion field, e.g. it will include EngineVersion=2.0 when PowerShell v2.0 is launched. We can look for this text and link it to a Service action (which can also be used to terminate processes).
Note: Since there is no way to correlate a Windows PowerShell event 400 with an active process (the 400 event doesn’t include a PID), we cannot just selectively kill version 2 powershell.exe processes. As such, when a PowerShell version instance is detected, all powershell.exe processes are terminated, version 5 instances. I personally don’t expect this to be a problem, since PowerShell processes usually only run for short periods of time, making it unlikely that a PowerShell v5 process is active while a PowerShell v2.0 process is (maliciously) being launched. But decide for yourself whether this is a practicable approach in your environment.
Command Line Parameters
Moving on to detection, where our objective is to detect potentially malicious uses of PowerShell. Due to the wide variety of abuse possibilities with PowerShell it’s somewhat difficult to detect every suspicious invocation of PowerShell, but there are a number of command line parameters that should almost always raise a red flag. In fact, I would recommend alerting or even terminating all powershell instances which include the following command line parameters:
Highly Suspicious PowerShell Parameters
Skip loading profile.ps1 and thus avoiding logging
Let a user run encoded PowerShell code
-ep bypass, -exp bypass, -exec bypass
Bypass any execution policy in place, may generate false positives
Prevents the creation of a window, may generate false positives
-v 2, -version 2.0
Forces PowerShell version 2
Any invocation of PowerShell that includes the above commands is highly suspicious
The advantage of analyzing command line parameters is that it doesn’t have to rely on PowerShell logging since we can evaluate the command line parameter of 4688 security events. EventSentry v220.127.116.11 and later can retrieve the command line of a process even when it’s not included in the 4688 event (if the process is active long enough). There is a risk of false positives with these parameters, especially the “windowStyle” option that is used by some Microsoft management scripts.
In addition to evaluating command line parameters we’ll also want to look out for modules that are predominantly used in attacks, such as .Download, .DownloadFile, Net.WebClient or DownloadString. This is a much longer list and will need to be updated on a regular basis as new toolkits and PowerShell functions are being made available.
Depening on the attack variant, module names can be monitored via security event 4688 or through PowerShell’s enhanced module logging (hence the importance of suppressing PowerShell v2.0!), like event 4103. Again, you will most likely get some false positives and have to setup a handful of exclusions.
Command / Code Obfuscation
But looking at the command line and module names still isn’t enough, since it’s possible to obfuscate PowerShell commands using the backtick character. For example, the command.
This means that just looking for DownloadString or Net.WebClient is not sufficient, and Daniel Bohannon devoted an entire presentation on PowerShell obfuscation that’s available here. Thankfully we can still detect tricks like this with regex patterns that look for a high number of single quotes and/or back tick characters. An example RegEx expression to detect 2 or more back ticks for EventSentry will look like this:
The above expression can be used in PowerShell Event ID 800 events, and will trigger every time a command which involves 2 or more back ticks is executed. To customize the trigger count, simply change the number 2 to something lower or higher. And of course you can look for characters other than the ` character as well, you can just substitute those in the above RegEx as well. Note that the character we look for appears three (3) times in the RegEx, so it will have to be substituted 3 times.
To make things easier for EventSentry users, EventSentry now offers a PowerShell event log package which you can download via the Packages -> Download feature. The package contains filters which will detect suspicious command line parameters (e.g. “-nop”), detect an excessive use of characters used for obfuscation (and likely not used in regular scripts) and also find the most common function names from public attack toolkits.
It’s still possible to avoid detection rules that focus on powershell.exe if the attacker manages to execute PowerShell code through a binary other than powershell.exe, because powershell.exe is essentially just the “default vehicle” that facilitates the execution of PowerShell code. The NPS (NotPowerShell) project is a good example and executes PS code through a binary named nps.exe (or whatever the attacker wants to call lit), but there are others. While the thought of running PowerShell code through any binary seems a bit concerning from a defenders perspective, it’s important to point out that downloading another binary negates the advantage of PowerShell being installed by default. I would only expect to see this technique in sophisticated, targeted attacks that possibly start the attack utilizing the built-in PowerShell, but then download a stealth app for all subsequent activity.
This attack can still be detected if we can determine that one of the following key DLLs from the Windows management framework are being loaded by a process other than powershell.exe:
You can detect this with Sysmon, something I will cover in a follow-up article.
Now, having traces of all PowerShell activity when doing forensic investigations is all well and good, and detecting malicious PowerShell activity after it happened is a step in the right direction. But if we can ascertain which commands are malicious, then why not stop & prevent the attack before it spreads and causes damage?
In addition to the obvious action of sending all logs to a central location, there are few things we can do in response to potentially harmful activity:
1. Send out an alert
2. Mark the event to require acknowledgment
3. Attempt to kill the process outright (the nuclear option)
4. A combination of the above
If the only source of the alert is from one of the PowerShell event logs then killing the exact offending PowerShell process is not possible, and all running powershell.exe processes have to be terminated. If we can identify the malicious command from a 4688 event however, then we can perform a surgical strike and terminate only the offending powershell.exe process – other potentially (presumably benign) powershell.exe processes will remain unharmed and can continue to do whatever they were supposed to do.
If you’re unsure as to how many PowerShell scripts are running on your network (and not knowing this is not embarrassing – many Microsoft products run PowerShell scripts on a regular basis in the background) then I recommend just sending email alerts initially (say for a week) and observe the generated alerts. If you don’t get any alerts or no legitimate PowerShell processes are identified then it should be safe to link the filters to a “Terminate PowerShell” action as shown in the screenshots above.
After downloading and deploying the PowerShell package I recommend executing a couple of offending PowerShell commands to ensure that EventSentry will detect them and either send out an alert or terminate the process (or both – depending on your level of conviction). The following commands should be alerted on and/or blocked:
Any detection rules you setup, whether with EventSentry or another product, will almost certainly result in false alerts – the amount of which will depend on your environment. Don’t let this dissuade you – simply identify the hosts which are “incompatible” with the detection rules and exclude either specific commands or exclude hosts from these specific rules. It’s better to monitor 98 out of 100 hosts than not monitor any host at all.
With EventSentry you have some flexibility when it comes to excluding rules from one or more hosts:
PowerShell is a popular attack vector on Windows-based systems since it’s installed by default on all recent versions of Windows. Windows admins need to be aware of this threat and take the appropriate steps to detect and mitigate potential attacks:
Disable or remove legacy versions of PowerShell (=PowerShell v2)
Enable auditing for both PowerShell and Process Creation
Collect logs as well as detect (and ideally prevent) suspicious activity
EventSentry users have an excellent vantage point since its agent-based architecture can not only detect malicious activity in real time, but also prevent it. The PowerShell Security event log package, which can be downloaded from the management console, offers a list of rules that can detect many PowerShell-based attacks.
Imagine someone getting the seemingly innocent ability to run a couple of commands on a machine on your network WITHOUT installing any new software, but those commands resulting in a reverse shell running on that same machine – giving the intruder a convenient outpost in your network. Now stretch your imagination even further and pretend that all of this happens without leaving any unusual traces in logs – leaving you completely in the dark. It’s like somebody living in your house or apartment yet you have no idea they’re there. Are you getting goose bumps yet?
Not too long ago I talked with Michael, the creator of the popular cheat sheets which cover PowerShell, the Windows Registry, Windows Logging and more. Michael ran a few scenarios by me that involved exploiting PowerShell and was curious how EventSentry could help detect those. This really sparked my interest in the topic, and after coming up with a few RegEx expressions that could be used in an EventSentry filter I decided to look more into this subject. I really have to take the opportunity to thank Michael here, whose cheat sheets and input helped me come up with this article and the new PowerShell Security event log package in EventSentry.
If you’re not an InfoSec professional then you may not be fully aware that PowerShell – you know, the language you’re supposed to be fluent in by now – is quite commonly used in attacks. In fact, InfoSec already reported back in 2016 that 38% of all attacks utilize PowerShell in one way or another. And let’s be honest – why wouldn’t you utilize a tool that is pretty much guaranteed to be installed while giving you full access to the .NET Framework and all Windows APIs? So if you haven’t already done so, then securing PowerShell in your environment is something you should think about sooner rather than later. This and the follow-up articles will assist you with this effort.
So what’s so potentially bad about PowerShell in particular? Now, Windows has always shipped with VBScript, a scripting language that’s easy to use for both simple and potentially more complex tasks. In fact, most of the things people do in PowerShell can be done with VBScript just the same. A key benefit of PowerShell however is the ability to utilize the .NET framework, something VBScript can’t since it can only interact with COM objects. And since PowerShell is, well, a shell, you get to pipe input/output and create powerful one-liners. On top of that, PowerShell contains some nifty features like encoding scripts, making it possible to run fairly complex code without ever having to use an actual .ps1 script file on disk. It’s VBScript on Steroids.
Here are some concrete examples as to what evildoers can do with PowerShell:
Remember when I talked about “without leaving a trace” above? That’s because Microsoft didn’t introduce the ability to log detailed PowerShell activity until version 5, although PowerShell 3 & 4 generate reasonably useful audit logs as well. In order to protect ourselves against PowerShell attacks we need to first detect it, which we can only do if PowerShell leaves traces. PowerShell’s ability to produce useful audit logs greatly depends on the version however, which the table below illustrates:
Which version of Windows ships with which version of PowerShell
What is the highest supported version of PowerShell for each version of Windows
Microsoft .NET Framework 4.5.2
(already installed on 2012 and later)
All versions of Windows since Server 2008 and the version of PowerShell that it included by default
Default PowerShell vs Highest Supported PowerShell Versions
Included With Windows
Windows Vista (SP2)
Windows Server 2008 (SP2)
Windows 7 (SP1)
Windows 2008 R2 (SP1)
Windows 2012 R2
Shows the versions of PowerShell that ship with Windows as well as the highest supported version of PowerShell
As you can see from the table above, thankfully most versions of Windows are compatible with PS v5, so unless you’re unfortunate enough to be running Server 2008 (or Vista), you should be able to deploy PowerShell 5.1 to most of your systems. I say most, because some Microsoft applications (e.g. Exchange Server 2010) aren’t compatible with PowerShell v5, so you’ll want to make sure you do some research on those machines that actively use PowerShell to prevent disruption.
Coexistence & Legacy
An important thing to note here is that PowerShell v1/v2 can peacefully coexist with PowerShell v3-v5, while versions 3 and later are always upgraded to the latest version. This means that you could have v2 and v4 installed (and many systems do), but not v3 and v5. What’s also interesting is that PS v2 is installed with every major version of Windows (including Server 2016!) although not usable until the .NET Framework v2.0.50727 is installed.
Starting with EventSentry v18.104.22.168 you can thankfully use EventSentry’s software inventory to determine which versions of PowerShell are installed on your network. If you haven’t manually deployed PS v5 yet and aren’t running Windows Server 2016 widely yet, then you will probably see PowerShell v2 and v4 installed on most hosts on your network. EventSentry’s grouping mechanism comes in real handy here.
Please note that even though PowerShell v2 may be installed on a machine it doesn’t necessarily mean that PowerShell v2 is actually usable. PowerShell relies on the .NET Framework being installed, and PowerShell v2 specifically relies on the .NET Framework 2.0.50727 (which is part of the 3.5 .NET Framework) – something that is usually not installed by default. I will explain later why this is a good thing.
OK, but enough about boring PowerShell versions. If you just remember one thing from the above tables and paragraphs it’s this:
Thankfully you don’t need version 5.x to get useful logging – even PowerShell v3 & v4 can log relevant details in the (Windows PowerShell) event log, e.g. the PowerShell command line or commands executed within the PowerShell shell. In fact, even the (decoded) commands are logged to the event log when obfuscated with the -encoded switch.
Logging can be enabled either through group policy or via registry settings. There are three general areas for logging available:
Script Block Logging
Since everything that is executed in PowerShell is essentially located in a module, module logging will at least generate a high-level audit trail of PowerShell activity and potentially malicious activity. At minimum this will show which commands were executed through PowerShell. This logging level should always be enabled and is useful starting with PS version 3.
Important: Module Logging only works if you specify at least one module to be monitored. Since it’s difficult and cumbersome to predict and edit a list of all modules that could potentially cause harm, I recommend just specifying the * wildcard characters as the module list – see screenshots below.
Script Block Logging Script Block Logging is more verbose than module logging and provides additional context and output, especially when functions are called and function output itself is invoked as a command. The amount of noise heavily depends on the type of PowerShell activity, but I’d recommend turning this option on as well. If it ends up producing too much noise / volume it can always be disabled or customized later.
This provides a full log of all input and output and requires additional considerations in regards to where the transcription files are placed. I’d only recommend this for high-secure environments, you can learn more about it here. Transcript files are stored in the file system, so it’s a little more work than just adding up a couple of registry values. If you enable this feature then you’ll need to make sure that the actual transcript files (which likely contain sensitive data) are protected from unauthorized access.
It’s definitely recommended to configure these options via Group Policy to ensure that all machines in the domain receive the settings. If changing group policy is not an option in the short term then you can at least set the registry options until you have an opportunity to set it via group policy. You can use a tool like the EventSentry Admin Assistant to push registry settings out to multiple hosts with just a few clicks.
Group Policy: Configuring this is unfortunately less straightforward than you’d think or expect, depending on the OS version of your domain controller. You can expect the “Module Logging” option to be available in the group policy editor on 2008 R2 and later, however “Script Block Logging” is only available on server 2016 or after manually updating ADMX files. See this thread on how to update your ADMX files. In my environment I just had to replace the PowerShellExecutionPolicy.admx and en-US\PowerShellExecutionPolicy.adml files in the %SYSTEMROOT%\SYSVOL\sysvol\[DOMAINNAME]\Policies\PolicyDefinitions directory with the newer versions after installing the latest version from here.
Registry: Only the HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\PowerShell key exists by default, the other two sub keys “ModuleLogging” and “ScriptBlockLogging” have to be created before you can add the “EnableModuleLogging” and “EnableScriptblockLogging” DWORD values inside those sub keys.
For Module Logging, as shown in the screenshot below, you’ll also need to create the “ModuleNames” sub key along with a list of modules that will be monitored. I recommend just using the asterisk character which monitors any module.
Configuring PowerShell Event Logging
Data: 1 (DWORD)Key: HKLM\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ModuleLogging\ModuleNames
Data: [ModulePattern] (REG_SZ)See screenshot above for example on module logging.
Policies\Administrative Templates\Windows Components\Windows PowerShell\Turn on Module Logging
You don’t need to restart after setting the registry values, they will become effective immediately. The same applies to group policy – as soon as the target host has applied the group policy settings, PowerShell will enforce the new logging options.
PowerShell logs a lot of different events to two different event logs, and the table below shows the events I have observed on test systems. Even though the table may not be 100% complete, it does list all the events that are relevant for threat detection. If an event is not listed below then it is likely not relevant for forensics. We will update the list if necessary.
What’s interesting to note is that newer versions of PowerShell will often log to both event logs simultaneously.
Security Event Log Auditing
PowerShell logging is great, but given the discrepancies between the different versions and the possibility to evade it (more on that later), I prefer to have as many methods as possible at my disposal that tell me what PowerShell is doing.
Since PowerShell code is usually invoked via powershell.exe (I’m point this out because you technically don’t have to use powershell.exe, and attackers are coming up with creative ways to launch it through other ways – more in part 2 of this series), and because we’re after that processes’ command line, it’s important to monitor Process Start (event id 4688) events from the security event log in addition to events logged by PowerShell itself. This means you’ll need audit the following sub categories from the Detailed Tracking category:
If you are not using EventSentry then I recommend collecting both 4688 and 4689 events so that you can not only determine whether a powershell.exe process was started, but also how long it remained active. If you are an EventSentry user then you just need to verify that Process Tracking (an object for Compliance Tracking) is enabled and configured to capture the command line of a process. EventSentry can automatically parse and correlate 4688 and 4689 events and provide a history of all processes on a monitored system.
EventSentry users can also utilize the Audit Policy Status page to verify that process creations are indeed being audited. You’ll also want to make sure that “Include command line in process creation events” is activated, so that Windows logs the command line of every process as part of 4688 events. After all it doesn’t help us that much just knowing that powershell.exe has been running, we need to know what exactly it has been running.
This can either be enabled via group policy (Administrative Templates\System\Audit Process Creation\Include command line in process creation events) or via the registry (set HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\Audit\ProcessCreationIncludeCmdLine_Enabled to 1).
Disclaimer: This option is available starting with Windows 7 / Server 2008 R2, earlier versions of Windows don’t support it. Things are a little easier for EventSentry users, which attempts to obtain the command line of a process if it’s not included in the 4688 event and subsequently makes it available as variable $STR9. But more on that in part 2 when we discuss ways to detect and mitigate attacks.
I hope I was able to convince you of the risks that PowerShell poses, what versions of PowerShell are out there, and what type of logging needs to be enabled in order to detect and stop malicious PowerShell in its tracks. In part 2 I’ll talk about how to actually mitigate PowerShell-based attacks – with specific instructions for EventSentry.
In Mr. Robot‘s episode 9 of season 2 (13:53), Angela Moss needs to obtain the Windows domain password of her superior, Joseph Green, in order to download sensitive documents that would potentially incriminate EvilCorp. Since her attack requires physical access to his computer, she starts with a good old-fashioned social engineering attack to get the only currently present employee in the office to leave.
Once in his office, she uses a USB Rubber Ducky, a fast and automated keyboard emulator, to obtain Joseph’s clear text password using mimikatz. Please note that there are some holes in this scene which I will get into later. For now we’ll assume that she was able to obtain his credentials by having physical access to his computer.
After she gets back to her workstation, she analyzes the capture which reveals Joseph’s password: holidayarmadillo. Not the best password, but for this particular attack the quality of the password wouldn’t have mattered anyways. Mimikatz was (is) able to get the password from memory without utilizing brute force or dictionary techniques. Once she has the password, she logs off and logs back on with Joseph’s user and downloads the documents she needs.
As somebody who helps our users improve the security of their networks, I of course immediately contemplated how this attack could have been detected with EventSentry. Since most users only log on to one workstation on any given day with their user account, Angela logging in with Joseph’s account (resulting in “joseph.green” logging on to two different workstations) would actually be an easy thing to detect.
Introduced with v3.4, collector-side thresholds allow the real-time detection of pretty much any user activity that originates from an event, for example user logons or process launches. You can tell EventSentry that (physical) logons for any user on more than one host (within a given time period – say 9 hours) should trigger an alert (aka as “lateral movement”). Had this been in place at EvilCorp, Angela logging in as Joseph would have immediately triggered an alert. With the right procedures in place, countermeasures could have been taken. Of course most viewers wouldn’t want Angela to be caught, so please consider my analysis strictly technical. Watch a short video on lateral detection with EventSentry here.
So what’s the hole? Well, the rubber ducky (mimikatz, really) requires access to an active logon session, which Angela most likely didn’t have. It looked like Joseph had been out of the office for a while, so his computer was likely either locked or turned off, rendering any attack based on mimikatz useless. Mimikatz – since it obtains passwords from memory – only works if the computer is unlocked. And had the computer been unlocked then she could have just downloaded the files from his computer – although this would have been even more risky with people walking around the office.
Cyber attacks are becoming more potent every year and are often sponsored by powerful criminal gangs and/or governments. It’s important that companies employ multiple layers of defense to protect themselves (and their customers) from these increasingly sophisticated and destructive attacks.
EventSentry is the only monitoring solution that utilizes robust agent-based technology that goes beyond logs, enabling the fusion of real-time log monitoring with in-depth system monitoring to not only detect but also react to attacks and anomalies. See for yourself and download a free evaluation of EventSentry.
Almost every company which runs Microsoft Exchange Server needs to make port 443 available to the Internet in order to provide their users access to email via their mobile devices or OWA.
Since both OWA & ActiveSync utilize Active Directory for authentication, exposing OWA/ActiveSync to the Internet indirectly exposes Active Directory as well. While user lockout policies provide some protection against brute force attacks, additional protection methods should be employed. Furthermore, password spraying attacks may be use to circumvent lockout policies – something that would be more likely to succeed in larger organizations.
With the proper auditing enabled (Logon/Logoff – Logon (Failure)) and EventSentry installed however, we can permanently block remote users / hosts who attempt to log on too many times with a wrong password. Setting this up is surprisingly simple:
Windows: Enable (or verify) Auditing
EventSentry: Setup action which creates firewall block rule
EventSentry: Setup filter looking for 4625 Audit Failure events
Bonus: This procedure works with the free version of EventSentry (EventSentry Light) and can be applied to any IIS-based web site which uses authentication.
In the group policy settings that affect the server running OWA, make sure that auditing for Failure events in the Audit Logon sub category of the Logon/Logoff category is enabled (of course you can audit success events as well). If you are running the full version of EventSentry v3.4 or later then you can verify all effective audit settings on the Audit Policy Status page for example.
Creating an Action
Since event 4625 contains the IP address of the remote host, the easiest way to subsequently block it is to run the netsh command. In the management console, create a new action by clicking on the “Action” header in the ribbon and selecting the process action as its type. See the screenshot below:
The following command line will work in EventSentry v3.4 and later:
The difference here is that v3.4 and later can refer to insertion string variables by name, making the action more universal and potentially applicable to any event that uses the same field name.
When this action is triggered, it will extract the IP address from the event and block it from the system entirely.
Creating a Filter
Create an event log filter which matches Audit Failure events from the Security event log with event id 4625, where insertion string 19 matches the w3wp.exe process (C:\Windows\System32\inetsrv\w3wp.exe). This ensures that only users accessing the host via the web will be subject to blocking. The screenshot below shows the configuration:
This filter can either be added to an existing package or added to a new package that is assigned only to the Exchange server. If the filter is added to an existing package that applies to servers other than the Exchange server, then the computer field of the filter can be used to ensure the filter is evaluated only on the desired host. Select the action created in the previous step.
Since users may occasionally enter an incorrect password I recommend setting up a threshold so that remote IPs are only blocked after 3 or more failed logon attempts. Threshold are configured by clicking on the “Threshold” tab (see the blue “i” above) and an example configuration is shown below. Feel free to adjust the threshold to match your users ability to enter their password correctly :-). Insertion string 20 – which represents the IP address of this event – was selected in the threshold matching section to ensure that each IP address has its own, unique threshold. Note: The event logging settings shown are optional.
Save/deploy or push the configuration to the mail server.
Triggering a system process from external input is something we should always do with caution. For example, if Windows has an upper limit to the maximum number of rules that can be added, then an attacker could launch a DoS attack IF they had the ability to launch attacks from different IP addresses. Launching a DoS attack from the same IP won’t be possible once they are blocked. You can mitigate this risk by applying a threshold to the EventSentry action calling netsh.exe, for example by limiting it to 100 / hour. This would still provide sufficient protection while also ensuring that only 100 rules could be added per hour (thresholds can be set by clicking on the “Options” button on an action). A regular audit of the netsh execution (e.g. via Process Tracking) would quickly show any sort of abuse.
Over time the number of firewall rules added to the mail server could become rather large, which is why the rules are created with a date appended. This makes managing these rules easier, and the name can also be adapted in the action by changing the “rule name” parameter. The screenshot below shows the inbound firewall rules after two IPs have been blocked:
If manual cleanup of firewall rules is not desirable or an option, then the netsh command can also be wrapped into a script which would erase the firewall rule again after a timeout (e.g. 15 minutes). The script could look like this:
In this case you would call the wrapper script instead of the netsh.exe process directly (General Options – Filename) and use the string below as the arguments:
To keep things simple you can just make the script an embedded script (Tools menu) and reference the script. The timeout value (120 in the above example) is the duration seconds the remote IP will be be blocked. If you want to block the IP for an hour then you would set the timeout value to 3600 instead. When going this route I strongly suggest clearing both event log check boxes in the Options dialog of the action.
Before I continue I need to point out that DNS auditing has become significantly easier starting with Windows 2012 R2. Not only is it enabled by default, but the generated audit data is also much more granular and easier to interpret. The logged events even distinguish between regular and dynamic updates, making it easy to filter out noise. So if you’re serious about DNS auditing and have the option to update then I recommend you do so.
If you’re running Windows 2008 (R2) or 2012 then setting up DNS auditing requires a few steps. Thankfully it’s a one-time process and shouldn’t take more than a few minutes. On the EventSentry side a pre-built package with all the necessary rules is available for download and included with the latest installer.
Please follow the steps outlined below exactly as described, auditing won’t work or will be incomplete if these steps aren’t followed exactly as described below.
Enabling Directory Service Auditing
Enabling Sub Category Auditing
We first need to make sure that the new subcategory-based audit settings are enabled in group policy. If you’ve already done that, then you can skip this step and jump to “ADSIEdit”.
Since most of the steps here involve domain controllers, I recommend that you make the changes in a group policy, e.g. in the “Default Domain Controller” policy. In Group Policy Management, find an existing group policy, or create a new one, and set Computer Configuration\Polices\Windows Settings\Security Settings\Local Polices\Security Options\Force audit policy subcategory settings (Windows Vista or later) to override audit policy category settings to Enabled.
Then, navigate to Computer Configuration\Polices\Windows Settings\Advanced Audit Policy Configuration\Audit Policies\DS Access and set Audit Directory Service Changes to Success.
Open ADSI Edit via Start -> Run -> “adsiedit.msc”. If your default naming context does not automatically appear OR if the listed naming context does not include dc=domaindnszones, then select “Action -> Connect To” and connect to the appropriate naming context, e.g.
Replace the dc components after “domaindnszones” with the actual DNS name of your domain. It’s important that “dc=domaindnszones” is part of the naming context.
Once connected, expand the naming context and locate the “CN=MicrosoftDNS” container, right-click it, and select Properties. Then select Security, Advanced, Auditing and click on Add. In the resulting dialog we’ll audit the built-in “Everyone” user so that DNS changes from everyone are audited:
Name (Principal): Everyone
Apply onto: This object and all descendent objects
Access: Create dnsZone objects
It may seem tempting to also check the “Delete dnsZone objects”, but resist the temptation. Don’t be fooled by the term “dnsZone”, the ACE entry we just added will audit the creation of all AD DNS objects (and not just DNS zones) and log event 5136 to the security event log. In order to audit deletions as well, click “Add” again but this time configure the dialog as shown below:
Name (Principal): Everyone
Apply onto: Descendent dnsZone objects
It’s important that the “Apply onto” is changed to Descendent dnsZone objects. This ACE entry will result in event 5141 being logged when a DNS-related directory service object is deleted. This is were things get a bit interesting though, since DNS records deleted from the DNS manager aren’t actually deleted. Instead, they are tombstoned (which is done internally by adding the dNSTombstoned attribute to the object). Only when the tombstoned object expires is it actually deleted. You can use ADSIEdit if you want to send a DNS object immediately to heaven and skip the graveyard stage.
Unlike Windows 2012 R2 and later, earlier versions of Windows are a little more verbose than you probably like when it comes to directory service auditing. For example, creating a new DNS A record in a zone will result in 4 different events with id 5136 being logged – and not just one. The events logged when adding or deleting a zone or A record are shown in the diagram below:
All events are logged under the “Directory Service Changes” category.
Before we start configuring EventSentry, we’ll want to make sure that auditing was setup correctly. On a domain controller, open the “DNS” application and either temporarily add a new A record or primary zone. You should either see a 5136 or 5137 events with the category “Directory Service Changes” logged to the security event log.
If you don’t see the events then walk through the above steps again, or reference this Microsoft article.
There are generally two things one will want to do with these audit events – store them in a database, email them or both. If you’re already consolidating Audit Success events in an EventSentry database then you shouldn’t have to do anything, all directory service change events will be written to the database automagically.
If you only want to store directory service change events in the database (opposed to storing all audit success events), then you can simply create an include filter with the following properties:
Setting up alerts using filters for when zones or records are added or removed is a little more involved than one would hope, thanks to Windows logging more than one event whenever such a change is done – as is depicted in the image above. For example, creating a new DNS record will result in 4 x 5136 events being logged, deleting will result in 3 events.
Lucky for you, we’ve analyzed the events and created a DNS Server Auditing package in EventSentry which will email you a single alert (=event) whenever a record or zone are added or removed. This package is included with all new installation of EventSentry, existing users running 3.4 or later can get it through the package update feature in the management console.
And that’s really all there is to it … if you have EventSentry installed then setting up auditing in Windows is really the only obstacle to auditing your DNS records!
Integrate with open source solutions (Graylog, ELK, Nagios Log Server & others)
and more. We’ve also been focusing on making the data EventSentry collects more actionable and subsequently more helpful, and as result you will see additional contextual data provided with some alerts & reports, and one new search page in EventSentry.
All in all, this upcoming release has a lot of improvements in store to help you do your job better by ensuring that your network is as reliable, secure and fast as possible.
While high-profile Ransomware attacks have slowed down somewhat in recent weeks, businesses – especially small businesses – are still hit with Ransomware infections every day. Even though EventSentry is not positioned as a AV or a AntiMalware software, it does include a variety of functionality which can detect Ransomware outbreaks.
What sets EventSentry apart from AV and most AntiMalware solutions is that it looks for pattern exhibited by the Ransomware – regardless of the variant. What’s new in version v3.4 is the ability to detect changes to the MBR and bootsector as well as the ability to calculate the entropy of (changed) files.
MBR/BootSector Monitoring & Backup
Some more recent Ransomware variants modify the MBR and/or boot sector and trigger a reboot to initiate an offline encryption process. EventSentry v3.4 can now monitor the MBR and detect changes in near real-time to alert the admin when suspicious activity is occurring.
By utilizing EventSentry’s advanced filtering engine it is also possible to potentially stop the encryption process, e.g. by hibernating the infected host. EventSentry even backs up the MBR and boot sector, making it possible to repair an infected system (with a boot disk) without having to perform a full restore from backup.
Entropy describes the randomness of a file, essentially a metric that can help detect compressed and encrypted files.
Since most Ransomware encrypts large amounts of files, EventSentry can utilize the entropy of a file, combined with event log thresholds, to make a determination that a Ransomware infection is in progress and take corrective measures.
Lateral Movement Detection with Thresholds
EventSentry has always included agent-side thresholds in order to throttle the alert volume or detect repeated events. Because these thresholds were executed on the agents, event patterns which involved more than one host could not be detected that way.
By adding a threshold component to collector – which analyzes and processes all events in real time – we can leverage this feature to new heights and detect network-wide event-based patterns – in real time!
Collector-side thresholds are configured exactly like agent-side thresholds with one the key difference – the threshold limit applies to all aggregated events sent to the collector. Collector-side thresholds also introduce the “Group By” setting that makes the lateral movement detection possible – the ability to count unique instances of an event property, and not just the total number of events.
Some of the event-based threat patterns you can detect:
The same user logging on to multiple hosts within a specific time frame
A process spreading (trickling) across multiple hosts within a specific time frame
A user running too many processes – either on a single or multiple hosts
Authentication failures of a user on too many hosts
Too many unique logon types used by a user account
Any event property and insertion string can be used to craft thresholds – the sky is the limit.
Software Version Checker
Earlier versions of EventSentry include a substantial hardware, software and patch inventory, making it extremely easy to find out which software packages are installed on your network, but also get alerted when software is installed or removed from a server/workstation.
In v3.4 we are taking this to the next level by providing the latest version available from the publisher for a growing list of 100+ software packages so that you can effortlessly identify outdated software on your network. You can now create reports listing any software on your network which is outdated, a dashboard tile is also available. The Web Reports refresh the version info list every 2 hours to ensure all reports are accurate.
If an application you are using on your network is not currently included then simply email us the name of the software as it is detected by EventSentry (and ideally the URL where we can obtain the current version) and we will add it to our list.
64-bit Web Reports for Windows
The EventSentry web reports are now available as a 64-bit application, and upgrading to v3.4 will automatically upgrade the existing v3.x 32-bit web reports to 64-bit on 64-bit when installed/upgraded on a 64-bit version of Windows. The new 64-bit web reports will allow you to run larger reports that would not run due to limits with the address space associated with 32-bit applications.
UPS & Battery Monitoring
Any UPS directly attached to a server/workstations that is detected by Windows can now be monitored by EventSentry. The status of the UPS will show up on the host inventory page, and alerts will be generated when a host is on battery power and back on AC power. EventSentry can also initiate a shutdown when the remaining run-time or charge level falls below a certain limit.
Batteries in laptops are also detected and listed on the host inventory page (battery capacity and current charge level), but generated alerts are informational only.
User Activity Tracking
While EventSentry provides its users with a wealth of information from multiple angles, it can be tedious to piece together data from multiple reports that is associated with a specific user. Data which can be linked to a user is scattered among process tracking, file access tracking, compliance logons and other pages.
The new “User Activity” page, which is located in the “Dashboard” menu, solves this problem by displaying data from the following pages on a single page:
Active Directory Changes
The user activity page makes seeing all activity by a user as easy as never before!
Integration with third-party log management solutions
A few months ago, one of our users approached with the need to integrate EventSentry into an existing log management system which was already in place at the location where EventSentry was to be deployed. While reviewing the request we recognized that even though we position EventSentry as a one-stop log management solution with a compelling and robust web-based reporting component, an integration with other products can be helpful in some cases.
Supplement EventSentry’s built-in reporting with additional reporting
Integrate EventSentry with an existing log management solution located in a different business unit
Integrate EventSentry’s sophisticated real-time agent and deployment infrastructure with a different log management back-end
In version 3.3 and earlier, EventSentry can be integrated with 3rd party products using the HTTP, process and Syslog action. The HTTP & process action are intended to be used with ticketing systems where only a low volume of alerts are submitted while the Syslog action obviously supports submitting a high volume of events. The Syslog format was however limited to the traditional RFC 3164 format, making an integration with other log management systems difficult.
Starting with version 3.4, EventSentry now supports the following formats in the Syslog action
RFC 3164 (legacy)
Nagios Log Server
Common Event Format (CEF)
If a log management server you need to integrate with is not listed above but supports the JSON format, then you can craft your own JSON packet with the JSON format, also introduced in v3.4.
Disk Space Alerts
Part of the effort to make EventSentry’s alerts more actionable is reflected in our improved disk space alerts which now list the 15 largest files and folders of the volume where disk space is low. The supplemental data will in many cases be enough to immediately identify the culprit so that corrective action can be taken immediately, without the need to run disk space analyzers on the volume.
Audit Policy Status
Since the introduction of the compliance tracking components, EventSentry has been recording all audit (and many other!) changes performed in Windows as part of the “Policy Changes” feature. It wasn’t however possible to see the current status of all audit categories and subcategories at a glance. Reviewing the current audit status of all monitored hosts can be important however, if only to verify that group policies are configured correctly.
In v3.4 we now have the new “Status” page available under “Compliance -> Audit Policy” which delivers information such as:
Compare/review audit settings of a particular sub category (e.g. “Registry”) among all monitored hosts
View all disabled audit settings across all or select hosts
(Re)view audit settings based on computer types (e.g. domain controllers, servers, workstations)
Our NetFlow component can now provide bandwidth visualization based on the collected NetFlow data. The information can either be accessed on the NetFlow page or as a dashboard tile. Even though bandwidth data can already be determined using SNMP, the data gathered by NetFlow should be preferred since it contains additional data not available via SNMP, such as:
Bytes per packet
Bytes per packet as well as packets sent received can be used to detect anomalies, e.g. when a host sends a large amount of network packets, or network packets with large/small content.
Auditing changes on Microsoft Windows DNS server is a common requirement and question, but it’s not immediately obvious which versions of Windows support DNS Auditing, how it’s enabled, and where the audit data (and what data) is available. Fortunately Microsoft has greatly simplified DNS Server auditing with the release of Windows Server 2012 R2.
In this post we’ll show how to enable DNS Auditing on 2012 R2 and later, and how to configure EventSentry to collect those audit events. In a future post we’ll also show how to do the same with older versions of Windows – 2008, 2008 R2 & 2012.
When configuration is finished you are going to be able to see when a zone/record is created/modified/deleted as well as by whom. The audit data will be available (and searchable) in the EventSentry Web Reports, and you’ll also be able to setup email alerts when some or all DNS entries are changed.
Since native DNS auditing was only introduced with Windows 2012 R2 or later you’ll need to run at least Windows Server 2012 R2 in order to follow this guide. The table below shows the types of DNS auditing available on Windows Server Operating Systems:
Active Directory – based auditing only
covered in future post
Windows 2008 R2
Active Directory – based auditing only
covered in future post
Active Directory – based auditing only
covered in future post
Windows 2012 R2
Native DNS Auditing
Available with hotfix 2956577
(automatically applied via windows update!)
Native DNS Auditing
enabled by default
Enhanced DNS logging and diagnostics are enabled by default in supported versions of Windows Server when the “DNS Server” role has been added to Windows, so there are no additional configuration steps that need to be done.
1. Package Creation
On the EventSentry machine we are going to add a package under Packages/Event Logs by right-clicking “Event Logs” and selecting “Add Package”. In this example we are going to call the package “Windows Server 2016”:
2. Adding a Filter
The next step is to add a filter to the previously created package “Windows Server 2016”. Right click the package and then select “Add Filter”.
Note: For a short tutorial on how to create a filter click here.
3. Filter Configuration
There are several ways to approach the filter configuration depending on your needs. As a reminder, a filter is a rule in EventSentry that determines to where an event is forwarded to, or how it is processed.
Collect all or select (e.g. creation only) DNS audit data in the database
Email alert on select audit data (e.g. email all deletions)
Email alert on all activity from a specific user
In this guide we will show how to accomplish (1) and (2) as a bonus.
On the right pane of the management console after the creation of the filter you will see the General tab of the new filter. We decided to configure it to log to the Primary Database, but the events can be sent to any action (Email, Syslog, …).
Under “Event Severity” we check all boxes since we want to log everything (it’s important to check “Information” since most of the creation/deletions/etc are logged at this level of severity).
4. Adding a custom event log
In the “Log” section click on “more” to jump to the “Custom Event Logs” tab (or, just click on that tab). Now we need to add the Microsoft-Windows-DNSServer/Audit event log to the list of custom event logs so that this filter picks up events from the DNS Audit event log. Click the save button in EventSentry Management Console title bar to save the changes we’ve made so far.
5a. Assigning the package (method A – manual assignment)
To assign the package, select the server you would like to assign it to and select “Assign Packages”. In the resulting dialog simply check the box next to the package we created in step 1. Alternatively you can also select the package and click “Assign” from the ribbon (or context menu) and select a group or host(s) to assign the package to.
5b. Assign the package (method B – dynamic activation)
Instead of assigning the package manually, the package can be assigned dynamically so that any host monitored by EventSentry running Windows Server 2012 R2 or later will automatically have this package assigned. To dynamically assign a package do the following:
Click the package and select “Properties” from the ribbon, or right-click.
In the “Dynamic Activation” section, check “Automatically activate …”
In the “Installed Services” field enter “DNS”
For the “Operating System”, select “at least” and “Windows 2012 R2”
Click the “Global” icon in the ribbon to make sure the package gets assigned to all hosts. Don’t worry, it will still only be activated on 2012 R2 or later hosts that have the DNS server running
After assigning the package and saving the configuration click “Save & Deploy” or push the configuration to all remote hosts. Please note that only new events generated in the DNS Audit Log will be processed, pre-existing log entries will be ignored.
Testing the Configuration
To test the configuration we will create a domain called “testzone.com” and add an A record called www on the monitored DNS server. We’ll then check if those modifications are visible in the EventSentry Web Reports. The screenshot below shows the new A record in the DNS console:
First, lets take a look to see what the actual DNS Audit entries look like (using the Windows Event Viewer: Applications and Services Logs/Microsoft/Windows/DNS-Server/Audit):
In the EventSentry Web Reports, navigate to Logs/Event Log and filter by the log “Microsoft-Windows-DNSServer” and then select “Detailed”. You should see all the modifications that were performed:
Bonus Track: Configuring alerts for a specific change
The first part of this post was purposely generic in order to understand the basics of monitoring your DNS Server. But as a bonus we’ll show how to monitor a specific change (in this case a creation) and trigger an email alert for that.
The process is the same as explained in the beginning:
Create a new filter and add it to the same package. The filter should be configured exactly the same way. To make things easier, you can also copy & paste the filter with the familiar Copy/Paste buttons in the ribbon or context menu.
This time however we specify the “Default Email” action in “General” tab so that we receive an email alert when the filter criteria matches an event.
In the “Details” area specify event id 515 in the “Event ID” field, which is the event id corresponding to the creation of a new record. This is how the filter would look like:
Filters can of course be more specific as well, it’s possible to filter based on the user or event content of the actual event. Below is a list of all audit events logged by the DNS Server:
Enlist directory partition
Prepare for demotion
Write root hints
Record create – dynamic update
Active refresh trust points
Record delete – dynamic update
Zone scope create
Zone scope delete
Update from DS
Write and notify
Key rollover start
Key rollover end
Transfer key master
Key rollover triggered
Key poke rollover
Modify SKD state
Create client subnet record
Delete client subnet record
Update client subnet record
Server scope create
Create server level policy
Server scope delete
Create zone level policy
Add trust point DNSKEY
Create forwarding policy
Add trust point DS
Delete server level policy
Remove trust point
delete zone level policy
Add trust point root
Delete forwarding policy
Clear debug logs
Write dirty zones
We hope that DNS changes will never remain a mystery after activating DNS auditing. Don’t fear if you’re running 2012 or earlier, the next post is on its way.
Today I have good news and bad news. You’d like to hear the good news first? OK. The bad news is that AutoAdministrator is being retired and will no longer be developed.
OK, now on to the good news. AutoAdministrator is of course not entirely history and is undergoing a similar transformation the NTToolkit did a few years back. AutoAdministrator is joining the “EventSentry” brand as the “EventSentry Admin Assistant”.
This brings the total number of software products under the EventSentry brand to three:
The debate as to whether agent-based or agent-less monitoring is “better” has been answered many times over the years in magazine / online articles, blog posts, vendor white papers and others. Unfortunately, most of these articles are often incomplete, inaccurate, biased, or a combination thereof.
To make things slightly more confusing, different ISV use different methods for monitoring servers and workstations. Some use agents, some don’t, and a small few offer both both methods. But what is ultimately the best method?
What are you monitoring?
First it’s important determine what is being monitored to determine whether an agent-based or agent-less approach is better. For example, collecting system metrics like performance data usually creates fewer challenges then transmitting large amounts of (event) log data. Furthermore, agent-based monitoring is not an option for devices which run a proprietary embedded OS (think switch, printer, …) where you can’t install an agent in the first place.
Consequently I’ll be focusing on monitoring (event) logs with an emphasis on Microsoft Windows in this post. Having developed both agent-based as well as agent-less components in C++ over the years I feel that I am in a good position to objectively compare agent-based with agent-less approaches.
Monitoring software is of course not the only type of software that uses agents, a lot of other enterprise software (backup, deployment, A/V …) uses agents as well. Below are some of the myths as to what (monitoring) using agents entails:
Agents may use up too many resources on the monitored hosts and slow down the monitored machines
Agents can become unstable and negatively affect the host OS
Deploying and managing agents is tedious and time-consuming
Installing agents may require the installation & deployment of dependencies the agents need (.NET, Java, …)
Installing third-party software will decrease the security of the monitored host
It’s understandable that software which is installed on potentially every server and workstation in a network undergoes some level of scrutiny, but would you be surprised to learn that agents excel in the following areas:
Better security since agents push data to a central component, instead of the monitored server being configured to allow remote collection.
Agents can temporarily store and cache monitored logs if connectivity to the central monitoring server is lost, even if local logs are no longer available. Agents can also take corrective actions more quickly because they can work in isolation (offline). Mobile devices cannot be monitored with agent-less solutions since they cannot be reached by the central monitoring component.
Agents can apply local filtering rules and only transmit data which is valuable, thus increasing throughput while decreasing network utilization.
They offer more capabilities since there are essentially no limits as to what type of information can be gathered by an agent since it has full access to the monitored system.
The Easy Way Out
Developing agents along with an easy-to-use deployment mechanism requires a lot of time and resources, so it doesn’t come as a surprise to learn that many vendors prefer to monitor hosts without agents. To compensate for the short-fall, ISV which solely have to rely on an agent-less approach will do their best to:
Emphasize that they do not use agents
Persuade you that agent-less monitoring is preferable
The irony, when promoting a solution as agent-less, is that even so-called agent-less solutions do in fact utilize and agent – the only difference being that the agent is (usually) integrated into Windows. Windows doesn’t just magically service remote clients asking for a boatload of WMI data – it processes these requests through the WMI service, which, for all intents and purposes, is an agent. For example, accessing the Windows event logs via WMI traverses significantly more layers than accessing the event logs directly.
With the exception of network devices where an agent cannot be installed, agent-based solutions will provide a more thorough monitoring experience 9 out of 10 times – assuming that the agent meets all the checklist requirements below.
Some event log monitoring vendors will try to convince you that agent-less monitoring is better & easier (easier for whom?) – but don’t fall for it. We’ve been tweaking and improving the EventSentry agent for more than 10 years, and as a result EventSentry offers one of the most advanced and efficient Windows agents for log monitoring on the market. Developing a rock-solid, secure and fast agent is hard, but it’s the only sensible approach which doesn’t cut corners.
There are situations when deploying a full-scale monitoring solution with agents is not possible, for example when you are tasked with monitoring a third-party network where installing any software is not an option. While unfortunate, an agent-less monitoring solution can fill the gap in this case.
EventSentry also utilizes SNMP (agent-less) to gather inventory, performance metrics as well as other system data from non-Windows devices, including Linux hosts. This collection method does suffer from the above limitations, but since log data is pushed from Non-Windows devices via the Syslog protocol, it’s an acceptable compromise.
Don’t compromise when it comes to monitoring the (event) logs of your Windows infrastructure and select an architecture which scales and offers security & performance.
The table below examines the difference between agent-based and agent-less solutions in greater detail.
Resource Utilization & Performance
Usually higher throughput since agents can analyze, filter and evaluate log entries before sending them across the network.
Local resource utilization depends on the implementation of the agent.
Agent can access (event) logs directly via efficient API access.
Network utilization is likely much higher since more logs have to be transmitted across the network before being evaluated. Local filtering capabilities are limited and depend on the protocol (usually WMI).
Network latency and utilization affect performance of monitoring solution. Network utilization cannot be controlled.
Accessing (event) log data remotely through WMI is much less efficient.
Over-saturation of central monitoring component can negatively affect monitoring of entire infrastructure.
Higher network utilization combined with the fact that remote log collection will still utilize CPU cycles on the remote host (e.g. through WMI provider) favors agent-based solutions.
Agent-less solutions have a single point of failure, while agents can filter & evaluate data locally before transmitting them to a central database.
EventSentry Agents are designed to be essentially invisible under normal operations and do not impact the host system negatively in any way.
Stability & Reliability
Failure of an agent does not affect monitoring of other hosts.
Locally collected data can be cached if central monitoring component is temporarily unavailable.
Failure of a central component may negatively affect deployed agents if they rely on the central component and cannot cache data.
Failure of central monitoring system will affect and potentially disable monitoring of all hosts.
Hosts which lose network connectivity (e.g. laptops) cannot be monitored while unreachable.
Agent-based solutions have an advantage since local data can be cached and corrective actions can be executed even when the central monitoring component is unavailable. Cache data & logs are re-transmitted – even if the local logs have been cleared or overwritten.
Agent-based solutions can monitor hosts even when disconnected from LAN.
The EventSentry Agent auto-recovers if the process aborts unexpectedly, and by default alerts the user when this occurs. When using the collector (default), the agent caches all data locally and retransmits when the network connection becomes available again.
Has to be deployed either with vendor management software and/or with third-party deployment software if vendor provides installation package (e.g. MSI).
Larger deployments will require multiple central monitoring components, potentially distributed over several LANs.
Only hosts in local LAN can be monitored.
Depends on deployment tools made available by vendor as well as management tools in place for configuring Windows settings. A poorly developed deployment tool would favor an agent-less solution.
EventSentry Agents can be deployed (multi-threaded) with the management console or through 3rd party deployment software by creating a MSI installer on the fly. When using the collector (default), agent updates (patches) can be deployed automatically.
Agent may have dependencies on third-party frameworks
Depends on whether the mechanism utilized by the monitoring software requires a Windows component to be added and/or configured.
Depends on whether agent has dependencies and whether configuration changes need to be made on the monitored hosts.
The EventSentry Agent does not depend on any 3rd party frameworks or libraries.
Potential security issues if installed agent exposes itself to the network (if not firewalled) and/or suffers from local vulnerabilities which can be exploited.
Remote log collection has to be enabled and at least the central monitoring component needs to have remote access.
Secure data transmission relies on protocols and settings from Windows.
Enabling multiple methods for gathering data remotely (e.g. WMI) provides additional attack vectors.
Credentials (usually Windows user/password) for remote systems must be stored in a central location so that the remote hosts can be queried. If the central system gets compromised, critical credentials can be exploited.
Since agent-based solutions do not require permanent remote access and monitored hosts can therefore be hardened more, they are inherently more secure IF the agent doesn’t suffer from an insecure design and/or vulnerabilities.
Agent-based solutions also have more control over how data is transmitted from the remote hosts.
If there is general concern against third-party software then the product in question should be researched in a vulnerability database like http://www.cvedetails.com.
The EventSentry Agent does not open any ports on a monitored host and resides in a secured location on disk. The agent transmits compressed data securely via TLS to the collector. No major security vulnerabilities have been discovered in the EventSentry agent since its first release in 2002.
Scope & Functionality
Agents have full access to the monitored system and can choose which technology to utilize to get the required data (API, WMI, registry, …)
Easily execute local corrective action like launching a script or process
Agent-less solutions are limited to remote APIs provided by the monitored host, most commonly WMI. While WMI does offer a lot of functionality, there are limitations.
Executing scripts on remote host is more involved and only possible when host is reachable.
Agent-based solutions have an advantage since they can utilize multiple technologies to obtain data, including highly efficient direct API access. Agents can also trigger (corrective) actions locally even while the agent is unreachable.
Agent-less solutions can only monitor data which is made available by the remote protocol.
The EventSentry Agent accesses log files, event logs and other system health data almost exclusively via direct API calls. The more resource-intensive WMI interface is only used minimally, for very specific purposes. Corrective action can be taken directly on the monitored host, often only in milliseconds after an error condition (event) has occurred.
When evaluating software that offers agents then you can utilize the check list below for evaluation purposes.
An agent needs to consume as little resources as possible under normal operations. With the exception of short (and unusual) peak periods, a user should never know that an agent is running on their server or workstation – period.
The thought of having a resource-hogging agent running on a server sends shivers down the backs of many SysAdmins, and the agents used by certain AntiVirus vendors that rhyme with Taffy didn’t set a good precedent.
Stability & Reliability
The agent needs to run at all times without crashing – the SysAdmin needs to be able to go to sleep knowing that his agents will reliably monitor all servers and workstations. Unstable agents are just no fun, especially when they negatively impact the host OS.
If an agent that encounters an issue, it needs to at least auto-recover and communicate the issue to the admin.
Agent deployment and management needs to be streamlined and easy – it shouldn’t be a burden on the end user. And while agent deployment is important, agent management, keeping the remote agent up-to-date, is equally important and should – ideally – be handled automatically.
Most SysAdmins have enough work the way it is, the last thing they need is baby-sitting agents of their monitoring solution.
The more dependencies an agent has, the more difficult it is to deploy the agent. Agents that rely on complex frameworks like .NET, Java or specific Visual Studio runtimes are difficult and time-consuming to deploy.
Furthermore, any third-party software that is installed as a dependency creates an additional attack vector and needs to then be kept up-to-date.
An agent needs to be 100% secure and cannot expose the monitored host to any additional security risks. I will explain below why using agents is actually more secure than not using an agent – even though this seems counter-intuitive at first glance.