When China built an entire emergency
hospital in a matter of days in Wuhan – a city about the size of NYC that most
of us had never heard of – the world was watching with concern, but somehow
still expected and hoped that the crisis would somehow remain contained to
China, or at least Asia. People in Europe and the U.S. continued to go on about
their busy lives, occasionally glancing at the headlines coming from China,
where the government was taking drastic measures to curtail the spread.
It now seems beyond naive, even
childish, to have thought that the virus wouldn’t spread to other parts of the
planet. The world we live in today, where between 8,000 and 20,000
planes fly across the sky every single day, is the perfect conductor for a
virus with an incubation period as high as two weeks. This gave the virus, which
had already started making its way through Wuhan and China back in December,
more than enough time to slowly travel to other countries on planes and ships.
Fast forward two months, and what is
currently taking place in many parts of the world is something we would have
only expected from a fatalistic science fiction novel or cheesy Hollywood
movie: a stock market crash on par with that of October 1929,
a large percentage of planes grounded either due to government mandates or lack
of business, borders shut, and almost 200 million people – from democratic countries
nonetheless – under a curfew that will likely last weeks. Empty shelves in
grocery stores, abandoned playgrounds and formerly busy streets are now empty
resulting in thousands of closed restaurants and stores, some of them possibly
The current global crisis feels like
a medley of 9/11, the 2008 recession and then some. Yet it’s not due to a war
or natural disaster but because of the respiratory disease that goes by the
catchy name of “Covid19.” This disease is caused by the SARS-CoV-2
virus that was at some point transmitted to humans from animals, as far as we
know from either bats, pangolins or a combination of the two (whatever happened
to eating tofu?).
What does all this have to do with
monitoring and network security?
But one country in Asia, located much closer to China than Italy and with a similar population density, has managed to avoid the disaster that is currently ravaging through Europe. That country is South Korea, where the number of new cases has slowed significantly since its peak at the end of February, without imposing curfews. South Korea has accomplished this with rigorous testing and isolation, including tracing contacts of infected people and quarantining them. Singapore, Taiwan and Hong Kong were similarly successful.
How did they do this? Data.
Since a large percentage of infected people show little to no symptoms –
particularly difficult to distinguish during flu season – the only way
to suppress the spread of the virus is to know who has the virus in the
first place. And then, once identified, immediately isolate the affected individuals
and people who had contact with them. If you wait until sick people show
up at the hospital, then you are already way behind the curve. For every person
that shows up at the hospital, you likely have twenty more walking around
Here at EventSentry we’re neither virologist nor pandemic experts. But
there are noticeable similarities between this outbreak and a
computer virus/malware infection. The purpose of monitoring after all is to be
aware of what is happening on the network so that organizations can take action
to stem the infection. You can only fight what you can see and measure.
The equivalent of Covid19 testing in
IT is monitoring. Monitoring only part of your infrastructure isn’t enough –
just like testing only 1% of the population isn’t sufficient. Yes, the infected
hosts will eventually reach the monitored ones, but at that point the majority
of your infrastructure may already have been compromised.
Many computer viruses, when
infecting a computer (host), first attempt to silently infect other hosts
before they do damage in one way or another. SARS-CoV-2 has similar
properties with an usually long incubation period. During that time, the host
is unaware that he or she is carrying the virus, potentially infecting others
through direct or indirect (e.g. surfaces like door knobs) transmission.
The SARS-CoV-2 virus is quite sneaky
and would likely do well in the popular “Plague” game, where the
player creates a virus with the goal of infecting and ultimately killing the
entire world population. One of the most important properties of a virus in the
game is that it’s highly contagious but not too deadly – otherwise it would kill
all of its hosts before it can spread.
Thankfully, SARS-CoV-2 is neither as deadly nor contagious enough to accomplish this, yet it’s second only to the Spanish Flu that killed between 20-40 million people almost 100 years ago. See this article for more information on how Covid19 compares to past outbreaks.
Finding patient zero – the first person to have contracted a virus, is similar to finding the source of a malware outbreak. In medicine it may provide important clues on how to come up with a cure, whereas in IT security it can provide important information on how an attacker penetrated a network. Monitoring software like EventSentry doesn’t just detect problems in real time, it also collects troves of important logs and other system data that can be of incredible value after a network has been compromised. China is still desperately trying to find and confirm patient zero, who may have been infected as early as October 2019.
On our never-ending quest to slash cost in order to maximize profits, manufacturing of both medicine and medical supplies has been outsourced to China, India and other countries. While there is nothing wrong with saving costs and manufacturing items where it costs less, it’s clear that there is a benefit of manufacturing certain products in the country where they are being used.
Similarly, cost savings in IT budgets that compromise the overall security of the IT infrastructure and with it the company itself, rarely pay off in the long turn. As you can see from the matrix below, even a very unlikely circumstance that will have a significant impact on a business has a medium risk and should be addressed.
But in the midst of the all the
chaos and uncertainty, there also upsides. The severe reduction of air traffic
and travel give our planet a long overdue breather, as satellite images in Italy have shown. It’s also noteworthy that air pollution (and smoking) make
the lungs much more susceptible to respiratory diseases like Covid-19.
We need to remind ourselves that we’re not robots and machines but mammals that live on a planet shared with nature – animals – their viruses included. As we humans continue to encroach on their habitats and land, the risk of another deadly virus spreading doesn’t go away. Watch this short 5-year old video about bats and the viruses they carry.
For me it’s still difficult to comprehend that the current pandemic is connected to consuming bats, pangolins (most of which are endangered) and other wildlife. Some risks are just not worth taking, and it would be prudent of the Chinese government to permanently ban this obviously dangerous practice.
In the meantime people will need to continue to isolate, self-quarantine or shelter in place until the number of new cases continues to decline and toilet paper is available again.
To keep an eye on Covid-19 cases in your country and/or state with EventSentry (v4.1), you can follow the instructions in this HowTo and view Covid-19 stats in any dashboard or performance chart.
As an IT professional I encourage you to stay alert, as many bad actors are exploiting the current chaos with phishing campaigns for a variety of nefarious reasons. We highly encourage you to consider monitoring workstations and laptops with EventSentry to ensure you have complete visibility and prevent a bad situation from becoming worse, we are offering discounts on a case-by case basis. In addition to monitoring all the things you’re familiar with from your servers, EventSentry monitors laptop batteries, Bitlocker status, outdated software and more.
Thank you for being an EventSentry customer, stay safe and positive during this difficult time.
EventSentry v4.1 builds on v4.0 released earlier this year and offers a lot of exciting new & improved features that enhance a variety of different monitoring scenarios. In this release we improved:
Laptop / Mobile Monitoring
Expiring passwords can often be an issue in larger networks, especially for mobile users whose passwords expire while they are out of the office. The new “Password Reminder” feature in ADMonitor alleviates this issue by giving you the ability to automatically send out password expiration reminders to your users before the password expires. The only requirement is that the ‘mail’ attribute is set for your users, or that it’s possible to dynamically determine the end user’s email address from one or more of its AD properties (e.g. first name, last name).
We also added new ADMonitor-related tiles that provide an overview of recent AD changes.
Laptop / Mobile Monitoring
With an increasing number of employees working remotely, ensuring that laptops are properly monitored and secure should remain a priority for any company that manages laptops. Starting with v4.1, EventSentry detects the BitLocker status of any host, allowing you to run reports to identify all laptops that pose a security risk due to their hard drive not being encrypted.
We also improved operational monitoring by tracking the health of laptop batteries; EventSentry can now tell you how healthy a laptop battery is based on the current capacity and the charge cycle count.
Up until now performance data could “only” be retrieved from Windows performance counters and SNMP-based counters, but obtaining data from other sources like web pages or utilities was not supported. This limitation is a thing of the past as you can now use the output of any executable or script as a data source – with practically unlimited possibilities. For example, numerical data from system tools, web pages and log files can now be visualized and alerted upon – all with the same familiar interface. An example of this new functionality can be seen on our live demo, where we’re displaying air pollution stats from 4 major cities in the US along with the global PPM (courtesy of the EPA).
EventSentry’s NetFlow implementation already includes two important security-related features: The ability to detect port scans and identifying traffic going to / coming from potentially malicious IP addresses (with support for AbuseIPDb).
One potential short-coming with the malicious IP detection is that any of your public-facing IP addresses will – sooner or later – be contacted by a remote IP address deemed malicious. These alerts often result in unwanted and unnecessary noise, especially if the port / service that the remote IP address tried to contact is blocked anyways.
To make these alerts more actionable, NetFlow v4.1 keeps track of the amount of traffic sent/received from a malicious IP, resulting in intelligent notifications that are only triggered if the amount of traffic exceeds a (configurable) limit. This means that you will only get an alert if a meaningful amount of data (e.g. 1Mb) was transferred to or from a malicious IP – for example if a APT is active on your network. Once identified, you can either get an alert and/or take corrective action by blocking the offending IP address.
EventSentry’s architecture already supports MSP-style scenarios well: Granular permissions and multi-tenant support in the web reports allow for multiple clients, and remote agents running on a customer network can securely transmit all data encrypted over the Internet to the central EventSentry collector.
Starting with V4.1, both the heartbeat and network services components also integrate with the collector and can transmit all collected data from the customer’s network directly to the collector – instead of requiring a direct database connection.
Helping you keep your network as secure as Fort Knox and assisting you with your compliance requirements remains a top priority for EventSentry.
Do you know how many servers and workstations on your network require a reboot to finish installing Windows updates or software? EventSentry now detects pending reboots as part of its inventory functionality – simply schedule a report on this new flag and you’ll never forget to reboot critical systems again.
BitLocker detection mentioned earlier also helps you secure your mobile workforce by quickly identifying laptops that do not use full disk encryption.
To aid troubleshooting and forensic analysis we added a “Changes” view that shows all permanent changes that occurred on a selected host – for example services being added/removed, critical system files, software installations and more. This is available on the Host Inventory page as well as the IP Activity page.
For users who need to be compliant with CJIS we also added CJIS reports to list of compliance reports.
Besides a UI refresh and easier access to event log data, Syslog messages can now also be acknowledged – just like event log records.
Since accumulating too much data is a common issue for our users, we improved the Database Summary dashboard tile which actively monitors the database size and detects failed purge jobs.
The IP Activity page has been improved and now offers more actionable information about IP and host activity.
Finally, for those concerned about Java(c)’s licensing, EventSentry now utilizes the OpenJDK.
We hope you find this additions useful, as always please don’t hesitate to send us feedback. I also encourage you to check out our system32.eventsentry.com site which has a ton of information about Windows events and more. It’s under constant development and tells you how events are related, whether you should monitor them, which audit settings are associated with the event and much more.
The recently discovered BlueKeep RDP vulnerability reminds us yet again (as if we needed to be reminded) that monitoring RDP is not a luxury but an absolute necessity.
Many organizations still expose RDP ports to the Internet, making it a prime target for attacks. But even when RDP is only available internally it can still pose a threat – especially for large networks.
So let’s start this off with some very basic best practices:
Make sure that RDP access is blocked from the Internet (e.g. only accessible via VPN)
RDP should be disabled on hosts where it’s not needed
All RDP access should be monitored (see below)
In this post you will see how EventSentry (and EventSentry Light) can be configured to automatically block remote hosts that have failed to log on via RDP after a certain number of times. Utilizing EventSentry offers a number of benefits over other approaches:
It works with any version of Windows, from Windows 2008 to Windows 2019
It works regardless of account lockout policies
The threshold and time period are fully configurable
The default action (block Windows firewall) can be substituted and/or supplemented with other actions
Before we delve into the nitty gritty details I need to level the playing field and explain why blocking remote RDP connection attempts is not as simple as linking event id 4625 with type 10 (failed RDP logon attempt) with an action. See, in the good old days security events logged by Windows mostly meant what they said. Failed logon events logged by Windows always included the correct logon type – all the way back to Windows Server 2003 (back then it was event 529). Having an event that included both the username, IP address and logon type made it straightforward to setup a rule:
If # of failed logons with type 10 of a [user] and/or from [IP address] > [threshold] then do [ABC].
All that changed with the introduction of NLA (Network Level Authentication), where the initial authentication of a RDP session is offloaded to another Windows subsystem, resulting in key information being lost in translation inside Windows.
The result is that starting with Windows 2008 and NLA enabled, event id 4625 always classify failed RDP logon attempts as logon type 3 instead of logon type 10. As a reminder, logon type indicates a network logon – not a RDP logon. It’s consequently impossible to use 4625 events as the sole indicator for a failed RDP logon.
How do you know if NLA is enabled? It’s usually pretty simple: If you are prompted for credentials when initiating a RDP connection before you see the Windows logon screen then NLA is enabled.
In an effort to better audit RDP connectivity events, Windows 2008 and later include a new event log, the Microsoft-Windows-RemoteDesktopServices-RdpCoreTS/Operational log, which logs some RDP activity. I say some because it cannot be used to solely detect failed RDP logins. While we have been able to consistently generate events when a remote client connects (event id 131), we have been unable to consistently generate the more important event id 140, which indicates a failed login (which could be used in place of the 4625 event to trigger an action).
So what are we to do? On the one hand we have an event telling us that a RDP connection has been initiated (although not fully logged on yet), and on the other hand we have a failed logon event that is virtually identical to hundreds or thousands of other failed logon events.
Thankfully there is an easy solution with EventSentry’s filter chaining feature, which allows us to correlate events from the security event log with the new RdpCoreTS event log. This allows us to correlate audit failure event 4625 from the Security event log with information event 131 that is logged in the Microsoft-Windows-RemoteDesktopServices-RdpCoreTS/Operational event log.
Filter chaining is activated on the package level, and can trigger an action (e.g. email, process, …) when all filters in a package match events in a certain time period. To make sure that the correct types of events are chained (correlated) together, insertion strings sharing the same data can be specified. And since both events include the IP address of the remote host connecting, they can be linked (chained) together if they occur within a certain time frame (e.g. 10 seconds).
When linking events via insertion strings it’s important that the strings match exactly, any deviation will break the chain. This turns out to be a potential issue since event id 131 doesn’t just log the remote IP address but also the remote source port in a single string (e.g. 192.168.1.1:33544). Event 4625 also logs the remote IP and source port, but in different fields.
To address this, EventSentry includes a feature that can override existing insertion strings (or create new ones if none already exist) which comes in handy in this scenario. In the case of the event 131 we can use a RegEx pattern to simply remove the remote port from the string so that we only end up with the IP address – as the only insertion string.
The diagram below visualizes how the failed RDP login detection works with EventSentry. When an unsuccessful login via RDP occurs (1), whether or not NLA is enabled (2) determines which type of 4625 event will be logged by Windows. The RDP subsystem logs event 131 either way (3), but we utilize it when NLA is active. Without NLA we simply utilize event 4625 (4) as the trigger for one or more actions, whereas with NLA being active we need to evaluate two different events.
With NLA enabled, event id 131 is evaluated first (5). Since event 131 is logged regardless of whether the subsequent authentication is successful or not, it needs to be correlated with a potential subsequent 4625 security event (8). In order to correlate those two events based on the IP address however, the remote port needs to be removed from event 131 so that only the IP address remains (6). Once event 131 is registered and reformatted, EventSentry will look for subsequent 4625 events (8) with a matching IP address (7).
Note: Since blocking every failed RDP-based authentication could lock out legitimate users that enter an incorrect password by accident, it’s highly recommend to add a threshold for event 4625 (8). When downloaded from EventSentry, our 4625 filter has a default threshold of 3 in 1 minute per IP address. This means that hosts will be blocked if an incorrect password is specified 4 times within 1 minute (from the same IP address, that’s what insertion string 20 is for).
Correlating multiple events is the nature of a filter chaining package (9), which requires that all events listed in the package match during a specified time interval. Once all filters (131 + 4625 in this case) match, EventSentry will log event id 10650 to the application event log, specifying the name of the filter chaining package along with the time span and insertion string(s), the ip address in this case (10). That event is then used as the trigger for one or more actions (11), such as blocking the remote IP using the Windows firewall and/or for sending an email alert.
Blocking an IP address with Windows Firewall is easy and can be done with the netsh.exe command, for example:
$YEAR, $MONTH and $DAY are variables that are generally always available in EventSentry, and $STR3 is the third insertion string from whichever event triggered the action. In our example we trigger netsh from event id 10650, which specifies the IP address in its insertion string %3:
The filter chain for event log package %1 is complete.
Below is the actual event as it would be found in the EventSentry event viewer. You can view the insertions strings with the EventSentry management console under Tools -> Utilities -> Event Message Browser or with the EventSentry SysAdmin Tools.
After we put everything together in EventSentry we end up with the following:
1. A Filter Chaining Package (“RDProtector”) which logs the above event when it detects failed RDP logons 2. A filter that triggers the firewall blocking from event 10650 (“Block Failed RDP IP”) 3. An action (“Block IP with Windows Firewall”) that calls netsh.exe to block an IP address
Newer EventSentry installations include the RDProtector package out of the box, but the package can also be downloaded through the Tools -> Download Packages feature. Keep in mind that both the “Corrective Actions” package and the “Block IP with Windows Firewall” action need to be created manually, their respective configuration is shown below.
The process command line (“Arguments”) should be: advfirewall firewall add rule name=”$STR3 $YEAR-$MONTH-$DAY — automatic block by EventSentry” dir=in interface=any action=block remoteip=$STR3/32
Rules added to the Windows firewall are perpetual of course, which – depending on the number of blocks – may result in a large number of Windows firewall rules. A somewhat easy work-around would be to launch a script that:
1. Creates the firewall rule 2. Waits a certain amount of time (e.g. 5 min) 3. Deletes the firewall rule again
A script with a 3-minute timeout would look slimilar to this:
Since Active Directory is the foundation of all Windows networks, monitoring Active Directory needs to be part of any comprehensive security strategy. Up to version 3.5, EventSentry utilized Windows auditing and the security event log to provide reports on:
User Account Changes
Computer Account Changes
While this functionality provides a good basis for monitoring the most relevant changes to Active Directory, we felt that a more comprehensive approach to monitoring Active Directory was needed – without the need to install & maintain yet another product!
ADMonitor is new (optional) component included in EventSentry that vastly improves Active Directory monitoring with these additional features:
Monitors changes to all objects (e.g. OUs) – not just users/groups/computers
Captures every attribute change made to an object, not just high level changes
Provides before & after values for all changes
Monitors Group Policy changes
User status reports (show idle users, users with non-expiring passwords, …)
Monitoring does not require auditing
We’re excited that we can now offer EventSentry ADMonitor to our users who are looking for a more in-depth Active Directory monitoring solution.
Active Directory is essentially a representation of the employees and their roles in your organization. But employees come and go, roles/responsibilities change, contractors get temporary access and so forth. But while adding users and additional access is usually reflected properly in Active Directory (otherwise IT would get a call because somebody presumably can’t do their job), removing access is often forgotten. As a result, users that should have been removed from AD a long time ago continue to exist. With ADMonitor it’s easy to identify orphaned user accounts (and many others) and keep your Active Directory lean and clean.
Since a significant development effort stands behind ADMonitor, it will be offered as an optional component that is licensed on a per-user basis. Pricing is very competitive with other solutions and we also offer bundle discounts to customers who already own or will purchase agent licenses; please request a quote here.
But enough theory, let’s look into the installation, configuration and reporting of ADMonitor.
Since ADMonitor is a component of EventSentry, it’s easily activated as part of the main EventSentry setup. Just like with other components of EventSentry (Heartbeat Monitor, Collector, …), users have the option to enable ADMonitor during the post installation setup procedure.
ADMonitor can be installed on any host that is part of the domain that needs to be monitored, it does not need to be installed on a domain controller.
Immediately following the initial installation, ADMonitor will initialize itself by creating an offline copy of all Active Directory objects. This process can take from a few seconds to a few minutes, depending on the number of objects in AD, connection speed to the domain controller as well as the overall performance of the host running ADMonitor.
The initial configuration of ADMonitor is simple and only requires you to pick a password for the ADMonitor service account. If you’re adding ADMonitor to an existing installation you may also need to select the appropriate EventSentry database action to which ADMonitor reports changes. Otherwise, ADMonitor is ready from the get go and will monitor all Active Directory changes.
ADMonitor provides three types of reports:
Group Policy Changes
Shows any change made to an AD objects. Reports can be filtered on the type of action performed (added, removed, modified), on the object type (user, group, organizationalUnit, …) and on the user who performed the action.
Note that the detailed changes to group policies are available in the “Group Policy Changes” report below. Of course you can expect the same type of summary view you’re already used to from most other EventSentry features and create reports like:
Show all changes to organizational units
Show all new objects created
Show all users that were changed
Group Policy Changes
When a group policy is changed, it is first indicated on the “Object Changes” report, since the versionNumber attribute of the AD object changes. The actual group policy settings themselves are available on the “Group Policy Changes” report however, since group policy settings are not stored in AD.
The screen shot below shows that the Default Domain Policy was changed, with the Specify traps for the public community setting being enabled.
The users report helps you identify potentially problematic user accounts such as idle users, users who haven’t change their passwords in years and others.
This report contains a list of all user objects in Active Directory including the following details:
Name, Full Name, SAM Account Name, Path, UPN
Administrative Account (yes/no)
Password Never Expires (yes/no)
Password Expired (yes/no)
Password must change (yes/no)
Locked Out (yes/no)
Password Last Set
Account Expiration Date
With ADMonitor you can now get detailed user stats with just a few clicks and quickly identify user accounts that need to be reviewed, changed or deleted. Of course you can also schedule all reports directly from the web reports and get daily/weekly AD status reports directly in your inbox, e.g.:
List of all Group Policy changes
List of all idle user accounts
List of all newly created users and/or groups
You can also create your own reports for just about anything that involves a change to an Active Directory object, for example all organizational units created in the last 24 hours.
ADMonitor also includes a number of stand-alone utilities that support advanced features such as filtering and email notifications that I will cover in a future post.
With ADMonitor, EventSentry users can now gain the additional visibility needed to fully audit all Active Directory & Group Policy changes. As a result, EventSentry users can more easily enhance compliance, security and accountability in their network without the need to install additional software – saving both time and money.
EventSentry v3.5 continues to increase visibility into networks with additional vantage points, making it easier for EventSentry users to reduce their attack surface as well as discover anomalies.
Process & Network Activity Tracking
One major focus of this release is process network activity, an important component in any monitoring strategy. Do you know which applications listen for incoming connections on your monitored machines – or when a new process suddenly starts accepting incoming traffic? Do you know which processes perform outgoing network connections, and to where? How much data are they transferring?
To help you (and possibly your overzealous auditors) answer these questions, EventSentry v3.5 takes the existing process tracking functionality to the next level by integrating with Sysmon and showing processes with active or listening network connections. With EventSentry deployed you can now see:
Complete Process Details (start & stop times, duration, caller, PID)
Process command line
All processes listening for incoming connections
All active processes
Network activity initiated by a process (Figure 1, requires Sysmon)
Correlation with EventSentry NetFlow (Figure 2, requires NetFlow component)
This means that you can easily see which network connections a host establishes if you have Sysmon installed, and can even correlate that information with the EventSentry NetFlow component with just a click (see below). This information is invaluable for forensics and troubleshooting alike.
But even without Sysmon, EventSentry can now show you every open TCP port on a monitored machine (optionally all active connections as well), making it easy to discover rogue services on a network – even if they are blocked by the firewall. Figure 3 below shows all active processes which are listening for incoming connections, grouped by host.
A new member in the compliance tracking features family is registry tracking. Similar to file access tracking, it normalizes all registry audit events on a monitored machine, making it much easier to report on the registry activity and changes. Configuring registry tracking to work with existing registry auditing is incredibly easy and can be enabled in 60 seconds if the proper audit settings are already in place. The screen shots below show a list of recent registry activity as well as the details of changes:
Users managing a large number of hosts will appreciate the new “Tags” feature which addresses a shortcoming with the existing flat group structure. Tags allow groups or hosts to be tagged with keywords (e.g. production, staging, development). The created tags can then be used in the web reports (e.g. Show me disk space from all hosts tagged with “development”) and for dynamic package assignments.
The file checksum monitoring component received a few enhancements to help reduce noise while also adding new functionality.
FIM can now verify the digital signature of executable files and optionally suppress alerts if a file is digitally signed – think Windows updates. This can reduce the number of alerts you get significantly and thus make the remaining alerts more meaningful. The digital signature status can also be displayed in the web reports as a new column.
FIM can also calculate the entropy (essentially a measure of randomness) of files with a scale from 0 to 10, with 10 being the maximum of a completely random file. This is useful for Ransomware detection, since encrypted (and compressed files as well) files have a higher entropy than regular files. Combined with a threshold filter this can detect when a large number of encrypted files are suddenly being processed in a given directory and thus indicate a Ransomware infection.
We also replaced the existing SHA-256 checksum algorithm with a faster version in 3.5 which should result in a lower CPU utilization on systems which need to calculate a large number of checksums.
Disk Space Monitoring
A common annoyance with disk space monitoring are large volumes where an otherwise useful limit of, say 5%, is just not useful. For example, 5% of a 2 Tb drive is still 100 Gb, and in most cases there is probably no reason to sound an alarm. Dynamic thresholds (a new feature) addresses this issue by automatically adjusting the limit based on the drive size. The result: Fewer alerts!
EventSentry will log an event to the application event log when dynamic thresholds are enabled AND the current settings warrant a change. An event will look something like this:
The percentage-based threshold on drive F:\ has been dynamically adjusted from 5 percent to 0.5 percent based on the total drive size of 999 GB. A low disk space alert will be triggered when the available space on this volume falls below 19 GB.
The software inventory page (detailed tab) will now show which hosts do NOT have a particular software installed when the search is restricted to a specific software product. Also related to processes, process tracking can now generate the checksum of all execute files, which can then be searched for at Malware databases like virustotal.com. If you utilize the maintenance mode feature in EventSentry then you can now see whether a host is in maintenance mode or not in the web reports. And last but not least, event logs can now be sent to a remote Syslog receiver via TLS.
Under the Hood
We always tweak and improve EventSentry to ensure it runs as efficiently as possible. In this release we replaced the SHA 256 algorithm with a more efficient version, resulting in less CPU usage by the agent when calculating SHA 256 checksums. As we gradually move to a full 64-bit monitoring suite, the Heartbeat Agent is the next component now also available as a 64-bit process so that all EventSentry services are now available in 64-bit. We plan on porting all executables over to 64-bit within the next 6-12 months. At that point you will not be able to run EventSentry on 32-bit platforms anymore; monitoring 32-bit hosts will of course still be supported for the foreseeable future.
To help with the stability of all EventSentry and simply troubleshooting, all server-side components will now automatically generate crash dumps if they encounters a problem. Finally, the management console includes additional context and ribbon buttons.
In part one I provided a high level overview of PowerShell and the potential risk it poses to networks. Of course we can only mitigate some PowerShell attacks if we have a trace, so going forward I am assuming that you followed part 1 of this series and enabled
Script Block Logging
Security Process Tracking (4688/4689)
I am dividing this blog post into 3 distinct sections:
We start by attempting to prevent PowerShell attacks in the first place, decreasing the attack surface. Next we want to detect malicious PowerShell activity by monitoring a variety of events produced by PowerShell and Windows (with EventSentry). Finally, we will mitigate and stop attacks in their tracks. EventSentry’s architecture involving agents that monitor logs in real time makes the last part possible.
But before we dive in … the
PowerShell Downgrade Attack
In the previous blog post I explained that PowerShell v2 should be avoided as much as possible since it offers zero logging, and that PowerShell v5.x or higher should ideally be deployed since it provides much better logging. As such, you would probably assume that basic script activity would end up in of the PowerShell event logs if you enabled Module & ScriptBlock logging and have at least PS v4 installed. Well, about that.
So let’s say a particular Windows host looks like this:
PowerShell v5.1 installed
Module Logging enabled
ScriptBlock Logging enabled
Perfect? Possibly, but not necessarily. There is one version of PowerShell that, unfortunately for us, doesn’t log anything useful whatsoever: PowerShell v2. Also unfortunately for us, PowerShell v2 is installed on pretty much every Windows host out there, although only activated (usable) on those hosts where it either shipped with Windows or where the required .NET Framework is installed. Unfortunately for us #3, forcing PowerShell to use version 2 is as easy as adding -version 2 to the command line. So for example, the following line will download some payload and save it as calc.exe without leaving a trace in any of the PowerShell event logs:
does the exact same thing. So when doing pattern matching we need to use something like -v* 2 to ensure we can catch this parameter.
Microsoft seems to have recognized that PowerShell is being exploited for malicious purposes, resulting in some of the advanced logging options like ScriptBlockLogging being supported in newer versions of PowerShell / Windows. At the same time, Microsoft also pads itself on the back by stating that PowerShell is – by far –the most securable and security-transparent shell, scripting language, or programming language available. This isn’t necessarily untrue – any scripting language (Perl, Python, …) can be exploited by an attacker just the same and would leave no trace whatsoever. And most interpreters don’t have the type of logging available that PowerShell does. The difference with PowerShell is simply that it’s installed by default on every modern version of Windows. This is any attackers dream – they have a complete toolkit at their fingertips.
So which Operating Systems are at risk?
PowerShell Version 2 Risk
Active By Default
Windows 2008 R2
Windows 8 & later
Potentially Vulnerable – depends on .NET Framework v2.0
Windows 2012 & later
Potentially Vulnerable – depends on .NET Framework v2.0
Versions of Windows susceptible to Downgrade Attack
OK, so that’s the bad news. The good news is that unless PowerShell v2 was installed by default, it isn’t “activated” unless the .NET Framework 2.0 is installed. And on many systems that is not the case. The bad news is that .NET 2.0 probably will likely be installed on some systems, making this downgrade attack feasible. But another good news is that we can detect & terminate PowerShell v2 instances with EventSentry (especially when 4688 events are enabled) – because PowerShell v2 can’t always be uninstalled (see table above). And since we’re on a roll here – more bad news is that you can install the required .NET Framework with a single command:
Of course one would need administrative privileges to run this command, something that makes this somewhat more difficult. But attacks that bypass UAC exist, so it’s feasible that an attacker accomplishes this if the victim is a local administrator.
According to a detailed (and very informative) report by Symantec, PS v2 downgrade attacks haven’t been observed in the wild (of course that doesn’t necessarily mean that they don’t exist), which I attribute to the fact that most organizations aren’t auditing PowerShell sufficiently, making this extra step for an attacker unnecessary. I do believe that we will start seeing this more, especially with targeted attacks, as organizations become more aware and take steps to secure and audit PowerShell.
Well, I think you get the hint: PowerShell v2 is bad news, and you’ll want to do one or all of the following:
Uninstall PowerShell v2 whenever possible
Prevent PowerShell v2 from running (e.g. via AppLocker)
Detect and terminate any instances of PowerShell v2
If you so wish, then you can read more about the PowerShell downgrade attack and detailed information on how to configure AppLocker here.
Uninstall PowerShell v2
Even if the .NET Framework 2.0 isn’t installed, there is usually no reason to have PowerShell v2 installed. I say usually because some Microsoft products like Exchange Server 2010 do require it and force all scripts to run against version 2. PowerShell version 2 can manually be uninstalled (Windows 8 & higher, Windows Server 2012 & higher) from Control Panel’s Program & Features or with a single PowerShell command: (why of course – we’re using PowerShell to remove PowerShell!):
While running this script is slightly better than clicking around in Windows, it doesn’t help much when you want to remove PowerShell v2.0 from dozens or even hundreds of hosts. Since you can run PowerShell remotely as well (something in my gut already tells me this won’t always be used for honorable purposes) we can use Invoke-Command cmdlet to run this statement on a remote host:
Just replace WKS1 with the host name from which you want to remove PowerShell v2 and you’re good to go. You can even specify multiple host names separated by a comma if you want to run this command simultaneously against multiple hosts, for example
Well congratulations, at this point you’ve hopefully accomplished the following:
Enabled ModuleLogging and ScriptBlockLogging enterprise-wide
Identified all hosts running PowerShell v2 (you can use EventSentry’s inventory feature to see which PowerShell versions are running on which hosts in a few seconds)
Uninstalled PowerShell v2 from all hosts where supported and where it doesn’t break critical software
Terminate PowerShell v2
Surgical Termination using 4688 events
If you cannot uninstall PowerShell v2.0, don’t have access to AppLocker or want to find an easier way than AppLocker then you can also use EventSentry to terminate any powershell.exe process if we detect that PowerShell v2.0 was invoked with the -version 2 command line argument. We do this by creating a filter that looks for 4688 powershell.exe events that include the -version 2 argument and then link that filter to an action that terminates that PID.
If an attacker tries to launch his malicious PowerShell payload using the PS v2.0 engine, then EventSentry will almost immediately terminate that powershell.exe process. There will be a small lag between the time the 4688 event is logged and when EventSentry sees & analyzes the event, so it’s theoretically possible that part of a script will begin executing. In all of the tests I have performed however, even a simple “Write-Host Test” PowerShell command wasn’t able to execute properly because the powershell.exe process was terminated before it could run. This is likely because the PowerShell engine does need a few milliseconds to initialize (after the 4688 event is logged), enough time for EventSentry to terminate the process. As such, any malicious script that downloads content from the Internet will almost certainly terminated in time before it can do any harm.
The above approach won’t prevent all instances of PowerShell v2.0 from running however, for example when the PowerShell v2.0 prompt is invoked through a shortcut. In order to prevent those instances of PowerShell from running we’ll need to watch out for Windows PowerShell event id 400, which is logged anytime PowerShell is launched. This event tells us which version of PowerShell was just launch via the EngineVersion field, e.g. it will include EngineVersion=2.0 when PowerShell v2.0 is launched. We can look for this text and link it to a Service action (which can also be used to terminate processes).
Note: Since there is no way to correlate a Windows PowerShell event 400 with an active process (the 400 event doesn’t include a PID), we cannot just selectively kill version 2 powershell.exe processes. As such, when a PowerShell version instance is detected, all powershell.exe processes are terminated, version 5 instances. I personally don’t expect this to be a problem, since PowerShell processes usually only run for short periods of time, making it unlikely that a PowerShell v5 process is active while a PowerShell v2.0 process is (maliciously) being launched. But decide for yourself whether this is a practicable approach in your environment.
Command Line Parameters
Moving on to detection, where our objective is to detect potentially malicious uses of PowerShell. Due to the wide variety of abuse possibilities with PowerShell it’s somewhat difficult to detect every suspicious invocation of PowerShell, but there are a number of command line parameters that should almost always raise a red flag. In fact, I would recommend alerting or even terminating all powershell instances which include the following command line parameters:
Highly Suspicious PowerShell Parameters
Skip loading profile.ps1 and thus avoiding logging
Let a user run encoded PowerShell code
-ep bypass, -exp bypass, -exec bypass
Bypass any execution policy in place, may generate false positives
Prevents the creation of a window, may generate false positives
-v 2, -version 2.0
Forces PowerShell version 2
Any invocation of PowerShell that includes the above commands is highly suspicious
The advantage of analyzing command line parameters is that it doesn’t have to rely on PowerShell logging since we can evaluate the command line parameter of 4688 security events. EventSentry v126.96.36.199 and later can retrieve the command line of a process even when it’s not included in the 4688 event (if the process is active long enough). There is a risk of false positives with these parameters, especially the “windowStyle” option that is used by some Microsoft management scripts.
In addition to evaluating command line parameters we’ll also want to look out for modules that are predominantly used in attacks, such as .Download, .DownloadFile, Net.WebClient or DownloadString. This is a much longer list and will need to be updated on a regular basis as new toolkits and PowerShell functions are being made available.
Depening on the attack variant, module names can be monitored via security event 4688 or through PowerShell’s enhanced module logging (hence the importance of suppressing PowerShell v2.0!), like event 4103. Again, you will most likely get some false positives and have to setup a handful of exclusions.
Command / Code Obfuscation
But looking at the command line and module names still isn’t enough, since it’s possible to obfuscate PowerShell commands using the backtick character. For example, the command.
This means that just looking for DownloadString or Net.WebClient is not sufficient, and Daniel Bohannon devoted an entire presentation on PowerShell obfuscation that’s available here. Thankfully we can still detect tricks like this with regex patterns that look for a high number of single quotes and/or back tick characters. An example RegEx expression to detect 2 or more back ticks for EventSentry will look like this:
The above expression can be used in PowerShell Event ID 800 events, and will trigger every time a command which involves 2 or more back ticks is executed. To customize the trigger count, simply change the number 2 to something lower or higher. And of course you can look for characters other than the ` character as well, you can just substitute those in the above RegEx as well. Note that the character we look for appears three (3) times in the RegEx, so it will have to be substituted 3 times.
To make things easier for EventSentry users, EventSentry now offers a PowerShell event log package which you can download via the Packages -> Download feature. The package contains filters which will detect suspicious command line parameters (e.g. “-nop”), detect an excessive use of characters used for obfuscation (and likely not used in regular scripts) and also find the most common function names from public attack toolkits.
It’s still possible to avoid detection rules that focus on powershell.exe if the attacker manages to execute PowerShell code through a binary other than powershell.exe, because powershell.exe is essentially just the “default vehicle” that facilitates the execution of PowerShell code. The NPS (NotPowerShell) project is a good example and executes PS code through a binary named nps.exe (or whatever the attacker wants to call lit), but there are others. While the thought of running PowerShell code through any binary seems a bit concerning from a defenders perspective, it’s important to point out that downloading another binary negates the advantage of PowerShell being installed by default. I would only expect to see this technique in sophisticated, targeted attacks that possibly start the attack utilizing the built-in PowerShell, but then download a stealth app for all subsequent activity.
This attack can still be detected if we can determine that one of the following key DLLs from the Windows management framework are being loaded by a process other than powershell.exe:
You can detect this with Sysmon, something I will cover in a follow-up article.
Now, having traces of all PowerShell activity when doing forensic investigations is all well and good, and detecting malicious PowerShell activity after it happened is a step in the right direction. But if we can ascertain which commands are malicious, then why not stop & prevent the attack before it spreads and causes damage?
In addition to the obvious action of sending all logs to a central location, there are few things we can do in response to potentially harmful activity:
1. Send out an alert
2. Mark the event to require acknowledgment
3. Attempt to kill the process outright (the nuclear option)
4. A combination of the above
If the only source of the alert is from one of the PowerShell event logs then killing the exact offending PowerShell process is not possible, and all running powershell.exe processes have to be terminated. If we can identify the malicious command from a 4688 event however, then we can perform a surgical strike and terminate only the offending powershell.exe process – other potentially (presumably benign) powershell.exe processes will remain unharmed and can continue to do whatever they were supposed to do.
If you’re unsure as to how many PowerShell scripts are running on your network (and not knowing this is not embarrassing – many Microsoft products run PowerShell scripts on a regular basis in the background) then I recommend just sending email alerts initially (say for a week) and observe the generated alerts. If you don’t get any alerts or no legitimate PowerShell processes are identified then it should be safe to link the filters to a “Terminate PowerShell” action as shown in the screenshots above.
After downloading and deploying the PowerShell package I recommend executing a couple of offending PowerShell commands to ensure that EventSentry will detect them and either send out an alert or terminate the process (or both – depending on your level of conviction). The following commands should be alerted on and/or blocked:
Any detection rules you setup, whether with EventSentry or another product, will almost certainly result in false alerts – the amount of which will depend on your environment. Don’t let this dissuade you – simply identify the hosts which are “incompatible” with the detection rules and exclude either specific commands or exclude hosts from these specific rules. It’s better to monitor 98 out of 100 hosts than not monitor any host at all.
With EventSentry you have some flexibility when it comes to excluding rules from one or more hosts:
PowerShell is a popular attack vector on Windows-based systems since it’s installed by default on all recent versions of Windows. Windows admins need to be aware of this threat and take the appropriate steps to detect and mitigate potential attacks:
Disable or remove legacy versions of PowerShell (=PowerShell v2)
Enable auditing for both PowerShell and Process Creation
Collect logs as well as detect (and ideally prevent) suspicious activity
EventSentry users have an excellent vantage point since its agent-based architecture can not only detect malicious activity in real time, but also prevent it. The PowerShell Security event log package, which can be downloaded from the management console, offers a list of rules that can detect many PowerShell-based attacks.
Imagine someone getting the seemingly innocent ability to run a couple of commands on a machine on your network WITHOUT installing any new software, but those commands resulting in a reverse shell running on that same machine – giving the intruder a convenient outpost in your network. Now stretch your imagination even further and pretend that all of this happens without leaving any unusual traces in logs – leaving you completely in the dark. It’s like somebody living in your house or apartment yet you have no idea they’re there. Are you getting goose bumps yet?
Not too long ago I talked with Michael, the creator of the popular cheat sheets which cover PowerShell, the Windows Registry, Windows Logging, and more. Michael ran a few scenarios by me that involved exploiting PowerShell and was curious how EventSentry could help detect those. This really sparked my interest in the topic, and after coming up with a few RegEx expressions that could be used in an EventSentry filter I decided to look more into this subject. I really have to take the opportunity to thank Michael here, whose cheat sheets and input helped me come up with this article and the new PowerShell Security event log package in EventSentry.
If you’re not an InfoSec professional then you may not be fully aware that PowerShell – you know, the language you’re supposed to be fluent in by now – is quite commonly used in attacks. In fact, InfoSec already reported back in 2016 that 38% of all attacks utilize PowerShell in one way or another. And let’s be honest – why wouldn’t you utilize a tool that is pretty much guaranteed to be installed while giving you full access to the .NET Framework and all Windows APIs? So if you haven’t already done so, then securing PowerShell in your environment is something you should think about sooner rather than later. This and the follow-up articles will assist you with this effort.
So what’s so potentially bad about PowerShell in particular? Now, Windows has always shipped with VBScript, a scripting language that’s easy to use for both simple and potentially more complex tasks. In fact, most of the things people do in PowerShell can be done with VBScript just the same. A key benefit of PowerShell, however, is the ability to utilize the .NET framework, something VBScript can’t since it can only interact with COM objects. And since PowerShell is, well, a shell, you get to pipe input/output and create powerful one-liners. On top of that, PowerShell contains some nifty features like encoding scripts, making it possible to run fairly complex code without ever having to use an actual .ps1 script file on disk. It’s VBScript on Steroids.
Here are some concrete examples as to what evildoers can do with PowerShell:
Remember when I talked about “without leaving a trace” above? That’s because Microsoft didn’t introduce the ability to log detailed PowerShell activity until version 5, although PowerShell 3 & 4 generate reasonably useful audit logs as well. In order to protect ourselves against PowerShell attacks, we need to first detect it, which we can only do if PowerShell leaves traces. PowerShell’s ability to produce useful audit logs greatly depends on the version, however, which the table below illustrates:
Which version of Windows ships with which version of PowerShell
What is the highest supported version of PowerShell for each version of Windows
Shows the versions of PowerShell that ship with Windows as well as the highest supported version of PowerShell
As you can see from the table above, thankfully most versions of Windows are compatible with PS v5, so unless you’re unfortunate enough to be running Server 2008 (or Vista), you should be able to deploy PowerShell 5.1 to most of your systems. I say most, because some Microsoft applications (e.g. Exchange Server 2010) aren’t compatible with PowerShell v5, so you’ll want to make sure you do some research on those machines that actively use PowerShell to prevent disruption.
Coexistence & Legacy
An important thing to note here is that PowerShell v1/v2 can peacefully coexist with PowerShell v3-v5, while versions 3 and later are always upgraded to the latest version. This means that you could have v2 and v4 installed (and many systems do), but not v3 and v5. What’s also interesting is that PS v2 is installed with every major version of Windows (including Server 2016!) although not usable until the .NET Framework v2.0.50727 is installed.
Starting with EventSentry v188.8.131.52 you can thankfully use EventSentry’s software inventory to determine which versions of PowerShell are installed on your network. If you haven’t manually deployed PS v5 yet and aren’t running Windows Server 2016 widely yet, then you will probably see PowerShell v2 and v4 installed on most hosts on your network. EventSentry’s grouping mechanism comes in real handy here.
Please note that even though PowerShell v2 may be installed on a machine it doesn’t necessarily mean that PowerShell v2 is actually usable. PowerShell relies on the .NET Framework being installed, and PowerShell v2 specifically relies on the .NET Framework 2.0.50727 (which is part of the 3.5 .NET Framework) – something that is usually not installed by default. I will explain later why this is a good thing.
OK, but enough about boring PowerShell versions. If you just remember one thing from the above tables and paragraphs it’s this:
Thankfully you don’t need version 5.x to get useful logging – even PowerShell v3 & v4 can log relevant details in the (Windows PowerShell) event log, e.g. the PowerShell command line or commands executed within the PowerShell shell. In fact, even the (decoded) commands are logged to the event log when obfuscated with the -encoded switch.
Logging can be enabled either through group policy or via registry settings. There are three general areas for logging available:
Script Block Logging
Module Logging Since everything that is executed in PowerShell is essentially located in a module, module logging will at least generate a high-level audit trail of PowerShell activity and potentially malicious activity. At minimum this will show which commands were executed through PowerShell. This logging level should always be enabled and is useful starting with PS version 3.
Important: Module Logging only works if you specify at least one module to be monitored. Since it’s difficult and cumbersome to predict and edit a list of all modules that could potentially cause harm, I recommend just specifying the * wildcard characters as the module list – see screenshots below.
Script Block Logging Script Block Logging is more verbose than module logging and provides additional context and output, especially when functions are called and function output itself is invoked as a command. The amount of noise heavily depends on the type of PowerShell activity, but I’d recommend turning this option on as well. If it ends up producing too much noise / volume it can always be disabled or customized later.
Transcription This provides a full log of all input and output and requires additional considerations in regards to where the transcription files are placed. I’d only recommend this for high-secure environments, you can learn more about it here. Transcript files are stored in the file system, so it’s a little more work than just adding up a couple of registry values. If you enable this feature then you’ll need to make sure that the actual transcript files (which likely contain sensitive data) are protected from unauthorized access.
It’s definitely recommended to configure these options via Group Policy to ensure that all machines in the domain receive the settings. If changing group policy is not an option in the short term then you can at least set the registry options until you have an opportunity to set it via group policy. You can use a tool like the EventSentry Admin Assistant to push registry settings out to multiple hosts with just a few clicks.
Group Policy: Configuring this is unfortunately less straightforward than you’d think or expect, depending on the OS version of your domain controller. You can expect the “Module Logging” option to be available in the group policy editor on 2008 R2 and later, however “Script Block Logging” is only available on server 2016 or after manually updating ADMX files. See this thread on how to update your ADMX files. In my environment I just had to replace the PowerShellExecutionPolicy.admx and en-US\PowerShellExecutionPolicy.adml files in the %SYSTEMROOT%\SYSVOL\sysvol\[DOMAINNAME]\Policies\PolicyDefinitions directory with the newer versions after installing the latest version from here.
Registry: Only the HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\PowerShell key exists by default, the other two sub keys “ModuleLogging” and “ScriptBlockLogging” have to be created before you can add the “EnableModuleLogging” and “EnableScriptblockLogging” DWORD values inside those sub keys.
For Module Logging, as shown in the screenshot below, you’ll also need to create the “ModuleNames” sub key along with a list of modules that will be monitored. I recommend just using the asterisk character which monitors any module.
Configuring PowerShell Event Logging
Key: HKLM\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ModuleLogging Name: EnableModuleLogging Data: 1 (DWORD)Key: HKLM\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ModuleLogging\ModuleNames Name: [ModulePattern] Data: [ModulePattern] (REG_SZ)See the screenshot above for example on module logging.
Policies\Administrative Templates\Windows Components\Windows PowerShell\Turn on Module Logging
You don’t need to restart after setting the registry values, they will become effective immediately. The same applies to group policy – as soon as the target host has applied the group policy settings, PowerShell will enforce the new logging options.
PowerShell logs a lot of different events to two different event logs, and the table below shows the events I have observed on test systems. Even though the table may not be 100% complete, it does list all the events that are relevant for threat detection. If an event is not listed below then it is likely not relevant for forensics. We will update the list if necessary.
What’s interesting to note is that newer versions of PowerShell will often log to both event logs simultaneously.
Security Event Log Auditing
PowerShell logging is great, but given the discrepancies between the different versions and the possibility to evade it (more on that later), I prefer to have as many methods as possible at my disposal that tell me what PowerShell is doing.
Since PowerShell code is usually invoked via powershell.exe (I’m point this out because you technically don’t have to use powershell.exe, and attackers are coming up with creative ways to launch it through other ways – more in part 2 of this series), and because we’re after that processes’ command line, it’s important to monitor Process Start (event id 4688) events from the security event log in addition to events logged by PowerShell itself. This means you’ll need audit the following sub categories from the Detailed Tracking category:
If you are not using EventSentry then I recommend collecting both 4688 and 4689 events so that you can not only determine whether a powershell.exe process was started, but also how long it remained active. If you are an EventSentry user then you just need to verify that Process Tracking (an object for Compliance Tracking) is enabled and configured to capture the command line of a process. EventSentry can automatically parse and correlate 4688 and 4689 events and provide a history of all processes on a monitored system.
EventSentry users can also utilize the Audit Policy Status page to verify that process creations are indeed being audited. You’ll also want to make sure that “Include command line in process creation events” is activated, so that Windows logs the command line of every process as part of 4688 events. After all it doesn’t help us that much just knowing that powershell.exe has been running, we need to know what exactly it has been running.
This can either be enabled via group policy (Administrative Templates\System\Audit Process Creation\Include command line in process creation events) or via the registry (set HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\Audit\ProcessCreationIncludeCmdLine_Enabled to 1).
Disclaimer: This option is available starting with Windows 7 / Server 2008 R2, earlier versions of Windows don’t support it. Things are a little easier for EventSentry users, which attempts to obtain the command line of a process if it’s not included in the 4688 event and subsequently makes it available as variable $STR9. But more on that in part 2 when we discuss ways to detect and mitigate attacks.
I hope I was able to convince you of the risks that PowerShell poses, what versions of PowerShell are out there, and what type of logging needs to be enabled in order to detect and stop malicious PowerShell in its tracks. In part 2 I’ll talk about how to actually mitigate PowerShell-based attacks – with specific instructions for EventSentry.
In Mr. Robot‘s episode 9 of season 2 (13:53), Angela Moss needs to obtain the Windows domain password of her superior, Joseph Green, in order to download sensitive documents that would potentially incriminate EvilCorp. Since her attack requires physical access to his computer, she starts with a good old-fashioned social engineering attack to get the only currently present employee in the office to leave.
Once in his office, she uses a USB Rubber Ducky, a fast and automated keyboard emulator, to obtain Joseph’s clear text password using mimikatz. Please note that there are some holes in this scene which I will get into later. For now we’ll assume that she was able to obtain his credentials by having physical access to his computer.
After she gets back to her workstation, she analyzes the capture which reveals Joseph’s password: holidayarmadillo. Not the best password, but for this particular attack the quality of the password wouldn’t have mattered anyways. Mimikatz was (is) able to get the password from memory without utilizing brute force or dictionary techniques. Once she has the password, she logs off and logs back on with Joseph’s user and downloads the documents she needs.
As somebody who helps our users improve the security of their networks, I of course immediately contemplated how this attack could have been detected with EventSentry. Since most users only log on to one workstation on any given day with their user account, Angela logging in with Joseph’s account (resulting in “joseph.green” logging on to two different workstations) would actually be an easy thing to detect.
Introduced with v3.4, collector-side thresholds allow the real-time detection of pretty much any user activity that originates from an event, for example user logons or process launches. You can tell EventSentry that (physical) logons for any user on more than one host (within a given time period – say 9 hours) should trigger an alert (aka as “lateral movement”). Had this been in place at EvilCorp, Angela logging in as Joseph would have immediately triggered an alert. With the right procedures in place, countermeasures could have been taken. Of course most viewers wouldn’t want Angela to be caught, so please consider my analysis strictly technical. Watch a short video on lateral detection with EventSentry here.
So what’s the hole? Well, the rubber ducky (mimikatz, really) requires access to an active logon session, which Angela most likely didn’t have. It looked like Joseph had been out of the office for a while, so his computer was likely either locked or turned off, rendering any attack based on mimikatz useless. Mimikatz – since it obtains passwords from memory – only works if the computer is unlocked. And had the computer been unlocked then she could have just downloaded the files from his computer – although this would have been even more risky with people walking around the office.
Cyber attacks are becoming more potent every year and are often sponsored by powerful criminal gangs and/or governments. It’s important that companies employ multiple layers of defense to protect themselves (and their customers) from these increasingly sophisticated and destructive attacks.
EventSentry is the only monitoring solution that utilizes robust agent-based technology that goes beyond logs, enabling the fusion of real-time log monitoring with in-depth system monitoring to not only detect but also react to attacks and anomalies. See for yourself and download a free evaluation of EventSentry.
Almost every company which runs Microsoft Exchange Server needs to make port 443 available to the Internet in order to provide their users access to email via their mobile devices or OWA.
Since both OWA & ActiveSync utilize Active Directory for authentication, exposing OWA/ActiveSync to the Internet indirectly exposes Active Directory as well. While user lockout policies provide some protection against brute force attacks, additional protection methods should be employed. Furthermore, password spraying attacks may be use to circumvent lockout policies – something that would be more likely to succeed in larger organizations.
With the proper auditing enabled (Logon/Logoff – Logon (Failure)) and EventSentry installed however, we can permanently block remote users / hosts who attempt to log on too many times with a wrong password. Setting this up is surprisingly simple:
Windows: Enable (or verify) Auditing
EventSentry: Setup action which creates firewall block rule
EventSentry: Setup filter looking for 4625 Audit Failure events
Bonus: This procedure works with the free version of EventSentry (EventSentry Light) and can be applied to any IIS-based web site which uses authentication.
In the group policy settings that affect the server running OWA, make sure that auditing for Failure events in the Audit Logon sub category of the Logon/Logoff category is enabled (of course you can audit success events as well). If you are running the full version of EventSentry v3.4 or later then you can verify all effective audit settings on the Audit Policy Status page for example.
Creating an Action
Since event 4625 contains the IP address of the remote host, the easiest way to subsequently block it is to run the netsh command. In the management console, create a new action by clicking on the “Action” header in the ribbon and selecting the process action as its type. See the screenshot below:
The following command line will work in EventSentry v3.4 and later:
The difference here is that v3.4 and later can refer to insertion string variables by name, making the action more universal and potentially applicable to any event that uses the same field name.
When this action is triggered, it will extract the IP address from the event and block it from the system entirely.
Creating a Filter
Create an event log filter which matches Audit Failure events from the Security event log with event id 4625, where insertion string 19 matches the w3wp.exe process (C:\Windows\System32\inetsrv\w3wp.exe). This ensures that only users accessing the host via the web will be subject to blocking. The screenshot below shows the configuration:
This filter can either be added to an existing package or added to a new package that is assigned only to the Exchange server. If the filter is added to an existing package that applies to servers other than the Exchange server, then the computer field of the filter can be used to ensure the filter is evaluated only on the desired host. Select the action created in the previous step.
Since users may occasionally enter an incorrect password I recommend setting up a threshold so that remote IPs are only blocked after 3 or more failed logon attempts. Threshold are configured by clicking on the “Threshold” tab (see the blue “i” above) and an example configuration is shown below. Feel free to adjust the threshold to match your users ability to enter their password correctly :-). Insertion string 20 – which represents the IP address of this event – was selected in the threshold matching section to ensure that each IP address has its own, unique threshold. Note: The event logging settings shown are optional.
Save/deploy or push the configuration to the mail server.
Triggering a system process from external input is something we should always do with caution. For example, if Windows has an upper limit to the maximum number of rules that can be added, then an attacker could launch a DoS attack IF they had the ability to launch attacks from different IP addresses. Launching a DoS attack from the same IP won’t be possible once they are blocked. You can mitigate this risk by applying a threshold to the EventSentry action calling netsh.exe, for example by limiting it to 100 / hour. This would still provide sufficient protection while also ensuring that only 100 rules could be added per hour (thresholds can be set by clicking on the “Options” button on an action). A regular audit of the netsh execution (e.g. via Process Tracking) would quickly show any sort of abuse.
Over time the number of firewall rules added to the mail server could become rather large, which is why the rules are created with a date appended. This makes managing these rules easier, and the name can also be adapted in the action by changing the “rule name” parameter. The screenshot below shows the inbound firewall rules after two IPs have been blocked:
If manual cleanup of firewall rules is not desirable or an option, then the netsh command can also be wrapped into a script which would erase the firewall rule again after a timeout (e.g. 15 minutes). The script could look like this:
In this case you would call the wrapper script instead of the netsh.exe process directly (General Options – Filename) and use the string below as the arguments:
To keep things simple you can just make the script an embedded script (Tools menu) and reference the script. The timeout value (120 in the above example) is the duration seconds the remote IP will be be blocked. If you want to block the IP for an hour then you would set the timeout value to 3600 instead. When going this route I strongly suggest clearing both event log check boxes in the Options dialog of the action.
Before I continue I need to point out that DNS auditing has become significantly easier starting with Windows 2012 R2. Not only is it enabled by default, but the generated audit data is also much more granular and easier to interpret. The logged events even distinguish between regular and dynamic updates, making it easy to filter out noise. So if you’re serious about DNS auditing and have the option to update then I recommend you do so.
If you’re running Windows 2008 (R2) or 2012 then setting up DNS auditing requires a few steps. Thankfully it’s a one-time process and shouldn’t take more than a few minutes. On the EventSentry side a pre-built package with all the necessary rules is available for download and included with the latest installer.
Please follow the steps outlined below exactly as described, auditing won’t work or will be incomplete if these steps aren’t followed exactly as described below.
Enabling Directory Service Auditing
Enabling Sub Category Auditing
We first need to make sure that the new subcategory-based audit settings are enabled in group policy. If you’ve already done that, then you can skip this step and jump to “ADSIEdit”.
Since most of the steps here involve domain controllers, I recommend that you make the changes in a group policy, e.g. in the “Default Domain Controller” policy. In Group Policy Management, find an existing group policy, or create a new one, and set Computer Configuration\Polices\Windows Settings\Security Settings\Local Polices\Security Options\Force audit policy subcategory settings (Windows Vista or later) to override audit policy category settings to Enabled.
Then, navigate to Computer Configuration\Polices\Windows Settings\Advanced Audit Policy Configuration\Audit Policies\DS Access and set Audit Directory Service Changes to Success.
Open ADSI Edit via Start -> Run -> “adsiedit.msc”. If your default naming context does not automatically appear OR if the listed naming context does not include dc=domaindnszones, then select “Action -> Connect To” and connect to the appropriate naming context, e.g.
Replace the dc components after “domaindnszones” with the actual DNS name of your domain. It’s important that “dc=domaindnszones” is part of the naming context.
Once connected, expand the naming context and locate the “CN=MicrosoftDNS” container, right-click it, and select Properties. Then select Security, Advanced, Auditing and click on Add. In the resulting dialog we’ll audit the built-in “Everyone” user so that DNS changes from everyone are audited:
Name (Principal): Everyone
Apply onto: This object and all descendent objects
Access: Create dnsZone objects
It may seem tempting to also check the “Delete dnsZone objects”, but resist the temptation. Don’t be fooled by the term “dnsZone”, the ACE entry we just added will audit the creation of all AD DNS objects (and not just DNS zones) and log event 5136 to the security event log. In order to audit deletions as well, click “Add” again but this time configure the dialog as shown below:
Name (Principal): Everyone
Apply onto: Descendent dnsZone objects
It’s important that the “Apply onto” is changed to Descendent dnsZone objects. This ACE entry will result in event 5141 being logged when a DNS-related directory service object is deleted. This is were things get a bit interesting though, since DNS records deleted from the DNS manager aren’t actually deleted. Instead, they are tombstoned (which is done internally by adding the dNSTombstoned attribute to the object). Only when the tombstoned object expires is it actually deleted. You can use ADSIEdit if you want to send a DNS object immediately to heaven and skip the graveyard stage.
Unlike Windows 2012 R2 and later, earlier versions of Windows are a little more verbose than you probably like when it comes to directory service auditing. For example, creating a new DNS A record in a zone will result in 4 different events with id 5136 being logged – and not just one. The events logged when adding or deleting a zone or A record are shown in the diagram below:
All events are logged under the “Directory Service Changes” category.
Before we start configuring EventSentry, we’ll want to make sure that auditing was setup correctly. On a domain controller, open the “DNS” application and either temporarily add a new A record or primary zone. You should either see a 5136 or 5137 events with the category “Directory Service Changes” logged to the security event log.
If you don’t see the events then walk through the above steps again, or reference this Microsoft article.
There are generally two things one will want to do with these audit events – store them in a database, email them or both. If you’re already consolidating Audit Success events in an EventSentry database then you shouldn’t have to do anything, all directory service change events will be written to the database automagically.
If you only want to store directory service change events in the database (opposed to storing all audit success events), then you can simply create an include filter with the following properties:
Setting up alerts using filters for when zones or records are added or removed is a little more involved than one would hope, thanks to Windows logging more than one event whenever such a change is done – as is depicted in the image above. For example, creating a new DNS record will result in 4 x 5136 events being logged, deleting will result in 3 events.
Lucky for you, we’ve analyzed the events and created a DNS Server Auditing package in EventSentry which will email you a single alert (=event) whenever a record or zone are added or removed. This package is included with all new installation of EventSentry, existing users running 3.4 or later can get it through the package update feature in the management console.
And that’s really all there is to it … if you have EventSentry installed then setting up auditing in Windows is really the only obstacle to auditing your DNS records!