Windows supports a code-signing feature called Authenticode, which allows a software publisher to digitally sign executable files (e.g. .exe, .msi, …) so that users can verify their autenticity. The digital signature of a file can be viewed in the file properties in Windows explorer on the “Digital Signature” tab.
Digital signature verification has been added to the checksum utility, which already calculates the checksum and entropy of a file. When using the new /s switch, checksum.exe will tell you whether:
the file is digitally signed
a counter signature exists
the digital signature is valid
the algorithm used (e.g. SHA 256)
who signed the file
who issued the certificate
when the file was signed
The utility also sets the ERRORLEVEL variable accordingly; if a signature check is requested with the /s switch but the file is unsigned, then checksum.exe will return %ERRORLEVEL% 2. Below is a sample output of the utility in action:
Digital signature verification will be added to EventSentry’s FIM monitoring component (“File Checksum Monitoring”) in the upcoming v3.4.3 release, which will give you the option to only get notified when unsigned files are changed, thus reducing overall noise.
You can download the latest version from here – enjoy!
Imagine someone getting the seemingly innocent ability to run a couple of commands on a machine on your network WITHOUT installing any new software, but those commands resulting in a reverse shell running on that same machine – giving the intruder a convenient outpost in your network. Now stretch your imagination even further and pretend that all of this happens without leaving any unusual traces in logs – leaving you completely in the dark. It’s like somebody living in your house or apartment yet you have no idea they’re there. Are you getting goose bumps yet?
Not too long ago I talked with Michael, the creator of the popular cheat sheets which cover PowerShell, the Windows Registry, Windows Logging and more. Michael ran a few scenarios by me that involved exploiting PowerShell and was curious how EventSentry could help detect those. This really sparked my interest in the topic, and after coming up with a few RegEx expressions that could be used in an EventSentry filter I decided to look more into this subject. I really have to take the opportunity to thank Michael here, whose cheat sheets and input helped me come up with this article and the new PowerShell Security event log package in EventSentry.
If you’re not an InfoSec professional then you may not be fully aware that PowerShell – you know, the language you’re supposed to be fluent in by now – is quite commonly used in attacks. In fact, InfoSec already reported back in 2016 that 38% of all attacks utilize PowerShell in one way or another. And let’s be honest – why wouldn’t you utilize a tool that is pretty much guaranteed to be installed while giving you full access to the .NET Framework and all Windows APIs? So if you haven’t already done so, then securing PowerShell in your environment is something you should think about sooner rather than later. This and the follow-up articles will assist you with this effort.
So what’s so potentially bad about PowerShell in particular? Now, Windows has always shipped with VBScript, a scripting language that’s easy to use for both simple and potentially more complex tasks. In fact, most of the things people do in PowerShell can be done with VBScript just the same. A key benefit of PowerShell however is the ability to utilize the .NET framework, something VBScript can’t since it can only interact with COM objects. And since PowerShell is, well, a shell, you get to pipe input/output and create powerful one-liners. On top of that, PowerShell contains some nifty features like encoding scripts, making it possible to run fairly complex code without ever having to use an actual .ps1 script file on disk. It’s VBScript on Steroids.
Here are some concrete examples as to what evildoers can do with PowerShell:
Remember when I talked about “without leaving a trace” above? That’s because Microsoft didn’t introduce the ability to log detailed PowerShell activity until version 5, although PowerShell 3 & 4 generate reasonably useful audit logs as well. In order to protect ourselves against PowerShell attacks we need to first detect it, which we can only do if PowerShell leaves traces. PowerShell’s ability to produce useful audit logs greatly depends on the version however, which the table below illustrates:
Which version of Windows ships with which version of PowerShell
What is the highest supported version of PowerShell for each version of Windows
Microsoft .NET Framework 4.5.2
(already installed on 2012 and later)
All versions of Windows since Server 2008 and the version of PowerShell that it included by default
Default PowerShell vs Highest Supported PowerShell Versions
Included With Windows
Windows Vista (SP2)
Windows Server 2008 (SP2)
Windows 7 (SP1)
Windows 2008 R2 (SP1)
Windows 2012 R2
Shows the versions of PowerShell that ship with Windows as well as the highest supported version of PowerShell
As you can see from the table above, thankfully most versions of Windows are compatible with PS v5, so unless you’re unfortunate enough to be running Server 2008 (or Vista), you should be able to deploy PowerShell 5.1 to most of your systems. I say most, because some Microsoft applications (e.g. Exchange Server 2010) aren’t compatible with PowerShell v5, so you’ll want to make sure you do some research on those machines that actively use PowerShell to prevent disruption.
Coexistence & Legacy
An important thing to note here is that PowerShell v1/v2 can peacefully coexist with PowerShell v3-v5, while versions 3 and later are always upgraded to the latest version. This means that you could have v2 and v4 installed (and many systems do), but not v3 and v5. What’s also interesting is that PS v2 is installed with every major version of Windows (including Server 2016!) although not usable until the .NET Framework v2.0.50727 is installed.
Starting with EventSentry v220.127.116.11 you can thankfully use EventSentry’s software inventory to determine which versions of PowerShell are installed on your network. If you haven’t manually deployed PS v5 yet and aren’t running Windows Server 2016 widely yet, then you will probably see PowerShell v2 and v4 installed on most hosts on your network. EventSentry’s grouping mechanism comes in real handy here.
Please note that even though PowerShell v2 may be installed on a machine it doesn’t necessarily mean that PowerShell v2 is actually usable. PowerShell relies on the .NET Framework being installed, and PowerShell v2 specifically relies on the .NET Framework 2.0.50727 (which is part of the 3.5 .NET Framework) – something that is usually not installed by default. I will explain later why this is a good thing.
OK, but enough about boring PowerShell versions. If you just remember one thing from the above tables and paragraphs it’s this:
Thankfully you don’t need version 5.x to get useful logging – even PowerShell v3 & v4 can log relevant details in the (Windows PowerShell) event log, e.g. the PowerShell command line or commands executed within the PowerShell shell. In fact, even the (decoded) commands are logged to the event log when obfuscated with the -encoded switch.
Logging can be enabled either through group policy or via registry settings. There are three general areas for logging available:
Script Block Logging
Since everything that is executed in PowerShell is essentially located in a module, module logging will at least generate a high-level audit trail of PowerShell activity and potentially malicious activity. At minimum this will show which commands were executed through PowerShell. This logging level should always be enabled and is useful starting with PS version 3.
Important: Module Logging only works if you specify at least one module to be monitored. Since it’s difficult and cumbersome to predict and edit a list of all modules that could potentially cause harm, I recommend just specifying the * wildcard characters as the module list – see screenshots below.
Script Block Logging Script Block Logging is more verbose than module logging and provides additional context and output, especially when functions are called and function output itself is invoked as a command. The amount of noise heavily depends on the type of PowerShell activity, but I’d recommend turning this option on as well. If it ends up producing too much noise / volume it can always be disabled or customized later.
This provides a full log of all input and output and requires additional considerations in regards to where the transcription files are placed. I’d only recommend this for high-secure environments, you can learn more about it here. Transcript files are stored in the file system, so it’s a little more work than just adding up a couple of registry values. If you enable this feature then you’ll need to make sure that the actual transcript files (which likely contain sensitive data) are protected from unauthorized access.
It’s definitely recommended to configure these options via Group Policy to ensure that all machines in the domain receive the settings. If changing group policy is not an option in the short term then you can at least set the registry options until you have an opportunity to set it via group policy. You can use a tool like the EventSentry Admin Assistant to push registry settings out to multiple hosts with just a few clicks.
Group Policy: Configuring this is unfortunately less straightforward than you’d think or expect, depending on the OS version of your domain controller. You can expect the “Module Logging” option to be available in the group policy editor on 2008 R2 and later, however “Script Block Logging” is only available on server 2016 or after manually updating ADMX files. See this thread on how to update your ADMX files. In my environment I just had to replace the PowerShellExecutionPolicy.admx and en-US\PowerShellExecutionPolicy.adml files in the %SYSTEMROOT%\SYSVOL\sysvol\[DOMAINNAME]\Policies\PolicyDefinitions directory with the newer versions after installing the latest version from here.
Registry: Only the HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\PowerShell key exists by default, the other two sub keys “ModuleLogging” and “ScriptBlockLogging” have to be created before you can add the “EnableModuleLogging” and “EnableScriptblockLogging” DWORD values inside those sub keys.
For Module Logging, as shown in the screenshot below, you’ll also need to create the “ModuleNames” sub key along with a list of modules that will be monitored. I recommend just using the asterisk character which monitors any module.
Configuring PowerShell Event Logging
Data: 1 (DWORD)Key: HKLM\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ModuleLogging\ModuleNames
Data: [ModulePattern] (REG_SZ)See screenshot above for example on module logging.
Policies\Administrative Templates\Windows Components\Windows PowerShell\Turn on Module Logging
You don’t need to restart after setting the registry values, they will become effective immediately. The same applies to group policy – as soon as the target host has applied the group policy settings, PowerShell will enforce the new logging options.
PowerShell logs a lot of different events to two different event logs, and the table below shows the events I have observed on test systems. Even though the table may not be 100% complete, it does list all the events that are relevant for threat detection. If an event is not listed below then it is likely not relevant for forensics. We will update the list if necessary.
What’s interesting to note is that newer versions of PowerShell will often log to both event logs simultaneously.
Security Event Log Auditing
PowerShell logging is great, but given the discrepancies between the different versions and the possibility to evade it (more on that later), I prefer to have as many methods as possible at my disposal that tell me what PowerShell is doing.
Since PowerShell code is usually invoked via powershell.exe (I’m point this out because you technically don’t have to use powershell.exe, and attackers are coming up with creative ways to launch it through other ways – more in part 2 of this series), and because we’re after that processes’ command line, it’s important to monitor Process Start (event id 4688) events from the security event log in addition to events logged by PowerShell itself. This means you’ll need audit the following sub categories from the Detailed Tracking category:
If you are not using EventSentry then I recommend collecting both 4688 and 4689 events so that you can not only determine whether a powershell.exe process was started, but also how long it remained active. If you are an EventSentry user then you just need to verify that Process Tracking (an object for Compliance Tracking) is enabled and configured to capture the command line of a process. EventSentry can automatically parse and correlate 4688 and 4689 events and provide a history of all processes on a monitored system.
EventSentry users can also utilize the Audit Policy Status page to verify that process creations are indeed being audited. You’ll also want to make sure that “Include command line in process creation events” is activated, so that Windows logs the command line of every process as part of 4688 events. After all it doesn’t help us that much just knowing that powershell.exe has been running, we need to know what exactly it has been running.
This can either be enabled via group policy (Administrative Templates\System\Audit Process Creation\Include command line in process creation events) or via the registry (set HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\Audit\ProcessCreationIncludeCmdLine_Enabled to 1).
Disclaimer: This option is available starting with Windows 7 / Server 2008 R2, earlier versions of Windows don’t support it. Things are a little easier for EventSentry users, which attempts to obtain the command line of a process if it’s not included in the 4688 event and subsequently makes it available as variable $STR9. But more on that in part 2 when we discuss ways to detect and mitigate attacks.
I hope I was able to convince you of the risks that PowerShell poses, what versions of PowerShell are out there, and what type of logging needs to be enabled in order to detect and stop malicious PowerShell in its tracks. In part 2 I’ll talk about how to actually mitigate PowerShell-based attacks – with specific instructions for EventSentry.
In Mr. Robot‘s episode 9 of season 2 (13:53), Angela Moss needs to obtain the Windows domain password of her superior, Joseph Green, in order to download sensitive documents that would potentially incriminate EvilCorp. Since her attack requires physical access to his computer, she starts with a good old-fashioned social engineering attack to get the only currently present employee in the office to leave.
Once in his office, she uses a USB Rubber Ducky, a fast and automated keyboard emulator, to obtain Joseph’s clear text password using mimikatz. Please note that there are some holes in this scene which I will get into later. For now we’ll assume that she was able to obtain his credentials by having physical access to his computer.
After she gets back to her workstation, she analyzes the capture which reveals Joseph’s password: holidayarmadillo. Not the best password, but for this particular attack the quality of the password wouldn’t have mattered anyways. Mimikatz was (is) able to get the password from memory without utilizing brute force or dictionary techniques. Once she has the password, she logs off and logs back on with Joseph’s user and downloads the documents she needs.
As somebody who helps our users improve the security of their networks, I of course immediately contemplated how this attack could have been detected with EventSentry. Since most users only log on to one workstation on any given day with their user account, Angela logging in with Joseph’s account (resulting in “joseph.green” logging on to two different workstations) would actually be an easy thing to detect.
Introduced with v3.4, collector-side thresholds allow the real-time detection of pretty much any user activity that originates from an event, for example user logons or process launches. You can tell EventSentry that (physical) logons for any user on more than one host (within a given time period – say 9 hours) should trigger an alert (aka as “lateral movement”). Had this been in place at EvilCorp, Angela logging in as Joseph would have immediately triggered an alert. With the right procedures in place, countermeasures could have been taken. Of course most viewers wouldn’t want Angela to be caught, so please consider my analysis strictly technical. Watch a short video on lateral detection with EventSentry here.
So what’s the hole? Well, the rubber ducky (mimikatz, really) requires access to an active logon session, which Angela most likely didn’t have. It looked like Joseph had been out of the office for a while, so his computer was likely either locked or turned off, rendering any attack based on mimikatz useless. Mimikatz – since it obtains passwords from memory – only works if the computer is unlocked. And had the computer been unlocked then she could have just downloaded the files from his computer – although this would have been even more risky with people walking around the office.
Cyber attacks are becoming more potent every year and are often sponsored by powerful criminal gangs and/or governments. It’s important that companies employ multiple layers of defense to protect themselves (and their customers) from these increasingly sophisticated and destructive attacks.
EventSentry is the only monitoring solution that utilizes robust agent-based technology that goes beyond logs, enabling the fusion of real-time log monitoring with in-depth system monitoring to not only detect but also react to attacks and anomalies. See for yourself and download a free evaluation of EventSentry.
Integrate with open source solutions (Graylog, ELK, Nagios Log Server & others)
and more. We’ve also been focusing on making the data EventSentry collects more actionable and subsequently more helpful, and as result you will see additional contextual data provided with some alerts & reports, and one new search page in EventSentry.
All in all, this upcoming release has a lot of improvements in store to help you do your job better by ensuring that your network is as reliable, secure and fast as possible.
While high-profile Ransomware attacks have slowed down somewhat in recent weeks, businesses – especially small businesses – are still hit with Ransomware infections every day. Even though EventSentry is not positioned as a AV or a AntiMalware software, it does include a variety of functionality which can detect Ransomware outbreaks.
What sets EventSentry apart from AV and most AntiMalware solutions is that it looks for pattern exhibited by the Ransomware – regardless of the variant. What’s new in version v3.4 is the ability to detect changes to the MBR and bootsector as well as the ability to calculate the entropy of (changed) files.
MBR/BootSector Monitoring & Backup
Some more recent Ransomware variants modify the MBR and/or boot sector and trigger a reboot to initiate an offline encryption process. EventSentry v3.4 can now monitor the MBR and detect changes in near real-time to alert the admin when suspicious activity is occurring.
By utilizing EventSentry’s advanced filtering engine it is also possible to potentially stop the encryption process, e.g. by hibernating the infected host. EventSentry even backs up the MBR and boot sector, making it possible to repair an infected system (with a boot disk) without having to perform a full restore from backup.
Entropy describes the randomness of a file, essentially a metric that can help detect compressed and encrypted files.
Since most Ransomware encrypts large amounts of files, EventSentry can utilize the entropy of a file, combined with event log thresholds, to make a determination that a Ransomware infection is in progress and take corrective measures.
Lateral Movement Detection with Thresholds
EventSentry has always included agent-side thresholds in order to throttle the alert volume or detect repeated events. Because these thresholds were executed on the agents, event patterns which involved more than one host could not be detected that way.
By adding a threshold component to collector – which analyzes and processes all events in real time – we can leverage this feature to new heights and detect network-wide event-based patterns – in real time!
Collector-side thresholds are configured exactly like agent-side thresholds with one the key difference – the threshold limit applies to all aggregated events sent to the collector. Collector-side thresholds also introduce the “Group By” setting that makes the lateral movement detection possible – the ability to count unique instances of an event property, and not just the total number of events.
Some of the event-based threat patterns you can detect:
The same user logging on to multiple hosts within a specific time frame
A process spreading (trickling) across multiple hosts within a specific time frame
A user running too many processes – either on a single or multiple hosts
Authentication failures of a user on too many hosts
Too many unique logon types used by a user account
Any event property and insertion string can be used to craft thresholds – the sky is the limit.
Software Version Checker
Earlier versions of EventSentry include a substantial hardware, software and patch inventory, making it extremely easy to find out which software packages are installed on your network, but also get alerted when software is installed or removed from a server/workstation.
In v3.4 we are taking this to the next level by providing the latest version available from the publisher for a growing list of 100+ software packages so that you can effortlessly identify outdated software on your network. You can now create reports listing any software on your network which is outdated, a dashboard tile is also available. The Web Reports refresh the version info list every 2 hours to ensure all reports are accurate.
If an application you are using on your network is not currently included then simply email us the name of the software as it is detected by EventSentry (and ideally the URL where we can obtain the current version) and we will add it to our list.
64-bit Web Reports for Windows
The EventSentry web reports are now available as a 64-bit application, and upgrading to v3.4 will automatically upgrade the existing v3.x 32-bit web reports to 64-bit on 64-bit when installed/upgraded on a 64-bit version of Windows. The new 64-bit web reports will allow you to run larger reports that would not run due to limits with the address space associated with 32-bit applications.
UPS & Battery Monitoring
Any UPS directly attached to a server/workstations that is detected by Windows can now be monitored by EventSentry. The status of the UPS will show up on the host inventory page, and alerts will be generated when a host is on battery power and back on AC power. EventSentry can also initiate a shutdown when the remaining run-time or charge level falls below a certain limit.
Batteries in laptops are also detected and listed on the host inventory page (battery capacity and current charge level), but generated alerts are informational only.
User Activity Tracking
While EventSentry provides its users with a wealth of information from multiple angles, it can be tedious to piece together data from multiple reports that is associated with a specific user. Data which can be linked to a user is scattered among process tracking, file access tracking, compliance logons and other pages.
The new “User Activity” page, which is located in the “Dashboard” menu, solves this problem by displaying data from the following pages on a single page:
Active Directory Changes
The user activity page makes seeing all activity by a user as easy as never before!
Integration with third-party log management solutions
A few months ago, one of our users approached with the need to integrate EventSentry into an existing log management system which was already in place at the location where EventSentry was to be deployed. While reviewing the request we recognized that even though we position EventSentry as a one-stop log management solution with a compelling and robust web-based reporting component, an integration with other products can be helpful in some cases.
Supplement EventSentry’s built-in reporting with additional reporting
Integrate EventSentry with an existing log management solution located in a different business unit
Integrate EventSentry’s sophisticated real-time agent and deployment infrastructure with a different log management back-end
In version 3.3 and earlier, EventSentry can be integrated with 3rd party products using the HTTP, process and Syslog action. The HTTP & process action are intended to be used with ticketing systems where only a low volume of alerts are submitted while the Syslog action obviously supports submitting a high volume of events. The Syslog format was however limited to the traditional RFC 3164 format, making an integration with other log management systems difficult.
Starting with version 3.4, EventSentry now supports the following formats in the Syslog action
RFC 3164 (legacy)
Nagios Log Server
Common Event Format (CEF)
If a log management server you need to integrate with is not listed above but supports the JSON format, then you can craft your own JSON packet with the JSON format, also introduced in v3.4.
Disk Space Alerts
Part of the effort to make EventSentry’s alerts more actionable is reflected in our improved disk space alerts which now list the 15 largest files and folders of the volume where disk space is low. The supplemental data will in many cases be enough to immediately identify the culprit so that corrective action can be taken immediately, without the need to run disk space analyzers on the volume.
Audit Policy Status
Since the introduction of the compliance tracking components, EventSentry has been recording all audit (and many other!) changes performed in Windows as part of the “Policy Changes” feature. It wasn’t however possible to see the current status of all audit categories and subcategories at a glance. Reviewing the current audit status of all monitored hosts can be important however, if only to verify that group policies are configured correctly.
In v3.4 we now have the new “Status” page available under “Compliance -> Audit Policy” which delivers information such as:
Compare/review audit settings of a particular sub category (e.g. “Registry”) among all monitored hosts
View all disabled audit settings across all or select hosts
(Re)view audit settings based on computer types (e.g. domain controllers, servers, workstations)
Our NetFlow component can now provide bandwidth visualization based on the collected NetFlow data. The information can either be accessed on the NetFlow page or as a dashboard tile. Even though bandwidth data can already be determined using SNMP, the data gathered by NetFlow should be preferred since it contains additional data not available via SNMP, such as:
Bytes per packet
Bytes per packet as well as packets sent received can be used to detect anomalies, e.g. when a host sends a large amount of network packets, or network packets with large/small content.
Today I have good news and bad news. You’d like to hear the good news first? OK. The bad news is that AutoAdministrator is being retired and will no longer be developed.
OK, now on to the good news. AutoAdministrator is of course not entirely history and is undergoing a similar transformation the NTToolkit did a few years back. AutoAdministrator is joining the “EventSentry” brand as the “EventSentry Admin Assistant”.
This brings the total number of software products under the EventSentry brand to three:
Almost everyone in IT has heard of VNC – which actually stands for “Virtual Network Computing”. The RFB (Remote Framebuffer) protocol which VNC relies on, was developed around 1998 by Olivetti & Oracle Research Labs. Olivetti (unlike Oracle) isn’t much known outside of Italy/Europe, and the ORL was ultimately closed in 2002 after being acquired by AT&T. But enough of the history.
When the need arises to remotely log into a (Windows) host on the network, Microsoft’s Remote Desktop application (which utilizes Microsoft’s RDP protocol – not RFB) is usually the default choice. And why wouldn’t it be? It’s built into Windows, there is no additional cost, and it’s usually quite efficient (=fast) – even over slower connections.
Remote Desktop has a few disadvantages though, especially when it comes to the IT help desk:
You cannot view the remote user’s current desktop
It’s not cross-platform
You can’t use RDP if it’s disabled or misconfigured
Especially when troubleshooting user problems, being able to see exactly what the user is doing is obviously very beneficial. VNC-based applications are a good alternative since they allow you to view the user’s desktop and subsequently interact with the user. This makes VNC viable for help desk as well as troubleshooting. Nevertheless, VNC-based solutions have their own shortcomings:
Free variations of VNC usually offer no deployment assistance
With over 10 variants available, finding the best VNC implementation is a daunting task
VNC is still deemed as somewhat insecure
VNC can be slow
We set out to solve these shortcomings by creating a number of scripts around UltraVNC that integrate with the EventSentry management console (although they’ll work well without EventSentry as well!). Using the QuickTools feature, you can then connect to a remote host via VNC with 2 clicks, even if the remote host doesn’t have VNC installed.
Important: The scripts only work in environments where you have administrative access to the remote hosts. The scripts need to copy files to the remote host’s administrative shares and control the remote VNC service.
Alternatively, you will also be able to start a VNC session by running the following command:
Even better, VNC can be automatically stopped and deactivated (until vnc_start.bat is run again) once the session is completed in order to reduce the attack surface.
As long as you have administrative access to the remote host(s), the script will remotely install VNC and even setup a firewall exclusion rule if necessary – although the UltraVNC installer takes care of this out of the box.
To reduce the attack surface of machines running VNC you can automatically stop the VNC service after you have disconnected from the remote host. Our connection script will automatically start the remote service again when you connect the next time.
For the utmost security you can also completely uninstall VNC when you are done, a script (vnc_uninstall.bat) is included for this purpose.
Even though VNC is generally not as fast as RDP, it’s usually sufficiently fast in LAN environments (especially for shorter trouble-shooting sessions) and the UltraVNC port which we’ll be covering in this post performs reasonably well even over slower WAN connections.
Integration with EventSentry
Monitoring workstations with EventSentry strengthens the capabilities of any IT helpdesk and IT support team with:
Software & Hardware Inventory
Access to process utilization and log consolidation
Enhanced security with security log & service monitoring
User console logon tracking
Pro-active troubleshooting with access to performance and other system health metrics
Remote desktop sharing is an additional benefit with the UltraVNC package which is included with the latest version of EventSentry (v18.104.22.168). Customizing the scripts and integrating them with EventSentry literally shouldn’t take more than 5 minutes, and once setup & configured will allow you to remotely control any monitored host with a couple of clicks. The scripts do not require EventSentry, but are included with the setup and integrate seamlessly into the EventSentry Management Console.
The EventSentry Management Console includes the “QuickTools” feature which allows you to link up to 8 commands to the context menu of a computer item. EventSentry ships with a few default QuickTools commands, for example to reboot a remote machine. Once configured, you simply right-click a computer icon in the EventSentry Management console and select one of the pre-configured applications from the QuickTools sub menu.
How does it work?
When you run the vnc_start.bat script, it will first check to see if UltraVNC is already installed on the remote host. If it is, it will skip the installation routine and bring up the local VNC viewer. If you configured the script to automatically stop the VNC service when not in use, it will start the service beforehand. When you disconnect, it will (optionally) stop the VNC service again so that VNC is not accessible remotely anymore.
If VNC is not installed, the script will remotely install & configure UltraVNC using psexec.
If you do not want to leave the UltraVNC service installed on the remote computer, the vnc_uninstall.bat script can be run when the remote session is done. Automatically stopping the remote VNC service is however sufficient in most cases.
The scripts need to be configured before they can be used in your environment, unless you are an EventSentry user, in which case you only need to download & install the prerequisites.
Super Quick Setup for EventSentry Users It’s no secret, we’re a little biased towards our EventSentry users, and as such setting this up with an existing EventSentry installation is rather easy:
Get psexec.exe and save it in C:\Program Files (x86)\EventSentry\resources.
Download the UltraVNC installers (they have 32-bit and 64-bit – download for the platforms you have on your network) and store them in the C:\Program Files (x86)\EventSentry\scripts\ultravnc folder.
Install UltraVNC on the computer where EventSentry is installed so that the VNC Viewer is available. It’s not necessary to install the whole package, only the viewer component is required.
If “VNC” is not listed in your QuickTools menu, then you will need to add it under Tools->Options->QuickTools. Simply enter “VNC” as the description and specify the path to the vnc_start utility, e.g. “C:\Program Files (x86)\EventSentry\scripts\ultravnc\vnc_start.bat $COMPUTER”. You can optionally check the “Hide” box to prevent the script output from being shown before you connect.
You’ll notice that no password was configured – that’s because you will be logging in with a Windows user and password – only allowing domain admins access by default. This can be configured in the authorized_acl.inf file, if you want to give additional groups and/or users access that are not domain admins.
That’s literally it – easy as pie. Even though we designed this thing to be easy peasy, since things do occasionally go wrong I recommend testing a first connection from the command line. Just open an administrative command prompt, navigate to C:\Program Files (x86)\EventSentry\scripts\ultravnc and type vnc_start somehost.
Now just right-click any host – or use the “Quicktools” button in the ribbon – and select the “VNC” menu option. Keep in mind that first-time connections will take longer since the VNC setup file has to be copied and installed on the remote computer. Subsequent connections should be faster.
Manual Normal-Speed Setup for Non-EventSentry Users
So you are not an EventSentry user but still want to utilize these awesome scripts? No problem – we won’t hold it against you. The setup is still easy – you’ll just need to customize a few variables in the variables.bat file.
In variables.bat, set the VNCSETUP_X86 and VNCSETUP_X64 to the setup file names you just downloaded.
Download the PSTools and extract psexec.exe into the working directory, or a directory of your choice.
In variables.bat, point the PSEXECFILE variable to the location where you just saved psexec.exe.
Optional: Edit the authorized_acl.inf file to specify which Windows group or user will have access to VNC. You can either change the first line, or add additional lines to give additional users and/or groups permission.
Install the respective version of UltraVNC on your workstation so that the VNC Viewer is available.
Open a command line window and navigate to the folder to which VNCSOURCE points to. Test the setup by running vnc_start hostname, replacing “hostname” with an actual host name of a remote host of course.
When presented with the login screen of the VncViewer, log in with a Windows domain admin user.
That wasn’t so bad now, was it? Just remember that you’ll need to initiate any VNC session with the vnc_start.bat file. Just launching the Viewer won’t work – even if VNC is already installed on the remote machine – since the VNC service is stopped by our scripts by default. To use the folder names we created, you’ll just run
Enjoy, and happy RFBing!
Configuration – variables.bat For the sake of completeness the variables.bat file is explained below:
VNCSETUP_X86: The file name of the 32-bit installer. This needs to only be changed whenever UltraVNC comes out with a new version. VNCSETUP_X64: The file name of the 64-bit installer. This needs to only be changed whenever UltraVNC comes out with a new version.
REMOTEINSTALLPATH: The directory where the script files will be copied to on the remote host.
VNCSOURCE: This is the folder where all the vnc-related files, including the setup executables, are located on the source host from where you initiate VNC connections – e.g. C:\Deployment\UltraVNC. VNCINSTALLDIR: The directory in which UltraVNC will be installed in (on the remote hosts).
VNCPASSWORD: This variable is not currently used since UltraVNC is automatically configured to authenticate against Windows, by default giving only Domain Admins access to VNC. This is generally more secure than using a password. You can edit the file authorized_acl.inf to give additional users and/or groups access to VNC. The file supports one ACL entry per line.
PSEXECFILE: Unfortunately we are not allowed to bundle the nifty psexec.exe file for license reasons, so you’ll have to download the PsTools and point this variable to wherever you end up copying the psexec.exe file to. If you already have psexec.exe installed then you can save yourself 2 minutes of time and just specify the path to the existing file here.
SET_VNC_SVC_TO_MANUAL: If you don’t entirely trust the security of VNC, maybe because you know what a brute force attack is, and you only want administrators to access VNC then you can set this variable to 1. As long as you only connect to the remote host(s) using the vnc_install.bat script, the scripts will ensure that the remote VNC service is started before you connect and stopped after you disconnect. Between the two of us, I’d always leave this set to 1 unless you have the desire to launch the VNC Viewer directly, or need non-administrators to be able to connect to the remote host(s).
ADD_FIREWALL_RULE: As the name (almost) implies, this will create a firewall exclusion rule on the remote host(s) if you’ve been doing your homework and enabled the Windows firewall. If you don’t like our boring firewall rule name then you can even change the name below by editing the FW_RULE_NAME variable. Enabling this is usually not necessary since the UltraVNC setup adds firewall exclusion rules by default.
VNCVIEWER: If you find that a different version of the VNC viewer works better than the version which we are shipping, then you can change the file name here.
In my previous post I talked about our new NetFlow component as well as the new agent management capabilities now available in EventSentry v3.3. In this post I’ll cover the remaining new features and improvements we’ve made in v3.3, starting with the web reports.
There are a number of new features and improvements in addition to NetFlow visualization. There are a few new dashboard tiles, including a “Recent Activity” tile which – as the name implies – shows recent relevant changes such as newly detected processes, software (un)installed, ping status or service status changes.
Anybody who works in a team of two or more Sysadmins should find the new notes feature incredibly helpful. It lets any web reports user add comments (=notes) which are subsequently visible to others. Notes can be associated with one or more hosts (ensuring they show up in the “Documentation” tab of the respective host status page) and can include documents as attachments as well! Do you have warranty documents or network diagrams you want to store in a central place – easily accessible? That’s what the notes are for.
The overall look and feel has also been refreshed, and we’ve reorganized the menu to make it faster to access dashboards and easier to find pages.
The visualization of data has been improved, since some chart types work better with certain features of EventSentry. You can now visualize grouped data using either pie charts, tree maps or column charts.
The security of the web reports has also improved with a lockout policy which will locking an account after too many unsuccessful logon events.
As mentioned in part 1, the EventSentry is agent is now available in 64-bit, making it possible to monitor 64-bit counters and easier to monitor files in 64-bit directories. For users upgrading from an earlier version, the EventSentry management console will automatically migrate any existing 32-bit agents on 64-bit versions of Windows.
Application & Services Event Logs
While monitoring Application & Services event logs, often referred to as “custom” event logs was possible, the way this needed to be configured in the management console was a common source of confusion. Some users also needed the ability to monitor more than 30 different logs. Consequently, monitoring additional event logs is now straightforward, and users can monitor as many event logs as they wish.
With thresholds, timers, schedules, insertion strings, EventSentry already offers a sophisticated engine for monitoring events in real time. New in this release is the ability to setup filter chaining. This makes it possible to trigger actions only when 2 or more events occur, and you can even link events together using insertion strings. Chaining is enabled on the package level, and every filter in a “chaining” package is automatically part of the filter chaining rules.
It happens frequently that we get alerts that require us to do additional research based on the information provided in the alert. For example, we may get an alert about an IP address for which we then need to do a reverse lookup or find the geoip location. Audit Success & Failure events from the security event log are another example, and often contain error codes and numbers which are not explained.
We set out to improve upon this, and starting with v3.3 EventSentry will annotate email alerts in a number of ways whenever possible:
IP addresses will include a reverse lookup
IP addresses will include a geoip location
Security events will have various error codes resolved
Please note that (1) and (2) are only supported for emails sent through the collector since it requires access to a local geoip database. (1) and (2) will need to be enabled in the email action “Options”, (3) is automatically enabled for all emails.
Insertion Strings & Regex
By making insertion strings from events accessible in filters and actions (e.g. through the $STR1, $STR2, … variables), it’s possible to create highly granular thresholds, customize emails, easily trigger corrective actions which utilize content from events and more. Based on our own requirements we took this capability a step further however, and you can now apply regex filters to events to define your own insertion strings. This is particularly useful for alerts which don’t use insertion strings or for events which contain log data. For those types of events, you can now parse parts of log strings and assign them to insertion strings. The previous blog article, Detecting Web Server Scan in Real-Time, shows a practical example of how to apply this new feature. It does require you to be a bit familiar with Regular Expressions, but the management console includes a handy dialog where you can test your regular expressions, shown below.
Faster is better! We’ve improved performance in a number of areas:
The database insert performance of the Syslog daemon has been improved for Microsoft SQL Server databases
The delimited log file feature now includes an additional index to increase database insert performance
The heartbeat agent now relies less on RPC-based agent status monitoring and can instead obtain the status of a remote agent either directly from the collector or the database, resulting in less network traffic and faster heartbeat monitoring cycles.
With new features & improvements in a variety of areas, this release should contain improvements for everyone. Remember that you can also submit feature requests here.
We are very excited to release EventSentry v3.3, a major update to our award-winning monitoring solution EventSentry, less than 10 months after the release of the previous major version 3.2. Version 3.2 included the collector component which supports secure and reliable communication with remote agents as well as better database throughput, switch port mapping and many improvements to the web reports.
I’d like to also thank everyone who took the time to fill out our annual survey – we read every single response in detail. If you haven’t taken it yet then you can still do so here.
The v3.3 release, which builds upon some of the architectural changes we have made in v3.2, and offers new functionality to help you:
Visualize, measure & investigate network traffic better with the new NetFlow component – with discounted introductory pricing until 12/31 2016!
Spend less time managing agents – the collector can now push configuration as well as agent updates automagically – think laptops!
Deployment via MSI is much easier – MSI file creation now only takes a few seconds
Investigate issues faster with email alerts which have geo location, reverse lookups as well complex security codes included inline
Visualize any data in the web reports more easily with additional dashboard tiles and treemaps throughout
Managing and using custom event logs is now more straightforward and scalable
Database throughput has been improved for Syslog data and delimited log files
Even more advanced filtering is possible with filter chaining and insertion string override via regular expressions
Communicating and documenting your network has just become a lot easier – add notes and/or upload documents in the web reports
Monitor 64-bit operating systems with a native 64-bit agent
With a brand new component and many new features in a variety of areas, v3.3 will have something of interest for everyone. Let’s dive in and look at the new features in more detail.
NetFlow NetFlow is a new component which is part of the “Network Services” service (along with Syslog, SNMP, ARP) and is licensed separately. Pricing is very competitive and an additional introductory discount will be available until the end of this year, 12/31 – including competitive upgrades. You can request a quote here.
Collecting NetFlow data allows you to see all traffic meta data which passes through network devices that support NetFlow, including:
Source IP, destination IP
Source host, destination IP (when resolvable)
Source port, destination port
Geo location (when available)
IP protocol used
Amount of traffic sent and received
Number of packets transmitted
EventSentry v3.3 currently supports the NetFlow v1, v5, v9 as well as sFlow flow protocols. NetFlow is usually supported by most commercial routers and firewalls whereas sFlow is most commonly supported by switches. NetFlow is generally preferable over sFlow – especially for forensic analysis since sFlow samples traffic and only sends every nth flow. sFlow can be preferable when dealing with large amounts of data, but EventSentry’s NetFlow implementation (as well as NetFlow itself) has a way to group flows and therefor condense traffic.
Do you need NetFlow, and is it worth looking into? Without NetFlow there is impossible to know which hosts communicate with each other (unless you capture network traffic). What traffic enters the network, and what traffic leaves it? Broadly speaking, implementing NetFlow lets you:
Visualize all network traffic in a variety of ways and reports
Analyze network data for forensic purposes
Utilize network traffic data for troubleshooting purposes
Map network traffic to geo location
Correlate network traffic with Active Directory users (requires workstation monitoring)
Measure bandwidth utilization
On the EventSentry side, setting up NetFlow should take less than 5 minutes; and setting it up on the network device side is generally just a matter of enabling NetFlow and pointing it to EventSentry.
EventSentry ships with the GeoLite geo database from MaxMind which does a good job of associating IP addresses with physical locations down to the city level. If you are looking for more accuracy however, then you can also purchase the full geo location database from MaxMind here.
Active Directory User Correlation
A unique feature of EventSentry’s NetFlow implementation is the ability to correlate workstation logins with network traffic, making it possible to associate network traffic with individual users. This requires that workstations are monitored with EventSentry and works best when users have a dedicated workstation.
Agent Management & Deployment
If you are utilizing the collector service then you have now a great time-saving feature available. Pushing a configuration update to remote hosts after you made a change or deploying agent updates after a patch installation are a thing of the past once you activate the respective options in the collector dialog.
Managing automatic configuration updates can be done in 2 ways: Either by automatically deploying a configuration update after you click “save”, or by deploying only approved configuration updates (recommended). If you select the latter, then you just have to click the new “Save & Deploy” sub-option on the ribbon and the collector will do the rest. It’s no longer necessary that the EventSentry agent is directly reachable from the management console; it will receive the latest configuration as soon as it connects to the collector.
Please note that you will still need to manually deploy a v3.3 agent once in order for automatic agent updates to work, since the self-update code is embedded in the new agents.
Creating MSI files has also been greatly simplified – a x86 and x64 agent MSI file is created with just a few mouse button clicks. Manually editing MSI files with tools like ORCA is a thing of the past. The only prerequisite is the (free) WiX Toolset which has to be installed only once.
In addition to saving most EventSentry users a lot of time, these new deployment features also make it possible to monitor laptops which aren’t permanently connected to the network. Simply deploy the agent MSI file with your favorite deployment tool (or deploy with the management console) and enable the configuration and agent management options in the collector. From that point on, any agent connecting to the collector will automatically receive the latest configuration AND any new agent updates – completely automatically – no matter where in the world they are located.
EventSentry v3.3 now ships with both a x86 and x64 agent, so that 64-bit editions of Windows can be monitored natively. The key benefit of this change is that 64-bit only performance counters can now be monitored, these counters were off limit with 32-bit agents. Utilizing 64-bit agents also results in the following changes:
Agents will be automatically converted to 64-bit when v3.3 is deployed. It is not possible to use a 32-bit v3.3 agent on a 64-bit version of Windows
File system redirection via “Sysnative” or in the File Checksum Monitoring packages is no longer necessary
Memory consumption will be slightly higher compared with 32-bit agents
Please note that EventSentry has not completely migrated to 64-bit yet, some components (management console, heartbeat agent, web reports) are still shipped as 32 bit executables. We plan on migrating all components to 64-bit by the end of 2017.
There are just too many new features in v3.3 to fit them all into one blog post, so stay tuned for part 2 which will follow shortly.
Since Ransomware is still all the rage – literally – I decided to write a 4th article with a potentially better method to stop an ongoing infection. In part 1, part 2 and part 3 we focused mostly on detecting an ongoing Ransomware infection and utilized the “nuclear” option to prevent it from spreading: stopping the “server” service which would prevent any client from accessing files on the affected server.
While these methods are certainly effective, there are other more targeted steps you can take instead of or in addition to shutting down the server service, provided that all hosts susceptible to a Ransomware infection are monitored by EventSentry.
When EventSentry detects an ongoing Ransomware infection, it can usually determine the infected user by extracting the domain user name from the 4663 event. Simply disabling the user is insufficient however, since a disabled user can continue to access the network (and wreak havoc) as long as he or she doesn’t log off. Any subsequent log on attempt would of course fail, but that provides little comfort when the user’s computer continues to plow through hundreds or thousands of documents, relentlessly encrypting everything in its path.
As such, the only reliable way to stop the ongoing infection, given only the user name, is to log off the user. While logging a user off remotely is possible using the query session and logoff.exe commands, I prefer to completely shut down the offending computer in order to reduce the risk of any future malicious activity. Logging the user off remotely may still be preferable in a terminal server environment (let me know if you want me to cover this in a future article).
Knowing the user name is of course great, but how do we find out which computer he or she is logged on to? If you have EventSentry deployed across your entire network – including workstations – then you can get this info by querying the console logon reports in the EventSentry web reports. If you are not so lucky to have EventSentry deployed in your entire environment (we offer significant discounts for large quantities of workstation licenses – you can request a quote here) then we can still obtain this information from the “net session” command in Windows.
We’ve created a little script named antiransom_shutdown.vbs which, given a user name, will report back from which remote IP this user most recently accessed the local server and optionally shut it down. Here are some usage examples:
Find out from which computer boris.johnson most recently accessed this server: cscript.exe C:\Scripts\antiransom_shutdown.vbs boris.johnson
Find out from which computer boris.johnson most recently accessed this server AND shut the remote host down (if found): cscript.exe C:\Scripts\antiransom_shutdown.vbs boris.johnson shutdown
The script uses only built-in Windows commands, as such there is no need to install anything else on the server where it’s run.
When executed with the “shutdown” parameter, the script will issue a shutdown command to the remote host, which will display a (customizable) warning message to the user indicating that the computer is being shutdown because of a potential infection. The timeout is 5 seconds by default but can be customized in the script. It’s recommended to keep the timeout short (5-10 seconds) in order to neutralize the threat as quickly as possible while still giving the user a few moments to know what is happening.
The overall setup of the Ransomware detection is still the same, we’re setting up a threshold filter to detect a higher than usual frequency of certain 4663 events and trigger an action in response. Only this time we don’t shut down the server service, but instead trigger this script. To properly execute the action, configure it as shown in the screenshot below. The executable is cscript.exe (the interpreter for .vbs files) and the command line parameters are the name of the script, $STR2 and “shutdown”.
So what’s the better and safer approach to freeze an ongoing Ransomware infection? Shutting down the server service is the most reliable approach – since it doesn’t require the workstation to be reachable and will almost certainly succeed. Remotely shutting down a workstation has minimal impact on operations but may not always succeed. See below for the pros and cons of each approach:
File Sharing Shutdown Pros: 100% effective Cons: Potentially larger disruption than necessary, false positive unnecessarily disrupts business
Remote Workstation Shutdown Pros: Only disables infected user/workstation, even if false positive Cons: Requires workstation to be reachable
This ends up being one of those “it depends” situations where you will have to decide what’s the best approach based on your environment. I would personally go with the remote workstation shutdown option in large networks where the vast majority of workstations are desktops reachable (and not firewalled) from the file server. In smaller, more distributed networks with a lot of laptops, I would go with the file service shutdown “nuclear” option.
A hybrid approach may also be an option for those opting for the remote workstation shutdown method: trigger a remote workstation shutdown during business hours when IT staff is available on short notice, but configure the file service shutdown after business hours when it’s safer and affects fewer people. All this can be configured in EventSentry by creating two filters which are identical except for the action and the day/time settings.
Prerequisites It’s important to point out that the EventSentry agent by default runs under the LocalSystem account, a built-in user account which does not have sufficient privileges on a remote host to issue the shutdown command. You can elevate the permissions of the EventSentry agent and work-around this limitation in 2 ways:
Change the service account (fast): Changing the service account the EventSentry service uses to a domain account with administrative permissions will allow the agent to remotely shut down a remote host. This will have to be done on every file server which may issue shut down commands (you can use AutoAdministrator to update multiple file servers if necessary).
Give the “Force shutdown from a remote system” user right: It’s not necessary to issue domain-wide admin rights to the EventSentry agent, the key right the agent needs is just the “Force shutdown from a remote system” user right. The quickest way to deploy this setting is of course through group policy:
a) Open the “Group Policy Management Editor”
b) Edit an existing policy (e.g. “Default Domain Policy”) or create a new group policy
c) Navigate to “Computer Configuration\Policies\Windows Settings\Security Settings\Local Policies\User Rights Assignment”
d) Double-click the “Force shutdown from a remote system” user right and add both “Administrators” and the computer accounts of the file servers to the list. Alternatively you can also create a group, add the file servers to the group, and add that group to the policy (keep in mind that you will need to restart the file servers if you go with the group method).
Once the group policy setting has propagated to the workstations, the remote shut down initiated from the file server(s) should succeed.
Good luck protecting your network against Ransomware infections, also remember to verify your backups – no protection is 100% effective.
Dashboards are a great way to visualize large amounts of information in a concise matter. In IT we usually display various types of network data from a monitoring software, but dashboards are used in all sorts of environments. You can visualize stock data or just show a map of all trucks in a fleet with their current position.
If you work for a large company with a dedicated NOC then you’ll likely have an integrated setup with 4 or more TVs, connected to hardware specialized for dashboards or, at the very least, a powerful PC with multiple PCI cards.
But not everybody has the budget or the need for a NOC like AT&Ts, and one or two TVs can be sufficient for most networks – provided the dashboard is well-designed and customizable of course.
Most dashboards require a fairly recent web browser (if you are unlucky even Adobe Flash), making some sort of a PC or Mac the preferred hardware to power that dashboard. Most IT departments have a plethora of old PCs sitting around, and it can be tempting to resurrect one of those boxes and give them a new life as a dashboard PC. After all, you’re “just” displaying a web page.
Low-cost integrated devices like the Raspberry Pi are tempting, but not perfect either. They’re not usually designed to be used with graphical interfaces, much less with memory and CPU hungry applications like web browsers displaying dashboards.
After trying everything from Raspberry Pi, old Mac Mini hardware and more, we finally found a solution for under $100 – which has now worked quite well for several months: The 1st generation Intel Compute Stick which you can get from online retailers like Amazon, NewEgg and others.
Even in its 1st generation (the one we tested) the Intel Compute Stick running Windows 10 Home performed surprisingly well. We’ve been running an EventSentry dashboard (which of course we’re hoping you are running as well) on it since February on Microsoft’s new Edge browser, and we’ve never had an issue.
The Intel Compute Stick features 2 Gb of RAM, is powered by a quad-core intel Atom processor and has 30 Gb of storage, of which more than half are available. This is of course not a machine you’ll want to render videos or play video games on, but plenty sufficient for a web browser from our experience. We were actually pleasantly surprised by how responsive the little device felt overall. Even though you cannot join a domain, you can still install the EventSentry agent on the machine to keep an eye on performance and other system metrics for example.
But there are of course some caveats, as is to be expected from a computer that costs less than $100 and is not much bigger than a USB memory stick. If you’re using Bluetooth and Wifi then you’ll only need to connect the power cord and the setup is clean. Since the stick also sports a single USB 2.0 port, we used a USB hub along with a USB-based Ethernet adapter to connect it to our LAN as well as connect a keyboard/mouse. USB 2.0 didn’t negatively affect performance in our limited use case scenario.
If you need more hardware, maybe because your dashboards are particularly taxing, then you can purchase a newer and faster model as well. The 2nd generation Intel Computer Sticks start around $149 and the high-end models include as much as 64Gb of disk space and 4Gb of RAM.
My first computer was a 80286 with 1Mb of RAM and a 20Mb hard drive, and it was about as big as two shoe boxes. It’s impressive to see a device this small perform that well. If you have the need to turn a TV into a full-blown desktop, then I’d definitely recommend the Intel Computer Stick(s)!