Today I have good news and bad news. You’d like to hear the good news first? OK. The bad news is that AutoAdministrator is being retired and will no longer be developed.
OK, now on to the good news. AutoAdministrator is of course not entirely history and is undergoing a similar transformation the NTToolkit did a few years back. AutoAdministrator is joining the “EventSentry” brand as the “EventSentry Admin Assistant”.
This brings the total number of software products under the EventSentry brand to three:
Almost everyone in IT has heard of VNC – which actually stands for “Virtual Network Computing”. The RFB (Remote Framebuffer) protocol which VNC relies on, was developed around 1998 by Olivetti & Oracle Research Labs. Olivetti (unlike Oracle) isn’t much known outside of Italy/Europe, and the ORL was ultimately closed in 2002 after being acquired by AT&T. But enough of the history.
When the need arises to remotely log into a (Windows) host on the network, Microsoft’s Remote Desktop application (which utilizes Microsoft’s RDP protocol – not RFB) is usually the default choice. And why wouldn’t it be? It’s built into Windows, there is no additional cost, and it’s usually quite efficient (=fast) – even over slower connections.
Remote Desktop has a few disadvantages though, especially when it comes to the IT help desk:
You cannot view the remote user’s current desktop
It’s not cross-platform
You can’t use RDP if it’s disabled or misconfigured
Especially when troubleshooting user problems, being able to see exactly what the user is doing is obviously very beneficial. VNC-based applications are a good alternative since they allow you to view the user’s desktop and subsequently interact with the user. This makes VNC viable for help desk as well as troubleshooting. Nevertheless, VNC-based solutions have their own shortcomings:
Free variations of VNC usually offer no deployment assistance
With over 10 variants available, finding the best VNC implementation is a daunting task
VNC is still deemed as somewhat insecure
VNC can be slow
We set out to solve these shortcomings by creating a number of scripts around UltraVNC that integrate with the EventSentry management console (although they’ll work well without EventSentry as well!). Using the QuickTools feature, you can then connect to a remote host via VNC with 2 clicks, even if the remote host doesn’t have VNC installed.
Important: The scripts only work in environments where you have administrative access to the remote hosts. The scripts need to copy files to the remote host’s administrative shares and control the remote VNC service.
Alternatively, you will also be able to start a VNC session by running the following command:
Even better, VNC can be automatically stopped and deactivated (until vnc_start.bat is run again) once the session is completed in order to reduce the attack surface.
As long as you have administrative access to the remote host(s), the script will remotely install VNC and even setup a firewall exclusion rule if necessary – although the UltraVNC installer takes care of this out of the box.
To reduce the attack surface of machines running VNC you can automatically stop the VNC service after you have disconnected from the remote host. Our connection script will automatically start the remote service again when you connect the next time.
For the utmost security you can also completely uninstall VNC when you are done, a script (vnc_uninstall.bat) is included for this purpose.
Even though VNC is generally not as fast as RDP, it’s usually sufficiently fast in LAN environments (especially for shorter trouble-shooting sessions) and the UltraVNC port which we’ll be covering in this post performs reasonably well even over slower WAN connections.
Integration with EventSentry
Monitoring workstations with EventSentry strengthens the capabilities of any IT helpdesk and IT support team with:
Software & Hardware Inventory
Access to process utilization and log consolidation
Enhanced security with security log & service monitoring
User console logon tracking
Pro-active troubleshooting with access to performance and other system health metrics
Remote desktop sharing is an additional benefit with the UltraVNC package which is included with the latest version of EventSentry (v220.127.116.11). Customizing the scripts and integrating them with EventSentry literally shouldn’t take more than 5 minutes, and once setup & configured will allow you to remotely control any monitored host with a couple of clicks. The scripts do not require EventSentry, but are included with the setup and integrate seamlessly into the EventSentry Management Console.
The EventSentry Management Console includes the “QuickTools” feature which allows you to link up to 8 commands to the context menu of a computer item. EventSentry ships with a few default QuickTools commands, for example to reboot a remote machine. Once configured, you simply right-click a computer icon in the EventSentry Management console and select one of the pre-configured applications from the QuickTools sub menu.
How does it work?
When you run the vnc_start.bat script, it will first check to see if UltraVNC is already installed on the remote host. If it is, it will skip the installation routine and bring up the local VNC viewer. If you configured the script to automatically stop the VNC service when not in use, it will start the service beforehand. When you disconnect, it will (optionally) stop the VNC service again so that VNC is not accessible remotely anymore.
If VNC is not installed, the script will remotely install & configure UltraVNC using psexec.
If you do not want to leave the UltraVNC service installed on the remote computer, the vnc_uninstall.bat script can be run when the remote session is done. Automatically stopping the remote VNC service is however sufficient in most cases.
The scripts need to be configured before they can be used in your environment, unless you are an EventSentry user, in which case you only need to download & install the prerequisites.
Super Quick Setup for EventSentry Users It’s no secret, we’re a little biased towards our EventSentry users, and as such setting this up with an existing EventSentry installation is rather easy:
Get psexec.exe and save it in C:\Program Files (x86)\EventSentry\resources.
Download the UltraVNC installers (they have 32-bit and 64-bit – download for the platforms you have on your network) and store them in the C:\Program Files (x86)\EventSentry\scripts\ultravnc folder.
Install UltraVNC on the computer where EventSentry is installed so that the VNC Viewer is available. It’s not necessary to install the whole package, only the viewer component is required.
If “VNC” is not listed in your QuickTools menu, then you will need to add it under Tools->Options->QuickTools. Simply enter “VNC” as the description and specify the path to the vnc_start utility, e.g. “C:\Program Files (x86)\EventSentry\scripts\ultravnc\vnc_start.bat $COMPUTER”. You can optionally check the “Hide” box to prevent the script output from being shown before you connect.
You’ll notice that no password was configured – that’s because you will be logging in with a Windows user and password – only allowing domain admins access by default. This can be configured in the authorized_acl.inf file, if you want to give additional groups and/or users access that are not domain admins.
That’s literally it – easy as pie. Even though we designed this thing to be easy peasy, since things do occasionally go wrong I recommend testing a first connection from the command line. Just open an administrative command prompt, navigate to C:\Program Files (x86)\EventSentry\scripts\ultravnc and type vnc_start somehost.
Now just right-click any host – or use the “Quicktools” button in the ribbon – and select the “VNC” menu option. Keep in mind that first-time connections will take longer since the VNC setup file has to be copied and installed on the remote computer. Subsequent connections should be faster.
Manual Normal-Speed Setup for Non-EventSentry Users
So you are not an EventSentry user but still want to utilize these awesome scripts? No problem – we won’t hold it against you. The setup is still easy – you’ll just need to customize a few variables in the variables.bat file.
In variables.bat, set the VNCSETUP_X86 and VNCSETUP_X64 to the setup file names you just downloaded.
Download the PSTools and extract psexec.exe into the working directory, or a directory of your choice.
In variables.bat, point the PSEXECFILE variable to the location where you just saved psexec.exe.
Optional: Edit the authorized_acl.inf file to specify which Windows group or user will have access to VNC. You can either change the first line, or add additional lines to give additional users and/or groups permission.
Install the respective version of UltraVNC on your workstation so that the VNC Viewer is available.
Open a command line window and navigate to the folder to which VNCSOURCE points to. Test the setup by running vnc_start hostname, replacing “hostname” with an actual host name of a remote host of course.
When presented with the login screen of the VncViewer, log in with a Windows domain admin user.
That wasn’t so bad now, was it? Just remember that you’ll need to initiate any VNC session with the vnc_start.bat file. Just launching the Viewer won’t work – even if VNC is already installed on the remote machine – since the VNC service is stopped by our scripts by default. To use the folder names we created, you’ll just run
Enjoy, and happy RFBing!
Configuration – variables.bat For the sake of completeness the variables.bat file is explained below:
VNCSETUP_X86: The file name of the 32-bit installer. This needs to only be changed whenever UltraVNC comes out with a new version. VNCSETUP_X64: The file name of the 64-bit installer. This needs to only be changed whenever UltraVNC comes out with a new version.
REMOTEINSTALLPATH: The directory where the script files will be copied to on the remote host.
VNCSOURCE: This is the folder where all the vnc-related files, including the setup executables, are located on the source host from where you initiate VNC connections – e.g. C:\Deployment\UltraVNC. VNCINSTALLDIR: The directory in which UltraVNC will be installed in (on the remote hosts).
VNCPASSWORD: This variable is not currently used since UltraVNC is automatically configured to authenticate against Windows, by default giving only Domain Admins access to VNC. This is generally more secure than using a password. You can edit the file authorized_acl.inf to give additional users and/or groups access to VNC. The file supports one ACL entry per line.
PSEXECFILE: Unfortunately we are not allowed to bundle the nifty psexec.exe file for license reasons, so you’ll have to download the PsTools and point this variable to wherever you end up copying the psexec.exe file to. If you already have psexec.exe installed then you can save yourself 2 minutes of time and just specify the path to the existing file here.
SET_VNC_SVC_TO_MANUAL: If you don’t entirely trust the security of VNC, maybe because you know what a brute force attack is, and you only want administrators to access VNC then you can set this variable to 1. As long as you only connect to the remote host(s) using the vnc_install.bat script, the scripts will ensure that the remote VNC service is started before you connect and stopped after you disconnect. Between the two of us, I’d always leave this set to 1 unless you have the desire to launch the VNC Viewer directly, or need non-administrators to be able to connect to the remote host(s).
ADD_FIREWALL_RULE: As the name (almost) implies, this will create a firewall exclusion rule on the remote host(s) if you’ve been doing your homework and enabled the Windows firewall. If you don’t like our boring firewall rule name then you can even change the name below by editing the FW_RULE_NAME variable. Enabling this is usually not necessary since the UltraVNC setup adds firewall exclusion rules by default.
VNCVIEWER: If you find that a different version of the VNC viewer works better than the version which we are shipping, then you can change the file name here.
In my previous post I talked about our new NetFlow component as well as the new agent management capabilities now available in EventSentry v3.3. In this post I’ll cover the remaining new features and improvements we’ve made in v3.3, starting with the web reports.
There are a number of new features and improvements in addition to NetFlow visualization. There are a few new dashboard tiles, including a “Recent Activity” tile which – as the name implies – shows recent relevant changes such as newly detected processes, software (un)installed, ping status or service status changes.
Anybody who works in a team of two or more Sysadmins should find the new notes feature incredibly helpful. It lets any web reports user add comments (=notes) which are subsequently visible to others. Notes can be associated with one or more hosts (ensuring they show up in the “Documentation” tab of the respective host status page) and can include documents as attachments as well! Do you have warranty documents or network diagrams you want to store in a central place – easily accessible? That’s what the notes are for.
The overall look and feel has also been refreshed, and we’ve reorganized the menu to make it faster to access dashboards and easier to find pages.
The visualization of data has been improved, since some chart types work better with certain features of EventSentry. You can now visualize grouped data using either pie charts, tree maps or column charts.
The security of the web reports has also improved with a lockout policy which will locking an account after too many unsuccessful logon events.
As mentioned in part 1, the EventSentry is agent is now available in 64-bit, making it possible to monitor 64-bit counters and easier to monitor files in 64-bit directories. For users upgrading from an earlier version, the EventSentry management console will automatically migrate any existing 32-bit agents on 64-bit versions of Windows.
Application & Services Event Logs
While monitoring Application & Services event logs, often referred to as “custom” event logs was possible, the way this needed to be configured in the management console was a common source of confusion. Some users also needed the ability to monitor more than 30 different logs. Consequently, monitoring additional event logs is now straightforward, and users can monitor as many event logs as they wish.
With thresholds, timers, schedules, insertion strings, EventSentry already offers a sophisticated engine for monitoring events in real time. New in this release is the ability to setup filter chaining. This makes it possible to trigger actions only when 2 or more events occur, and you can even link events together using insertion strings. Chaining is enabled on the package level, and every filter in a “chaining” package is automatically part of the filter chaining rules.
It happens frequently that we get alerts that require us to do additional research based on the information provided in the alert. For example, we may get an alert about an IP address for which we then need to do a reverse lookup or find the geoip location. Audit Success & Failure events from the security event log are another example, and often contain error codes and numbers which are not explained.
We set out to improve upon this, and starting with v3.3 EventSentry will annotate email alerts in a number of ways whenever possible:
IP addresses will include a reverse lookup
IP addresses will include a geoip location
Security events will have various error codes resolved
Please note that (1) and (2) are only supported for emails sent through the collector since it requires access to a local geoip database. (1) and (2) will need to be enabled in the email action “Options”, (3) is automatically enabled for all emails.
Insertion Strings & Regex
By making insertion strings from events accessible in filters and actions (e.g. through the $STR1, $STR2, … variables), it’s possible to create highly granular thresholds, customize emails, easily trigger corrective actions which utilize content from events and more. Based on our own requirements we took this capability a step further however, and you can now apply regex filters to events to define your own insertion strings. This is particularly useful for alerts which don’t use insertion strings or for events which contain log data. For those types of events, you can now parse parts of log strings and assign them to insertion strings. The previous blog article, Detecting Web Server Scan in Real-Time, shows a practical example of how to apply this new feature. It does require you to be a bit familiar with Regular Expressions, but the management console includes a handy dialog where you can test your regular expressions, shown below.
Faster is better! We’ve improved performance in a number of areas:
The database insert performance of the Syslog daemon has been improved for Microsoft SQL Server databases
The delimited log file feature now includes an additional index to increase database insert performance
The heartbeat agent now relies less on RPC-based agent status monitoring and can instead obtain the status of a remote agent either directly from the collector or the database, resulting in less network traffic and faster heartbeat monitoring cycles.
With new features & improvements in a variety of areas, this release should contain improvements for everyone. Remember that you can also submit feature requests here.
We are very excited to release EventSentry v3.3, a major update to our award-winning monitoring solution EventSentry, less than 10 months after the release of the previous major version 3.2. Version 3.2 included the collector component which supports secure and reliable communication with remote agents as well as better database throughput, switch port mapping and many improvements to the web reports.
I’d like to also thank everyone who took the time to fill out our annual survey – we read every single response in detail. If you haven’t taken it yet then you can still do so here.
The v3.3 release, which builds upon some of the architectural changes we have made in v3.2, and offers new functionality to help you:
Visualize, measure & investigate network traffic better with the new NetFlow component – with discounted introductory pricing until 12/31 2016!
Spend less time managing agents – the collector can now push configuration as well as agent updates automagically – think laptops!
Deployment via MSI is much easier – MSI file creation now only takes a few seconds
Investigate issues faster with email alerts which have geo location, reverse lookups as well complex security codes included inline
Visualize any data in the web reports more easily with additional dashboard tiles and treemaps throughout
Managing and using custom event logs is now more straightforward and scalable
Database throughput has been improved for Syslog data and delimited log files
Even more advanced filtering is possible with filter chaining and insertion string override via regular expressions
Communicating and documenting your network has just become a lot easier – add notes and/or upload documents in the web reports
Monitor 64-bit operating systems with a native 64-bit agent
With a brand new component and many new features in a variety of areas, v3.3 will have something of interest for everyone. Let’s dive in and look at the new features in more detail.
NetFlow NetFlow is a new component which is part of the “Network Services” service (along with Syslog, SNMP, ARP) and is licensed separately. Pricing is very competitive and an additional introductory discount will be available until the end of this year, 12/31 – including competitive upgrades. You can request a quote here.
Collecting NetFlow data allows you to see all traffic meta data which passes through network devices that support NetFlow, including:
Source IP, destination IP
Source host, destination IP (when resolvable)
Source port, destination port
Geo location (when available)
IP protocol used
Amount of traffic sent and received
Number of packets transmitted
EventSentry v3.3 currently supports the NetFlow v1, v5, v9 as well as sFlow flow protocols. NetFlow is usually supported by most commercial routers and firewalls whereas sFlow is most commonly supported by switches. NetFlow is generally preferable over sFlow – especially for forensic analysis since sFlow samples traffic and only sends every nth flow. sFlow can be preferable when dealing with large amounts of data, but EventSentry’s NetFlow implementation (as well as NetFlow itself) has a way to group flows and therefor condense traffic.
Do you need NetFlow, and is it worth looking into? Without NetFlow there is impossible to know which hosts communicate with each other (unless you capture network traffic). What traffic enters the network, and what traffic leaves it? Broadly speaking, implementing NetFlow lets you:
Visualize all network traffic in a variety of ways and reports
Analyze network data for forensic purposes
Utilize network traffic data for troubleshooting purposes
Map network traffic to geo location
Correlate network traffic with Active Directory users (requires workstation monitoring)
Measure bandwidth utilization
On the EventSentry side, setting up NetFlow should take less than 5 minutes; and setting it up on the network device side is generally just a matter of enabling NetFlow and pointing it to EventSentry.
EventSentry ships with the GeoLite geo database from MaxMind which does a good job of associating IP addresses with physical locations down to the city level. If you are looking for more accuracy however, then you can also purchase the full geo location database from MaxMind here.
Active Directory User Correlation
A unique feature of EventSentry’s NetFlow implementation is the ability to correlate workstation logins with network traffic, making it possible to associate network traffic with individual users. This requires that workstations are monitored with EventSentry and works best when users have a dedicated workstation.
Agent Management & Deployment
If you are utilizing the collector service then you have now a great time-saving feature available. Pushing a configuration update to remote hosts after you made a change or deploying agent updates after a patch installation are a thing of the past once you activate the respective options in the collector dialog.
Managing automatic configuration updates can be done in 2 ways: Either by automatically deploying a configuration update after you click “save”, or by deploying only approved configuration updates (recommended). If you select the latter, then you just have to click the new “Save & Deploy” sub-option on the ribbon and the collector will do the rest. It’s no longer necessary that the EventSentry agent is directly reachable from the management console; it will receive the latest configuration as soon as it connects to the collector.
Please note that you will still need to manually deploy a v3.3 agent once in order for automatic agent updates to work, since the self-update code is embedded in the new agents.
Creating MSI files has also been greatly simplified – a x86 and x64 agent MSI file is created with just a few mouse button clicks. Manually editing MSI files with tools like ORCA is a thing of the past. The only prerequisite is the (free) WiX Toolset which has to be installed only once.
In addition to saving most EventSentry users a lot of time, these new deployment features also make it possible to monitor laptops which aren’t permanently connected to the network. Simply deploy the agent MSI file with your favorite deployment tool (or deploy with the management console) and enable the configuration and agent management options in the collector. From that point on, any agent connecting to the collector will automatically receive the latest configuration AND any new agent updates – completely automatically – no matter where in the world they are located.
EventSentry v3.3 now ships with both a x86 and x64 agent, so that 64-bit editions of Windows can be monitored natively. The key benefit of this change is that 64-bit only performance counters can now be monitored, these counters were off limit with 32-bit agents. Utilizing 64-bit agents also results in the following changes:
Agents will be automatically converted to 64-bit when v3.3 is deployed. It is not possible to use a 32-bit v3.3 agent on a 64-bit version of Windows
File system redirection via “Sysnative” or in the File Checksum Monitoring packages is no longer necessary
Memory consumption will be slightly higher compared with 32-bit agents
Please note that EventSentry has not completely migrated to 64-bit yet, some components (management console, heartbeat agent, web reports) are still shipped as 32 bit executables. We plan on migrating all components to 64-bit by the end of 2017.
There are just too many new features in v3.3 to fit them all into one blog post, so stay tuned for part 2 which will follow shortly.
Since Ransomware is still all the rage – literally – I decided to write a 4th article with a potentially better method to stop an ongoing infection. In part 1, part 2 and part 3 we focused mostly on detecting an ongoing Ransomware infection and utilized the “nuclear” option to prevent it from spreading: stopping the “server” service which would prevent any client from accessing files on the affected server.
While these methods are certainly effective, there are other more targeted steps you can take instead of or in addition to shutting down the server service, provided that all hosts susceptible to a Ransomware infection are monitored by EventSentry.
When EventSentry detects an ongoing Ransomware infection, it can usually determine the infected user by extracting the domain user name from the 4663 event. Simply disabling the user is insufficient however, since a disabled user can continue to access the network (and wreak havoc) as long as he or she doesn’t log off. Any subsequent log on attempt would of course fail, but that provides little comfort when the user’s computer continues to plow through hundreds or thousands of documents, relentlessly encrypting everything in its path.
As such, the only reliable way to stop the ongoing infection, given only the user name, is to log off the user. While logging a user off remotely is possible using the query session and logoff.exe commands, I prefer to completely shut down the offending computer in order to reduce the risk of any future malicious activity. Logging the user off remotely may still be preferable in a terminal server environment (let me know if you want me to cover this in a future article).
Knowing the user name is of course great, but how do we find out which computer he or she is logged on to? If you have EventSentry deployed across your entire network – including workstations – then you can get this info by querying the console logon reports in the EventSentry web reports. If you are not so lucky to have EventSentry deployed in your entire environment (we offer significant discounts for large quantities of workstation licenses – you can request a quote here) then we can still obtain this information from the “net session” command in Windows.
We’ve created a little script named antiransom_shutdown.vbs which, given a user name, will report back from which remote IP this user most recently accessed the local server and optionally shut it down. Here are some usage examples:
Find out from which computer boris.johnson most recently accessed this server: cscript.exe C:\Scripts\antiransom_shutdown.vbs boris.johnson
Find out from which computer boris.johnson most recently accessed this server AND shut the remote host down (if found): cscript.exe C:\Scripts\antiransom_shutdown.vbs boris.johnson shutdown
The script uses only built-in Windows commands, as such there is no need to install anything else on the server where it’s run.
When executed with the “shutdown” parameter, the script will issue a shutdown command to the remote host, which will display a (customizable) warning message to the user indicating that the computer is being shutdown because of a potential infection. The timeout is 5 seconds by default but can be customized in the script. It’s recommended to keep the timeout short (5-10 seconds) in order to neutralize the threat as quickly as possible while still giving the user a few moments to know what is happening.
The overall setup of the Ransomware detection is still the same, we’re setting up a threshold filter to detect a higher than usual frequency of certain 4663 events and trigger an action in response. Only this time we don’t shut down the server service, but instead trigger this script. To properly execute the action, configure it as shown in the screenshot below. The executable is cscript.exe (the interpreter for .vbs files) and the command line parameters are the name of the script, $STR2 and “shutdown”.
So what’s the better and safer approach to freeze an ongoing Ransomware infection? Shutting down the server service is the most reliable approach – since it doesn’t require the workstation to be reachable and will almost certainly succeed. Remotely shutting down a workstation has minimal impact on operations but may not always succeed. See below for the pros and cons of each approach:
File Sharing Shutdown Pros: 100% effective Cons: Potentially larger disruption than necessary, false positive unnecessarily disrupts business
Remote Workstation Shutdown Pros: Only disables infected user/workstation, even if false positive Cons: Requires workstation to be reachable
This ends up being one of those “it depends” situations where you will have to decide what’s the best approach based on your environment. I would personally go with the remote workstation shutdown option in large networks where the vast majority of workstations are desktops reachable (and not firewalled) from the file server. In smaller, more distributed networks with a lot of laptops, I would go with the file service shutdown “nuclear” option.
A hybrid approach may also be an option for those opting for the remote workstation shutdown method: trigger a remote workstation shutdown during business hours when IT staff is available on short notice, but configure the file service shutdown after business hours when it’s safer and affects fewer people. All this can be configured in EventSentry by creating two filters which are identical except for the action and the day/time settings.
Prerequisites It’s important to point out that the EventSentry agent by default runs under the LocalSystem account, a built-in user account which does not have sufficient privileges on a remote host to issue the shutdown command. You can elevate the permissions of the EventSentry agent and work-around this limitation in 2 ways:
Change the service account (fast): Changing the service account the EventSentry service uses to a domain account with administrative permissions will allow the agent to remotely shut down a remote host. This will have to be done on every file server which may issue shut down commands (you can use AutoAdministrator to update multiple file servers if necessary).
Give the “Force shutdown from a remote system” user right: It’s not necessary to issue domain-wide admin rights to the EventSentry agent, the key right the agent needs is just the “Force shutdown from a remote system” user right. The quickest way to deploy this setting is of course through group policy:
a) Open the “Group Policy Management Editor”
b) Edit an existing policy (e.g. “Default Domain Policy”) or create a new group policy
c) Navigate to “Computer Configuration\Policies\Windows Settings\Security Settings\Local Policies\User Rights Assignment”
d) Double-click the “Force shutdown from a remote system” user right and add both “Administrators” and the computer accounts of the file servers to the list. Alternatively you can also create a group, add the file servers to the group, and add that group to the policy (keep in mind that you will need to restart the file servers if you go with the group method).
Once the group policy setting has propagated to the workstations, the remote shut down initiated from the file server(s) should succeed.
Good luck protecting your network against Ransomware infections, also remember to verify your backups – no protection is 100% effective.
Dashboards are a great way to visualize large amounts of information in a concise matter. In IT we usually display various types of network data from a monitoring software, but dashboards are used in all sorts of environments. You can visualize stock data or just show a map of all trucks in a fleet with their current position.
If you work for a large company with a dedicated NOC then you’ll likely have an integrated setup with 4 or more TVs, connected to hardware specialized for dashboards or, at the very least, a powerful PC with multiple PCI cards.
But not everybody has the budget or the need for a NOC like AT&Ts, and one or two TVs can be sufficient for most networks – provided the dashboard is well-designed and customizable of course.
Most dashboards require a fairly recent web browser (if you are unlucky even Adobe Flash), making some sort of a PC or Mac the preferred hardware to power that dashboard. Most IT departments have a plethora of old PCs sitting around, and it can be tempting to resurrect one of those boxes and give them a new life as a dashboard PC. After all, you’re “just” displaying a web page.
Low-cost integrated devices like the Raspberry Pi are tempting, but not perfect either. They’re not usually designed to be used with graphical interfaces, much less with memory and CPU hungry applications like web browsers displaying dashboards.
After trying everything from Raspberry Pi, old Mac Mini hardware and more, we finally found a solution for under $100 – which has now worked quite well for several months: The 1st generation Intel Compute Stick which you can get from online retailers like Amazon, NewEgg and others.
Even in its 1st generation (the one we tested) the Intel Compute Stick running Windows 10 Home performed surprisingly well. We’ve been running an EventSentry dashboard (which of course we’re hoping you are running as well) on it since February on Microsoft’s new Edge browser, and we’ve never had an issue.
The Intel Compute Stick features 2 Gb of RAM, is powered by a quad-core intel Atom processor and has 30 Gb of storage, of which more than half are available. This is of course not a machine you’ll want to render videos or play video games on, but plenty sufficient for a web browser from our experience. We were actually pleasantly surprised by how responsive the little device felt overall. Even though you cannot join a domain, you can still install the EventSentry agent on the machine to keep an eye on performance and other system metrics for example.
But there are of course some caveats, as is to be expected from a computer that costs less than $100 and is not much bigger than a USB memory stick. If you’re using Bluetooth and Wifi then you’ll only need to connect the power cord and the setup is clean. Since the stick also sports a single USB 2.0 port, we used a USB hub along with a USB-based Ethernet adapter to connect it to our LAN as well as connect a keyboard/mouse. USB 2.0 didn’t negatively affect performance in our limited use case scenario.
If you need more hardware, maybe because your dashboards are particularly taxing, then you can purchase a newer and faster model as well. The 2nd generation Intel Computer Sticks start around $149 and the high-end models include as much as 64Gb of disk space and 4Gb of RAM.
My first computer was a 80286 with 1Mb of RAM and a 20Mb hard drive, and it was about as big as two shoe boxes. It’s impressive to see a device this small perform that well. If you have the need to turn a TV into a full-blown desktop, then I’d definitely recommend the Intel Computer Stick(s)!
Our latest patch for EventSentry v3.2 (v18.104.22.168) requires some additional information in addition to the release notes.
Heartbeat Monitoring (Agent Status) By default, the EventSentry Heartbeat Monitor ensures that all remote agents are running by querying the status of the remote “EventSentry” service. While is an accurate way to ensure the remote agent is running, the Microsoft RPC mechanism isn’t very efficient when connecting to remote hosts across a slow (WAN) link, and concurrently checking the service status of 100+ hosts at the same time can on occasion also cause issues. In these situations, the heartbeat agent may not be able to monitor all hosts in the configured monitoring interval. Furthermore, querying the remote status of a service requires that the EventSentry Heartbeat Agent run under a domain account, otherwise the dreaded “Access Denied” error appears on the heartbeat status page in the web reports.
To address these issues for larger EventSentry deployments (500+ hosts) and deployments where the remote agents are connected through a slower WAN link, we have added the ability to query the remote agent status through the EventSentry database where the remote agents periodically check in. This check is enabled by default for new installations, but existing installations will need to make a database permission change in order to give the heartbeat agent permission to query the agent status. More information can be found here.
In the next release of EventSentry (v3.3), this functionality will be configurable, and the heartbeat agent will also be able to determine the current agent status by communicating directly with the collector service (when enabled) for even better accuracy. The Heartbeat Monitor will always attempt to revert back to the legacy method of checking the service status directly if it cannot obtain the status through other means.
Service Monitoring: Configuration Changes EventSentry distinguishes between three types of service changes: Status changes (e.g. Running to Stopped), service configuration changes (e.g. changes to the startup type) and services being added or removed. Up until release 22.214.171.124, all status changes and service configuration changes were logged with the same event severity, which we didn’t think was very fitting since the status change of a service is very different to a change of the service itself. As such, starting with 126.96.36.199, only service status changes will be logged under the severity configured under “Monitor Service Status Changes” category. All other service changes will be logged under the severity configured under “Monitor Service Addition / Removal” category.
Management Console: Quicktools
The EventSentry QuickTools allow you to run an application/script against a server or workstation in your EventSentry configuration. EventSentry includes a few default QuickTools entries, such as “Reboot”, “Remote Desktop” and others. Starting with the latest release we added a new “Hide” option, which will not show the executed application on the desktop. This will be useful for integrating our upcoming VNC wrapper scripts (Blog article coming soon), which will allow you to install & launch a (Tiger)VNC client directly from the EventSentry management console.
EventSentry Light 3.2 Starting with this release, EventSentry Light v3.2 will also be available. We have good news for all EventSentry Light users: We have increased the number of full hosts you can remotely manage to 5, and also increased the number of network devices you can monitor to 5. As such you can now monitor up to 10 hosts with EventSentry Light completely for free.
There seems to be a new variant of ransomware popping up somewhere every few months (Locky being the most recent one), with every new variation targeting more users / computers / networks and circumventing protections put in place by the defenders for their previous counterparts. The whole thing has turned into a cat and mouse game, with an increasing number of software companies and SysAdmins attempting to come up with effective countermeasures.
I’ve already proposed two ways to counteract ransomware on file servers with EventSentry in part 1 and part 2, both of which take a little bit of time to implement (although I’d argue less than it would take to restore all of your files from backups). The last part of the series, remediation, offers a way to remotely log off an infected user. In this post I’m proposing a third, and better, method with the following improvements:
In the first article we configured file integrity monitoring on a volume, and if the number of file modifications occurring during a certain time interval exceeded a preset threshold, the ransomware would be stopped in its tracks. In the seconds article we used bait (canary) files to accomplish the same thing.
In this third installment we’ll keep track of the number of file modifications made by a user to detect if an infection is underway. To effectively defeat ransomware, we have to be able to distinguish between legitimate user activity and an infection. To date we know this:
Users add/change/remove files, but the number of changes made by a user in a short amount of time (say 15 min) is generally small
Ransomware always runs in the context of a user, and as such an infection will usually come from one user (unless things go really awry and multiple users are infected). The approach here will work equally well, regardless of the number of infections.
Thus, to detect an infection, EventSentry will be counting the number of file modifications (event 4663) with its advanced threshold capabilities. If the threshold is exceeded, EventSentry will trigger an action of your choice (e.g. disable the user, remove a file share, stop the server service, …) to limit the damage of the ransomware.
Here is what you need:
Object Access / File System Auditing enabled
Auditing enabled on the files which are to be protected
EventSentry installed on the server which needs to be protected
This KB article explains how to configure EventSentry and enable auditing (preferably through group policy) on one or more directories. I recommend referencing the KB article when you’re ready to configure everything. Pretty much everything in the KB article applies here, although we will make a small change to the threshold settings of the filter (last paragraph of section (4)).
Once auditing is setup, Windows will log event 4663 for every write access which is performed by a user. An example event looks like this:
The default behavior of a filter threshold in EventSentry is to simply count every filter match towards the threshold. In our case, every 4663 event encountered would count towards the threshold. You can think of there being one bucket for all 4663 events, with the bucket being emptied whenever the threshold period expires, say every 5 minutes. If the bucket fills up we can trigger an alert.
This doesn’t work so well on a file server, where potentially hundreds of users are constantly modifying files. It would take some time to come up with a good baseline (how many file modifications are considered “normal”) that we could use as a threshold, and there would still be a chance for a false positive. For example, a lot of 4663 events could be generated during a busy day at the office, thus causing the threshold to reach its limit.
A better way is to assign each user their own “personal” threshold which we can then monitor. Think of it like each user having their own bucket. If a user writes to a file, EventSentry adds the 4663 event only to that user’s bucket. Subsequently, an alert is only triggered when a user’s bucket is full. Any insertion string of an event can be used to create a new bucket.
We can do this by utilizing the insertion string capabilities of the filter threshold feature. Setting this up is surprisingly easy – all we have to do is change the Threshold Options to “Event”, click the “Insertion Strings” button and select the correct insertion string. What is the correct insertion string? The short answer is #1.
The long answer lies in the “Event Message Browser”, which you can either find through the Tools – Utilities menu in the EventSentry Management Console or in the EventSentry SysAdmin Tools. Once in there, click on “Security”, then “Microsoft-Windows-Security-Auditing”, then 4663. You will see that the number next to the field identifying the calling user (“Security ID”) is %1.
Enough with the theory, here is what you need to implement it (assuming EventSentry is already installed on the servers hosting the file share(s)):
Create a package & filter looking for 4663 events. See section 4 of KB 279 and review the additional threshold settings below.
Customizing the threshold Once you have the package & threshold filter for 4663 events in place, we need to modify the threshold settings as explained above. Edit the filter, click the threshold tab and make sure your filter looks like the one shown below:
The only variable setting is the actual threshold, since it depends on how fast the particular variant of ransomware would be modifying files. A couple of things to keep in mind:
The interval shouldn’t be too long, otherwise it will take too long before the infection is detected.
Make sure the actual event log filter is only looking at 4663 events, no other event ids.
With the above example, any user modifying any file (on a given server) more than 30 times in 3 minutes will trigger any action associated with the filter, e.g. shutting down the server service. Note that the action listed in the General tab will be triggered as soon as the threshold is met. If 30 4663 events for a single user are generated within 45 seconds, the action will be triggered after 45 seconds, it won’t wait 3 minutes.
Bonus – Disabling a user One advantage of intercepting 4663 events is that we can extract information from them and pass them to commands. While shutting down the Server service is pretty much essential, there are a few other things you can do once you have data from the events, e.g. the username, available. You can now do things like:
Disabling the user
Removing the user from the share permissions
Revoking access to select folders for the user
There are a couple of caveats when (trying to) disable a user however:
The user account (usually the computer account) under which the EventSentry service runs under (usually LocalSystem) needs to be part of the Account Operators group so that it has permission to disable a user
Disabling a user is usually not enough though, since Windows won’t automatically disconnect the user or revoke access. As such, any ransom/crypto process already running will continue to run – even if the user has been disabled.
Disabling a user account from the command line is surprisingly simple (leave Powershell in the drawer). To disable the user john.doe, simply run this command:
net user john.doe /domain /active:no
Note that since “net user” doesn’t support a domain prefix (MYDOMAIN\john.doe won’t work), we need to make sure that we pass only the username (which is insertion string %2) and the /domain switch to ensure the user is disabled on the domain controller. Of course you would need to omit the /domain switch if the users connecting to the share are local users. The action itself would look like the screenshot below, where $STR2 will be substituted by EventSentry with the actual user listed in the event 4663:
That’s it, now just push the configuration and you should be much better prepared to take any ransomware attacks heading your users way.
I am e-x-c-i-t-e-d to announce the availability of EventSentry v3.2 and tell you more about the new features and improvements. So, if you’re looking for a little bit more than the release notes then read on!
Collector The biggest new feature in 3.2 is the collector, a new central component which enables a 3-tier architecture in EventSentry. Traditionally, the EventSentry agents have been communicating directly with email servers, databases and other services. While this usually worked well – and is still desirable in many setups – it does impose a limitation in some scenarios:
The SMTP server cannot be configured to allow relaying and/or accepting SMTP connections from remote clients
The central database cannot be configured to allow connections from remote clients
Agents need to communicate over an insecure medium like the Internet
The collector addresses the above limitations by acting similar to a proxy between the remote agents and the service (e.g. database). In a nutshell, it provides the following benefits:
Agents only communicate with the collector over a single port
All traffic can be encrypted and compressed
Database connection details do not need to be stored on the agents anymore
All collected data is cached on the agents if and while the collector is unreachable
Whether you will need the collector or not will largely depend on your network setup. If all of your hosts are in the same data center and/or the same LAN, the collector may provide little benefit. If you are a MSP and monitoring remote sites and laptops however, then the collector is probably what you have been waiting for!
When upgrading (or installing from scratch), the post-installer configuration assistant will ask you whether you are interested in enabling the collector.
If you are installing from scratch, then enabling the collector during the installation is all you need to do. When upgrading, an additional step is required – an action needs to be configured to use the collector. While the collector service is installed & started during the upgrade when selected, it will not enable any of the existing actions to use the collector. As such, if you want to route data for a specific action through the collector, that needs to be configured. Simply edit the action and click the “Use Collector” check box on the bottom left and push the configuration.
In version 3.2.1, the following actions can be routed through the collector:
Since the collector, when enabled, is a critical component, we recommend monitoring the collector stats either through the collector status page (Maintenance -> Collector Status) or by adding the collector status tile to one of your dashboards.
There is one other advantage the collector can bring when routing emails through it:
Emails from multiple hosts can be grouped together (if the action polling interval is sufficiently high)
Action thresholds can now be applied centrally
Both features can help reduce the number of emails you receive from EventSentry, usually a popular thing to do!
EventSentry has always included the compliance tracking components which monitor and interpret Windows security events. Compliance tracking provides process, console, account management and other tracking reports. While popular and extremely useful, the compliance reports themselves don’t tell the user which particular compliance requirement they address.
Say Hello to the new compliance modules, which provide detailed, out-of-the-box reports for:
Once a compliance module is enabled, it will install a number of reports that pertain to the specific compliance requirement that was enabled. Every report will be associated with a specific control (e.g. PCI 10.2.2) and allow you to setup a required review, job and more.
(Network) Switch Mapping Finding the port on a switch to which a server, workstation or network device is connected is often a time-consuming and annoying process for most SysAdmins. Starting with version 3.2, EventSentry tries to ease that pain by showing exactly to which switch – and port – a host is connected to. All you need to do is add the switch to the EventSentry configuration, make sure that it can be monitored via SNMP and that it provides the MAC to port mappings via SNMP (OID 188.8.131.52.184.108.40.206.3.1.2 – iso.identified-organization.dod.internet.mgmt.mib-2.bridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbPort). This feature should work well with all mainstream managed switched, and we haven’t run into a switch yet where this feature wasn’t provided or did not work.
Once EventSentry pulls the MAC to Port mappings, you will be able to retrieve the collected information in two ways:
Through the Inventory – Switch page, which will show all monitored switches and connected devices
Through the Inventory – Host page. If the switch port can be detected, it will be displayed next to the IP address of the network card
Since switches only provide MAC addresses, EventSentry attempts to map MAC addresses to host names and IP addresses by analyzing the hardware inventory details as well as the ARP status table when available. As such, it is recommended to enable the ARP component of the network services if the results on your switch inventory page are incomplete.
Web Reports Improvements
Starting with a visual overhaul of the interface, the web reports also received an internal overhaul to improve overall performance, especially when using multiple profiles. The performance trends page can now display multiple charts on a single page, and the host inventory page now shows the highest supported USB version on that host.
Managing multiple reports is easier now through the ability to bulk-edit reporting settings. Reports can now also be saved to a folder instead of being emailed.
Finally, the web reports are now also officially available in 6 additional languages: French, Spanish, Polish, Portuguese, Dutch and Italian. This brings the total number of supported languages in the web reports to 9!
Management Console Improvements
Improvements in the management console pertain mostly to remote update and computer management. Hosts can now be imported from a network scan, which is particularly useful when managing network devices which often don’t show up in Active Directory. The network scan is multi-threaded and can scan a class C subnet in a few seconds and even supports checking TCP ports for hosts which have ICMP disabled.
Remote update can now store the result of every activity in CSV file(s), and output from remote update can be toggled with the context menu to apply remote update actions to a sub-set of hosts easily.
Also new is the ability to export all event log filters to a CSV file allowing you to analyze the results in your favorite spreadsheet application to identify issues, duplicates etc.
I’m excited to announce a new version of our free EventSentry SysAdmin Tools which, in addition to bug fixes and minor improvements, also includes a new command-line tool: snmptool. This brings the total number of utilities in the toolkit to thirty (30)!
Free SNMP tools for Windows® are not easy to find and often require you to memorize the various OIDs in order to test a remote host’s SNMP functionality, or to get useful information back.
Our free snmptool utility solves that problem by giving you a simple utility which downloads a variety of stats, depending on what the remote host provides via SNMP, and displays it to the user. For example, if you are querying a VMWare® ESXi™ host with the snmptool, it will – among other stats – enumerate all VMs configure on the host, whereas it will display switch port mappings when querying a switch.
The snmptool currently retrieves the following:
System Description string
Current CPU usage
Network interfaces (name, MAC address, IP if available)
Virtual Machines (ESXi™ only)
Switch port mappings
Running the utility is incredibly easy, simply specify the SNMP credentials and the remote host, and the utility will do the rest on its own:
C:\>snmptool /u public linuxserver
System Description: Linux openvas.netikus.local 4.32.22-573.7.1.el6.x86_128 #1 SMP Tue Sep 22 22:00:00 UTC 2019 x86_128
OS Info: Linux 4.32.22-573.7.1.el6.x86_128 #1
Current Uptime: 3 years, 321 days, 3 hours and 52 minutes
CPU Usage: 0%