Securing Exchange Server OWA & ActiveSync – Proactive Security with EventSentry

Almost every company which runs Microsoft Exchange Server needs to make port 443 available to the Internet in order to provide their users access to email via their mobile devices or OWA.

Since both OWA & ActiveSync utilize Active Directory for authentication, exposing OWA/ActiveSync to the Internet indirectly exposes Active Directory as well. While user lockout policies provide some protection against brute force attacks, additional protection methods should be employed. Furthermore, password spraying attacks may be use to circumvent lockout policies – something that would be more likely to succeed in larger organizations.

With the proper auditing enabled (Logon/Logoff – Logon (Failure)) and EventSentry installed however, we can permanently block remote users / hosts who attempt to log on too many times with a wrong password. Setting this up is surprisingly simple:

  1. Windows: Enable (or verify) Auditing
  2. EventSentry: Setup action which creates firewall block rule
  3. EventSentry: Setup filter looking for 4625 Audit Failure events

Bonus: This procedure works with the free version of EventSentry (EventSentry Light) and can be applied to any IIS-based web site which uses authentication.

Windows Auditing

In the group policy settings that affect the server running OWA, make sure that auditing for Failure events in the Audit Logon sub category of the Logon/Logoff category is enabled (of course you can audit success events as well). If you are running the full version of EventSentry v3.4 or later then you can verify all effective audit settings on the Audit Policy Status page for example.

Enabling correct auditing for logon failures

Creating an Action

Since event 4625 contains the IP address of the remote host, the easiest way to subsequently block it is to run the netsh command. In the management console, create a new action by clicking on the “Action” header in the ribbon and selecting the process action as its type. See the screenshot below:

Triggering netsh to block an IP address

The following command line will work in EventSentry v3.4 and later:

advfirewall firewall add rule name="$STRIpAddress $YEAR-$MONTH-$DAY -- automatic block by EventSentry" dir=in interface=any action=block remoteip=$STRIpAddress/32

If you are running EventSentry v3.3 or earlier then you will need to use the $STR20 variable instead:

advfirewall firewall add rule name="$STR20 $YEAR-$MONTH-$DAY -- automatic block by EventSentry" dir=in interface=any action=block remoteip=$STR20/32

The difference here is that v3.4 and later can refer to insertion string variables by name, making the action more universal and potentially applicable to any event that uses the same field name.

When this action is triggered, it will extract the IP address from the event and block it from the system entirely.

Creating a Filter

Create an event log filter which matches Audit Failure events from the Security event log with event id 4625, where insertion string 19 matches the w3wp.exe process (C:\Windows\System32\inetsrv\w3wp.exe). This ensures that only users accessing the host via the web will be subject to blocking. The screenshot below shows the configuration:

Adding a firewall rule from a 4625 audit failure event

This filter can either be added to an existing package or added to a new package that is assigned only to the Exchange server. If the filter is added to an existing package that applies to servers other than the Exchange server, then the computer field of the filter can be used to ensure the filter is evaluated only on the desired host. Select the action created in the previous step.

Since users may occasionally enter an incorrect password I recommend setting up a threshold so that remote IPs are only blocked after 3 or more failed logon attempts. Threshold are configured by clicking on the “Threshold” tab (see the blue “i” above) and an example configuration is shown below. Feel free to adjust the threshold to match your users ability to enter their password correctly :-). Insertion string 20 – which represents the IP address of this event – was selected in the threshold matching section to ensure that each IP address has its own, unique threshold. Note: The event logging settings shown are optional.

Trigger process after 3 failed logon attempts.

Save/deploy or push the configuration to the mail server.

 

Considerations

Triggering a system process from external input is something we should always do with caution. For example, if Windows has an upper limit to the maximum number of rules that can be added, then an attacker could launch a DoS attack IF they had the ability to launch attacks from different IP addresses. Launching a DoS attack from the same IP won’t be possible once they are blocked. You can mitigate this risk by applying a threshold to the EventSentry action calling netsh.exe, for example by limiting it to 100 / hour. This would still provide sufficient protection while also ensuring that only 100 rules could be added per hour (thresholds can be set by clicking on the “Options” button on an action). A regular audit of the netsh execution (e.g. via Process Tracking) would quickly show any sort of abuse.

Over time the number of firewall rules added to the mail server could become rather large, which is why the rules are created with a date appended. This makes managing these rules easier, and the name can also be adapted in the action by changing the “rule name” parameter. The screenshot below shows the inbound firewall rules after two IPs have been blocked:

List of inbound firewall rules

If manual cleanup of firewall rules is not desirable or an option, then the netsh command can also be wrapped into a script which would erase the firewall rule again after a timeout (e.g. 15 minutes). The script could look like this:

"C:\windows\system32\netsh.exe" advfirewall firewall add rule name="%1 %2 -- automatic block by EventSentry" dir=in interface=any action=block remoteip=%1/32
timeout /t 900
"C:\Windows\system32\netsh.exe" advfirewall firewall delete rule name="%1 %2 -- automatic block by EventSentry"

In this case you would call the wrapper script instead of the netsh.exe process directly (General Options – Filename) and use the string below as the arguments:

$STRIpAddress $YEAR-$MONTH-$DAY

To keep things simple you can just make the script an embedded script (Tools menu) and reference the script. The timeout value (120 in the above example) is the duration seconds the remote IP will be be blocked. If you want to block the IP for an hour then you would set the timeout value to 3600 instead. When going this route I strongly suggest clearing both event log check boxes in the Options dialog of the action.

Happy Blocking!

Auditing DNS Server Changes on Windows 2008/2008R2/2012 with EventSentry

In part one of this blog series we showed how to monitor DNS audit logs in Windows 2012 R2 and higher with EventSentry.

Before I continue I need to point out that DNS auditing has become significantly easier starting with Windows 2012 R2. Not only is it enabled by default, but the generated audit data is also much more granular and easier to interpret. The logged events even distinguish between regular and dynamic updates, making it easy to filter out noise. So if you’re serious about DNS auditing and have the option to update then I recommend you do so.

If you’re running Windows 2008 (R2) or 2012 then setting up DNS auditing requires a few steps. Thankfully it’s a one-time process and shouldn’t take more than a few minutes. On the EventSentry side a pre-built package with all the necessary rules is available for download and included with the latest installer.

Please follow the steps outlined below exactly as described, auditing won’t work or will be incomplete if these steps aren’t followed exactly as described below.

Enabling Directory Service Auditing

Enabling Sub Category Auditing

We first need to make sure that the new subcategory-based audit settings are enabled in group policy. If you’ve already done that, then you can skip this step and jump to “ADSIEdit”.

Since most of the steps here involve domain controllers, I recommend that you make the changes in a group policy, e.g. in the “Default Domain Controller” policy. In Group Policy Management, find an existing group policy, or create a new one, and set Computer Configuration\Polices\Windows Settings\Security Settings\Local Polices\Security Options\Force audit policy subcategory settings (Windows Vista or later) to override audit policy category settings to Enabled.

Then, navigate to Computer Configuration\Polices\Windows Settings\Advanced Audit Policy Configuration\Audit Policies\DS Access and set Audit Directory Service Changes to Success.

ADSI Edit

Open ADSI Edit via Start -> Run -> “adsiedit.msc”. If your default naming context does not automatically appear OR if the listed naming context does not include dc=domaindnszones, then select “Action -> Connect To” and connect to the appropriate naming context, e.g.

dc=domaindnszones,dc=yourdomain,dc=com

Replace the dc components after “domaindnszones” with the actual DNS name of your domain. It’s important that “dc=domaindnszones” is part of the naming context.

Once connected, expand the naming context and locate the “CN=MicrosoftDNS” container, right-click it, and select Properties. Then select Security, Advanced, Auditing and click on Add. In the resulting dialog we’ll audit the built-in “Everyone” user so that DNS changes from everyone are audited:

Name (Principal): Everyone
Apply onto:       This object and all descendent objects
Access:           Create dnsZone objects

 

Enabling auditing with ADSI Edit (1)

It may seem tempting to also check the “Delete dnsZone objects”, but resist the temptation. Don’t be fooled by the term “dnsZone”, the ACE entry we just added will audit the creation of all AD DNS objects (and not just DNS zones) and log event 5136 to the security event log. In order to audit deletions as well, click “Add” again but this time configure the dialog as shown below:

Name (Principal): Everyone
Apply onto:       Descendent dnsZone objects
Access:           Delete
Enabling auditing with ADSI Edit (2)

It’s important that the “Apply onto” is changed to Descendent dnsZone objects. This ACE entry will result in event 5141 being logged when a DNS-related directory service object is deleted. This is were things get a bit interesting though, since DNS records deleted from the DNS manager aren’t actually deleted. Instead, they are tombstoned (which is done internally by adding the dNSTombstoned attribute to the object). Only when the tombstoned object expires is it actually deleted. You can use ADSIEdit if you want to send a DNS object immediately to heaven and skip the graveyard stage.

Unlike Windows 2012 R2 and later, earlier versions of Windows are a little more verbose than you probably like when it comes to directory service auditing. For example, creating a new DNS A record in a zone will result in 4 different events with id 5136 being logged – and not just one. The events logged when adding or deleting a zone or A record are shown in the diagram below:

Directory Service Changes events

All events are logged under the “Directory Service Changes” category.

Testing

Before we start configuring EventSentry, we’ll want to make sure that auditing was setup correctly. On a domain controller, open the “DNS” application and either temporarily add a new A record or primary zone. You should either see a 5136 or 5137 events with the category “Directory Service Changes” logged to the security event log.

If you don’t see the events then walk through the above steps again, or reference this Microsoft article.

Configuring EventSentry

There are generally two things one will want to do with these audit events – store them in a database, email them or both. If you’re already consolidating Audit Success events in an EventSentry database then you shouldn’t have to do anything, all directory service change events will be written to the database automagically.

Database

If you only want to store directory service change events in the database (opposed to storing all audit success events), then you can simply create an include filter with the following properties:

Log:      Security
Severity: Audit Success
Source:   Microsoft-Windows-Security-Auditing
Category: Directory Service Changes

… and assign your database action to it.

Alerts

Setting up alerts using filters for when zones or records are added or removed is a little more involved than one would hope, thanks to Windows logging more than one event whenever such a change is done – as is depicted in the image above. For example, creating a new DNS record will result in 4 x 5136 events being logged, deleting will result in 3 events.

Lucky for you, we’ve analyzed the events and created a DNS Server Auditing package in EventSentry which will email you a single alert (=event) whenever a record or zone are added or removed. This package is included with all new installation of EventSentry, existing users running 3.4 or later can get it through the package update feature in the management console.

DNS Server Auditing Package
DNS Server Auditing Package

And that’s really all there is to it … if you have EventSentry installed then setting up auditing in Windows is really the only obstacle to auditing your DNS records!

EventSentry v3.4: New Security Features, Software Version Checker, Better Performance & more!

We’re again excited to announce the availability of EventSentry v3.4, the latest release of our hybrid SIEM monitoring suite.

EventSentry v3.4 delivers a number of new features to

  • Protect yourself against ransomware attacks
  • Detect lateral movement on a network with collector thresholds
  • Identify outdated software on your network
  • View detailed bandwidth utilization (requires NetFlow)
  • Monitor attached UPS devices
  • Integrate with open source solutions (Graylog, ELK, Nagios Log Server & others)

and more. We’ve also been focusing on making the data EventSentry collects more actionable and subsequently more helpful, and as result you will see additional contextual data provided with some alerts & reports, and one new search page in EventSentry.

All in all, this upcoming release has a lot of improvements in store to help you do your job better by ensuring that your network is as reliable, secure and fast as possible.

Audit Subcategories with audit success enabled, grouped by host

Ransomware

While high-profile Ransomware attacks have slowed down somewhat in recent weeks, businesses – especially small businesses – are still hit with Ransomware infections every day. Even though EventSentry is not positioned as a AV or a AntiMalware software, it does include a variety of functionality which can detect Ransomware outbreaks.

What sets EventSentry apart from AV and most AntiMalware solutions is that it looks for pattern exhibited by the Ransomware – regardless of the variant. What’s new in version v3.4 is the ability to detect changes to the MBR and bootsector as well as the ability to calculate the entropy of (changed) files.

MBR/BootSector Monitoring & Backup
Some more recent Ransomware variants modify the MBR and/or boot sector and trigger a reboot to initiate an offline encryption process. EventSentry v3.4 can now monitor the MBR and detect changes in near real-time to alert the admin when suspicious activity is occurring.

By utilizing EventSentry’s advanced filtering engine it is also possible to potentially stop the encryption process, e.g. by hibernating the infected host. EventSentry even backs up the MBR and boot sector, making it possible to repair an infected system (with a boot disk) without having to perform a full restore from backup.

MBR & Bootloader Backup

File Entropy
Entropy describes the randomness of a file, essentially a metric that can help detect compressed and encrypted files.

Since most Ransomware encrypts large amounts of files, EventSentry can utilize the entropy of a file, combined with event log thresholds, to make a determination that a Ransomware infection is in progress and take corrective measures.

Lateral Movement Detection with Thresholds

Lateral Network Movement
Lateral movement through a network

EventSentry has always included agent-side thresholds in order to throttle the alert volume or detect repeated events. Because these thresholds were executed on the agents, event patterns which involved more than one host could not be detected that way.

By adding a threshold component to collector – which analyzes and processes all events in real time – we can leverage this feature to new heights and detect network-wide event-based patterns – in real time!

Collector-side thresholds are configured exactly like agent-side thresholds with one the key difference – the threshold limit applies to all aggregated events sent to the collector. Collector-side thresholds also introduce the “Group By” setting that makes the lateral movement detection possible – the ability to count unique instances of an event property, and not just the total number of events.

 

 

Some of the event-based threat patterns you can detect:

  • The same user logging on to multiple hosts within a specific time frame
  • A process spreading (trickling) across multiple hosts within a specific time frame
  • A user running too many processes – either on a single or multiple hosts
  • Authentication failures of a user on too many hosts
  • Too many unique logon types used by a user account

Any event property and insertion string can be used to craft thresholds – the sky is the limit.

Software Version Checker

Earlier versions of EventSentry include a substantial hardware, software and patch inventory, making it extremely easy to find out which software packages are installed on your network, but also get alerted when software is installed or removed from a server/workstation.

In v3.4 we are taking this to the next level by providing the latest version available from the publisher for a growing list of 100+ software packages so that you can effortlessly identify outdated software on your network. You can now create reports listing any software on your network which is outdated, a dashboard tile is also available. The Web Reports refresh the version info list every 2 hours to ensure all reports are accurate.

If an application you are using on your network is not currently included then simply email us the name of the software as it is detected by EventSentry (and ideally the URL where we can obtain the current version) and we will add it to our list.

Software Version Check Report

64-bit Web Reports for Windows

The EventSentry web reports are now available as a 64-bit application, and upgrading to v3.4 will automatically upgrade the existing v3.x 32-bit web reports to 64-bit on 64-bit when installed/upgraded on a 64-bit version of Windows. The new 64-bit web reports will allow you to run larger reports that would not run due to limits with the address space associated with 32-bit applications.

UPS & Battery Monitoring

Any UPS directly attached to a server/workstations that is detected by Windows can now be monitored by EventSentry. The status of the UPS will show up on the host inventory page, and alerts will be generated when a host is on battery power and back on AC power. EventSentry can also initiate a shutdown when the remaining run-time or charge level falls below a certain limit.

UPS Alert

Batteries in laptops are also detected and listed on the host inventory page (battery capacity and current charge level), but generated alerts are informational only.

UPS Inventory & Monitoring

User Activity Tracking

While EventSentry provides its users with a wealth of information from multiple angles, it can be tedious to piece together data from multiple reports that is associated with a specific user. Data which can be linked to a user is scattered among process tracking, file access tracking, compliance logons and other pages.

User activity tracking

The new “User Activity” page, which is located in the “Dashboard” menu, solves this problem by displaying data from the following pages on a single page:

  • Logons
  • Processes
  • File Access
  • Active Directory Changes
  • Tasks
  • Events

The user activity page makes seeing all activity by a user as easy as never before!

Integration with third-party log management solutions

A few months ago, one of our users approached with the need to integrate EventSentry into an existing log management system which was already in place at the location where EventSentry was to be deployed. While reviewing the request we recognized that even though we position EventSentry as a one-stop log management solution with a compelling and robust web-based reporting component, an integration with other products can be helpful in some cases.

  • Supplement EventSentry’s built-in reporting with additional reporting
  • Integrate EventSentry with an existing log management solution located in a different business unit
  • Integrate EventSentry’s sophisticated real-time agent and deployment infrastructure with a different log management back-end

In version 3.3 and earlier, EventSentry can be integrated with 3rd party products using the HTTP, process and Syslog action. The HTTP & process action are intended to be used with ticketing systems where only a low volume of alerts are submitted while the Syslog action obviously supports submitting a high volume of events. The Syslog format was however limited to the traditional RFC 3164 format, making an integration with other log management systems difficult.

Starting with version 3.4, EventSentry now supports the following formats in the Syslog action

  1. RFC 3164 (legacy)
  2. Snare
  3. RFC 5424
  4. GELF (Graylog)
  5. Nagios Log Server
  6. Common Event Format (CEF)
  7. JSON (customizable)

If a log management server you need to integrate with is not listed above but supports the JSON format, then you can craft your own JSON packet with the JSON format, also introduced in v3.4.

Disk Space Alerts

Part of the effort to make EventSentry’s alerts more actionable is reflected in our improved disk space alerts which now list the 15 largest files and folders of the volume where disk space is low. The supplemental data will in many cases be enough to immediately identify the culprit so that corrective action can be taken immediately, without the need to run disk space analyzers on the volume.

Diskspace Alert with embedded file/folder size info

Audit Policy Status

Since the introduction of the compliance tracking components, EventSentry has been recording all audit (and many other!) changes performed in Windows as part of the “Policy Changes” feature. It wasn’t however possible to see the current status of all audit categories and subcategories at a glance. Reviewing the current audit status of all monitored hosts can be important however, if only to verify that group policies are configured correctly.

Hosts with disabled audit subcategories
Hosts with disabled audit subcategories

In v3.4 we now have the new “Status” page available under “Compliance -> Audit Policy” which delivers information such as:

  • Compare/review audit settings of a particular sub category (e.g. “Registry”) among all monitored hosts
  • View all disabled audit settings across all or select hosts
  • (Re)view audit settings based on computer types (e.g. domain controllers, servers, workstations)

NetFlow Bandwidth

Our NetFlow component can now provide bandwidth visualization based on the collected NetFlow data. The information can either be accessed on the NetFlow page or as a dashboard tile. Even though bandwidth data can already be determined using SNMP, the data gathered by NetFlow should be preferred since it contains additional data not available via SNMP, such as:

  • Packets sent/received
  • Bytes sent/received
  • Bytes per packet
  • % Utilization

Bytes per packet as well as packets sent received can be used to detect anomalies, e.g. when a host sends a large amount of network packets, or network packets with large/small content.

NetFlow Bandwidth

Auditing DNS Server Changes on Windows 2012 R2 and later with EventSentry

Auditing changes on Microsoft Windows DNS server is a common requirement and question, but it’s not immediately obvious which versions of Windows support DNS Auditing, how it’s enabled, and where the audit data (and what data) is available. Fortunately Microsoft has greatly simplified DNS Server auditing with the release of Windows Server 2012 R2.

In this post we’ll show how to enable DNS Auditing on 2012 R2 and later, and how to configure EventSentry to collect those audit events. In a future post we’ll also show how to do the same with older versions of Windows – 2008, 2008 R2 & 2012.

When configuration is finished you are going to be able to see when a zone/record is created/modified/deleted as well as by whom. The audit data will be available (and searchable) in the EventSentry Web Reports, and you’ll also be able to setup email alerts when some or all DNS entries are changed.

Prerequisites

Since native DNS auditing was only introduced with Windows 2012 R2 or later you’ll need to run at least Windows Server 2012 R2 in order to follow this guide. The table below shows the types of DNS auditing available on Windows Server Operating Systems:

Windows OS Auditing Type Comments
Windows 2008 Active Directory – based auditing only covered in future post
Windows 2008 R2 Active Directory – based auditing only covered in future post
Windows 2012 Active Directory – based auditing only covered in future post
Windows 2012 R2 Native DNS Auditing Available with hotfix 2956577
(automatically applied via windows update!)
Windows 2016 Native DNS Auditing enabled by default

Configuration

Enhanced DNS logging and diagnostics are enabled by default in supported versions of Windows Server when the “DNS Server” role has been added to Windows, so there are no additional configuration steps that need to be done.

1. Package Creation

On the EventSentry machine we are going to add a package under Packages/Event Logs by right-clicking “Event Logs” and selecting “Add Package”. In this example we are going to call the package “Windows Server 2016”:

Package Creation
Package Creation

2. Adding a Filter

The next step is to add a filter to the previously created package “Windows Server 2016”. Right click the package and then select “Add Filter”.

Adding a DNS Audit filter
Adding the DNS Audit filter

Note: For a short tutorial on how to create a filter click here.

3. Filter Configuration

There are several ways to approach the filter configuration depending on your needs. As a reminder, a filter is a rule in EventSentry that determines to where an event is forwarded to, or how it is processed.

  1. Collect all or select (e.g. creation only) DNS audit data in the database
  2. Email alert on select audit data (e.g. email all deletions)
  3. Email alert on all activity from a specific user

In this guide we will show how to accomplish (1) and (2) as a bonus.

On the right pane of the management console after the creation of the filter you will see the General tab of the new filter. We decided to configure it to log to the Primary Database, but the events can be sent to any action (Email, Syslog, …).

Under “Event Severity” we check all boxes since we want to log everything (it’s important to check “Information” since most of the creation/deletions/etc are logged at this level of severity).

Editing the DNS Audit filter
Editing the DNS Audit filter

4. Adding a custom event log

In the “Log” section click on “more” to jump to the “Custom Event Logs” tab (or, just click on that tab). Now we need to add the Microsoft-Windows-DNSServer/Audit event log to the list of custom event logs so that this filter picks up events from the DNS Audit event log. Click the save button in EventSentry Management Console title bar to save the changes we’ve made so far.

Adding an Application & Services event log
Adding an Application & Services event log

5a. Assigning the package (method A – manual assignment)

To assign the package, select the server you would like to assign it to and select “Assign Packages”. In the resulting dialog simply check the box next to the package we created in step 1. Alternatively you can also select the package and click “Assign” from the ribbon (or context menu) and select a group or host(s) to assign the package to.

Assigning a package
Assigning a package

5b. Assign the package (method B – dynamic activation)

Instead of assigning the package manually, the package can be assigned dynamically so that any host monitored by EventSentry running Windows Server 2012 R2 or later will automatically have this package assigned. To dynamically assign a package do the following:

  • Click the package and select “Properties” from the ribbon, or right-click.
  • In the “Dynamic Activation” section, check “Automatically activate …”
  • In the “Installed Services” field enter “DNS”
  • For the “Operating System”, select “at least” and “Windows 2012 R2”
  • Click the “Global” icon in the ribbon to make sure the package gets assigned to all hosts. Don’t worry, it will still only be activated on 2012 R2 or later hosts that have the DNS server running

6. Saving

After assigning the package and saving the configuration click “Save & Deploy” or push the configuration to all remote hosts. Please note that only new events generated in the DNS Audit Log will be processed, pre-existing log entries will be ignored.

Testing the Configuration

To test the configuration we will create a domain called “testzone.com” and add an A record called www on the monitored DNS server. We’ll then check if those modifications are visible in the EventSentry Web Reports. The screenshot below shows the new A record in the DNS console:

testzone.com with the new A record "www"
testzone.com with the new A record “www”

First, lets take a look to see what the actual DNS Audit entries look like (using the Windows Event Viewer: Applications and Services Logs/Microsoft/Windows/DNS-Server/Audit):

DNS Server Audit Event Log
DNS Server Audit Event Log

In the EventSentry Web Reports, navigate to Logs/Event Log and filter by the log “Microsoft-Windows-DNSServer” and then select “Detailed”. You should see all the modifications that were performed:

Detailed audit log in the web reports
Detailed audit log in the web reports

Bonus Track: Configuring alerts for a specific change

The first part of this post was purposely generic in order to understand the basics of monitoring your DNS Server. But as a bonus we’ll show how to monitor a specific change (in this case a creation) and trigger an email alert for that.

The process is the same as explained in the beginning:

  1. Create a new filter and add it to the same package. The filter should be configured exactly the same way. To make things easier, you can also copy & paste the filter with the familiar Copy/Paste buttons in the ribbon or context menu.
  2. This time however we specify the “Default Email” action in “General” tab so that we receive an email alert when the filter criteria matches an event.
  3. In the “Details” area specify event id 515 in the “Event ID” field, which is the event id corresponding to the creation of a new record. This is how the filter would look like:
Filter for receiving an alert on record creation

Filters can of course be more specific as well, it’s possible to filter based on the user or event content of the actual event. Below is a list of all audit events logged by the DNS Server:

Event ID Type Event ID Type
512 Zone added 551 Clear statistics
513 Zone delete 552 Start scavenging
514 Zone updated 553 Enlist directory partition
515 Record create 554 Abort scavenging
516 Record delete 555 Prepare for demotion
517 RRSET delete 556 Write root hints
518 Node delete 557 Listen address
519 Record create – dynamic update 558 Active refresh trust points
520 Record delete – dynamic update 559 Pause zone
521 Record scavenge 560 Resume zone
522 Zone scope create 561 Reload zone
523 Zone scope delete 562 Refresh zone
525 Zone sign 563 Expire zone
526 Zone unsign 564 Update from DS
527 Zone re-sign 565 Write and notify
528 Key rollover start 566 Force aging
529 Key rollover end 567 Scavenge servers
530 Key retire 568 Transfer key master
531 Key rollover triggered 569 Add SKD
533 Key poke rollover 570 Modify SKD
534 Export DNSSEC 571 Delete SKD
535 Import DNSSEC 572 Modify SKD state
536 Cache purge 573 Add delegation
537 Forwarder reset 574 Create client subnet record
540 Root hints 575 Delete client subnet record
541 Server setting 576 Update client subnet record
542 Server scope create 577 Create server level policy
543 Server scope delete 578 Create zone level policy
544 Add trust point DNSKEY 579 Create forwarding policy
545 Add trust point DS 580 Delete server level policy
546 Remove trust point 581 delete zone level policy
547 Add trust point root 582 Delete forwarding policy
548 Restart server
549 Clear debug logs
550 Write dirty zones

We hope that DNS changes will never remain a mystery after activating DNS auditing. Don’t fear if you’re running 2012 or earlier, the next post is on its way.

Mariano + Ingmar.

AutoAdministrator: Chapter 3

Today I have good news and bad news. You’d like to hear the good news first? OK. The bad news is that AutoAdministrator is being retired and will no longer be developed.

OK, now on to the good news. AutoAdministrator is of course not entirely history and is undergoing a similar transformation the NTToolkit did a few years back. AutoAdministrator is joining the “EventSentry” brand as the “EventSentry Admin Assistant”.


This brings the total number of software products under the EventSentry brand to three:

  1. EventSentry + EventSentry Light (free)
  2. EventSentry SysAdmin Tools (free)
  3. EventSentry Admin Assistant (free)

To celebrate this transformation we’ve made a number of improvements in the new release:

  1. The EventSentry Admin Assistant is now available as a native 64-bit application
  2. REG_MULTI_SZ and REG_QWORD data types are now supported (for “read” actions)
  3. WMI queries can be filtered (aka “condition”) to allow for things like checking whether a Microsoft Windows patch is installed

The last enhancement is particularly useful if you need to check whether a particular Windows update is installed on your network – see the screenshot below.

Checking installed patches with WMI

Please note that there is no upgrade path from AutoAdministrator to the EventSentry Admin Assistant – they are treated as separate pieces of software and will install side-by-side. You can download the latest version the EventSentry Admin Assistant here.

Enjoy!

Agent vs Agentless: Why you should monitor (event) logs with an agent-based log monitoring solution

The debate as to whether agent-based or agent-less monitoring is “better” has been answered many times over the years in magazine / online articles, blog posts, vendor white papers and others. Unfortunately, most of these articles are often incomplete, inaccurate, biased, or a combination thereof.

To make things slightly more confusing, different ISV use different methods for monitoring servers and workstations. Some use agents, some don’t, and a small few offer both both methods. But what is ultimately the best method?

What are you monitoring?

First it’s important determine what is being monitored to determine whether an agent-based or agent-less approach is better. For example, collecting system metrics like performance data usually creates fewer challenges then transmitting large amounts of (event) log data. Furthermore, agent-based monitoring is not an option for devices which run a proprietary embedded OS (think switch, printer, …) where you can’t install an agent in the first place.

Consequently I’ll be focusing on monitoring (event) logs with an emphasis on Microsoft Windows in this post. Having developed both agent-based as well as agent-less components in C++ over the years I feel that I am in a good position to objectively compare agent-based with agent-less approaches.

The Myths

Monitoring software is of course not the only type of software that uses agents, a lot of other enterprise software (backup, deployment, A/V …) uses agents as well. Below are some of the myths as to what (monitoring) using agents entails:

  • Agents may use up too many resources on the monitored hosts and slow down the monitored machines
  • Agents can become unstable and negatively affect the host OS
  • Deploying and managing agents is tedious and time-consuming
  • Installing agents may require the installation & deployment of dependencies the agents need (.NET, Java, …)
  • Installing third-party software will decrease the security of the monitored host

The Reality

It’s understandable that software which is installed on potentially every server and workstation in a network undergoes some level of scrutiny, but would you be surprised to learn that agents excel in the following areas:

1. Security: Better security since agents push data to a central component, instead of the monitored server being configured to allow remote collection.
2. Reliability: Agents can temporarily store and cache monitored logs if connectivity to the central monitoring server is lost, even if local logs are no longer available. Agents can also take corrective actions more quickly because they can work in isolation (offline). Mobile devices cannot be monitored with agent-less solutions since they cannot be reached by the central monitoring component.
3. Performance: Agents can apply local filtering rules and only transmit data which is valuable, thus increasing throughput while decreasing network utilization.
4. Functionality: They offer more capabilities since there are essentially no limits as to what type of information can be gathered by an agent since it has full access to the monitored system.

The Easy Way Out

Developing agents along with an easy-to-use deployment mechanism requires a lot of time and resources, so it doesn’t come as a surprise to learn that many vendors prefer to monitor hosts without agents. To compensate for the short-fall, ISV which solely have to rely on an agent-less approach will do their best to:

  • Emphasize that they do not use agents
  • Persuade you that agent-less monitoring is preferable

The irony, when promoting a solution as agent-less, is that even so-called agent-less solutions do in fact utilize and agent – the only difference being that the agent is (usually) integrated into Windows. Windows doesn’t just magically service remote clients asking for a boatload of WMI data – it processes these requests through the WMI service, which, for all intents and purposes, is an agent. For example, accessing the Windows event logs via WMI traverses significantly more layers than accessing the event logs directly.

Conclusion

With the exception of network devices where an agent cannot be installed, agent-based solutions will provide a more thorough monitoring experience 9 out of 10 times – assuming that the agent meets all the checklist requirements below.

Some event log monitoring vendors will try to convince you that agent-less monitoring is better & easier (easier for whom?) – but don’t fall for it. We’ve been tweaking and improving the EventSentry agent for more than 10 years, and as a result EventSentry offers one of the most advanced and efficient Windows agents for log monitoring on the market. Developing a rock-solid, secure and fast agent is hard, but it’s the only sensible approach which doesn’t cut corners.

There are situations when deploying a full-scale monitoring solution with agents is not possible, for example when you are tasked with monitoring a third-party network where installing any software is not an option. While unfortunate, an agent-less monitoring solution can fill the gap in this case.

EventSentry also utilizes SNMP (agent-less) to gather inventory, performance metrics as well as other system data from non-Windows devices, including Linux hosts. This collection method does suffer from the above limitations, but since log data is pushed from Non-Windows devices via the Syslog protocol, it’s an acceptable compromise.

Don’t compromise when it comes to monitoring the (event) logs of your Windows infrastructure and select an architecture which scales and offers security & performance.

Technical Comparison

The table below examines the difference between agent-based and agent-less solutions in greater detail.

Resource Utilization & Performance
Agent-Based
Agent-Less
Verdict
  • Usually higher throughput since agents can analyze, filter and evaluate log entries before sending them across the network.
  • Local resource utilization depends on the implementation of the agent.
  • Agent can access (event) logs directly via efficient API access.
  • Network utilization is likely much higher since more logs have to be transmitted across the network before being evaluated. Local filtering capabilities are limited and depend on the protocol (usually WMI).
  • Network latency and utilization affect performance of monitoring solution. Network utilization cannot be controlled.
  • Accessing (event) log data remotely through WMI is much less efficient.
  • Over-saturation of central monitoring component can negatively affect monitoring of entire infrastructure.
Higher network utilization combined with the fact that remote log collection will still utilize CPU cycles on the remote host (e.g. through WMI provider) favors agent-based solutions.
Agent-less solutions have a single point of failure, while agents can filter & evaluate data locally before transmitting them to a central database.
EventSentry Agents are designed to be essentially invisible under normal operations and do not impact the host system negatively in any way.
Stability & Reliability
Agent-Based
Agent-Less
Verdict
  • Failure of an agent does not affect monitoring of other hosts.
  • Locally collected data can be cached if central monitoring component is temporarily unavailable.
  • Failure of a central component may negatively affect deployed agents if they rely on the central component and cannot cache data.
  • Failure of central monitoring system will affect and potentially disable monitoring of all hosts.
  • Hosts which lose network connectivity (e.g. laptops) cannot be monitored while unreachable.
Agent-based solutions have an advantage since local data can be cached and corrective actions can be executed even when the central monitoring component is unavailable. Cache data & logs are re-transmitted – even if the local logs have been cleared or overwritten.
Agent-based solutions can monitor hosts even when disconnected from LAN.
The EventSentry Agent auto-recovers if the process aborts unexpectedly, and by default alerts the user when this occurs. When using the collector (default), the agent caches all data locally and retransmits when the network connection becomes available again.
Deployment
Agent-Based
Agent-Less
Verdict
  • Has to be deployed either with vendor management software and/or with third-party deployment software if vendor provides installation package (e.g. MSI).
  • Larger deployments will require multiple central monitoring components, potentially distributed over several LANs.
  • Only hosts in local LAN can be monitored.
Depends on deployment tools made available by vendor as well as management tools in place for configuring Windows settings. A poorly developed deployment tool would favor an agent-less solution.
EventSentry Agents can be deployed (multi-threaded) with the management console or through 3rd party deployment software by creating a MSI installer on the fly. When using the collector (default), agent updates (patches) can be deployed automatically.
Dependencies
Agent-Based
Agent-Less
Verdict
  • Agent may have dependencies on third-party frameworks
  • Depends on whether the mechanism utilized by the monitoring software requires a Windows component to be added and/or configured.
Depends on whether agent has dependencies and whether configuration changes need to be made on the monitored hosts.
The EventSentry Agent does not depend on any 3rd party frameworks or libraries.
Security
Agent-Based
Agent-Less
Verdict
  • Potential security issues if installed agent exposes itself to the network (if not firewalled) and/or suffers from local vulnerabilities which can be exploited.
  • Remote log collection has to be enabled and at least the central monitoring component needs to have remote access.
  • Secure data transmission relies on protocols and settings from Windows.
  • Enabling multiple methods for gathering data remotely (e.g. WMI) provides additional attack vectors.
  • Credentials (usually Windows user/password) for remote systems must be stored in a central location so that the remote hosts can be queried. If the central system gets compromised, critical credentials can be exploited.
Since agent-based solutions do not require permanent remote access and monitored hosts can therefore be hardened more, they are inherently more secure IF the agent doesn’t suffer from an insecure design and/or vulnerabilities.
Agent-based solutions also have more control over how data is transmitted from the remote hosts.
If there is general concern against third-party software then the product in question should be researched in a vulnerability database like http://www.cvedetails.com.
The EventSentry Agent does not open any ports on a monitored host and resides in a secured location on disk. The agent transmits compressed data securely via TLS to the collector. No major security vulnerabilities have been discovered in the EventSentry agent since its first release in 2002.
Scope & Functionality
Agent-Based
Agent-Less
Verdict
  • Agents have full access to the monitored system and can choose which technology to utilize to get the required data (API, WMI, registry, …)
  • Easily execute local corrective action like launching a script or process
  • Agent-less solutions are limited to remote APIs provided by the monitored host, most commonly WMI. While WMI does offer a lot of functionality, there are limitations.
  • Executing scripts on remote host is more involved and only possible when host is reachable.
Agent-based solutions have an advantage since they can utilize multiple technologies to obtain data, including highly efficient direct API access. Agents can also trigger (corrective) actions locally even while the agent is unreachable.
Agent-less solutions can only monitor data which is made available by the remote protocol.
The EventSentry Agent accesses log files, event logs and other system health data almost exclusively via direct API calls. The more resource-intensive WMI interface is only used minimally, for very specific purposes. Corrective action can be taken directly on the monitored host, often only in milliseconds after an error condition (event) has occurred.

 

Appendix: Checklist

When evaluating software that offers agents then you can utilize the check list below for evaluation purposes.

Resource Utilization
An agent needs to consume as little resources as possible under normal operations. With the exception of short (and unusual) peak periods, a user should never know that an agent is running on their server or workstation – period.
The thought of having a resource-hogging agent running on a server sends shivers down the backs of many SysAdmins, and the agents used by certain AntiVirus vendors that rhyme with Taffy didn’t set a good precedent.
Stability & Reliability
The agent needs to run at all times without crashing – the SysAdmin needs to be able to go to sleep knowing that his agents will reliably monitor all servers and workstations. Unstable agents are just no fun, especially when they negatively impact the host OS.
If an agent that encounters an issue, it needs to at least auto-recover and communicate the issue to the admin.
Deployment
Agent deployment and management needs to be streamlined and easy – it shouldn’t be a burden on the end user. And while agent deployment is important, agent management, keeping the remote agent up-to-date, is equally important and should – ideally – be handled automatically.
Most SysAdmins have enough work the way it is, the last thing they need is baby-sitting agents of their monitoring solution.
Dependencies
The more dependencies an agent has, the more difficult it is to deploy the agent. Agents that rely on complex frameworks like .NET, Java or specific Visual Studio runtimes are difficult and time-consuming to deploy.
Furthermore, any third-party software that is installed as a dependency creates an additional attack vector and needs to then be kept up-to-date.
Security
An agent needs to be 100% secure and cannot expose the monitored host to any additional security risks. I will explain below why using agents is actually more secure than not using an agent – even though this seems counter-intuitive at first glance.

Remote Support with VNC – The Easy & Secure Way!

Almost everyone in IT has heard of VNC – which actually stands for “Virtual Network Computing”. The RFB (Remote Framebuffer) protocol which VNC relies on, was developed around 1998 by Olivetti & Oracle Research Labs. Olivetti (unlike Oracle) isn’t much known outside of Italy/Europe, and the ORL was ultimately closed in 2002 after being acquired by AT&T. But enough of the history.

When the need arises to remotely log into a (Windows) host on the network, Microsoft’s Remote Desktop application (which utilizes Microsoft’s RDP protocol – not RFB) is usually the default choice. And why wouldn’t it be? It’s built into Windows, there is no additional cost, and it’s usually quite efficient (=fast) – even over slower connections.

Remote Desktop has a few disadvantages though, especially when it comes to the IT help desk:

  • You cannot view the remote user’s current desktop
  • It’s not cross-platform
  • You can’t use RDP if it’s disabled or misconfigured

Especially when troubleshooting user problems, being able to see exactly what the user is doing is obviously very beneficial. VNC-based applications are a good alternative since they allow you to view the user’s desktop and subsequently interact with the user. This makes VNC viable for help desk as well as troubleshooting. Nevertheless, VNC-based solutions have their own shortcomings:

  • Free variations of VNC usually offer no deployment assistance
  • With over 10 variants available, finding the best VNC implementation is a daunting task
  • VNC is still deemed as somewhat insecure
  • VNC can be slow

We set out to solve these shortcomings by creating a number of scripts around UltraVNC that integrate with the EventSentry management console (although they’ll work well without EventSentry as well!). Using the QuickTools feature, you can then connect to a remote host via VNC with 2 clicks, even if the remote host doesn’t have VNC installed.

Important: The scripts only work in environments where you have administrative access to the remote hosts. The scripts need to copy files to the remote host’s administrative shares and control the remote VNC service.

Alternatively, you will also be able to start a VNC session by running the following command:

vnc_start.bat remotehost.yourdomain.com

Even better, VNC can be automatically stopped and deactivated (until vnc_start.bat is run again) once the session is completed in order to reduce the attack surface.

VNC Deployment
As long as you have administrative access to the remote host(s), the script will remotely install VNC and even setup a firewall exclusion rule if necessary – although the UltraVNC installer takes care of this out of the box.

Security
To reduce the attack surface of machines running VNC you can automatically stop the VNC service after you have disconnected from the remote host. Our connection script will automatically start the remote service again when you connect the next time.

For the utmost security you can also completely uninstall VNC when you are done, a script (vnc_uninstall.bat) is included for this purpose.

Speed
Even though VNC is generally not as fast as RDP, it’s usually sufficiently fast in LAN environments (especially for shorter trouble-shooting sessions) and the UltraVNC port which we’ll be covering in this post performs reasonably well even over slower WAN connections.

Integration with EventSentry
Monitoring workstations with EventSentry strengthens the capabilities of any IT helpdesk and IT support team with:

  • Software & Hardware Inventory
  • Access to process utilization and log consolidation
  • Enhanced security with security log & service monitoring
  • User console logon tracking
  • Pro-active troubleshooting with access to performance and other system health metrics

Remote desktop sharing is an additional benefit with the UltraVNC package which is included with the latest version of EventSentry (v3.3.1.42). Customizing the scripts and integrating them with EventSentry literally shouldn’t take more than 5 minutes, and once setup & configured will allow you to remotely control any monitored host with a couple of clicks. The scripts do not require EventSentry, but are included with the setup and integrate seamlessly into the EventSentry Management Console.

The EventSentry Management Console includes the “QuickTools” feature which allows you to link up to 8 commands to the context menu of a computer item. EventSentry ships with a few default QuickTools commands, for example to reboot a remote machine. Once configured, you simply right-click a computer icon in the EventSentry Management console and select one of the pre-configured applications from the QuickTools sub menu.

EventSentry QuickTools
EventSentry QuickTools

How does it work?
When you run the vnc_start.bat script, it will first check to see if UltraVNC is already installed on the remote host. If it is, it will skip the installation routine and bring up the local VNC viewer. If you configured the script to automatically stop the VNC service when not in use, it will start the service beforehand. When you disconnect, it will (optionally) stop the VNC service again so that VNC is not accessible remotely anymore.

If VNC is not installed, the script will remotely install & configure UltraVNC using psexec.

If you do not want to leave the UltraVNC service installed on the remote computer, the vnc_uninstall.bat script can be run when the remote session is done. Automatically stopping the remote VNC service is however sufficient in most cases.

Prerequisites
There is not much you need:

Installation
The scripts need to be configured before they can be used in your environment, unless you are an EventSentry user, in which case you only need to download & install the prerequisites.

Super Quick Setup for EventSentry Users
It’s no secret, we’re a little biased towards our EventSentry users, and as such setting this up with an existing EventSentry installation is rather easy:

  1. Get psexec.exe and save it in C:\Program Files (x86)\EventSentry\resources.
  2. Download the UltraVNC installers (they have 32-bit and 64-bit – download for the platforms you have on your network) and store them in the C:\Program Files (x86)\EventSentry\scripts\ultravnc folder.
  3. Install UltraVNC on the computer where EventSentry is installed so that the VNC Viewer is available. It’s not necessary to install the whole package, only the viewer component is required.
  4. If “VNC” is not listed in your QuickTools menu, then you will need to add it under Tools->Options->QuickTools. Simply enter “VNC” as the description and specify the path to the vnc_start utility, e.g. “C:\Program Files (x86)\EventSentry\scripts\ultravnc\vnc_start.bat $COMPUTER”. You can optionally check the “Hide” box to prevent the script output from being shown before you connect.

You’ll notice that no password was configured – that’s because you will be logging in with a Windows user and password – only allowing domain admins access by default. This can be configured in the authorized_acl.inf file, if you want to give additional groups and/or users access that are not domain admins.

That’s literally it – easy as pie. Even though we designed this thing to be easy peasy, since things do occasionally go wrong I recommend testing a first connection from the command line. Just open an administrative command prompt, navigate to C:\Program Files (x86)\EventSentry\scripts\ultravnc and type vnc_start somehost.

Now just right-click any host – or use the “Quicktools” button in the ribbon – and select the “VNC” menu option. Keep in mind that first-time connections will take longer since the VNC setup file has to be copied and installed on the remote computer. Subsequent connections should be faster.

VNC Viewer Connect Dialog
VNC Viewer Connect Dialog

Manual Normal-Speed Setup for Non-EventSentry Users
So you are not an EventSentry user but still want to utilize these awesome scripts? No problem – we won’t hold it against you. The setup is still easy – you’ll just need to customize a few variables in the variables.bat file.

  1. Download the package from here.
  2. Create a local folder for this project, e.g. C:\Deployment\UltraVNC.
  3. Copy all the scripts to this folder, e.g. you should end up with C:\Deployment\UltraVNC\vnc_start.bat
  4. Open the file variables.bat in a text editor and keep it open as you will be making a few modifications to this file.
  5. In variables.bat, set the VNCSOURCE variable to the directory you just created.
  6. Download the latest version of both the 32-bit and 64-bit UltraVNC installers.
  7. In variables.bat, set the VNCSETUP_X86 and VNCSETUP_X64 to the setup file names you just downloaded.
  8. Download the PSTools and extract psexec.exe into the working directory, or a directory of your choice.
  9. In variables.bat, point the PSEXECFILE variable to the location where you just saved psexec.exe.
  10. Optional: Edit the authorized_acl.inf file to specify which Windows group or user will have access to VNC. You can either change the first line, or add additional lines to give additional users and/or groups permission.
  11. Install the respective version of UltraVNC on your workstation so that the VNC Viewer is available.
  12. Open a command line window and navigate to the folder to which VNCSOURCE points to. Test the setup by running vnc_start hostname, replacing “hostname” with an actual host name of a remote host of course.
  13. When presented with the login screen of the VncViewer, log in with a Windows domain admin user.

That wasn’t so bad now, was it? Just remember that you’ll need to initiate any VNC session with the vnc_start.bat file. Just launching the Viewer won’t work – even if VNC is already installed on the remote machine – since the VNC service is stopped by our scripts by default. To use the folder names we created, you’ll just run

C:\Deployment\UltraVNC\vnc_start hostname

Enjoy, and happy RFBing!

Connecting to remote host
Connecting to remote host

Configuration – variables.bat
For the sake of completeness the variables.bat file is explained below:

VNCSETUP_X86: The file name of the 32-bit installer. This needs to only be changed whenever UltraVNC comes out with a new version.
VNCSETUP_X64: The file name of the 64-bit installer. This needs to only be changed whenever UltraVNC comes out with a new version.

REMOTEINSTALLPATH: The directory where the script files will be copied to on the remote host.

VNCSOURCE: This is the folder where all the vnc-related files, including the setup executables, are located on the source host from where you initiate VNC connections – e.g. C:\Deployment\UltraVNC.
VNCINSTALLDIR: The directory in which UltraVNC will be installed in (on the remote hosts).

VNCPASSWORD: This variable is not currently used since UltraVNC is automatically configured to authenticate against Windows, by default giving only Domain Admins access to VNC. This is generally more secure than using a password. You can edit the file authorized_acl.inf to give additional users and/or groups access to VNC. The file supports one ACL entry per line.

PSEXECFILE: Unfortunately we are not allowed to bundle the nifty psexec.exe file for license reasons, so you’ll have to download the PsTools and point this variable to wherever you end up copying the psexec.exe file to. If you already have psexec.exe installed then you can save yourself 2 minutes of time and just specify the path to the existing file here.

SET_VNC_SVC_TO_MANUAL: If you don’t entirely trust the security of VNC, maybe because you know what a brute force attack is, and you only want administrators to access VNC then you can set this variable to 1. As long as you only connect to the remote host(s) using the vnc_install.bat script, the scripts will ensure that the remote VNC service is started before you connect and stopped after you disconnect. Between the two of us, I’d always leave this set to 1 unless you have the desire to launch the VNC Viewer directly, or need non-administrators to be able to connect to the remote host(s).

ADD_FIREWALL_RULE: As the name (almost) implies, this will create a firewall exclusion rule on the remote host(s) if you’ve been doing your homework and enabled the Windows firewall. If you don’t like our boring firewall rule name then you can even change the name below by editing the FW_RULE_NAME variable. Enabling this is usually not necessary since the UltraVNC setup adds firewall exclusion rules by default.

VNCVIEWER: If you find that a different version of the VNC viewer works better than the version which we are shipping, then you can change the file name here.

 

EventSentry v3.3 Part 2: Event annotation, Filter Chaining, RegEx and more

In my previous post I talked about our new NetFlow component as well as the new agent management capabilities now available in EventSentry v3.3. In this post I’ll cover the remaining new features and improvements we’ve made in v3.3, starting with the web reports.

Web Reports
There are a number of new features and improvements in addition to NetFlow visualization. There are a few new dashboard tiles, including a “Recent Activity” tile which – as the name implies – shows recent relevant changes such as newly detected processes, software (un)installed, ping status or service status changes.

Viewing recent activity on the dashboard
Viewing recent activity on the dashboard

Anybody who works in a team of two or more Sysadmins should find the new notes feature incredibly helpful. It lets any web reports user add comments (=notes) which are subsequently visible to others. Notes can be associated with one or more hosts (ensuring they show up in the “Documentation” tab of the respective host status page) and can include documents as attachments as well! Do you have warranty documents or network diagrams you want to store in a central place – easily accessible? That’s what the notes are for.

Adding a note to the web reports
Adding a note to the web reports

The overall look and feel has also been refreshed, and we’ve reorganized the menu to make it faster to access dashboards and easier to find pages.

The visualization of data has been improved, since some chart types work better with certain features of EventSentry. You can now visualize grouped data using either pie charts, tree maps or column charts.

The security of the web reports has also improved with a lockout policy which will locking an account after too many unsuccessful logon events.

Monitoring Improvements
As mentioned in part 1, the EventSentry is agent is now available in 64-bit, making it possible to monitor 64-bit counters and easier to monitor files in 64-bit directories. For users upgrading from an earlier version, the EventSentry management console will automatically migrate any existing 32-bit agents on 64-bit versions of Windows.

Application & Services Event Logs
While monitoring Application & Services event logs, often referred to as “custom” event logs was possible, the way this needed to be configured in the management console was a common source of confusion. Some users also needed the ability to monitor more than 30 different logs. Consequently, monitoring additional event logs is now straightforward, and users can monitor as many event logs as they wish.

Filter Chaining
With thresholds, timers, schedules, insertion strings, EventSentry already offers a sophisticated engine for monitoring events in real time. New in this release is the ability to setup filter chaining. This makes it possible to trigger actions only when 2 or more events occur, and you can even link events together using insertion strings. Chaining is enabled on the package level, and every filter in a “chaining” package is automatically part of the filter chaining rules.

Event Annotation
It happens frequently that we get alerts that require us to do additional research based on the information provided in the alert. For example, we may get an alert about an IP address for which we then need to do a reverse lookup or find the geoip location. Audit Success & Failure events from the security event log are another example, and often contain error codes and numbers which are not explained.

Green line shows reverse lookup, blue line geo location
Green line shows reverse lookup, blue line geo location

We set out to improve upon this, and starting with v3.3 EventSentry will annotate email alerts in a number of ways whenever possible:

  • IP addresses will include a reverse lookup
  • IP addresses will include a geoip location
  • Security events will have various error codes resolved

Please note that (1) and (2) are only supported for emails sent through the collector since it requires access to a local geoip database. (1) and (2) will need to be enabled in the email action “Options”, (3) is automatically enabled for all emails.

Insertion Strings & Regex
By making insertion strings from events accessible in filters and actions (e.g. through the $STR1, $STR2, … variables), it’s possible to create highly granular thresholds, customize emails, easily trigger corrective actions which utilize content from events and more. Based on our own requirements we took this capability a step further however, and you can now apply regex filters to events to define your own insertion strings. This is particularly useful for alerts which don’t use insertion strings or for events which contain log data. For those types of events, you can now parse parts of log strings and assign them to insertion strings. The previous blog article, Detecting Web Server Scan in Real-Time, shows a practical example of how to apply this new feature. It does require you to be a bit familiar with Regular Expressions, but the management console includes a handy dialog where you can test your regular expressions, shown below.

Regex preview & test utility in management console
Regex preview & test utility in management console

Performance
Faster is better! We’ve improved performance in a number of areas:

  • The database insert performance of the Syslog daemon has been improved for Microsoft SQL Server databases
  • The delimited log file feature now includes an additional index to increase database insert performance
  • The heartbeat agent now relies less on RPC-based agent status monitoring and can instead obtain the status of a remote agent either directly from the collector or the database, resulting in less network traffic and faster heartbeat monitoring cycles.

With new features & improvements in a variety of areas, this release should contain improvements for everyone. Remember that you can also submit feature requests here.

EventSentry v3.3 Part 1: NetFlow, Easier Deployment & Laptop Monitoring

We are very excited to release EventSentry v3.3, a major update to our award-winning monitoring solution EventSentry, less than 10 months after the release of the previous major version 3.2. Version 3.2 included the collector component which supports secure and reliable communication with remote agents as well as better database throughput, switch port mapping and many improvements to the web reports.

I’d like to also thank everyone who took the time to fill out our annual survey – we read every single response in detail. If you haven’t taken it yet then you can still do so here.

The v3.3 release, which builds upon some of the architectural changes we have made in v3.2, and offers new functionality to help you:

  • Visualize, measure & investigate network traffic better with the new NetFlow component – with discounted introductory pricing until 12/31 2016!
  • Spend less time managing agents – the collector can now push configuration as well as agent updates automagically – think laptops!
  • Deployment via MSI is much easier – MSI file creation now only takes a few seconds
  • Investigate issues faster with email alerts which have geo location, reverse lookups as well complex security codes included inline
  • Visualize any data in the web reports more easily with additional dashboard tiles and treemaps throughout
  • Managing and using custom event logs is now more straightforward and scalable
  • Database throughput has been improved for Syslog data and delimited log files
  • Even more advanced filtering is possible with filter chaining and insertion string override via regular expressions
  • Communicating and documenting your network has just become a lot easier – add notes and/or upload documents in the web reports
  • Monitor 64-bit operating systems with a native 64-bit agent

With a brand new component and many new features in a variety of areas, v3.3 will have something of interest for everyone. Let’s dive in and look at the new features in more detail.

NetFlow
NetFlow is a new component which is part of the “Network Services” service (along with Syslog, SNMP, ARP) and is licensed separately. Pricing is very competitive and an additional introductory discount will be available until the end of this year, 12/31 – including competitive upgrades. You can request a quote here.

Collecting NetFlow data allows you to see all traffic meta data which passes through network devices that support NetFlow, including:

  • Source IP, destination IP
  • Source host, destination IP (when resolvable)
  • Source port, destination port
  • Geo location (when available)
  • IP protocol used
  • Amount of traffic sent and received
  • Number of packets transmitted
NetFlow Dashboard
Dashboard for NetFlow

EventSentry v3.3 currently supports the NetFlow v1, v5, v9 as well as sFlow flow protocols. NetFlow is usually supported by most commercial routers and firewalls whereas sFlow is most commonly supported by switches. NetFlow is generally preferable over sFlow – especially for forensic analysis since sFlow samples traffic and only sends every nth flow. sFlow can be preferable when dealing with large amounts of data, but EventSentry’s NetFlow implementation (as well as NetFlow itself) has a way to group flows and therefor condense traffic.

Do you need NetFlow, and is it worth looking into? Without NetFlow there is impossible to know which hosts communicate with each other (unless you capture network traffic). What traffic enters the network, and what traffic leaves it? Broadly speaking, implementing NetFlow lets you:

  • Visualize all network traffic in a variety of ways and reports
  • Analyze network data for forensic purposes
  • Utilize network traffic data for troubleshooting purposes
  • Map network traffic to geo location
  • Correlate network traffic with Active Directory users (requires workstation monitoring)
  • Measure bandwidth utilization
NetFlow Summary
NetFlow Summary

On the EventSentry side, setting up NetFlow should take less than 5 minutes; and setting it up on the network device side is generally just a matter of enabling NetFlow and pointing it to EventSentry.

Geo Location
EventSentry ships with the GeoLite geo database from MaxMind which does a good job of associating IP addresses with physical locations down to the city level. If you are looking for more accuracy however, then you can also purchase the full geo location database from MaxMind here.

Blocked ports by origin country
Blocked ports by origin country

Active Directory User Correlation
A unique feature of EventSentry’s NetFlow implementation is the ability to correlate workstation logins with network traffic, making it possible to associate network traffic with individual users. This requires that workstations are monitored with EventSentry and works best when users have a dedicated workstation.

Agent Management & Deployment
If you are utilizing the collector service then you have now a great time-saving feature available. Pushing a configuration update to remote hosts after you made a change or deploying agent updates after a patch installation are a thing of the past once you activate the respective options in the collector dialog.

Managing automatic configuration updates can be done in 2 ways: Either by automatically deploying a configuration update after you click “save”, or by deploying only approved configuration updates (recommended). If you select the latter, then you just have to click the new “Save & Deploy” sub-option on the ribbon and the collector will do the rest. It’s no longer necessary that the EventSentry agent is directly reachable from the management console; it will receive the latest configuration as soon as it connects to the collector.

Configuring Agent Management
Configuring Agent Management

Please note that you will still need to manually deploy a v3.3 agent once in order for automatic agent updates to work, since the self-update code is embedded in the new agents.

Creating MSI files has also been greatly simplified – a x86 and x64 agent MSI file is created with just a few mouse button clicks. Manually editing MSI files with tools like ORCA is a thing of the past. The only prerequisite is the (free) WiX Toolset which has to be installed only once.

Monitoring Laptops
In addition to saving most EventSentry users a lot of time, these new deployment features also make it possible to monitor laptops which aren’t permanently connected to the network. Simply deploy the agent MSI file with your favorite deployment tool (or deploy with the management console) and enable the configuration and agent management options in the collector. From that point on, any agent connecting to the collector will automatically receive the latest configuration AND any new agent updates – completely automatically – no matter where in the world they are located.

64-Bit Agents
EventSentry v3.3 now ships with both a x86 and x64 agent, so that 64-bit editions of Windows can be monitored natively. The key benefit of this change is that 64-bit only performance counters can now be monitored, these counters were off limit with 32-bit agents. Utilizing 64-bit agents also results in the following changes:

  • Agents will be automatically converted to 64-bit when v3.3 is deployed. It is not possible to use a 32-bit v3.3 agent on a 64-bit version of Windows
  • File system redirection via “Sysnative” or in the File Checksum Monitoring packages is no longer necessary
  • Memory consumption will be slightly higher compared with 32-bit agents

Please note that EventSentry has not completely migrated to 64-bit yet, some components (management console, heartbeat agent, web reports) are still shipped as 32 bit executables. We plan on migrating all components to 64-bit by the end of 2017.

There are just too many new features in v3.3 to fit them all into one blog post, so stay tuned for part 2 which will follow shortly.

Your NETIKUS.NET team.

Detecting Web Server Scans in Real-Time

Any web site exposed to the Internet is constantly being probed by bots, malicious hackers and other evildoers in an attempt to take over the machine, gain access to unauthorized data, install back doors and so forth. Detecting probing attempts as early as possible and taking corrective action as soon possible is key to maintaining a secure network.

Manual probing usually involves investigating the HTTP headers to determine the type of web server (e.g. IIS, Apache, Nginx), viewing HTML sources and possibly attempt to access well-known pages in order to determine whether any well-known web-based software (WordPress, CRM, OWA, …) is installed.

IIS Email Alert
Email alert from an IIS web site scan

If the attacker prefers the sledgehammer approach then he or she may also point a vulnerability scanner such as OpenVAS at the web server, which will reveal vulnerabilities with a minimum amount of work. Automated systems aren’t as surgical and will usually just look for specific vulnerabilities by checking for the existence of various URLs on the web server.

But whether it’s a manual probe, a vulnerability scan or a bot, all methods usually result in a non-existent page (URL) being attempted to be accessed, resulting in a “Page Not Found”, 404 error at some point. As such, a larger than usual amount of 404 errors can be a good indicator that suspicious activity is occurring on your web server. If you are a little paranoid like me then you could even look for every single 404 error that occurs on your web server. The same technique can be applied to other errors as well, such as “Access Denied” errors for example if the web site is secured by ACLs.

EventSentry’s log file monitoring feature can monitor Windows-based log files in real time and trigger alerts and/or corrective actions by applying sophisticated rulesets to all parsed text.

Log File Flow
Log File Flow (Icon made by Freepik from www.flaticon.com)

I’ll explain how this can be setup based on an IIS web server, but the same generic steps would apply to other web servers as well.

  1. Define the log file
    The first step is to tell EventSentry which log file you’d like to monitor in the management console. Using the ribbon click on “Packages”, “Log Files” and “Define Files”. In the “Log Files” section on top, click on the plus icon (+) and define the log file. Give the file a descriptive name, specify the path to the log file and select “Non-Delimited” as the file type. Make sure to utilize wild cards or variables for the log file path if the name of the log file is dynamic, as shown in the screenshot below:

    IIS Log File Setup
    IIS Log File Setup

    If you plan on storing contents in the EventSentry database as well then you can also select a matching log file definition (such as IIS 7) as the log file type. More information on log file types can be found in our IIS Log File Monitoring with EventSentry screen cast.

  2. Setup a log file filter
    A log file filter defines where content from the log file is routed to. In this example we’ll route 404 errors to the Application event log. Using the ribbon again, and while still in the log file context, click on “Add Package” on the top left to create a new package – give the package a descriptive name. Then, click the “Assign” button to assign this package globally, to a group or individual host. (remember that you can also assign packages dynamically). Now click “Add” in the “Log File” section to add the previously configured log file to the package.

    Log File Filter
    Specify how log file content is routed

    In the resulting dialog we can configure the log file filter to send log file contents to the database, the event log or both. For the purpose of this example we will only log certain lines of the log file to the event log – those matching the wildcard filter * 404* (note the space between the first * and 404) as shown in the screenshot above. You can also use a regex expression for a more sophisticated match type.

  3. Setup Event Log Filter
    At this point EventSentry will log an informational event every time the text ” 404″ is logged to the specified log file. In order to dispatch (e.g. to an email recipient) this event however, an include filter needs to be setup which should look similar to the screenshot below:

    Event Log Filter for Log File Alert
    Event Log Filter

That’s all that is required to trigger an email or process every time a 404 error is triggered on your web server. Read on to refine this setup and only get alerted when the same remote IP address triggers a certain number of 404 errors within a certain time period – fun!

Additional Resources
Owasp.org is a great resource for web developers which provides a plethora of information to help keep web sites secure. The Owasp Top 10 document illustrates what the most critical web application security flaws are.

Tutorial: Delimited Log File Monitoring
Screen Cast: Log File Monitoring with EventSentry

Bonus for Advanced Users (requires EventSentry v3.3 or later)
Getting alerts whenever specific text – like a 404 error – are logged is quite useful, but utilizing EventSentry’s advanced event log filter & thresholds features can reduce noise and make monitoring log file contents even more actionable.

EventSentry supported utilizing insertion strings from events for quite some time, allowing you to use those insertion strings either in actions (e.g. an email subject, a parameter for a script) or thresholds. Since events don’t always utilize insertion strings properly, or custom content in events needs to be parsed separately, EventSentry v3.3 and later let you define insertion strings based on regular expressions. The screenshot below shows insertion strings before and after a regular expression fitted for IIS 7.5 is applied to EventSentry’s log file monitoring alert:

Apply RegEx to Event
Overriding insertion strings by applying a regular expression

You can learn more about insertion strings here, and view insertion strings either with the Event Message Browser or the EventSentry Management Console (Tools -> Utilities). The regular expression for a default IIS 7.5 setup is as follows:

([0-9]{4}-[0-9]{2}-[0-9]{2}) ([0-9]{2}:[0-9]{2}:[0-9]{2}) ([0-9\\.]*) ([A-Z]*) (.*?) (.*?) ([0-9]*) (.*?) ([0-9\\.]*) (.*?) ([0-9]*) ([0-9]*) ([0-9]*) ([0-9]*)

Since insertion strings can be used in variables (e.g. $STR1 … $STR14) and thresholds, overriding insertion strings in an event has two main benefits:

  • Use any field from the log file in the email subject and other action fields
  • Create thresholds based on log file content – e.g. create dynamic run-time thresholds for each IP address

Regular expressions are set using the “Advanced” button on the “Generic” tab of an event log filter. In the advanced dialog, simply click “Edit” in the “Insertion String Override” section.

Using insertion strings in emails
The generic EventSentry email subject is nice, but a customized subject reflecting the type of alert would certainly be better:

Red Alert: IIS scan detected from IP $STR9

This is possible with the redefined insertion strings, since #9 (=$STR9) is the remote host’s IP address. To set a custom subject, click the “Advanced” button on the “Generic” tab of an event log filter.

Using insertion strings in thresholds
By default, threshold counters are increased every time an event matches the corresponding filter. To stick with our example here, we could configure EventSentry to let us know if more than three 404 errors occur within 5 minutes. But we’d essentially be throwing all events into the same bucket. If you look at it in detail however, you realize that it makes a difference whether three 404 errors are a result of activity from the same remote host, or three different remote hosts.

Since events are often a result of specific activity by something or somebody, it’s important that we can correlate multiple events. In our example, the “something” is the remote host, which is represented by insertion string $STR9. As such, we can configure our threshold to use $STR9 as the common identifier, and create unique thresholds based on the run-time value of $STR9. By doing that, we will trip the threshold only if the same remote host accesses a non-existing URL three times, but not if three different remote hosts only access one non-existent URL each.

Event Log Filter Threshold
Event Log Filter Threshold

The same technique can be applied to thresholds for failed logon events. It’s usually acceptable if a user types the wrong password a few times, but a large number of failed logons from the same user are not. Just applying a threshold to all 4625 events is usually not practicable since many users occasionally type a wrong password. But by tying the threshold to the insertion string representing the user name (they are 6 & 7 in case you are curious), we can create a separate threshold for every user and avoid false positives.