EventSentry v6: Azure Logs, HEC, Sigma, Log Signing & More

ingmar.koecher 0 Comment Announcements, EventSentry, Security, Tools & Utilities

Azure Logs

Even though the shift to the cloud has slowed recently as many businesses are moving certain workloads back on-premise, Microsoft Exchange remains one cloud-based service that most organizations continue to embrace – despite its frequent outages. This doesn’t come as a surprise, as Microsoft has successfully devolved on-prem Exchange Server – the only viable alternative – into an unfriendly dragon that even experienced sysadmins won’t touch with a 10 ft pole. I’d rather compile the kernel of a rare Linux distro on a mobile phone than administer Exchange 2019.

Importing Azure logs was already possible in EventSentry v5.2 with a manual setup, but EventSentry v6 makes this significantly easier where this functionality is now fully integrated. Setting it up is easy and straightforward and utilizes the already-existing delimited log file feature that you may already be familiar with (which has also been simplified!):

  • Create Microsoft Cloud credentials
  • Create or reuse a log file definition (e.g. Azure Sign-In Logs)
  • Create a log file + assign the previously created credentials

It’s possible to assign multiple credentials to a single log file, for example if you’re a MSP that needs to monitor Azure Sign-In Logs from multiple clients.

Logs are downloaded by the new EventSentry Sync service which also converts the JSON-based logs to CSV format so they can be imported by the agent. You can configure how often logs are downloaded and whether temporary files are kept or deleted. Accessing the logs works just like accessing other on-prem delimited logs like DHCP, IIS etc. logs – via FEATURES -> LOGS -> Delimited Log Files in the web reports.

HEC: HTTP Event Collector

EventSentry already supports a variety of log sources like log files, Syslog, SNMP and more. The new HTTP Receiver – a new component in the network services – can receive remote logs via the HTTP protocol in either CSV or JSON format. The process is similar to the aforementioned Sync Service, only that the HTTP Receiver saves the received logs in a temporary directory where they are then picked up, converted (if necessary) and processed by the EventSentry agent.

Many network devices support exporting logs via HTTP, including Palo Alto, Fortinet, SonicWall and more. To enable the HTTP Receiver, simply navigate to “Network Services -> HTTP Receiver” in the management console.

Sigma

Sigma is a vendor-agnostic YAML-based format for the purpose of describing threat detection rules. This essentially allows security researches to write threat detection rules once – instead of having to convert them into one or more proprietary formats. You can then use commercial services like SOC Prime or tools to convert the Sigma rules into the format your SIEM uses. This allows security researches to focus on what they do best – identifying and documenting threats.

Here at EventSentry we think that converting Sigma rules into event log filters is boring, so we added native Sigma support to EventSentry. This means that you can take a Sigma rule “as is” and simply paste it into an event log filter. There are a number of benefits with this:

  • Sigma rules are evaluated at the agent – in real time
  • No need to convert Sigma rules
  • Ability to create complex detection rules in Sigma format

We hate to admit it, but Sigma rules can actually support more complex rules than EventSentry’s content filters. For example, with Sigma you can match conditions like

(event id = 1 and text includes “myapp.exe”) or (event id = 2 and text includes “thisprogram.exe”)

Log Signing

Cryptographically signing logs is sometimes necessary for regulatory compliance and in high-security environments. Log signing essentially ensures that collected log data has not been removed or tampered with. Implementing log signing is a fairly complex endeavor and we solved it in a fairly straight-forward, simple way in this first implementation.

Event log & Syslog data can now be cryptographically signed by storing log data in a text file (“File Action”) and signing the created files using OpenSSL, which is now included in the latest installer. Signing log data in files has a number of advantages:

  • The existing mechanism for consolidating log data in the database remains unaffected
  • Signed logs can be stored on any medium that exposes NTFS/SMB, including WORM
  • Log destination can be in a completely different security context – blocking EventSentry admins from modify access

Even when the signed logs are stored on the same server where EventSentry is installed, timestamping ensures that the logs cannot be modified & resigned without detection. By default, new log files are created & signed every minute, reducing overhead and making it easy to detect deleted logs.

It’s important to note that viewing & verifying logs is a manual process and requires the use of an (included) command-line script.

Anomaly Baseline

Anomaly detection not only helps detect anomalies like new processes, but can also visualize the collected data to and provide insight into activity on servers and endpoints. For example, anomaly baseline can show:

  • all parent/child process relationships
  • all user/process relationships
  • all DLLs loaded by a process (requires Sysmon)
  • all User/Pipe connections

Anomaly baseline data is automatically populated if an anomaly filter is configured to use the collector, as such it’s recommended to enable “Filter on Collector” for all anomaly rules.

Web Reports

You can now activate HTTPS in the web reports without having to run scripts and edit configuration files – simply enable HTTPS while setting up the web reports and the rest is done automagically for you!

All hosts can now be assigned a physical location, which can then be used to filter search results on all search pages. Additional map-based pages will be added in future releases.

A new auto-complete menu makes writing search queries significantly easier and more convenient:

Switch port mapping now includes an Overview tab which visualizes all switch port connections with an intuitive map that takes the new location settings into consideration as well.

In an effort to modernize the UI in the web reports and take advantage of new web technologies, support for Internet Explorer had to be dropped – as such you won’t be able to use IE to access the web reports anymore.

Other improvements include support for secondary LDAP servers, an improved collector status which now shows the TLS version, the last time the BIOS was updated and more.

Other Improvements & Features

OAuth support was added to email and HTTP actions – emails can now be sent via Office 365 and Google’s GMail without using a SMTP server (the email dialog as been optimized as well). HTTP actions can authenticate with OAuth now as well.

EventSentry now also includes a secondary installer which can be used to install the collector, network services and heartbeat services on remote hosts. In the past, setting up any of these services required a lot of manual steps which are now automated with the new installer (ADMonitor is not included as it requires additional steps unique to ADMonitor).

Deploying EventSentry has also been simplified by supporting an unattended installation of the main EventSentry installer, including the web reports. Agent-based installations now also support an alias option (intended to be used with MSI-based agent installations), allowing the assignment of virtual host names at MSI installation-time to any agent.

Performance of the large file enumeration feature, which is part of disk space monitoring, has been vastly improved by scanning the MFT directly. This feature scans every monitored volume to find the 250 largest files, which can be time-consuming and large volumes and/or slower hosts. Scanning the MFT instead of the file system only takes a fraction of the time compared with earlier versions.

Sync Service

EventSentry v6 also sports a new service, the “EventSentry Sync” service. In addition to downloading Azure-based logs, the Sync service can also download validation script updates in the background and ensure that core EventSentry services are always running.

Users with an active maintenance agreement can download EventSentry v6 either from within the EventSentry Management Console, or by downloading the installer from the customer area. If you’re not an EventSentry customer yet then you can start an EventSentry evaluation.

EventSentry v5.2: Processes, Security & Inventory

ingmar.koecher 0 Comment Announcements, EventSentry , , , , , , ,

The latest iteration of EventSentry adds many powerful security features, continuing to enhance EventSentry’s ability to improve the security of Windows-based networks by strengthening its foundation and detecting suspicious behavior.

Local Inventory

EventSentry already provides an impressive inventory of monitored end points, including software, browser extensions, hardware, services, tasks and much more. Being added to the mix are:

  • Local User
  • Local Groups
  • (File) Shares

As such, you can now review all local users & groups that exist across your entire infrastructure, making it easy to identify administrative as well as suspicious or unneeded user accounts. Providing insight into all shared resources across the network again lets Sysadmins spot unnecessary or insecure local file shares.

Sysmon Management

The Sysmon utility, part of the Microsoft Sysinternals Suite, is a powerful, free add-on which is already tightly integrated in EventSentry via the Process Sysmon feature, which tracks network activity by individual processes. And while Sysmon is a great utility, it does have a major flaw: Deployment (or lack thereof). Installing and maintaining Sysmon is a manual process that doesn’t scale well across larger enterprises. But EventSentry users neither need to worry about maintaining the latest version of Sysmon nor keeping the Sysmon configuration updated – the new Sysmon management features takes care of both! Simply point EventSentry to the Sysmon URL (or a local copy), specify a configuration file (of course we ship one) and move on!

Tackle Compromised & Duplicate Active Directory Passwords

ADMonitor can now take advantage of two powerful features to strengthen the security of their Active Directory domains:

  • Identify Compromised Passwords
  • Identify Duplicate Passwords

ADMonitor can now query the haveibeenpwned.com web site to identify AD user accounts with a password that has been previously compromised in a data breach. A user account flagged by ADMonitor indicates that the password of this particular user has been part of a breach at some point – it doesn’t mean that this particular AD user account has been breached.

Multiple user accounts sharing the same password creates a number of security risks, but usually remains undetected. ADMonitor in v5.2 can now detect duplicate passwords, that is more than one AD user accounts using the same exact password.

Taking advantage of these new features – combined with the standard ADMonitor functionality and validation scripts – significantly strengthens the security of your Active Directory domain.

Threat Scoring

Events can now be tagged with a “threat score”, usually a number between 0 and 100. The EventSentry agent then keeps track of these threat scores, adding them up as they occur and issuing alerts if the threat score exceeds a certain threshold within a configurable time period. This is somewhat similar to the already-existing “Filter Chaining” feature, albeit more flexible since it doesn’t require creating separate packages. Filter chaining is still around however, especially since it allows specifying the order in which events occur.

A practical example of threat scoring is the detection of admin tools that are often utilized by attackers when they gather information from a compromised host. We’re talking about apps like whoami.exe, ipconfig.exe, reg.exe, wmic.exe, systeminfo.exe and the likes. Occasionally used by System Administrators and in isolation they are harmless – but more than one of these EXEs run in close succession can be suspicious and should trigger a review. Furthermore, some EXEs are more dangerous than others which the threat score can reflect. Running ipconfig.exe should have a lower threat score than wmic.exe.

The most obvious application for threat scoring will be event id 4688 which is logged when a new process is started, but it can be applied to any type of event – e.g. Logon events, Sysmon events and others.

Anomaly Detection v2: Fewer false positives

Anomaly detection, introduced in v5.1 is an easy way to flag potentially suspicious activity or detect previously unseen patterns: Logons from new users, new processes and more. For some use cases, like detecting new processes, anomaly detection could result in false positives. For example, a never-before seen process on Host A could be triggered again on Host B – since the anomaly engine run independently on each end point (agent).

In v5.2 anomalies can now be filtered on the collector, reducing most false positives if they occur on more than one host. Taking the previous example, the collector would suppress the “anomaly” flag on Host B since it was already registered by Host A earlier.

Searching Process Activity

Since EventSentry collects various process activity and information on multiple pages (process tracking, Sysmon, process status w/ netstat, services, …), adding a hybrid “Process Activity” page – similar to the current IP Activity & User Activity pages – became a necessity. This makes identifying (suspicious) processes extremely easy and fast – even stepping backwards through the parent processes. Process Activity supports searching through common process properties like process executable, PID and searches the following areas:

  • Currently Active Processes
  • Process Activity (Tracking & Sysmon)
  • Processing listening for incoming network connections
  • Services

Other New Features

  • Newly attached USB/BT keyboards are now detected and listed on the host inventory
  • Collector Client supports port knocking
  • Collector support for Process action
  • Process Spoofing Detection (identifying processes pretending/masquerading to be part of Windows)
Process Spoofing / Masquerading

Web Reports

PWA Support
On Chrome and Edge web browsers, the web reports can now be run as a “Progressive Web Application”, which delivers an app-like experience of the EventSentry web reports on desktops.

Dashboard Tiles
The Acknowledge Tile has been improved to support different output styles as well as queries, allowing users to create multiple tiles for different purposes.

Syslog RegEx Support
Since Syslog messages tend to be unformatted and, as such, sometimes difficult to interpret, the Syslog search now supports overlaying RegEx queries to format Syslog messages. This was previously only supported in the dashboard.

NetFlow Top Connections
This new NetFlow tile vividly displays your network’s key host connections. Uncover traffic patterns effortlessly and group data by IP address, hostname, country, and more for actionable insights.

Other Web Reports Improvements

  • Improved Dashboard Manager with Sorting
  • 2FA Email Authentication

EventSentray

The task-bar utility which can be automatically deployed with the agent and is also available in the free Sysadmin Tools, now includes an activity dialog which displays real-time information about process, file, scheduled tasks & service activity. But that’s not all – the new process/service dialog allows for conveniently terminating processes and controlling services.

Capturing Network Traffic anytime

ingmar.koecher 0 Comment EventSentry, Monitoring, Security, Tools & Utilities , , ,

Capturing network traffic is usually done either for security reasons or to troubleshoot networking issues. But by the time you initiate a network capture (either manually or automatically) it’s often too late already – the train has already left the station.

Point in case: Say your SIEM (obviously EventSentry) detects abnormal or suspicious behavior in a log and a network capture is initiated. By the time the capture is started, the most relevant network traffic will likely have already occurred. The same applies to network problems – they are often spurious and starting a capture when you notice the issue will often be too late.

So what can you do? Capturing – and storing – network traffic on a regular basis is often not feasible due to the large amounts of data generated. Permanently writing all network traffic to disks (even if you were to rotate the collected pcap files) creates an enormous amount of disk I/O – something to avoid especially on SSD drives.

But what if you could – at any time – simply dump the last 100 Mb (or more …) worth of network packets with a single command to a .pcap file? Instead of installing or starting an app like Wireshark manually and initiating a capture, you simply issue a single command and done. What’s even better is that it dumps the last 100 Mb of data traffic – you’re capturing the past!

Well, now you can do exactly that with the EventSentry Network Capture service, a new component that is part of the free EventSentry SysAdmin Tools. This service continuously captures all network traffic in a memory-based ring buffer in the background (the size of the buffer is configurable) and dumps the captured network packets to a directory when needed. Better yet, the service is extremely light on resources and uses <1% of CPU time, no permanent disk I/O and most of its memory usage is for the network packet buffer.

The service is easily customized via command line parameters and logs all relevant information to the application event log.

The service does require that a Winpcap-compatible network driver (e.g. npcap) is installed. Out of the box it uses a 50 Mb memory buffer, capturing traffic on the first active NIC with a valid IP address. The default location for .pcap files is the %TEMP% directory.

NOTE: If you already have an earlier version of the EventSentry SysAdmin Tools installed, then you’ll need to uninstall & reinstall to see the new component in the installer.

You can get started in 3 quick steps:

  1. Install a WinPcap-compatible driver, like Npcap
  2. Install the EventSentry SysAdmin Tools and make sure the optional EventSentry Network Capture service is checked
  3. Review & customize settings

Once everything is setup, simply run a single command to capture the most recent network traffic.

If the dump was successful, event id 120 will be immediately logged to the Application event log:

This is also useful on systems where you frequently need to capture network traffic. Instead of loading up an app like Wireshark and waiting, simply dump the buffer and you’re done!

Can the EventSentry Agents cause the same outage & disruption like the CrowdStrike Falcon sensor did?

ingmar.koecher 0 Comment EventSentry, Pure Knowledge, Security ,

The faulty Rapid Response Content CrowdStrike update that disabled millions of Windows machines across the globe on 7/19/2024 was any IT professional’s nightmare. Having to manually visit and restore each affected machine (further complicated by BitLocker) severely limited the recovery speed, especially for businesses with remote locations, TVs, kiosks, etc.

Of course, we’re all used to seeing bugs in the OS, our phones, and third-party software—but the impact is usually much less severe than what was observed on that fateful Black Friday. Prior to this incident, the majority of IT staff probably never imagined that an outage like this was even possible (and it appears as if CrowdStrike’s management team didn’t either).

It doesn’t come as a surprise, then, that many who were directly affected by this bug are seriously concerned about the software running on their networks—especially monitoring and security software like EventSentry that sits on most endpoints. How many more ticking time bombs are there that can take out everything within minutes? Many customers who use CrowdStrike also use EventSentry and are naturally wondering whether the EventSentry agents also have the “capability” to cause a BSOD.

The good news is that, since EventSentry is a user-mode service that does not directly run any code inside the Windows Kernel, it cannot cause a system crash like the CrowdStrike Falcon sensor did. A similar bug in the EventSentry agent would “merely” cause the EventSentry agent to terminate (“crash”) and normally restart automatically.

All EventSentry updates (including patches) are tested on our supported operating systems with a variety of configurations prior to release. This ensures that potential bugs affecting OS stability are identified before they are released to customers.

However, EventSentry is a highly configurable monitoring suite that provides users with myriad ways to customize it—for example, through the ability to call custom monitoring scripts for monitoring and remediation. As such, it’s vital that access to the EventSentry management console and configuration is monitored and restricted via Windows permissions. By default, EventSentry:

* only gives local Administrators access to the configuration
* logs every launching of the management console
* logs every time the configuration is changed
* agents log an event when a new configuration is applied

Nevertheless, bugs in user-mode software can still negatively impact a monitored system. For example, a user-mode process can use up all CPU time or exhaust all available memory, slowly causing a system crash. User-mode processes can also impact the kernel indirectly, for example, by opening (and not closing) extremely large numbers of thread handles that are allocated in the kernel nonpaged pool.

Exhausting the nonpaged pool (which can only be stored in physical RAM) can also cause a BSOD. For example, the C++ code below, when executed on a system, will slowly bring it to its knees until it’s either unresponsive or crashed. Windows Server has no built-in protections to prevent this from happening.

unsigned int ThreadTest(void *dummy)
{
    while (1)
        { Sleep(1000); }
}

int main(int argc, char **argv)
{
    std::vector<HANDLE> threadHandles;

    while (true)
    {
        HANDLE hThread = CreateThread(NULL, 10240, (LPTHREAD_START_ROUTINE)ThreadTest, NULL, 0, NULL);
        if (hThread != NULL)
            threadHandles.push_back(hThread);
    }
}

These type of “subtle” bug takes time, however, and EventSentry’s extensive performance monitoring features would detect such abnormal resource usage on a monitored system in various ways.

EventSentry also doesn’t use content files like CrowdStrike does. Instead, rules are shipped through package updates which:

* Do not get shipped automatically and require the user to open the management console.
* Do not have the ability to crash the agent or the system.

It’s important to understand that developing software products—especially when used within the increasingly complicated Windows ecosystem—is a complex and intricate process. Supporting multiple platforms, languages, options, and configurations further complicates it. While it’s unfortunately impossible to write perfect, bug-free software, working only with experienced developers and utilizing both automated and manual testing procedures can minimize the risk of disruptive bugs without affecting the evolution of the software product.

CrowdStrike’s Falcon sensor, due to its operation in kernel mode, is both a powerful and high-risk software product—similar to hardware drivers. It appears that CrowdStrike’s QA efforts for its “Rapid Response Content” were not proportional to the risk posed by the Falcon sensor, with extensive QA being applied only to Falcon sensor software updates. It is surprising and disappointing that a large software corporation like CrowdStrike, which sells an expensive software product, did not anticipate this risk ahead of time and instead offloaded the risk to its users. Software vendors can all learn from this disconnect and expand their QA efforts beyond just software code.

Adding insult to injury is the fact that the current CEO of CrowdStrike, George Kurtz, was CTO of McAfee when that software also crippled millions of computers back in 2010. It did this by marking a core Windows executable – svchost.exe – as being infected with a virus and deleting it.

EventSentry 5.1.1.104: Security, Security, Security!

ingmar.koecher 0 Comment Uncategorized , ,

Everybody wants to have a more secure network – and everybody has various tools at their disposal to at least improve the security of their network. But which tool is the best for the job, and where do you start? The answer to this question is somewhat easier (and more structured) for organizations that have to adhere to compliance frameworks (ISO, CMMC, PCI, SOC, …), but a little harder for business that have no such requirements.

EventSentry has long included many tools to increase the security of your network, and in the latest 5.1.1.104 update we made it significantly easier to increase the security of your infrastructure with 3 new dashboards:

The security dashboards will guide you through the process of:

  • Ensuring your audit settings are correct
  • Identifying insecure settings on your network
  • Illustrating significant changes that should be reviewed

So how do you get started? After applying the latest patch, head over to the web reports and load any dashboard and hit the SPACE bar – the new security dashboards will show up in the list and you can import them there.

After the dashboards are imported, click on the Dashboard in the menu and select the first one: “Dashboard [1] Foundation”. This dashboard simply evaluates all of your audit policies to make sure they are adequate. Proper auditing is crucial, and is the foundation for many other monitoring initiatives.

Important: You can click on the header of each imported dashboard to navigate to a KB article, which explains the purpose of the dashboard in detail and how to resolve any identified issues.

It’s recommended to configure audit settings via group policy, to ensure that settings are always enforced and apply network-wide. You will know that your audit settings are correct when all tiles on this dashboard are “OK”. Please keep in mind that it may take an hour before new audit settings are pushed by GPO to your end points and subsequently picked up by EventSentry. Detailed information about this dashboard is available here.

Once your audit settings are in the clear you can graduate to dashboard #2 – Attack Surface. This dashboard utilizes various validation scripts to ensure that your monitored hosts follow best practices and/or pass security and compliance requirements. The checks are generally divided into different categories like

  • Best Practices
  • Security
  • Privacy
  • CIS Critical Security Controls

Most findings on this dashboard can usually be resolved via group policy and/or registry settings. The end result is a reduced attack surface of your network since your monitored hosts will now adhere to recommended best practices and security recommendations. More information about this dashboard can be found here.

The last dashboard, “Security [3] Critical Changes” illustrates critical changes that have occurred on your network (and domain) recently. This dashboard should be reviewed daily and will reveal important changes like

  • Services/Drives/Scheduled Tasks added
  • Software Installed
  • Recent logon failures
  • Important AD changes

Patch 104 also includes improvements to the names of built-in event log packages, which have been consolidated and given better, more descriptive names. As such, expect a lot of changes if you do a package update in the management console.

Another useful improvement is that you can now easily access information about security events with a new “info” button in the management console.

If you’re running EventSentry 5.1 then download the latest patch and make sure to install & review the new security dashboards!