Our latest patch for EventSentry v3.2 (v184.108.40.206) requires some additional information in addition to the release notes.
Heartbeat Monitoring (Agent Status) By default, the EventSentry Heartbeat Monitor ensures that all remote agents are running by querying the status of the remote “EventSentry” service. While is an accurate way to ensure the remote agent is running, the Microsoft RPC mechanism isn’t very efficient when connecting to remote hosts across a slow (WAN) link, and concurrently checking the service status of 100+ hosts at the same time can on occasion also cause issues. In these situations, the heartbeat agent may not be able to monitor all hosts in the configured monitoring interval. Furthermore, querying the remote status of a service requires that the EventSentry Heartbeat Agent run under a domain account, otherwise the dreaded “Access Denied” error appears on the heartbeat status page in the web reports.
To address these issues for larger EventSentry deployments (500+ hosts) and deployments where the remote agents are connected through a slower WAN link, we have added the ability to query the remote agent status through the EventSentry database where the remote agents periodically check in. This check is enabled by default for new installations, but existing installations will need to make a database permission change in order to give the heartbeat agent permission to query the agent status. More information can be found here.
In the next release of EventSentry (v3.3), this functionality will be configurable, and the heartbeat agent will also be able to determine the current agent status by communicating directly with the collector service (when enabled) for even better accuracy. The Heartbeat Monitor will always attempt to revert back to the legacy method of checking the service status directly if it cannot obtain the status through other means.
Service Monitoring: Configuration Changes EventSentry distinguishes between three types of service changes: Status changes (e.g. Running to Stopped), service configuration changes (e.g. changes to the startup type) and services being added or removed. Up until release 220.127.116.11, all status changes and service configuration changes were logged with the same event severity, which we didn’t think was very fitting since the status change of a service is very different to a change of the service itself. As such, starting with 18.104.22.168, only service status changes will be logged under the severity configured under “Monitor Service Status Changes” category. All other service changes will be logged under the severity configured under “Monitor Service Addition / Removal” category.
Management Console: Quicktools
The EventSentry QuickTools allow you to run an application/script against a server or workstation in your EventSentry configuration. EventSentry includes a few default QuickTools entries, such as “Reboot”, “Remote Desktop” and others. Starting with the latest release we added a new “Hide” option, which will not show the executed application on the desktop. This will be useful for integrating our upcoming VNC wrapper scripts (Blog article coming soon), which will allow you to install & launch a (Tiger)VNC client directly from the EventSentry management console.
EventSentry Light 3.2 Starting with this release, EventSentry Light v3.2 will also be available. We have good news for all EventSentry Light users: We have increased the number of full hosts you can remotely manage to 5, and also increased the number of network devices you can monitor to 5. As such you can now monitor up to 10 hosts with EventSentry Light completely for free.
There seems to be a new variant of ransomware popping up somewhere every few months (Locky being the most recent one), with every new variantion targetting more users / computers / networks and circumventing protections put in place by the defenders for their previous counterparts. The whole thing has turned into a cat and mouse game, with an increasing number of software companies and SysAdmins attempting to come up with effective countermeasures.
I’ve already proposed two ways to counteract ransomware on file servers with EventSentry part 1 and part 2, both of which take a little bit of time to implement (although I’d argue less than it would take to restore all of your files from backups). In this post I’m proposing a third, and better, method with the following improvements:
In the first article we configured file integrity monitoring on a volume, and if the number of file modifications occurring during a certain time interval exceeded a preset threshold, the ransomware would be stopped in its tracks. In the seconds article we used bait (canary) files to accomplish the same thing.
In this third installment we’ll keep track of the number of file modifications made by a user to detect if an infection is underway. To effectively defeat ransomware, we have to be able to distinguish between legitimate user activity and an infection. To date we know this:
Users add/change/remove files, but the number of changes made by a user in a short amount of time (say 15 min) is generally small
Ransomware always runs in the context of a user, and as such an infection will usually come from one user (unless things go really awry and multiple users are infected). The approach here will work equally well, regardless of the number of infections.
Thus, to detect an infection, EventSentry will be counting the number of file modifications (event 4663) with its advanced threshold capabilities. If the threshold is exceeded, EventSentry will trigger an action of your choice (e.g. disable the user, remove a file share, stop the server service, …) to limit the damage of the ransomware.
Here is what you need:
Object Access / File System Auditing enabled
Auditing enabled on the files which are to be protected
EventSentry installed on the server which needs to be protected
This KB article explains how to configure EventSentry and enable auditing (preferably through group policy) one or more directories. I recommend referencing the KB article when you’re ready to configure everything. Pretty much everything in the KB article applies here, although we will make a small change to the threshold settings of the filter (last paragraph of section (4)).
Once auditing is setup, Windows will log event 4663 for every write access which is performed by a user. An example event looks like this:
The default behavior of a filter threshold in EventSentry is to simply count every filter match towards the threshold. In our case, every 4663 event encountered would count towards the threshold. You can think of there being one bucket for all 4663 events, with the bucket being emptied whenever the threshold period expires, say every 5 minutes. If the bucket fills up we can trigger an alert.
This doesn’t work so well on a file server, where potentially hundreds of users are constantly modifying files. It would take some time to come up with a good baseline (how many file modifications are considered “normal”) that we could use as a threshold, and there would still be a chance for a false positive. For example, a lot of 4663 events could be generated during a busy day at the office, thus causing the threshold to reach its limit.
A better way is to assign each user their own “personal” threshold which we can then monitor. Think of it like each user having their own bucket. If a user writes to a file, EventSentry adds the 4663 event only to that user’s bucket. Subsequently, an alert is only triggered when a user’s bucket is full. Any insertion string of an event can be used to create a new bucket.
We can do this by utilizing the insertion string capabilities of the filter threshold feature. Setting this up is surprisingly easy – all we have to do is change the Threshold Options to “Event”, click the “Insertion Strings” button and select the correct insertion string. What is the correct insertion string? The short answer is #1.
The long answer lies in “Event Message Browser”, which you can either find through the Tools – Utilities menu in the EventSentry Management Console or in the EventSentry SysAdmin Tools. Once in there, click on “Security”, then “Microsoft-Windows-Security-Auditing”, then 4663. You will see that the number next to the field identifying the calling user (“Security ID”) is %1.
Enough with the theory, here is what you need to implement it (assuming EventSentry is already installed on the servers hosting the file share(s)):
Create a package & filter looking for 4663 events. See section 4 of KB 279 and review the additional threshold settings below.
Customizing the threshold Once you have the package & threshold filter for 4663 events in place, we need to modify the threshold settings as explained above. Edit the filter, click the threshold tab and make sure your filter looks like the one shown below:
The only variable setting is the actual threshold, since it depends on how fast the particular variant of ransomware would be modifying files. A couple of things to keep in mind:
The interval shouldn’t be too long, otherwise it will take too long before the infection is detected.
Make sure the actual event log filter is only looking at 4663 events, no other event ids.
With the above example, any user modifying any file (on a given server) more than 30 times in 3 minutes will trigger any action associated with the filter, e.g. shutting down the server service. Note that the action listed in the General tab will be triggered as soon as the threshold is met. If 30 4663 events for a single user are generated within 45 seconds, the action will be triggered after 45 seconds, it won’t wait 3 minutes.
Bonus – Disabling a user One advantage of intercepting 4663 events is that we can extract information from them and pass them to commands. While shutting down the Server service is pretty much essential, there are a few other things you can do once you have data from the events, e.g. the username, available. You can now do things like:
Disabling the user
Removing the user from the share permissions
Revoking access to select folders for the user
There are a couple of caveats when (trying to) disable a user however:
The user account (usually the computer account) under which the EventSentry service runs under (usually LocalSystem) needs to be part of the Account Operators group so that it has permission to disable a user
Disabling a user is usually not enough though, since Windows won’t automatically disconnect the user or revoke access. As such, any ransom/crypto process already running will continue to run – even if the user has been disabled.
Disabling a user account from the command line is surprisingly simple (leave Powershell in the drawer). To disable the user john.doe, simply run this command:
net user john.doe /domain /active:no
Note that since “net user” doesn’t support a domain prefix (MYDOMAIN\john.doe won’t work), we need to make sure that we pass only the username (which is insertion string %2) and the /domain switch to ensure the user is disabled on the domain controller. Of course you would need to omit the /domain switch if the users connecting to the share are local users. The action itself would look like the screenshot below, where $STR2 will be substituted by EventSentry with the actual user listed in the event 4663:
That’s it, now just push the configuration and you should be much better prepared to take any ransomware attacks heading your users way.
I am e-x-c-i-t-e-d to announce the availability of EventSentry v3.2 and tell you more about the new features and improvements. So, if you’re looking for a little bit more than the release notes then read on!
Collector The biggest new feature in 3.2 is the collector, a new central component which enables a 3-tier architecture in EventSentry. Traditionally, the EventSentry agents have been communicating directly with email servers, databases and other services. While this usually worked well – and is still be desirable in many setups – it does impose a limitation in some scenarios:
The SMTP server cannot be configured to allow relaying and/or accepting SMTP connections from remote clients
The central database cannot be configured to allow connections from remote clients
Agents need to communicate over an insecure medium like the Internet
The collector addresses the above limitations by acting similar to a proxy between the remote agents and the service (e.g. database). In a nutshell, it provides the following benefits:
Agents only communicate with the collector over a single port
All traffic can be encrypted and compressed
Database connection details do not need to be stored on the agents anymore
All collected data is cached on the agents if and while the collector is unreachable
Whether you will need the collector or not will largely depend on your network setup. If all of your hosts are in the same data center and/or the same LAN, the collector may provide little benefit. If you are a MSP and monitoring remote sites and laptops however, then the collector is probably what you have been waiting for!
When upgrading (or installing from scratch), the post-installer configuration assistant will ask you whether you are interested in enabling the collector.
If you are installing from scratch, then enabling the collector during the installation is all you need to do. When upgrading, an additional step is required – an action needs to be configured to use the collector. While the collector service is installed & started during the upgrade when selected, it will not enable any of the existing actions to use the collector. As such, if you want to route data for a specific action through the collector, that needs to be configured. Simply edit the action and click the “Use Collector” check box on the bottom left and push the configuration.
In version 3.2.1, the following actions can be routed through the collector:
Since the collector, when enabled, is a critical component, we recommend monitoring the collector stats either through the collector status page (Maintenance -> Collector Status) or by adding the collector status tile to one of your dashboards.
There is one other advantage the collector can bring when routing emails through it:
Emails from multiple hosts can be grouped together (if the action polling interval is sufficiently high)
Action thresholds can now be applied centrally
Both features can help reduce the number of emails you receive from EventSentry, usually a popular thing to do!
EventSentry has always included the compliance tracking components which monitor and interpret Windows security events. Compliance tracking provides process, console, account management and other tracking reports. While popular and extremely useful, the compliance reports themselves don’t tell the user which particular compliance requirement they address.
Say Hello to the new compliance modules, which provide detailed, out-of-the-box reports for:
Once a compliance module is enabled, it will install a number of reports that pertain to the specific compliance requirement that was enabled. Every report will be associated with a specific control (e.g. PCI 10.2.2) and allow you to setup a required review, job and more.
(Network) Switch Mapping Finding the port on a switch to which a server, workstation or network device is connected is often a time-consuming and annoying process for most SysAdmins. Starting with version 3.2, EventSentry tries to ease that pain by showing exactly to which switch – and port – a host is connected to. All you need to do is add the switch to the EventSentry configuration, make sure that it can be monitored via SNMP and that it provides the MAC to port mappings via SNMP (OID 22.214.171.124.126.96.36.199.3.1.2 – iso.identified-organization.dod.internet.mgmt.mib-2.bridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbPort). This feature should work well with all mainstream managed switched, and we haven’t run into a switch yet where this feature wasn’t provided or did not work.
Once EventSentry pulls the MAC to Port mappings, you will be able to retrieve the collected information in two ways:
Through the Inventory – Switch page, which will show all monitored switches and connected devices
Through the Inventory – Host page. If the switch port can be detected, it will be displayed next to the IP address of the network card
Since switches only provide MAC addresses, EventSentry attempts to map MAC addresses to host names and IP addresses by analyzing the hardware inventory details as well as the ARP status table when available. As such, it is recommended to enable the ARP component of the network services if the results on your switch inventory page are incomplete.
Web Reports Improvements
Starting with a visual overhaul of the interface, the web reports also received an internal overhaul to improve overall performance, especially when using multiple profiles. The performance trends page can now display multiple charts on a single page, and the host inventory page now shows the highest supported USB version on that host.
Managing multiple reports is easier now through the ability to bulk-edit reporting settings. Reports can now also be saved to a folder instead of being emailed.
Finally, the web reports are now also officially available in 6 additional languages: French, Spanish, Polish, Portuguese, Dutch and Italian. This brings the total number of supported languages in the web reports to 9!
Management Console Improvements
Improvements in the management console pertain mostly to remote update and computer management. Hosts can now be imported from a network scan, which is particularly useful when managing network devices which often don’t show up in Active Directory. The network scan is multi-threaded and can scan a class C subnet in a few seconds and even supports checking TCP ports for hosts which have ICMP disabled.
Remote update can now store the result of every activity in CSV file(s), and output from remote update can be toggled with the context menu to apply remote update actions to a sub-set of hosts easily.
Also new is the ability to export all event log filters to a CSV file allowing you to analyze the results in your favorite spreadsheet application to identify issues, duplicates etc.
I’m excited to announce a new version of our free EventSentry SysAdmin Tools which, in addition to bug fixes and minor improvements, also includes a new command-line tool: snmptool. This brings the total number of utilities in the toolkit to thirty (30)!
Free SNMP tools for Windows® are not easy to find and often require you to memorize the various OIDs in order to test a remote host’s SNMP functionality, or to get useful information back.
Our free snmptool utility solves that problem by giving you a simple utility which downloads a variety of stats, depending on what the remote host provides via SNMP, and displays it to the user. For example, if you are querying a VMWare® ESXi™ host with the snmptool, it will – among other stats – enumerate all VMs configure on the host, whereas it will display switch port mappings when querying a switch.
The snmptool currently retrieves the following:
System Description string
Current CPU usage
Network interfaces (name, MAC address, IP if available)
Virtual Machines (ESXi™ only)
Switch port mappings
Running the utility is incredibly easy, simply specify the SNMP credentials and the remote host, and the utility will do the rest on its own:
C:\>snmptool /u public linuxserver
System Description: Linux openvas.netikus.local 4.32.22-573.7.1.el6.x86_128 #1 SMP Tue Sep 22 22:00:00 UTC 2019 x86_128
OS Info: Linux 4.32.22-573.7.1.el6.x86_128 #1
Current Uptime: 3 years, 321 days, 3 hours and 52 minutes
CPU Usage: 0%
Emails are still the alert type of choice for most network administrators, but they come with a major pitfall: They either rely on your local email server, the Internet connection, or worse: both.
So if you need to get notified when your email server or Internet connection are down – and most likely you will want to get notified – text messages (aka as SMS) are a convenient way out of this monitoring conundrum.
Email to SMS, or web-based SMS services can help in some situations, but you need a different kind of beast when the Internet connection is down.
This is where hardware solutions like the SMSEagle come in. The SMSEagle is essentially a cell phone with an Ethernet jack, a web server + API, allowing you to send text messages either through its web interface or API (preferred).
The SMSEagle and similar devices usually only require three things:
A valid SIM card
A physical location with coverage from your provider
An Ethernet connection to your network
With the SMSEagle in place, you no longer have to fear loss of Internet connectivity or mail server downtime – a text alert is just a few seconds away when combined with a monitoring solution like EventSentry. In fact, the SMSEagle is somewhat unique in that it actually offers its own basic network monitoring capabilities which can be used in addition to EventSentry – or to monitor the host running the EventSentry Heartbeat Monitor.
Since the SMSEagle features a web-based API, EventSentry can submit certain alerts, e.g. pertaining to Internet connectivity or the availability of a specific host and/or service, directly to the SMSEagle through its HTTP action, which will then send a SMS alert directly to your mobile device of choice.
The SMSEagle accepts mini SIM cards and works with the GSM/GPRS/EGPRS 900/1800/1900 MHz wave bands. For users in the U.S. this unfortunately means that the Verizon & Sprint customers will not be able to utilize this device. The setup of the device is straightforward: You first insert the SIM card, connect it to the Ethernet and power on the device. The device automatically assigns itself IP address 192.168.0.101 but can also retrieve its IP address via DHCP. More instructions & details are available in the manual.
Once you’ve setup the SMSEagle and verified that SMS messages can be sent and received, you can create an EventSentry HTTP action and point it to your SMSEagle device. The HTTP action includes a number of templates to quickly load all required fields for a specific HTTP API; simply select the SMSEagle from the drop-down and specify all the required fields such as host name and so forth. Use the test button to ensure the action is setup correctly.
Once the action is setup it can be referenced by one or more filters so that it is triggered under the right circumstances. Assuming that you are already monitoring a host outside your network (e.g. your ISPs DNS server, a public web site) to determine whether your Internet connection is available or not, you would want to look for EventSentry event 11000, which indicates when a host changes its availability status, such as:
Host ispdns (Internet) changed its PING status from OK to ERROR. The reason for the status change was: “100% packets lost”.
A filter setup for this event would look like something like this:
Here, we are looking for an application error generated by the Heartbeat Monitoring category of the EventSentry source. We’re also further restricting the filter to only alert us on status changes of the “ispdns” host when it goes off-line. Since the SMSEagle is listed as the action, this particular event (alert) will be sent to the SMSEagle action.
The insertion strings can be determined by either clicking on the “Lookup” button on the filter dialog, or by clicking on the “Preview” button when adding a content filter.
If you’re located in the Europe then the SMSEagle is a non-brainer, but it’s also a good option for US-based customers who work with a GSM-based cell provider like T-Mobile or AT&T. Otherwise, there are a few other devices out there who work similarly , and as long as they offer a HTTP-based API, integrating EventSentry with them should be easy.
A final note for users in the U.S.: The device comes with a European power connector (“Europlug”) which won’t work in the U.S., you will need to purchase a simple adapter either at your local electronics store (if such a thing exists) or online retailer. A voltage transformer is not necessary since the device operates from 100V to 240V.
Trello is a simple yet powerful and innovative task management / collaboration platform for teams. With Trello, the developers have basically taken the familiar concept of traditional white boards where you add and remove tasks (by writing on them), and moved it to an easy-to-use online tool.
While Trello doesn’t attempt to replace the more complex project management and collaboration tools available (including its own FogBugz platform), it makes keeping track of small ToDo lists and tasks surprisingly simple, while still supporting advanced features such as due dates, attachments, assignments and more. Of course, Trello also includes a very capable mobile app for iOS and Android (I only tested the iOS version).
And best of all, it’s completely free if you stick with the basic (and for most people completely sufficient) functionality. But what does Trello have to do with EventSentry and cutting down on emails?
We’re always looking for innovative ways to make managing alerts easier and more productive, especially in larger teams. While email alerts certainly serve a purpose and can be quite useful, alerts dispatched via email suffer from a few disadvantages:
Emails sent to multiple recipients make it difficult for the recipient to know whether the alert has been acted upon or not
Alerts which have already been resolved by a team member still remain in your inbox
Emails often get lost amidst other emails and potentially critical alerts may get overlooked
How Trello Works
Trello is organized into boards, each of which can have one or more lists, each of which have multiple cards. Since Trello offers an API, you can use EventSentry’s HTTP action to submit events (alerts) directly to one (or more) Trello lists.
And this is where the fun starts. Once in Trello, alerts (now cards, or “alert cards”) can be acted upon in a variety of creative and useful ways. You can:
Receive alerts in your browser when a card is created
Move a card to a different list (e.g. “Resolved”, “Under Investigation”, …)
Assign one or more people to a card
Add comments to a card
Assign a due date to a card
Mark a card as important (you can even define your own color codes)
Receive periodic summary emails if you don’t visit the board
All of these features make managing alerts in teams with multiple SysAdmins much easier. When an alert comes in, anybody can act on it (e.g. add themselves) or assign it another team member. Any changes are immediately visible to all other team members in real-time (and we at NETIKUS love anything real-time).
Integrating EventSentry with Trello is a 3-step process:
Sign up for Trello, create a board and customize the associated lists
Get an API & access key & determine ID of your list
Setup HTTP action in EventSentry and create/modify rules
Signing up for Trello
To get started, navigate to http://www.trello.com and sign up with an email address. After you log in for the first time, you will automatically get the “Welcome Board” which will show you all the things you can do with Trello. Since we don’t want to use the default board, we click the big PLUS icon on the top right instead and select “New Board”.
Give the board a descriptive name, e.g. “EventSentry Alerts”. Once created, the board will contain three default lists. You can either leave the list names as they are, or customize them as shown in the screen shot below. I chose “Active”, “Working on” and “Resolved”.
Getting an API and access key
Now that you’ve signed up, the next logical step is to get the API key so that EventSentry can start submitting events to Trello. So while you are logged in, navigate to https://trello.com/1/appKey/generate and note down (aka copy & paste) the first value “Key”, a 32 character-long hexadecimal value. This is the “main” key for your user account, and will be used whenever you (or EventSentry) make an API request.
The API key doesn’t actually let us access data from the boards, for which we’ll need an access key. There are different types of access keys with customizable expiration dates available, but in this case we’ll just get a read/write key without an expiration date. Navigate to the following URL to get a universal read/write access key and substitute APIKEY with the key you obtained just before:
You will end up with a dialog similar to the one shown above, where you need to click the green “Allow” button. This will issue another hexadecimal key, this time 64 characters in length. Note this key down as well. Of course you can be less generous and issue keys which expire automatically, e.g. after 30 days. See the Trello docs for more details on the different “expiration” options available.
Getting the list ID
Our end goal is to submit cards to the “Active” list on our “EventSentry Alerts” board. In order to add a new card to this list however, we’ll need the list’s ID. Equipped with our main key and access key, we’re almost there. First, navigate to your “EventSentry Alerts” board in Trello (or whichever board you want to submit cards to) and note down the URL. For example, if the URL is https://trello.com/b/gePT9Wax/eventsentry-alerts, then you’ll want to extract the text between the /b/ and the board name, gePT9Wax in this case. Now, navigate to the URL below, and replace APIKEY with the API key, and ACCESSKEY with the access key:
What we are interested in is the list id of our “Active” list, 561e92617481e9a123aef3b00 in the example above. With the last missing piece of the puzzle in our hands, we’re now ready to setup a HTTP action in EventSentry.
Right-click the actions container or utilize the ribbon to create a new HTTP action. In the action dialog, specify the following URL, replacing LISTID with the list id we just obtained:
In addition to the URL, we’ll need to specify at least 4 form fields:
The key and token fields need to be replaced with your API key and access key, whereas the name and desc fields can be customized to suit your needs: what I have shown above is just an example which should work reasonably well in most cases. You can add or remove other event variables as you wish. The upcoming v3.1 will include Trello in the template list to make this a bit easier.
Once the action is configured, click the Test button to ensure that all IDs have been specified correctly. If the test succeeds, then you should see a new card in the “Active Alerts” list in the EventSentry Alerts board.
Of course an action alone will not forward any alerts to Trello, so you will need to make some changes to your filters and packages. You can either modify existing filters / event log packages and replace the email action with the new Trello HTTP action, or add the Trello action to existing event log packages / filters. Remember that actions can be defined on a package-level through the package properties as well which can help save time.
Managing Alert Cards
Once your first alert card arrives in the “Active” lists and is analyzed by a team member, a few actions can be taken:
You can add a team member to the card, essentially assigning the alert to them. You can add multiple team members as well
If the event is a false alert, it can be moved to a “False Alert” list, which would indicate that an exclusion filter should be setup in EventSentry
You can assign a due date, if the alert requires a resolution by a specific date
You can add a comment to the card
You can label the card (e.g. “Important”)
You can archive & delete the card
As you can see, despite its simplicity, Trello offers quite a few features to manage and collaborate. This ensures that alerts don’t disappear in an email inbox somewhere and instead are acted upon, while also allowing collaboration with comments, due dates and such.
Additional Tips & Tricks for Trello
In order to get alerted when a new alert card is created in the EventSentry Alert boards, you’ll need to subscribe to the board. This ensures that you will get a notification on your mobile phone, browser (when enabled http://blog.trello.com/how-to-use-trello-like-a-pro/) or email every time there is activity on a board. Activities include new cards being created, cards being moved to a different list, users being added to cards and so forth.
Note: You will not get a notification if the EventSentry Agent is submitting new cards while using your access key (only other users will see the alerts). This is because Trello assumes that you are creating the cards, and subsequently not notifying you about them.
One way to circumvent this restriction is to create a “service” account (e.g. email@example.com) and issue the access token under this user. Then, everybody will see the alerts.
But don’t stop there!
Of course you can use Trello for what it was originally designed to do as well – manage tasks. We’ve found it to be a great and easy way to handle ToDo lists for teams, resulting in more transparency and efficiency. Assigning a task is quick and easy, and team members can easily track progress with projects – without pesky emails floating around between team members.
Now you just have to get all your To-Do items actually done too. But at least I can now move my “Create Trello Blog Post” card into the “Done” list. And that feels good.
Every Windows server runs a seemingly ever increasing number of services which range from built-in services providing core Windows functionality (e.g. Print Spooler, Bitlocker, WMI) to 3rd party services added when installing 3rd party software (e.g. various software update services, MySQL) – all of which run in the context of a specific user account.
For example, Windows Server 2012 includes more than 300 services, about half of which are automatically running (this particular server has SQL Server installed as well):
That user account is either a built-in security principal of Windows (e.g. NetworkService), a user account specifically created for that service, or another user account from the server or domain.
Services should always run under a user account which has the least amount of privileges necessary to do its job. It’s common, and often tempting, to run a service an administrative account like “Administrator”. While this often the easiest way to “get it working”, it’s also the least secure.
When a service runs under the “Administrator” account – especially if it’s the domain Administrator account – the service has almost unrestricted access to all resources on the host or, in case of a domain admin, on the domain. This is not something a service usually needs nor you want. It also means that the service will stop working whenever the password of the Administrator account is changed (the service will continue to work until it is restarted).
Less is Better
Whenever possible, try to use one of the built-in security principals available in Windows to run a service under, or create a specific user account for the service. For example, if you have a file synchronization app which runs as a service, create a “ServiceFileSync” or similar account and configure the service to run under that account. Carefully examine the rights the service requires, and only assign those privileges to the user account which the service actually needs.
When creating the user account, give it a very strong & complex password. Users won’t have to log on with that user account, so the password can be complex and long. You can optionally check the “password does not expire” option if you feel that the password is sufficiently secure and you have a short password expiration policy on your domain which could interfere with the service starting after the password expired.
In domain environments I also recommend giving those user accounts (since you will most likely end up with more than one) either a common prefix or suffix (e.g. svc_mysql) and/or moving the accounts into a specific OU. This makes managing and distinguishing these accounts easier – especially in teams with more than one SysAdmin.
The quick way: Local Services grouped by User Account
To view all locally installed services grouped by the user account they are running under, download the EventSentry SysAdmin Tools and just run srvsec.exe. This will show you all locally running services, and group the output by the user account they are running under. Srvsec can also be pointed at a remote host, and can also change the passwords stored in services. Click here for more information on srvsec.
Srvsec is a great tool to quickly see what’s going on a single host, but to manage services on an entire domain effectively a more scalable solution is available: EventSentry + AutoAdministrator – the dynamic duo!
The right way: Making sense of ALL installed services
Even when passwords for service accounts are sufficiently strong, they should still be changed on a regular basis. But which services are installed where and are using which service account?
If this is your first time examining service accounts on your network, you should first identify which services run under which user accounts. EventSentry’s service monitoring feature combined with the web-based reporting really makes this a breeze. Assuming that you have a service monitoring system health package assigned to all of your servers, you can simply open the web reports and navigate to Status – Services and get a birds-eye view of all installed services.
In the Overview view, all installed services are grouped by common attributes, including startup type (automatic startup services vs manual startup services), current status, service name and, most importantly for this post, the service user account.
Click the “Show All” link to see all user accounts, or click on a specific user account (e.g. “LocalSystem”) to filter the list and only show services running under this specific user account. In most cases you will want to click on “Detailed” to see a list of all services with more detail.
In addition to filtering and viewing details, you can also click on the header of the
username (or any other) column to see a chart depicting all user accounts used by services from all monitored servers and workstations.
Any report viewed in the web reports can also be scheduled with a job, e.g. a list of all user accounts used by services could be emailed daily/weekly. Simply click the “Save as Report” link to create a report and setup a job.
The standard way to configure the user account and password used by a service is through the “Services” application in Windows. This works well for one or two servers, but not when you need to update the password for a service on multiple hosts.
This is where AutoAdministrator comes in: A free graphical tool which lets you do just that (and quite a bit more): Update the username and/or password of a service on multiple servers in a domain or work-group. Since AutoAdministrator is multi-threaded, even tasks affecting a large amount of hosts usually only take a few seconds.
To update the stored password of a service, open AutoAdministrator and select “Services” from the drop-down list on the top left.
Next, select the service you wish to update from the “Service key / display name” drop-down. If the service is not listed, simply specify the service key name in the service field. The key name is the internal name used by the service and can be obtained by double-clicking a service name in the “Services” MMC application in Windows.
Next, click on the “Set logon” tab and specify the new username and/or password. Of course you can also specify other service actions, such as restarting the service or changing the start-up type.
As the next step, select the hosts you wish to apply the selected changes to. You can select hosts from Active Directory, EventSentry, custom groups or work groups (Microsoft Windows Network).
Once the correct hosts are selected, click the “Start” button. The number of hosts which will be affected by any action is always shown on the bottom right of the application.
Over the past couple months, we’ve taken time to go through the various EventSentry SysAdmin Tools, one by one, and show you how they can benefit your environment in powerful ways. We’ve talked about the security tools, the networking tools, and the “check” monitoring utilities. As you know, the SysAdmin Tools offer a set of graphical and command-line utilities designed to help you with your daily administrative tasks. These tools are always being honed to provide simple yet powerful functionality.
This month, let’s take a look at the extremely beneficial file-system utilities: ADSList, CheckSum, DirMon, DirectorySize, FileReplace, PurgeTemp, and SuperDel. Here’s what they can do.
ADSList scans a folder structure to find any alternate data streams (aka “hidden” data streams). Alternate date streams are a feature of the NTFS file system in which you can hide payload (additional files) inside existing files. The jury is still out about whether malware uses these streams, but it’s always a good idea to make sure nobody has hidden something malicious in alternate data streams, because the Windows Explorer and directory listings don’t show them.
ADSList lists any alternate data streams that are associated with a file. When the tool finds an alternate data stream, it displays the name of the stream along with the regular file the stream is associated with. The output will also show a summary that lists the number of files analyzed, the number of files that have an alternate data stream associated with them, the number of alternate data streams that have been found, and the elapsed time.
The main purpose of ADSList is to give you a command-line utility that can be run/scheduled on a regular basis to reveal any hidden streams on a server or workstation. The /s option lets you include subdirectories.
CheckSum generates a one-way checksum (error detection scheme) of a file with a configurable algorithm and displays it onscreen. This capability is useful for ensuring the integrity of a file and making sure that it hasn’t been modified. CheckSum not only supports the SHA set of cryptographic hash functions (e.g., SHA256, SHA512), but also less secure hash functions (e.g., MD5).
To display and create a file’s checksum, simply supply the filename as the first argument. Keep in mind that generating checksums of large files (e.g., greater than 100Mb) can take a significant amount of time and CPU time.
The CheckSum utility is also included in EventSentry as an add-on to the File Monitoring feature, which can automatically generate SHA checksums and detect file modifications based on checksum changes.
Directory Monitor (DirMon) is a useful troubleshooting tool that monitors a directory (and optionally subdirectories) and displays all file changes in real-time. You simply run it on the command line, and it displays any file activity occurring on a given folder (or subfolder).
DirMon will show you when files are added, deleted, or modified. DirMon also lets you specifically include or exclude filters, so you can skip files that you aren’t interested in or show only files that you are interested in. The /I (/includefiles) option includes only files that match a wildcard filter, and the /e (/exclude) option does the opposite. The /s (/subdirectories) option includes subdirectories.
The DirectorySize (dirsize.exe) utility calculates the current size of a directory, including subdirectories, and displays it onscreen. The output shows the number of files and directories searched, and the total size in physical (actual size taken up on the disk) and logical (actual file size) bytes.
DirectorySize will process the current directory if you pass no command-line arguments.
PurgeTemp is a new and exciting tool that lets you purge files that are older than a certain number of days. The tool traverses the %TEMP% directory (or a manually specified directory) and deletes files that have not been modified in 120 days (by default). Because it scans the temp folder by default, you can incorporate PurgeTemp into a login script or run it with Task Scheduler to clean up temp files, for example. It’s a great way to keep users’ temp folders small.
You can customize and configure all of PurgeTemp’s parameters, including /t (time in days) and /p (path). When called without arguments, PurgeTemp simply shows the configured temp directory, the number of files in the directory, and their cumulative size.
SuperDelete (superdel.exe) essentially deletes all instances of a specific file. It parses a directory (including subdirectories) and deletes multiple occurrences of one file.
Suppose you have a thumbs.db file that Windows Explorer creates in every folder containing images, and you want to remove that from every folder on a drive. You can use SuperDelete for that purpose, using the <directory> variable to specify the directory to search (subdirectories are included), and the <fileToDelete> variable to find all occurrences of a file in the directory (wildcards are supported).
FileReplace is a command-line utility that parses a directory (including subdirectories) and replaces multiple occurrences of one template file with a template file of the same name.
Suppose you have 50 instances of various myfile.txt files scattered on your computer. You can quickly replace them all with a new myfile.txt file.
Another useful example is this: You have file C:\WebSite\Default\index.html and want to replace all other index.html files in the directory D:\WWW (including subdirectories) with C:\WebSite\Default\index.html. FileReplace lets you accomplish that with one command.
Streamline Your File System!
This is just another taste of the free, constantly evolving tools available in EventSentry SysAdmin Tools. Give them a try—they’re all free and will help you manage your IT infrastructure more effectively.
We’ve already talked about the security-focused and “check” monitoring utilities included in the freeware EventSentry SysAdmin Tools, part of the larger EventSentry network-management solution. The SysAdmin Tools offer a set of graphical and command-line utilities designed to help you with your daily administrative tasks. These tools are always being honed to provide simple yet powerful functionality.
Now let’s take a look at the extremely beneficial network monitoring utilities: Fping, Gethttp, IPMon+, Ntpclient, Pagesnpp, and WakeOnLan. Here’s what they can do.
Fast Ping (Fping)
NETIKUS.NET introduced Fast Ping (fping.exe) years ago as part of the NTToolkit. The tool was developed as a way to offer a faster way to ping remote hosts. Frankly, we were annoyed by the built-in Windows ping, which is far slower than its Linux and Apple OS X counterparts. That’s right, on non-Windows OSs pinging a remote host (especially one that is online) is a lightning-fast prospect—so why not on Windows? Fping solves the problem.
Fping also offers some fun options. For example, you can use the Solaris-style syntax, which shows you only whether a host is up or down. You can check a TCP port instead of doing an Internet Control Message Protocol (ICMP)-based ping. You can play a sound on successful or failed ping—a more useful capability than you might think! You can also see silly comments, and you can save your presets—something you can’t do with any other ping utility.
The parameters of this command-line utility are straightforward: The required <host> variable identifies the host name or IP address to ping; the /brief (/b) parameter performs a quick ping and only indicates whether the host is up or down; the /count (/c) parameter determines the number of packets to send; the /defaultset (/w) parameter sets the current options as the default; the /comment (/u) parameter shows unhelpful comments when performing a brief ping; the /playok (/p) and /playfailure (/f) configure sounds; the /loop (/l) parameter pings indefinitely, allowing an abort with Control + C.
As part of the SysAdmin Tools, this utility is better than it’s ever been. In addition to those fun options, it is a fully customizable tool. You can control the number of packets, the packet size, sound, display mode, and the delay. You can even set your preferences and store them as the default. To check the TCP port, simply append a colon and the port number to the host name (e.g. fping www.eventsentry.com:80).
GetHTTP (gethttp.exe) is a simple command-line utility to download files from a website through the HTTP protocol. Mostly useful for scripts, it supports HTTPS and proxy servers and shows the progress of the download in the command-line window. If you’re familiar with Curl (curl.exe), you have an idea what Get HTTP does.
The parameters of this command-line utility are straightforward: The /usewininet parameter utilizes the Windows proxy engine; the /proxyport parameter determines the IP port of the proxy server; the /proxyhost parameter determines the host name or IP address of the proxy server; the /quiet (/q) parameter specifies quiet output; among others, include username and password authentication parameters.
An excellent troubleshooting utility, IPMon+ is a GUI tool that shows all TCP, UDP, ICMP, and ARP connection endpoints between the local computer (default) and remote hosts. It’s the graphical version of IPMon, offering functionality that isn’t available in the command-line version.
IPMon+ is terrific for troubleshooting network connections and revealing incoming and outgoing network traffic for those situations where you don’t need to see every packet detail. The tool monitors all network traffic on the specified interface and shows which hosts communicate with the local host, how much data is being transferred through the IP connection, the direction of traffic, and which UDP/TCP ports are used in the communication. If IPMon+ runs in promiscuous mode, traffic from non-local hosts is also displayed. IPMon+ and IPMon both require the free WinPcap.
A simple but essential tool, NTP Client (ntpclient.exe) checks the local time against an NTP server, and optionally updates the local time to match that of the server. NTP Client supports the Network Time Protocol (NTP) up to version 3 and takes network latency into consideration when setting the local time. (Note that NTP Client doesn’t run as a service, and as such will have to be called repeatedly if you want to keep the time of a computer synchronized.)
Network latency is taken into consideration when calculating the clock offset, providing precision down to milliseconds. The primary parameter of this command-line utility does all the work: The /set (/s) parameter sets the time according to the time retrieved from the NTP server.
PageSNPP (pagesnpp.exe) sends a message to a pager using the internet-based Simple Network Paging Protocol (SNPP). The tool has a message limit of 1500 characters, but you can check with your paging provider to determine the maximum supported message length for your plan and device (usually less than 500). PageSNPP returns an %ERRORLEVEL% of 0 when the message was sent successfully, and an %ERRORLEVEL% greater than 0 when the message could not be sent.
The primary parameters of this command-line utility do all the work: The <SNPP_HOST> variable identifies the host name or IP address of the SNPP host, the <SNPP_PORT> variable identifies the ICP port used, and the <MESSAGE> variable displays the message to send, enclosed in quotes. (The maximum is 2,048 characters.)
The WakeOnLAN (WOL) utility sends a “magic” packet to a remote network interface card (NIC), based on the MAC address. If the NIC supports the Wake On LAN feature (and the feature is enabled in the computer BIOS of the computer), the computer will power on automatically after receiving the packet. You can also send the magic packet to a router, if the router supports direct broadcasts.
The primary parameters of this command-line utility do all the work: The required <MAC Address> variable identifies the MAC address without delimiters, and the /IP Address (/ip) parameter identifies the IP address to send the packet to (usually a router) if the remote host is not in the local subnet.
More to Come!
This is just a taste of the free, constantly evolving tools available in EventSentry SysAdmin Tools. Give them a try—you won’t be able to stop with just one.
Last week, we talked about the security-focused utilities of the freeware EventSentry SysAdmin Tools, part of the larger EventSentry network-management solution. The SysAdmin Tools offer a set of graphical and command-line utilities designed to help you with your daily administrative tasks. These tools are constantly under development, always being honed to provide simple yet powerful functionality. Three of these tools are vital monitoring utilities: CheckDB, CheckTCP, and CheckURL. Here’s what they can do.
CheckDB verifies a database connection through the Open Database Connectivity (ODBC) interface. With this capability, you can not only verify that a database server is up and running, you can also check that a database is online. You can optionally run a SQL statement of your choice. CheckDB is particularly useful because it doesn’t merely verify that a database server is online (e.g. through a port check), it also verifies that a SQL statement was successful. That capability improves the usability of this tool because it verifies that the SQL server is accepting logins and is working correctly (at least as far as that statement is concerned). Also, this tool will work with any database that supplies ODBC drivers, so it will work with MySQL, MSSQL, and so on. You can schedule CheckDB from within EventSentry (“Application Scheduler”), and even time it. The scheduling capability is a bit advanced, and the setup requires a few steps, but after getting it up and running, you can easily schedule a statement and configure it to notify you if it takes more than two seconds, for example. The parameters of this command-line utility are straightforward: The <DSN/Connectionstring> parameter is the DSN or connection strong to connect to; the /q (or /query) parameter is the SQL query you can run upon successful connection; the /u (or /username) parameter is the DSN unsername to connect as; and the /p (or /password) parameter is the password for “username.” CheckDB can log output either to the console or to the event log, making it easy to receive alerts from the utility through EventSentry or any other log monitoring software. The /I (/logToLog) and /c (/logToConsole) parameters take care of this functionality.
CheckTCP is another command-line application, this one letting you quickly determine whether a TCP port on a host is open. Additionally, you can receive initial data sent from the remote host through an open TCP connection, such as when connecting to most SMTP hosts. CheckTCP exists because Windows doesn’t really offer a built-in way to check whether a TCP port is open. Yes, Nmap is a powerful utility, but you probably often just want to know whether a server that you rebooted is available for remote desktop login. For that, you can simply run “checktcp server123 3389.” It’s not fancy, but it accomplishes a vital task. If you use the /s switch, you can get only the first line of the response. For example, if you use it against a mail server, you would get this:
checktcp /s mymailserver 25
Data: 220 mx.somedomain.com Microsoft ESMTP MAIL Service ready at Fri, 25 Apr 2014 15:07:33 -0500
The parameters of this command-line utility are straightforward: The /s parameter, as mentioned, gets initial data from the remote port (for example, when connecting to an SMTP port); the <Port> parameter displays the TCP port to connect to; and <Hostname> identifies the IP address of hostname to connect to. Although you can use this utility to display any data sent by the remote host over the established connection, CheckTCP is not intended to be used as a port scanner.
CheckURL is the HTTP version of CheckDB, and it lets you detect changes in web pages (through checksums) and look for text inside web pages. With CheckURL you’ll know when a web page changes or when a particular string is or isn’t included in a page. You might use this tool to monitor your corporate pages (at least those which are static and don’t have dynamic content) and also development pages to ensure that they don’t return a HTTP error. This is beneficial because you can have CheckURL look for specific text on the pages. The checksum feature is cool, too, because it lets you know when a page changes. As with with CheckDB, you can schedule CheckURL from within EventSentry (“Application Scheduler”), and even time it. The scheduling capability is a bit advanced, and the setup requires a few steps, but after getting it up and running, you can easily schedule a statement and configure it to notify you if it takes more than two seconds, for example. At NETIKUS.NET, we monitor our online store that way. If the store takes more than three seconds to load, we get an alert. Like CheckDB, CheckURL can log output either to the console or to the event log, making it easy to receive alerts from the utility through EventSentry or any other log monitoring software. CheckURL supports SSL as well as proxy servers.
More to Come!
This is just a taste of the free, constantly evolving tools available in EventSentry SysAdmin Tools. Give them a try—you won’t be able to stop with just one.