Appendix B: Best Practices for File Integrity Monitoring
This appendix provides best-practice suggestions for efficiently deploying and configuring Halo File Integrity Monitoring in small to very large-scale environments. The five steps below contain recommendations for designing server groups, creating policies, applying baselines, performing ongoing monitoring, and handling server upgrades in deployments that can scale from a few servers up to thousands of servers.
1. Organize servers into groups according to geography or function
The basic purpose of a server group is to hold similarly configured and purposed servers, so that the same set of Halo policies can be applied to all of them. In general, take these considerations into account when designing your groups.
- When setting up your server groups, especially in a very large deployment, we recommend that you adopt a practical naming strategy that identifies the type of servers in each group and possibly also where they are geographically located—for example, "US-East-Web-Servers".
- If your deployment includes, for example, very similarly configured web servers but with different applications running on top of them, you might append an application identifier to the server group name, to clarify which policies should be applied to each group—for example, "US-East-Web-Servers-Login" and "US-East-Web-Servers-Messaging". Being this specific and granular with groups and their names can make the assignment of policies and the generation of baselines simpler to manage.
- Servers within a given group should have the same function and very similar configurations. Usually, that means the same O.S. and the same applications. Often, the servers in a group are all clones of the same gold master image.
- You can have multiple versions of the same operating system or application in a single group. The multiple baselines feature of File Integrity Monitoring allows you to accommodate this kind of departure from absolute uniformity.
- You can also have more than one operating-system platform in the same server group. File Integrity Monitoring allows you to assign both a Windows and a Linux policy to a single server group. Halo automatically applies the appropriate policy and its baselines to each server, depending on its platform.
Naming groups according to function complements the modular policy-design best practice described below, in which you create broad policies that cover the common operating systems and applications, plus additional focused policies for the specific applications or data that might be unique to certain subsets of servers.
2. Create narrowly targeted file integrity policies
When developing file integrity policies, try to be as specific as possible when choosing your file and directory targets. For example:
- Be judicious about where to use recursion on a target directory. If you’re concerned about the state of just a few files in a directory, it is more efficient to use them as separate specific targets, rather than setting up their parent directory for recursion.
- On the other hand, if you are more concerned about detecting the addition or removal of files inside a directory, marking the directory target for recursion will detect those additions or deletions (although it will also result in all existing files in the directory and its subdirectories being examined for changes.)
- We do not recommend targeting any files or directories that you expect to change dynamically over time, such as log files, temporary directories, upload directories or
For Linux policies:
- Monitor the operating system binary files in
/usr/sbinand anywhere else appropriate if you have deployed code for your running services.
- Monitor specific configuration files (typically found in
/etcor in its subdirectories).
- Monitor any static custom code or content that you have deployed to your servers and that should not change except when you explicitly update it.
For Windows policies:
- Monitor the
System32folders, and specific applications in the
Program Filesfolder (as analogous to the operating system files in Linux).
- Monitor portions of HKEY_LOCAL_MACHINE in the Windows registry.
- Also monitor any static custom code or static content that you have deployed.
Since you can assign more than one file integrity policy to a server group, consider creating individual, modular file integrity policies for all of the various file types (O.S.-specific binary files, application-specific binary files, configuration files, static content, and so on) that you wish to monitor. It can be easier and more efficient to maintain these focused policies than it is to keep one monolithic policy up-to-date. And you can conveniently assemble them into a custom policy set for each server group, depending on its O.S. platform and its specific applications.
Note: Halo currently provides default O.S.-level file integrity policies for Windows Server platforms.
Assigning a policy to more than one server group
If you have several server groups with identical or very similar configurations, you might wish to assign the same file integrity policy to all of them. Instead, because of the way that file integrity target comparisons are made, it may be more efficient to clone the policy and rename it for each new group that you assign it to.
The reason for this is that, if you assign one file integrity policy to multiple groups, all baselines generated for that policy—from any of the groups—will be included in the scans of all of the groups. A large number of baselines that don't apply to a particular group would unnecessarily be included in the processing of that group's scans, leading to wasted time making the unneeded comparisons.
3. Generate and actively manage baselines
Every unique server image used in your fleet requires a file integrity baseline that describes the configuration's canonical, secure state with respect to a given file integrity policy. If you wish to allow a single server group to accommodate more than one canonical image (for example, you might want it to support two or more upgrade levels of a given O.S. or application), you can do that by defining multiple baselines for the group's file integrity policy.
- Use multiple baselines for server groups whose software you are upgrading. For example, if you have three common "gold master" server images at different upgrade levels, from which you launch anywhere from a few to perhaps hundreds of servers in a given server group, you'll need to generate a baseline for each of those master images when editing the file integrity policy assigned to the server group.
- If you subsequently add new software to any server in that group to allow it to support a new function—or if you upgrade one to the master images and instantiate new servers from it—you'll then need to generate a baseline for that new configuration and assign it to the group's file integrity policy, so that future scans do not report the configuration as a (false positive) policy violation.
- Note that removing unused baselines is just as important as creating new ones. If you have an old baseline for a server that no longer reflects any of the machines running in your environment, leaving that baseline assigned to a policy is inefficient; it can lead to extra comparison time spent during a scan for no useful result. Make a point of manually (or automatically, through automation scripts) removing baselines from policies if the particular server configurations they represent are no longer running.
Alternatively, when you create each baseline, set its expiration so that it will expire shortly after you will have upgraded all of your servers to a new configuration. (You can do this if you upgrade on a regular schedule.) If you set a baseline's expiration to "never", you'll have to remember to manually delete it at the proper time.
- Creating modular, narrow policies and defining multiple baselines can go a long way toward reducing the potential “alert load” of false positives that can can occur when software changes are introduced. By updating the individual affected policies (easier than searching for the affected targets in a monolithic policy) and generating new baselines, you can eliminate or greatly reduce those false positives across your fleet of servers.
Note: Adding and removing baselines in relation to software upgrades should be considered in the context of a "baseline lifecycle". See the discussion and diagram under Step 5. Upgrade servers without disrupting file integrity scans.
Planning out your initial file integrity monitoring deployment ahead of time, including all unique server builds in your baselines, and modularizing your policies should help your ongoing operations run more smoothly.
Add comments to your baselines
When you create a baseline in the Halo portal, you should always put in a meaningful comment to help keep track of the baseline and remember what configuration it represents. At the very least include the date that the baseline was generated and the name of the user that created it. This helps capture the institutional knowledge and fosters coordination with the responsible party (or parties) as policies change, or as baselined servers are retired or removed from your environment.
If you use an IT automation tool like Puppet, Chef, or Ansible in your environment, consider leveraging the Halo REST API in your scripts or recipes.
For example, if you have an automation script that handles things like new package installs or file content updates, have the script create a new a file integrity baseline of the new or updated server configuration, so that the server group's file integrity policy can automatically adapt to the changed environment, instead of reporting false positives when the next file integrity scan is run.
4. Keep your ongoing monitoring efficient
To maintain maximum efficiency and effectiveness of your File Integrity Monitoring implementation, consider these recommendations for common practices and specific configuration settings that you can make. Making these settings adjustments is another way to significantly reduce your file integrity "event load" .
In the Site Administration > Scanner Settings tab you can configure Halo’s behavior for the File Integrity Monitoring module (and others). The following are useful recommendations at any scale, but especially for large deployments:
- Scanning Frequency. For maximum security during an investigation, you might temporarily set the file integrity scan frequency to as often as "hourly". For compliance monitoring, where immediate response is not as important, "daily" or even "weekly" might be sufficient. For general security purposes, "daily" for large installations and "twice daily" for smaller installations is probably a good compromise between maximum security and minimum load.
You can always run an ad-hoc or on-demand file integrity scan from the Halo portal or through the API at any time.
- Scanning on agent Start. If you are going to be launching a large number of servers regularly in your environment we recommend that you uncheck the checkbox “Execute scan on agent start” for the File Integrity Monitoring scanner. It's possible that a scan could start before your orchestration scripts have finished configuring the server, which could make that initial scan invalid.
- Aggregate file integrity events. Halo's default settings aggregate all changes detected in a single file integrity target into a single "file integrity change detected" event. This keeps the volume of file integrity events in check, while the details about the changed objects are available in the historical scan results. We strongly recommend that you leave the checkbox "Generate a separate event for every change detected by a file integrity monitoring rule" unselected at all times, except for temporary situations in which the each changed object changed must have its own event.
5. Upgrade servers without disrupting file integrity scans
Depending on your organization's procedures for applying operating system and software patches and updates, you should take into consideration when and how often you should baseline or re-baseline the unique server configurations in your fleet, and when to expire older baselines. Plan your upgrades, baselining, and retirement of baselines to follow a consistent "baseline lifecycle" throughout your server fleet.
File integrity baseline lifecycle
A. If you re-instantiate servers from a patched gold master...
If your organization's policy is to perform updates only on a “gold master” image that you then use to instantiate all subsequent servers, you can
- Launch that image and apply the updates.
- Create a new baseline from that image. Do not remove the old baseline yet.
This is a useful preventative measure especially for large deployments, because the next file integrity scan could possibly begin before you have had a chance to upgrade all of your servers form the upgraded gold master. Having the old servers covered by the pre-existing baseline and the newly upgraded ones covered by the new baseline will prevent false alerts caused by the older configuration.
- Shut down the existing, unpatched servers according to schedule and bring up the new, fully patched servers from the image that you have updated. You should not receive false-positive events from your file integrity scans.
- Once the update is completed across all servers, the old baseline should automatically expire (if you have set its expiration date appropriately—typically 60 days or less if you apply updates regularly).Otherwise, manually remove the unneeded baselines.
B. If you patch your servers in-place...
If you will be upgrading systems in place, you should
- Make sure that the first servers you upgrade each time are the ones you have built your file integrity baselines from.
- Once those upgrades are complete, instead of just re-baselining the original baselined servers, create new, additional baselines for those upgraded servers—because the next file integrity scan could possibly begin before you have had a chance to upgrade all of your servers in place. Having the old servers covered by the pre-existing baseline and the newly upgraded ones covered by the new baseline will prevent false alerts caused by the older configuration.
- Once all your servers are upgraded, the old baseline is no longer be needed and you can remove from the policies—or, if you have set its expiration appropriately, it will expire automatically. For example, if you update your servers every 60 days, set your baselines to expire at 60 days or sooner.
As previously mentioned, applying descriptive comments when creating a new baseline or re-baselining a server will help you keep everything clarified, making it easier to manage updates and baseline retirement.
Also keep in mind that you can automate the process of baselining and integrate it into your upgrade process by inserting Halo API client scripts into your server orchestration tools.
For More Information
For more information about how Halo File Integrity Monitoring works and how you can configure it to give you the best results, see the following discussions:
- Setting Up Server Groups in the Halo Operations Guide.
- Creating a File integrity Policy in this document.
- Specifying a Baseline Server and Running a Baseline Scan in this document.
- Specifying File Integrity Monitoring Settings in this document.