In cybersecurity, analysts want to gather just the right information from their logs to protect against sophisticated threats. More than 70% of analysts are drowning in alerts due to their overwhelming volume, according to recent research. So mastering efficient log collection is vital, helping analysts keep up with the constant battle against cyber threats. Efficiency in cybersecurity isnât just about cutting costs; itâs about investing wisely. By collecting the right logs and configuring the right alerts in the SOC, youâre not just saving money; youâre investing in precision defense against evolving threats
SOC analystâs struggle, with overwhelming logs and alerts, mental toll
In cybersecurity, effective data collection and log management are important for detecting and responding to potential threats. This blog post delves into the utilization of AMA for data collection in Azure Sentinel, focusing on filtering custom logs using XPath queries, sending logs to destinations using Data Collection Rules (DCR), and best practices for collecting crucial event IDs. My aim is to provide actionable tips, real-life examples, and expert insights that will aid readers in optimizing their security operations while managing costs effectively.
Understanding Data Collection with AMA
What is AMA?
The Azure Monitor Agent (AMA) is a tool designed to collect monitoring data from the operating system and network assets of various cloud and on-premises environments. It supports a range of data collection capabilities, including the collection of system logs and performance data. The Azure Monitor agent uses rules called data collection rules (DCR) to set up what data to gather from each agent. These rules make it easier to manage collection settings for various groups of environments or machines, which ultimately saves money and reduces the number of events. To configure AMA you can visit Microsoft documentation Manage Azure Monitor Agent but we are more focusing on how to filter custom logs using DCR.
DCR flow â Microsoft
What is AMA?
Centralized Management:Â AMA allows for the centralized management of data collection policies across multiple machines.
Flexibility:Â It supports various configurations tailored to specific monitoring needs and environments.
Security:Â Enhanced security features ensure that data is collected and transmitted securely.
Filtering Custom Logs with XPath Query
XPath Query Basics
XPath queries are used to filter logs based on specific criteria, ensuring that only relevant data is collected. This not only optimizes performance but also reduces costs by minimizing the volume of data stored and processed.
Creating and Testing XPath Queries
Event Viewer:
Use Windows Event Viewer to create and test XPath queries. This tool allows you to specify the event IDs and other criteria, and it generates the corresponding XPath query. Open the Event Viewer and select the log file. Choose Filter Current log and enter the Event IDs you want to collect. Click on XML for opening the Xpath structure.
PowerShell:
Test the validity of an XPath query locally using the Get-WinEvent cmdlet with the FilterXPath parameter. In PowerShell, you can filter logs further based on specific criteria. This capability becomes particularly useful when you want to refine your data collection for subsequent processing. For instance:
Examples of an XPath Query
Collect only event 4625 (failed sign-in):Â Filters to include only security events with Event ID 4625, which indicates a failed sign-in attempt. Query: Security!*[System[(EventID=4625)]]
Collect all critical events from the system log except Event ID 6:Filters to include all events from the system log with severity levels of critical events excluding those with Event ID 6. Query: System!*[System[(Level=1 or Level=2 or Level=3) and (EventID != 6)]]
Sending Logs to Destination Using DCR
What is a DCR?
A Data Collection Rule (DCR) in Azure defines how data should be collected, transformed, and sent to a destination, such as a Log Analytics workspace.
Creating a DCR
Define the Rule:Â Specify what data to collect, which can include logs from Windows Events, Syslog, or performance counters.
- Open Microsoft Sentinel
- In the menu select Data connectors (1)
- Select the Windows Security event via AMA connector (2)
- Open the connector page (3)
Now from the connector page configure the new data sources. Make sure you have read and write permissions. For collecting security events from Windows agents and installing the AMA agent. Start with creating a new data collection rule (DCR). For creating the new rule click the button Create data collection rule.
The Data Collection Rule determines the destination for data transmission. In this blog, we utilize the Microsoft Sentinel Log Analytics workspace for this purpose. Add the following details and click create:
- Rule Name:Â Name for the specific Data Collection Rule
- Subscription:Â Choose the appropriate subscription with region.
- Resource Group:Â Select the relevant resource group & platform type.
For the remaining steps, you can follow a step-by-step approach for how to configure a data collection rule (DCR) from Microsoft documentation (Configure DCR). However, today we are more focused on custom logs filtering using DCR, so letâs skip ahead to the last step where we tell the DCR how to collect and filter logs using an XPath query.
For collecting events select one of the event groups:
1. All Events: This category encompasses all Windows security and AppLocker events.
2. Common Events: This standard collection includes essential events for auditing purposes, providing a comprehensive user audit trail. It covers activities such as user sign-in and sign-out (event IDs 4624, 4634), security group modifications, key domain controller Kerberos operations, and other events aligned with best practices. While primarily designed to streamline event volume, it still ensures complete audit trail capability, even including less frequent events.
3. Minimal Events: This minimalistic set focuses on potential threat indicators without providing a full audit trail. It concentrates solely on events signaling a possible breach and other critical occurrences with low frequency. For instance, it includes successful and failed user logons (event IDs 4624, 4625) but excludes sign-out information (4634), which, though significant for auditing, isnât as pertinent for breach detection and tends to generate higher volume. Most data within this set pertains to sign-in events and process creation events (event ID 4688).
4. Custom Events: This category enables users to define their own event sets through a data collection rule utilizing XPath queries. Users can tailor this set to specific needs and preferences.
Create Custom Collection:
Now this is the step where we can finally implement XPath queries to create custom collections and the event of interests. After finalizing the setup, you can change the rule with more events of different types. The selection custom enables the custom event configuration.
Collect only event 4625 (failed sign-in)
Filters to include only security events with Event ID 4625, which indicates a failed sign-in attempt.
Query: Security!*[System[(EventID=4625)]]
Collect event 4625 (failed sign-in) and 4624 (Successfully logged on)
Filters to include security events with Event IDs 4624 and 4625, representing successful and failed sign-in attempts, respectively.
Query: Security!*[System[(EventID=4624) or (EventID=4625)]]
All Network Logon Events
Find all failed network logon attempts:
*[System[(EventID='4625')] and EventData[Data[@Name='LogonType']='3']]
Collect event 1, 299, 4624, 4625, 4661, 4662, 4663, 4664, 4665
Filters to include security events with any of the specified Event IDs, covering a range of system and security-related activities
Query: Security!*[System[(EventID=1) or (EventID=299) or (EventID=4624) or (EventID=4625) or (EventID=4661) or (EventID=4662) or (EventID=4663) or (EventID=4665)]]
Collect all critical, Error, Warning, and Information events from the system log except Event ID 6
Filters to include all events from the system log with severity levels of critical, error, warning, and information, excluding those with Event ID 6.
Query: System!*[System[(Level=1 or Level=2 or Level=3) and (EventID != 6)]]
For validating rules before adding in Sentinel, PowerShell can be used for validating XPath queries. use the Get-WinEvent cmdlet to validate the XPath query, for example:
$XPathQuery = "*[System[(EventID=4625)]]" Get-WinEvent -LogName Security -FilterXPath $XPathQuery
Best Practices for Collecting Event IDs
To optimize costs and performance in Azure Sentinel, itâs essential to collect only the necessary events that are relevant to your monitoring objectives. This can be achieved by using XPath queries to specify the events precisely.
1- Collect Only Necessary Events
To collect only necessary events, you should focus on specific Event IDs that are relevant to your security monitoring objectives. For example, if youâre interested in monitoring for failed logins, you can use an XPath query like this: Security*[System[EventID=4625]]. This query collects only events with EventID 4625, which represents a failed sign-in attempt.
By collecting only these specific events, you can reduce the volume of data and optimize your costs. While the specific event IDs to monitor may vary depending on your organizationâs environment and security requirements, there are certain critical events that should be prioritized for collection and analysis. These include, but are not limited to:
Failed logon attempts (Event ID 4625): Monitor for potential brute-force attacks or unauthorized access attempts.
Successful logon events (Event ID 4624): Track user activity and identify potential account compromises.
Process creation events (Event ID 4688): Detect the execution of suspicious or malicious processes.
File creation/modification events (Event IDs 4663, 4660): Monitor for potential data exfiltration or malware activity.
Firewall and network events: Capture events related to network connections, blocked traffic, and firewall policy changes.
By focusing on these critical event IDs, you can gain valuable insights into potential security incidents, enabling timely detection and response.
Regular Review and Update
Regularly reviewing and updating your collection policies and XPath queries is crucial to adapt to changing security landscapes and operational needs. This ensures that your monitoring setup remains effective and efficient over time.
For instance, if you initially collect events for failed logins (EventID 4625), you might need to update your query to include additional Event IDs as new threats emerge. You could add more Event IDs to the query like this:Security*[System[(EventID=4624 or EventID=4625 or EventID=4626)]]
This updated query collects events for successful logins (EventID 4624), failed logins (EventID 4625), and other relevant events (EventID 4626). By regularly reviewing and updating your queries, you can ensure that your monitoring setup remains effective and efficient in response to changing security threats.
Conclusion
Effective data collection and log management are critical components of a robust security strategy. By leveraging Azure Sentinelâs capabilities, such as XPath queries and Data Collection Rules, organizations can optimize their log collection processes, reduce operational costs, and enhance the efficiency of their security monitoring efforts. By following best practices and focusing on critical event IDs, security teams can gain valuable insights and improve their ability to detect and respond to potential threats. Remember, the key to success lies in striking the right balance between comprehensive log coverage and selective filtering to ensure that only the most relevant data is collected and analyzed. By implementing the techniques and best practices outlined in this blog post, you can embark on a journey towards a more efficient and effective security operations center, empowering your organization to stay ahead of evolving cyber threats.
Precision in data collection is the shield against digital threats
Latest News
Every week we publish exclusive content on various topics.
