Why is Alert Management Rule Processing important?

Alert Management Rule Processing plays a vital role in enhancing the efficiency and effectiveness of incident management processes. By defining specific rules for processing alerts, organizations can automate and streamline the entire incident response lifecycle.

Alert Management rules are created using the Alert Management Rule form, which provides a user-friendly interface for configuring rules. This form allows users to specify the conditions that must be met for an alert to trigger the rule, as well as the actions that should be taken when the rule is triggered.

Overall, Alert Management is a critical component of ServiceNow’s Event Management platform, allowing users to quickly and efficiently manage alerts generated by events. By configuring and managing Alert Management rules, organizations can ensure that critical alerts are addressed promptly, minimizing the impact of events on their business operations.

Key benefits of Alert Management Rule Processing include

1. Automated event correlation: By defining rules based on specific fields and conditions, the system automatically correlates related alerts and groups them together. This helps eliminate alert noise and reduces the number of duplicate incidents, allowing teams to focus on resolving critical issues.

2. Escalation and notification: The rule processing engine enables organizations to configure rules that determine the appropriate escalation and notification actions for specific types of events. For example, high-severity incidents can be automatically escalated to the appropriate teams or individuals, ensuring timely response and resolution.

3. Intelligent routing: Alert Management Rule Processing allows for intelligent routing of alerts based on predefined attributes, such as location, business service, or impact. This ensures that alerts are assigned to the most appropriate support groups or individuals, reducing response time and enhancing customer satisfaction.

4. Event enrichment: With the help of rules, organizations can enrich incoming alerts by adding additional contextual information or populating specific fields. This additional information can be critical when investigating and diagnosing incidents, as it provides teams with a complete picture of the event.

5. Advanced filtering and prioritization: Alert Management Rule Processing enables organizations to filter and prioritize alerts based on specific criteria. This ensures that critical or high-priority alerts are immediately brought to the attention of the appropriate teams, reducing response time and minimizing the impact of major incidents.

How does Alert Management Rule Processing work?

Alert Management Rule Processing in ServiceNow Event Management operates on a set of predefined rules and conditions defined within the system. These rules utilize conditions, filters, and actions to determine how incoming alerts are processed.

The typical workflow of Alert Management Rule Processing involves the following steps:

1. Alert ingestion: Incoming alerts from various sources are ingested into the ServiceNow Event Management system.

2. Condition evaluation: The system evaluates each incoming alert against the predefined rule conditions. These conditions can be based on specific fields, textual matching, patterns, or time-based criteria.

3. Rule matching: Once the conditions are evaluated, the system matches the alert against the rules defined in the system. Each rule consists of one or more conditions.

4. Rule actions: When a rule is matched, the system executes the specified actions. These can include grouping related alerts, escalating incidents, routing alerts, sending notifications, or enriching alert information.

5. Incident creation: If the rule specifies creating an incident, a new incident record is generated and associated with the relevant alert. This incident record is then further processed as per the organization’s incident management processes.

6. Escalation and resolution: Based on the specified conditions and actions, alerts can be escalated to higher support groups or individuals. Escalation policies can be defined to ensure timely resolution of critical incidents.

Alert Management Rule Configuration

ServiceNow Event Management provides a flexible platform for creating and managing alert rules. Alert rules are used to define the conditions that trigger an alert and the actions that are taken when an alert is triggered. The Alert Management module provides a user-friendly interface for creating and managing alert rules.

Alert Info Tab

The Alert Info component of Alert Management Rules in ServiceNow includes settings such as the order of rule processing and options to continue or stop searching for additional rules after processing a current rule.

It is important to activate the rule and specify the execution order of each rule. If the Multiple alert rules option is set to “Search for additional rules”, the execution of this rule will not stop the evaluation of the other rules.

Alert Filter Tab

This component comprises the rule activation conditions and the alert filter, which is activated every time the filter criteria are met.

Alert rules are triggered according to the conditions defined in this tab. ServiceNow Event Management offers a wide range of conditions that can be used.

Actions Tab

Actions within Alert Management Rules include options like remediation sub-flows, which can be automatic, manual, or both, execution limits, and launch application options. These actions are triggered when the specified conditions are met.

Alert rules are processed in a specific order based on their precedence. The higher the precedence, the earlier the rule is processed. If multiple rules are triggered at the same time, the rule with the highest precedence is processed first. The user can specify the precedence of each rule to ensure that they are processed in the correct order.

In conclusion, ServiceNow Event Management provides a powerful platform for creating and managing alert rules. By using the Alert Management module, users can easily create and manage alert rules that trigger based on specific conditions and take specific actions when triggered.

Best practices for implementing Alert Management Rule Processing

To make the most of the Alert Management Rule Processing feature in ServiceNow Event Management, consider the following best practices:

1. Clearly define rules: Take the time to clearly define and document the rules based on your organization’s specific needs. Establish a clear understanding of the expected behavior and actions for different types of alerts.

2. Regularly review and update rules: It is essential to review and update rules periodically to ensure they align with changing business requirements and evolving alert patterns. Regular maintenance helps optimize rule efficiency and relevance.

3. Collaborate with stakeholders: Involve relevant stakeholders such as IT operations, support teams, and business owners in the rule definition process. This collaboration ensures that rules are aligned with the organization’s goals and priorities.

4. Start with simple rules: Begin by implementing simple rules and gradually expand the complexity as needed. This iterative approach helps in understanding the impact of rules and fine-tuning them based on real-world scenarios.

5. Leverage data analytics: Utilize data analytics capabilities to gain insights into alert patterns, trends, and performance metrics. Analyzing this data helps identify opportunities for refining rules and enhancing the overall incident management process.

By honing your Alert Management Rules, you can significantly impact your IT operations in several ways:

• Reduced Noise: Only meaningful alerts reach your dashboard, which reduces overwhelm and focuses your team’s efforts.

• Faster Resolution: Prioritization and automation mean that issues are resolved more quickly, often before users are even aware of a problem.

• Improved Monitoring: With effective rules in place, your event monitoring becomes more targeted and effective, helping you to better understand the health of your IT environment.

Effective Event Management Alert Rule Management processing is essential for maintaining operational resilience and mitigating the impact of IT incidents on business operations. By understanding the core components of alert rule management and adhering to best practices, organizations can streamline incident response workflows, enhance situational awareness, and deliver superior service quality.

In our next article, we'll focus on best practices when implementing Event Management in your organization. Until then, we wish you a good weekend and an excellent week.

Glossary

The post Alert Management Rules Processing appeared first on ITSC.

Introduction

Event Rules are one of the main components in event processing, and there are usually predefined ones for various monitoring systems, which you can use as examples to help you configure your own. Event Rules have the following purposes:

1. Process raw event data
2. Convert events into alerts
3. Bind CIs more efficiently than default binding

NB: By default, events are automatically bind to the node that generated them; after processing, the Event Rule can more effectively bind to the node to which the event should indeed be binded. Let’s take the case of an Oracle database server installed on a Linux server. When an event is generated on the server, the default binding may link the event to the Linux server, but after processing the Event Rule, and depending on the criteria we’ve defined, the event may ultimately be bind to the Oracle database server, if that’s the one presenting the dysfunction. In this way, the alert and/or incident can be assigned to the right support group (Oracle team) rather than the wrong one (Linux team) due to the default binding.

Events come from various monitoring systems through MID servers and Connectors or REST APIs. An event record is inserted into the event table, where Event Rules and Event Field Mappings are applied to generate alerts. Alerts Management Rules are applied to the alerts generated by the Event Rules, in order to prioritize alerts for remediation and/or root cause analysis.

Event Processing Detail

1: As events arrive in the event table, the system identifies the Event Rules to be applied to them. Event Rules are evaluated according to the order assigned in the configuration, to check if the source and optional filters match. If the event source and optional filters match those of the Rule, then the Event Rule will be executed.

NB: If no Rule matches, the default binding using the Node field to locate or match the CI hostname, FQDN, IP or MAC address is implemented and processing continues.

2: If the Ignore Event Rule checkbox is selected, no alert is generated. The event is left to be examined and remedied.

3: If Transformation and Composition Fields have been defined in the Event Rule, they will be applied.

4: If a Threshold is set, all events are accumulated until the threshold is reached, then processed.

NB: If the “Apply Additional Matching Rule” option is checked, the Threshold is deactivated and Event Rule processing continues.

5: Then apply the corresponding Event Field Mapping even if there is no corresponding Event Rule.

6: If no Severity is defined at this time, keep the event and set its status to Error, but do not generate an alert.

7: If the Severity is defined, search the alert table for the corresponding Message Key. If a match exists, update the alert and associate the event with the unique alert. If there is no match, create a new alert for root cause analysis purposes, and associate the event with the unique alert.

Processing Notes

After processing an event, to view how the processing was applied, consult the Processing Notes field. It indicates the Event Rules and Field Mappings that have been applied to the event, the associated CI, the link to the executed Event Rule and also the rules that have not been applied.

As mentioned in the previous article, the life cycle of an event is defined by its state, which can be Ready, Processed, Ignored or Error.

Event Rules

Event Rules are used to :

1. Bind alerts to specific CIs in the CMDB, showing their impact on the correct Application Service in the Service Operator Workspace and in Event Dashboards.
2. Transform and Compose alert information from relevant data in event fields.
3. Ignore events that match the filter conditions and reduce incident noise until a defined threshold is reached.

NB: Since the Tokyo version, it is possible to apply multiple event rules by selecting Apply Additional Matching Rules to Event Rules. To simplify the creation of Event Rules, the Event Management application provides recommended rules based on tracked events. The recommended rules engine analyzes and groups events, and provides the Regex expressions used to create event groups.

The configuration of an Event Rule is performed in several steps on the Event Rules Form.

Event Rule Info

Name: Give an appropriate name to the Event Rule.

Source: Refers to the Source of the event. If this field is empty, all sources will be taken into account.

Order: Represents the execution order of the Rules. This is very important, as Rules are processed in the sequence specified, from the lowest to the highest.

Description: Used to add additional relevant information that can be used during processing or investigations.

Event Filter

The Filter defines the conditions that must be met for the Rule to be applied. You can also ignore events based on the Filter criteria.

Transform and Compose Alert Output

In this section, we define the transformation rules for obtaining the data required to generate alerts.

It’s important to note that all the data available to us are those received from the source. Based on that data and the Transformation Rules, we’ll deduce the useful values for alert generation. If we need more information than we’ve received, we may have to manage it directly on the source.

To compose the alert details, we need to drag the data from the data piles (on the left in the image) to the alert fields. Regexes can be used for complex compositions.

Threshold

Thresholds allow events to be processed according to a threshold metric and volume over a given period; if the threshold is reached, an alert is generated.

By default, the active field is set to false; once set to true, the threshold can be defined as required.

Create Alert Operator: allows you to define the threshold operation you wish to implement. The form varies according to the selected operation.

Field name: defines the field to which the threshold will be applied.

Threshold value: defines the expected value of the field to trigger threshold calculation.

Occurs: defines the number of occurrences of the event before the alert is generated.

Over(seconds): define the frequency with which to check whether an event corresponding to the rule has occurred.

Close Alert Operator: allows you to define what is considered “quiet” and indicate that there is no longer a problem, then close the alert.

NB: The threshold is disabled for rules using “Apply Additional Matching Rules”.

Binding

By default, events are bind to CIs based on the Node value, which is either a host name, IP address, MAC address or FQDN. To better bind alerts to the CIs concerned, the Event Rule uses this section to better identify the CI concerned in the CMDB. Two types of Binding are available:

CI Identification: attribute criteria (CMDB attributes specific to the CI class) matched to identify a CI class on the host. This is a unique identification process for a CI class. It uses the Identification, and Reconciliation Engine (IRE), based on Identification Rules, to find the appropriate CIs.

CI Field Matching: use of additional information (the pair fields : values) to identify the specific CI.

CI Binding Process Flow

1. When an event arrives, Event Management checks the node or CI Identifiers.
2. If no Node exists, the generated alert can bind to the CI using the alert Type, Additional information, or Configuration item identifier fields.
3. If the event has a node value, search for a valid host.
4. If the event has a host and a CI type, try to bind to a device CI.
5. If the event has a host, try to bind to the application CI.

Event Field Mapping

During Event Processing, after the Event Rules have been executed, the process continues with Event Field Mapping. This step aims to replace the information received from the event source with more meaningful and/or standardized information, before generating the alerts resulting from event processing.

By default, some Event Mapping Fields are provided for common monitoring systems. This allows us to start from an existing rule base that we can use to define our own rules. It is recommended to clone the default rules before modifying them.

In our next article, we'll look at the final step in the event management process. We'll be talking about triggering proactive or remediation activities. In the meantime, I wish you an excellent day and a wonderful week!

Glossary

• CI: Configuration Items
• MAC: Media Access Control
• FQDN: Fully Qualified Domain Name
• REST: REpresentational State Transfer

The post Event Management Event Rules Processing appeared first on ITSC.

Main tables and configuration

The Event Management application is built around three main platform tables:

Event table (em_event): the event table receives events from monitoring tools in near-real time, with relevant information about the issue.

Alerts table (em_alert): the alerts table receives alerts generated by event processing. Alerts are action triggers that can be associated with equipment or processes.

Service table (cmdb_ci_service): the service table is the parent table of the “Business”, “Technical” and “Applications” services, as well as other derived services. This table makes it possible to precisely identify the service concerned by alerts, and thus map its health status in detail.

Example of services derived from application services

• Manual service: cmdb_ci_service_manual
• Tag-based service: cmdb_ci_service_by_tag
• Dynamic CI group: cmdb_ci_query_based_service
• Discovered application: cmdb_ci_service_discovered

NB: Events older than five days are removed from the event table using the table rotation function. It is recommended to not change the table rotation value, as this may have an impact on system performance.

Event Records Overview

Source: refers to the monitoring tool that generated the event.

Node: This is the configuration item on which the event occurred. It can be a CI name, MAC address, IP address or Fully Qualified Domain Name (FQDN).

Type: indicates the type of metric, e.g. disk or memory.

Resource: this refers to the resource within a node, such as the C drive, process name, etc.

Metric name: this is the name or attribute of the metric in the monitoring system.

Message key: a unique identifier for a specific event on a configuration item. The message key is responsible for event deduplication.

Severity: this is the criticality of the event, based on the values defined in the monitoring system. Available values are Critical, Major, Minor, Warning, OK and Clear.

Processing notes: provide feedback on processing. It shows event rules and other steps performed during processing.

Status: This represents the life cycle of an Event. Its values are Ready, Processed, Ignored or Error.

• Ready – when the event has not yet been processed by the system.
• Processed – when the event has been successfully processed by the system.
• Ignored – when the event has been ignored by an event rule.
• Error – when the event has encountered an error during processing.

NB: if the event’s “Severity” is empty, the event cannot be processed and the status will be “Error“.

Message key

The message key is unique and identifies the event. It determines whether an existing alert should be updated or not.

If the event has no key provided at the source, the key is automatically populated with: Source, Node, Type, Host resource and Metric name (Source_Node_Type_Resource on host_Metric name).

NB: repetitive events use the same key, the alert uses the same key as the event, and an alert can have more than one event attached.

Event Processing Overview

Event processing is carried out in three steps:

1 – Event reception (anomalies)

This step involves receiving events from monitoring tools. These events are collected from several sources via the MID Servers. The MID Server collects events at a frequency of 120 seconds by default. This frequency can be modified by changing the value of the event collection property (evt_mgmt.connector.minimum_schedule).

Event sources

There are many different sources that can be configured to create events on ServiceNow.

• Configure the SNMP Trap function to send events to ServiceNow. For this method, we need to enable the SNMP Trap Listener on the MID Server.

• Use the APIs to send events via custom scripts (Scripted REST APIs) or using the REST API of the ServiceNow event table, or the REST API in the import set.

• Use vendor-based monitoring software to send an e-mail notification that can be treated as an event by Emails Actions or by Flows.
• Use vendor-based ServiceNow connectors to collect events through the ServiceNow MID Server.

2 – Event filtering and transformation

In this step, events received from monitoring tools are filtered and transformed by event rules to generate meaningful alerts. This operation reduces the noise generated by events, binds events to configuration items and generates alerts for events requiring action.

3 – Launching proactive or resolution activities

At this step, we can configure the system to perform proactive or resolution tasks by creating incidents, launching remediation processes or recommending knowledge articles to problem-solving members.

NB: By default, the tasks generated at this step is incidents, and the severity of the alert will determine the priority of these incidents. If we don’t want to generate incidents all the time, we can create a custom table extended from the tasks table to store that information.

For the moment, I suggest we leave it there. In future articles on Event Management, we'll cover more in-depth the filtering and transformation of events, followed by the launching of proactive or resolution activities. In the meantime, I wish you an excellent day and a wonderful week!

Glossary

The post Event Management Processing appeared first on ITSC.

MID Server

As we mentioned in the previous article, the ServiceNow Event Management process is based on information collected from monitoring platforms within the organization. This information is collected through various methods (Push Connectors, Pull Connectors, Emails …), some of them pass through what we call the MID Server.

MID for Management, Instrumentation and Discovery is a Java application that can be installed on a Windows or Linux server. This application establishes secure communication between your ServiceNow instance and the organization’s local network. Communication is outbound between the MID and your instance, and uses the SOAP method on port 443 (HTTPS).

The MID Server is also involved in several other ServiceNow platform processes, as shown in the image below.

To install the MID Server, we recommend that you follow the ITOM Guide:

• All > Guided Setup – Legacy > ITOM Guided Setup > Get Started > MID Server
• Follow the three MID installation steps

1. Create a MID server account on the instance and give it the mid_server role.
2. Download and install the appropriate MID application file on the host server.
3. Run the application and validate the MID to ensure it is reliable.

NB: The MID Server must be installed in the DMZ or directly behind a firewall and must be able to communicate with the servers to be discovered, as well as with the supervision tools from which events are collected.

Communication between the MID server and the instance takes place via the ECC Queue table. The ServiceNow instance inserts records (jobs) destined for the MID server into the ECC Queue table (outpout record), and the MID continuously polls this table to identify unprocessed records, an operation known as Polling. By default, polling is performed every 40 seconds. The frequency can be modified by the mid.poll.time parameter.

Discovery and Service Mapping

Discovery

Discovery is the process of identifying and collecting all the applications and devices installed on your network, and then updating the CMDB based on that collected information. ServiceNow discovery is agentless and allows you to discover many different types of elements, such as Operating Systems, Applications, Running Processes, Application Supporting CIs and many more.

Discovery is carried out through the MID servers, which execute a series of commands (Probes) received from the ServiceNow platform to the local network, and return the result of these commands to the instance in XML format (input records in the ECC Queue table). The XML result is then processed by the Sensors, which update the CMDB according to the existence or non-existence of the collected devices. The discovery process is divided into four phases:

Service Mapping

Also known as Top-Down discovery, it covers only the service components and ignores everything else. It enables the Operations team to become aware of the services by providing targeted discovery of the IT infrastructure directly linked to the supported services.

Dependency View Map vs. Application Service Map

Event Management includes both the Dependency View and the Application Services Map.

Dependency View Map

1. Displays all dependency relationships between configuration items based on network traffic.
2. The selected CI or starting point is called Root CI with a pulsing effect.
3. Upstream and Downstream relationships are visible and, by default, up to three levels of upstream and downstream relationships are visible.

Application Service Map

1. The application Services Map provides a topological view of a specific application service.
2. The application service entry point is the highest layer, followed by the downstream relationship of consecutive configuration items.
3. Event Management highlights the service map by linking alerts to these configuration items to represent the health of the service.

Common Services Data Model

Event Management and ITOM Health improve the ability to realize the impact of problems on the company’s services, and to prioritize accordingly. The better the service organization, the more effective the impact calculation, triage and problem resolution.

Service: a Service is a way of delivering value to customers, facilitating the results they want to achieve, without them having to assume specific costs and risks.

Baseline Services :

1. Business Services: These are generally ordered by business users. The user can select the derived service offering and service commitment levels via the service catalog.
2. Application Service: This is the logical representation of deployed application stacks, such as a single instance of an application.
3. Technical Services: These are published for service owners, and generally underpin one or more enterprise or application services.

Application Service Creation

Automated creation:

Service Mapping enables you to discover all the services in an organization and draw up a complete map of all the devices, applications and configuration profiles used in these services.

1. Agentless: no additional software is required on target devices.
2. Service-centric: it discovers only CIs related to the defined service.
3. Configuration-based: does not rely on network traffic (netstat).

Manual creation :

Event Management allows you to manually create an application service using existing CIs and their relationships in the CMDB.

Required role: app_service_admin

In our next article, we'll talk about the Event Management Process and how it works. In the meantime, we wish you an excellent day and a wonderful week!

Glossary

• AMB : Asynchronous Messaging Bus
• ECC : External Communication Channel
• MID : Management, Instrumantation and Discovery

The post Event Management Architecture appeared first on ITSC.

IT Operation Management (ITOM): This is the suite of ServiceNow products used to manage IT operations. The suite currently comprises four main applications:

• ITOM Visibility: Discovery + Service Mapping
• ITOM Health: Event Management + Operational Intelligence
• ITOM Optimization: Cloud Management + Cloud Insights

Discovery (Horizontal discovery): Horizontal discovery is a technique used by Discovery to scan your network, find computers and devices, and then populate the CMDB with the CIs it finds. Horizontal discovery creates direct relationships between CIs, such as a “run on” relationship between an application CI and the computer CI on which it is running. Horizontal search is not aware of Business Services, and does not create relationships between CIs based on the Business Service in which they are located.

Service Mapping (Top-down discovery): Top-down discovery is a technique used by Service Mapping to find and map critical infrastructures that are part of Business Services, such as an e-mail service. For example, top-down discovery can be used to map an enterprise web site service by showing the relationships between an Apache Tomcat web server service, a Windows server and the MSSQL database that stores the Business Service data.

NB: It’s important to note that if the CMDB data is not correctly organized, the Event Management application won’t bring us any benefits. That’s why we invite you to follow the CSDM standard for properly structuring your CMDB.

The benefits of Event Management

Before going deeper, it is important to note that the Event Management application does not replace the monitoring tools in place within the organization, and is therefore not a monitoring tool. Its mission is to :

1. Consolidate events from different sources
2. Filter events to reduce noise and generate qualified events
3. Link events to CIs (Configuration Items) in the CMDB (Configuration Management Database)
4. Identifies alerts created due to planned maintenance (change)
5. Performs service impact analysis
6. Provides a Service Operator Workspace and Dashboard to understand service health and view service statistics.
7. Create ServiceNow tasks for alerts
8. Help remediate alerts, automatically or manually
9. Bridge the gap between infrastructure and application

As mentioned above, these benefits derive not only from the application of Event Management, but also from the implementation of a good CMDB with the CSDM standard.

In our next article, we'll be talking about event management architecture. In the meantime, we wish you an excellent day and a wonderful week!

Glossary

• Service Operations Workspace: This is a dashboard that provides a view of the impact of alerts on applications and services.
• Event Dashboard: Provides a quick view of active alerts and affected services.
• Event: A notable occurrence or notification from the infrastructure.
• Metric: This is a measure of a device’s operating characteristic over time.

The post Event Management Introduction appeared first on ITSC.