Alerting

• The alerting feature notifies you when data from one or more database meets certain conditions. For example, you might want to notify in email if your application logs more than five HTTP 503 errors in one hour.
• We are supporting e-mail as a notification channel to send an alert.
• This advanced section is only for users who are familiar with scripting and Linux fundamentals.

Key Terms

Predefined Alerts

• Some categories of alerts are pre-defined for ease of use. A monitor is pre-created with defined trigger condition and an e-mail is sent to notify the user.
• Predefined monitors will monitor the alerts for every 15 minutes and will send an alert once the conditions are met.
• The notification in email contains following details like tenant name from which the notification is sent, device IP address or host name, and message.

1. Autoflow – Defines a rule to generate an alert for network balancing rule for autoflow controller.
2. Netbalancer – Defines a rule to generate an alert for failover monitor and gateway monitor. Netbalancer rule will contain jitter, latency and packet loss conditions.
3. IPS Alert – Defines a rule to detect network attacks like DDoS
4. IPS Flow – Defines a rule to generate an alert as to which server to client the attack was detected.
5. CPU and Memory Threshold – Defines a rule when CPU and memory met certain condition. It generates the alert when CPU utilization reaches 80% or above and free memory is less than 500 MBs.
6. Firmware Update – Defines a rule when the device is updated with latest firmware image successfully.
7. Interface status – Defines a rule when the device interface status is changed, like LAN or WAN is UP or Down.
8. Reboot – Defines a rule to alert the user whenever the device reboots.
9. UnCleanShutDown – Defines a rule to generate the alert whenever the device was shutdown unexpectedly following the last power cycle.
10. StartUp – Defines a rule to generate an alert when the device started successfully.
11. Tunnel Status – Defines a rule to generate an alert whenever the SSL/IPSEC tunnel created or deleted. Tunnel is an medium to communicate between the sender and receiver.
12. LTE – Defines a rule to generate an alert with signal quality details like signal to noise ratio, signal strength and the interface name with LTE support.

Steps to configure alerts

1. Create destination as e-mail.
2. Create sender and receiver group in e-mail destination. Configure sender details in keystore file and load the database.
3. Create monitor to monitor the scheduled job and mention the period of monitoring.
4. Create trigger which contains some conditions.
5. Create action which contains the information that you want the monitor to send out after being triggered. Actions have a destination, subject, and a body.

Configuration

Create destination

• Navigate to Alerting –> Destinations –> Add destination.
• Specify a name for the destination so that you can identify it later.
• Select e-mail type as destination and choose Add sender >> New sender >> enter a unique name. Enter the e-mail address, SMTP host and the port.
• Choose an encryption method, or use the default value of none. However, most email providers require SSL or TLS, which requires a username and password in keystore file. In our case we are using TLS as encryption method and port as 587.
• Choose Save to save the configuration.
• Choose Manage e-mail groups >> Add email group >> New email group >> Enter a unique name >> Save. For recipient e-mails, enter any number of e-mail addresses.
• Click Create button to create a destination.

Steps to configure SMTP sender details in keystore file

Modify keystore database to add the user/e-mail credentials, if you have selected TLS or SSL in the previous step.

1. Login to provider with admin permission.
2. Login ELK VM with admin permission using SSH client like Putty.
   1. Steps to login to ELK VM, enter password when promted:
      sh $ ssh router@ELK $ sudo su
   2. OR in single server deployment, use the following commands:
      sh $ cd /root/VMELK $ vagrant ssh $ sudo su
3. Login to odfe-node docker container as admin user:
   sh $ docker exec -u 0 -it odfe-node /bin/bash
4. Follow the below steps to configure username and password
   1. The is the name you entered for Sender earlier.
   2. Set username using below command:
      sh $ ./bin/elasticsearch-keystore add opendistro.alerting.destination.email.<sender_name>.username
   3. Enter the sender username
   4. Set password using below command:
      sh $ ./bin/elasticsearch-keystore add opendistro.alerting.destination.email.<sender_name>.password
   5. Enter the sender password.
   6. Exit from the container using below command
      sh $ exit
5. To update user’s credential, call the reload API update those credentials using below command.
   sh $ curl -XPOST -k -u <admin_username_from_provider>:<admin_password_from_provider> "https://127.0.0.1:9200/_nodes/reload_secure_settings"

Note:
- You can reuse senders across many different destinations, but each destination only supports one sender.
- Keystore settings are node-specific. You must run these commands on each node and need to reload all the nodes.

Create monitors

• Navigate to Alerting -> Monitors -> Create monitor.
• Specify a name for the monitor. Choose one or more indices. You can also use * as a wildcard to specify an index pattern.
• Choose “Define using visual graph”. Then choose an aggregation (for example, count() or average()), a set of documents, and a timeframe. Visual definition works well for most monitors.
• Advance users familiar with DSL queries can go with using
  Extraction Query as a monitor type.
• Choose a frequency and timezone for your monitor. Note that you can only pick a timezone if you choose Daily, Weekly, Monthly, or custom cron expression for frequency.
• Choose Create.

Note:
- Define the monitor in one of two ways either using visual graphs or using extraction query.
- We are using an DSL query for predefined alert categories.

Create triggers

• These steps differ based on whether you chose “Define using visual graph” or “Define using extraction query” or “Define using Anomaly detector” when you created the monitor.
• Specify a name and severity level for the trigger. For Trigger condition, specify a threshold for the aggregation and timeframe you chose earlier, such as “is below 1,000” or “is exactly 10 or ”is above 500”.
• We can configure multiple triggers for one monitor.
• Add an action and specify a name for the action. Choose a destination which was created under “Destination” tab.
• Add a subject and body for the message.
• Change the mustache templates, for example:

{"text":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.
- Trigger: {{ctx.trigger.name}}
- Severity: {{ctx.trigger.severity}}
- Period start: {{ctx.periodStart}}
- Period end: {{ctx.periodEnd}}"} 

• Choose Create.

Note:
- You can add variables to your messages using Mustache templates.
- (Optional) Use action throttling to limit the number of notifications you receive within a given span of time.