Create Alerts in Splunk Observability

Varunkumar Inbaraj
3 min readJun 6, 2024

The threshold for Pub/Sub message count has been reached.

Here are the detailed steps to create alerts in Splunk Observability:

  1. Go to Splunk Observability:

— Log in to your Splunk Observability account.

2. Navigate to Metric Finder:

— In the left navigation pane, click on **Metric Finder**.

3. Search for Metric:
— In the search bar, type `subscription/num_undelivered_messages`.
— Add filters or exclusions as needed. For example, filter by `project_id` and `subscription_id` to narrow down the results.

4. Select Metric:
— Click on the `subscription/num_undelivered_messages` metric from the search results.

5. Create New Alert from Chart:
— Click on the **Alerts** icon (bell icon) located on the top right corner of the chart.
— Select **New detector from chart**.

6. Name the Detector:
— Give your detector a meaningful name.
— Click the **Create alert rule** button.

7. Verify and Proceed:
— In the pop-up window, verify that the `project_id` and `subscription` filters are correctly applied.
— Click the **Proceed To Alert Condition** button.

8. Set Alert Condition:
— Choose between a static threshold or a custom threshold for the alert condition.
— Click the **Proceed to alert settings** button.

9. Set Threshold Value:
— Set the threshold value to `10000`.
— Click the **Proceed to alert message** button.

10. Configure Alert Message:
— Provide a dashboard link or any additional information that should be included in the alert message.
— Click the **Proceed to alert recipients** button.

11. Add Alert Recipients:
— Add recipients by specifying email addresses, teams, or webhook URLs.
— Click the **Proceed** button.

12. Verify and Activate:
— Verify the rule name and other details.
— Click the **Activate alert rule** button.

13. Save the Detector:
— Save your detector configuration.

14. Verification:
— To verify, go to the **Detectors & SLOs** menu from the main navigation.
— Check that your new alert rule is listed and correctly configured.

Following these steps will help you set up alerts in Splunk Observability to monitor the `subscription/num_undelivered_messages` metric.

--

--