I have to give props to the excellent blog article done by Pete Zerger that talks about using an Orchestrator Runbook in conjunction with Operations Manager to help with service recovery on a global scale.
We took this 1 step further and integrated the Operations Manager alert connector into Service Manager. This allows for service monitoring to trigger an alert from Operations Manager. That alert now becomes an incident within Service Manager notifying the applicable support group. Orchestrator then executes its magic recovering the service automatically. Service Manager then in turn automatically resolves the incident. All without any analyst intervention!
First, head to the URL above and get Pete’s Runbook off the ground. Here’s a modified version with looped recovery, e-mail alerts, etc.
In this example we’ll kill the Print Spooler service on a server being monitored.
Within a few minutes the alert is thrown in Operations Manager
Huston we have a problem!
This in turn kicks off the Runbook for automatic recovery. The trick here is to setup a slight delay in the execution of the Runbook, in this case 5 minutes.
This gives time for Service Manager to pickup the NEW Active Alert from Operations Manager and turn it into an Incident. Service Manager synchronizes alerts from Operations Manager every 3 minutes.
Incident has been synced into Service Manager, and the Analysts have been notified of the issue.
Within 5 minutes the Runbook will execute & the service will be recovered notifying the Analysts.
E-mail notification on service recovery
Within 3 minutes of the alert disappearing from Operation Manager, Service Manager will automatically resolve the Incident, leaving a note “Service Restarted by Orchestrator”
All of this without an analyst lifting a finger to act on remediation or interaction with Service Manager to resolve incidents. Service Recovery using Orchestrator is excellent, integrating Service Manager into the process is even better! Now you have a detailed record of each and every service recovery. Show everyone how hard you “don’t” have to work.