Posted on:
Categories: System Center
Description:

I have to give props to the excellent blog article done by Pete Zerger that talks about using an Orchestrator Runbook in conjunction with Operations Manager to help with service recovery on a global scale.

http://www.systemcentercentral.com/how-to-restart-any-windows-service-on-scom-alert-from-a-single-orchestrator-runbook/

We took this 1 step further and integrated the Operations Manager alert connector into Service Manager. This allows for service monitoring to trigger an alert from Operations Manager. That alert now becomes an incident within Service Manager notifying the applicable support group. Orchestrator then executes its magic recovering the service automatically. Service Manager then in turn automatically resolves the incident. All without any analyst intervention!

First, head to the URL above and get Pete’s Runbook off the ground. Here’s a modified version with looped recovery, e-mail alerts, etc.

Orchestrator
Runbook executing service recoveries from Operations Manager

 

servicedown
Print Spooler Down!

In this example we’ll kill the Print Spooler service on a server being monitored.

Within a few minutes the alert is thrown in Operations Manager

spoolerdown 

Huston we have a problem!
This in turn kicks off the Runbook for automatic recovery. The trick here is to setup a slight delay in the execution of the Runbook, in this case 5 minutes.

ExecutionDelay
Delay in the “Pass Alert Data” to allow time for Incident Creation.

This gives time for Service Manager to pickup the NEW Active Alert from Operations Manager and turn it into an Incident. Service Manager synchronizes alerts from Operations Manager every 3 minutes.

NewIncident
Incident created from Operations Manager alert

Incident has been synced into Service Manager, and the Analysts have been notified of the issue.

IncidentNotification
E-Mail notification on Incident Creation

Within 5 minutes the Runbook will execute & the service will be recovered notifying the Analysts.RecoveryEmail

E-mail notification on service recovery

Within 3 minutes of the alert disappearing from Operation Manager, Service Manager will automatically resolve the Incident, leaving a note “Service Restarted by Orchestrator”

AlertResolved
Incident auto resolution

All of this without an analyst lifting a finger to act on remediation or interaction with Service Manager to resolve incidents. Service Recovery using Orchestrator is excellent, integrating Service Manager into the process is even better! Now you have a detailed record of each and every service recovery. Show everyone how hard you “don’t” have to work.