Troubleshooting in ServiceNow
Areas of Concern
ServiceNow has a number of different components that can each warrant its own troubleshooting steps depending on what is happening. The following ‘areas of concern’ are in the order that you would most commonly experience them (you can’t raise Incidents or tasks before you have your alerts imported for example).
Alerts Not Being Created
This is more likely to be an issue on the SCOM side or in transport (proxies or firewalls getting in the way) than the ServiceNow side, however, if you have permission, it’s only a quick check to look at the import set, which can highlight issues. On rare occasions, you may see things like a large number of skipped records and even more occasional still, errors. If you dig into the related import set row in these cases you can see the alert being processed, and any errors that may have been generated. If you have errors you should check and act on the error message given, if you have skipped records check the age of the alert you might be in a scenario where Connection Center is checking its work and reprocessing old alerts (in which case it should catch up given some time).
It is also worth ensuring that you have an active Outbound Notification connection in SCOM that can successfully connect to your instance.
Search for ‘SCOM Alert Import’ in the ServiceNow menu
Open this table and check to see if there are any items in this table
If this all looks fine, check the 'Transform History' link at the bottom of the table and see if there are any errors reported.
If there are errors click on the set with errors (not the history itself) then check over any Logs or Errors that may be present.
In our experience issues with Alerts not being created in this manner are related to hardened ServiceNow instances not allowing the service account access to part of the import set functionality. For example, we have seen service accounts that were allowed to create records but not insert values. In this example, the error was around coalescing on Alert Id (which was always null because the service account could not write to it).
A good quick check would be to use the service account and then an Admin account to try and push sample data into the import set. In the following example, we use PowerShell, however, you could easily use tools such as Postman or curl if preferred.
If the admin account succeeds and the service account does not you should check over your ACLs paying specific attention to the following tables:
The x_oklt_cookdown_sc_scom_alert_import table extends the sys_import_set_row table, so ACL entries on this table should take preference.
Incidents and/or Tasks Not Being Raised
In order to raise Incidents or other types of tasks, you need to have an Alert and a matching Creation Rule.
Firstly you should make sure that you have Alerts available in the 'SCOM Alerts' table by pushing them from SCOM or by creating them manually using the minimum requirements as described here
Next, you should check that you have a ‘Creation Rule' in place to raise incidents by going to the 'Creation Rules’ table
Make sure you have at least one Creation Rule
Make sure that the Creation Rule is active
Make sure that the Creation Rule has a filter (Alert Condition)
Copy the text version of the alert criteria from the main table view
Navigate to the SCOM Alerts table
Add your search query to the end of your URL to check that you have matching alerts
Clear out any existing query after x_oklt_cookdown_sc_scom_alerts_list.do
Add the query that you copied in step 4
If you do not see any alerts in the table at this stage you may need to adjust your filter
If you have alerts matching the criteria check to see if they have matched the creation rule at the bottom of the alert:
If it has not matched a Creation Rule at all
Try to ‘re-run’ the alert as described here
If it has matched a Creation Rule but has not raised an alert
Check the history tab to see if there are any messages and investigate any errors
Investigate any errors seen such as the one described here
If disabling Post-Processing or other scripts has allowed you to generate an Incident/Task proceed to Custom Script Issues below
If you are not able to determine what is going wrong consider reaching out to our support team
Alerts Updates Not Being Returned to SCOM
Much like alerts not being created, it’s more likely to be an issue caused by the SCOM side or in transport (proxies or firewalls getting in the way) than the ServiceNow side. Make sure that you do have an active Inbound Notification connection that can connect to ServiceNow.
In order to sync back to SCOM the incident needs to be linked to the SCOM alert, the SCOM alert needs to be linked to a Creation Rule, and the alert needs to have the 'Updates for SCOM?' option checked (should be automatically set when a linked incident is updated and removed when SCOM pulls the updates, though you can set this manually for testing/troubleshooting purposes):
The first thing to do is hurry up and wait
When Connection Center successfully pulls the alert it will mark the alert as polled (by default every 60s)
The alert will remain checked for a further 4 minutes in case a re-run needs to occur for some reason
The Checkmark will be removed and the alert will no longer be available to Connection Center to poll unless another change happens to reset the checkmark
If this does not occur the next thing to check is that the API is successfully presenting the alert to the Service Account
Impersonate the service account (not required in all scenarios, but is the most accurate representation)
Use the API Explorer (or another service such as PowerShell, Postman, or curl) to call our API
Select API Explorer
Set Namespace to x_oklt_cookdown_sc
Send the request
Verify that the status is 200 and that the response body contains your alert/incident in the body:
If it does not, repeat the steps using an admin account to check for permissions issues
If the admin account returns the details as expected, verify that the service account has the correct roles assigned
If you do not have access to the Rest API explorer and/or impersonation in ServiceNow you can also try troubleshooting with PowerShell. You can adjust the script described here to suit your needs.
Alerts not being closed in SCOM on Resolved in ServiceNow
Usually, this is caused by using Advanced Alert Updates on your creation rules. By default, this does not include any closure logic and also does not set custom field 5 in SCOM.
If you aren’t customizing your alert updates we would recommend that you disable the Advanced Alert Updates feature. Otherwise, you can put in logic to replicate the default closure behavior:
Custom Script Issues
Alerts in ServiceNow Show a Different Time/Timezone to the Alert Itself in the SCOM Console
The timestamps on SCOM alerts in ServiceNow don't match those on the alert in SCOM itself, making it hard to understand when an alert was truly raised from within ServiceNow.
In ServiceNow, the time of each alert is displayed in the timezone that you (the user logging into ServiceNow to view the alert) as.
Change the timezone of your ServiceNow user account in your profile to the correct one:
Navigate to your Profile
Change the Timezone
Creation Rule Users See Strange UI Behavior
A non-admin user of ServiceNow is reporting strange UI behavior when using creation rules. This typically includes the Configuration Item picker and script-able elements not displaying as expected.
This is generally caused by the user not having access to certain tables within ServiceNow. Typically Configuration Items. Granting access to these elements resolves the issue (typically through the ITIL role). If you do not require this functionality Alert Sync will continue to operate as normal without them and the role can be left off if desired.
Further information on Connection Center and any related roles can be found here.
When Using a Caller That Is Not the Service Account, Alert Closed Incident State Will Not Resolve or Close an Incident
You have set your creation rule to use a caller other than your service account and the ‘Alert Closed Incident State' from the 'Advanced' tab is set to Resolved, Closed, or Cancelled. When a SCOM alert is closed the Incident remains on its previous status and does not get moved to the desired state.
Cookdown and ServiceNow both adhere to ITIL best practices by default. They allow a caller to resolve or close their own Incidents but require resolution/closure notes and codes when resolving Incidents for another caller. If at the time of SCOM alert resolution the Incident is linked to a different caller (set either by the creation rule or through other means) by default the Incident will not close. You may also see this behavior using the service account as the caller in a customized instance.
Use the service account as the caller where possible
Rather than using ‘Alert Closed Incident State’ in your creation rules use 'Advanced Incident/Task Updates' to script the resolution of the ticket, setting the required fields as appropriate.
In ServiceNow create or adjust your ‘Data Policy' and 'Data Policy Rules’ to exempt Incidents raised by creation rules from needing to fill in these fields.
Specifically how you approach this would be down to your organizational requirements, however, a suggestion might be to exempt any Incident created/updated by your service account, with a certain contact type, or a combination of both.
Creation Rule Fails to Create Task, Catalog Task, and Service Task
When a new SCOM alert comes in, a creation rule set to create a Task, Catalog Task, or Service Task fails to raise a task of this type
By default, ServiceNow does not provide users or the ITIL role with access to these tables. You may find that there are no ACL entries for these roles or you may find that there are not any ACL entries for those tables at all…
Provide the Service account with appropriate access to these tables to use these task types. For example, you could provide the ‘catalog_admin' role to the service account to provide access to create Catalog Tasks. However, rather than doing this, we strongly recommend that you create your own ACL to provide access to the required table via the 'x_oklt_cookdown_sc.scom_sync_service’ role or another role of your choice.
Alert updates continue without a creation rule
You remove a creation rule that was associated with a number of alerts with linked tasks. Alert updates continue to occur after the creation rule is removed.
This is by design, alerts will continue to sync using the default settings. If you need advanced alert updates, you should disable the creation rule until the alerts/tasks have all been groomed at which point you should be able to remove it without falling back on default behaviors.
Testing Get/Post from a script
Uncomment sections of the script as you require them: