Troubleshooting in ServiceNow

Areas of Concern

ServiceNow has a number of different components that can each warrant their own troubleshooting steps depending on what is happening. The following ‘areas of concern’ are in the order that you would most commonly experience them in (you can’t raise Incidents or tasks before you have your alerts imported).

Alerts Not Being Created

This is more likely to be an issue on the SCOM side or in transport (proxies or firewalls getting in the way) than the ServiceNow side, however, it’s only a quick check to look at the import set, which can highlight issues. Though it is also worth making sure that you have an active Outbound Notification connection in SCOM that can successfully connect to your instance.

  1. Search for ‘SCOM Alert Import’ in the ServiceNow menu

  2. Open this table and check to see if there are any items in this table

  3. If this all looks fine, check the 'Transform History' link at the bottom of the table and see if there are any errors reported.

  4. If there are errors click on the set with errors (not the history itself) then check over any Logs or Errors that may be present.

In our experience issues with Alerts not being created in this manner are related to hardened ServiceNow instances not allowing the service account access to part of the import set functionality. For example, we have seen service accounts that were allowed to create records but not insert values. In this example, the error was around coalescing on Alert Id (which was always null because the service account could not write to it).

A good quick check would be to use the service account and then an Admin account to try and push sample data into the import set. In the following example, we use PowerShell, however, you could easily use tools such as Postman or curl if preferred.

$Instance = 'yourinstancenamehere'
$Credential = Get-Credential -Message 'Enter some valid ServiceNow credentials for the scenario you are testing'

If ([Net.ServicePointManager]::SecurityProtocol -notlike '*Tls12*'){[Net.ServicePointManager]::SecurityProtocol += [Net.SecurityProtocolType]::Tls12}

$Headers = @{
    Accept = 'application/json'
    'Content-Type' = 'application/json'
} 

$Bareminimumalert = @{
    u_alertid = 'testdata'
}

Invoke-RestMethod -Method Post -Credential $Credential -Uri "https://$Instance.service-now.com/api/now/table/x_oklt_cookdown_sc_scom_alert_import" -Body ($Bareminimumalert | ConvertTo-Json) -Headers $Headers

If the admin account succeeds and the service account does not you should check over your ACLs paying specific attention to the following tables:

  • x_oklt_cookdown_sc_scom_alert_import

  • sys_import_set_row

  • sys_import_set

The x_oklt_cookdown_sc_scom_alert_import table extends the sys_import_set_row table, so ACL entries on this table should take preference.

If these rules are in place and active it would be worth engaging in ACL debugging and contacting support.

Back to top

Incidents and/or Tasks Not Being Raised

In order to raise Incidents or other types of tasks, you need to have an Alert and a matching Creation Rule.

  1. Firstly you should make sure that you have Alerts available in the 'SCOM Alerts' table by pushing them from SCOM or by creating them manually using the minimum requirements as described here

  2. Next, you should check that you have a ‘Creation Rule' in place to raise incidents by going to the 'Creation Rules’ table

    1. Make sure you have at least one Creation Rule

    2. Make sure that the Creation Rule is active

    3. Make sure that the Creation Rule has a filter (Alert Condition)

  3. If you have Post-Processing or other advanced settings enabled consider disabling them temporarily

  4. Copy the text version of the alert criteria from the main table view

  5. Navigate to the SCOM Alerts table

  6. Add your search query to the end of your URL to check that you have matching alerts

    1. Clear out any existing query after x_oklt_cookdown_sc_scom_alerts_list.do

    2. Add %3Fsysparm_query%3D

    3. Add the query that you copied in step 4

    4. Press Enter

  7. If you do not see any alerts in the table at this stage you may need to adjust your filter

  8. If you have alerts matching the criteria check to see if they have matched the creation rule at the bottom of the alert:

    1. If it has not matched a Creation Rule at all

      1. Try to ‘re-run’ the alert as described here

    2. If it has matched a Creation Rule but has not raised an alert

      1. Check the history tab to see if there are any messages and investigate any errors

      2. Impersonate the service account and then try to re-run the alert

      3. Investigate any errors seen such as the one described here

      4. If disabling Post-Processing or other scripts has allowed you to generate an Incident/Task proceed to Custom Script Issues below

  9. If you are not able to determine what is going wrong consider reaching out to our support team

Back to top

Alerts Updates Not Being Returned to SCOM

Much like alerts not being created, it’s more likely to be an issue caused by the SCOM side or in transport (proxies or firewalls getting in the way) than the ServiceNow side. Make sure that you do have an active Inbound Notification connection that can connect to ServiceNow.

In order to sync back to SCOM the incident needs to be linked to the SCOM alert, the SCOM alert needs to be linked to a Creation Rule, and the alert needs to have the 'Updates for SCOM?' option checked (should be automatically set when a linked incident is updated and removed when SCOM pulls the updates, though you can set this manually for testing/troubleshooting purposes):

  1. The first thing to do is hurry up and wait

    1. When Connection Center successfully pulls the alert it will mark the alert as polled (by default every 60s)

    2. The alert will remain checked for a further 4 minutes in case a re-run needs to occur for some reason

    3. The Checkmark will be removed and the alert will no longer be available to Connection Center to poll unless another change happens to reset the checkmark

  2. If this does not occur the next thing to check is that the API is successfully presenting the alert to the Service Account

    1. Impersonate the service account

    2. Use the API Explorer (or another service such as PowerShell, Postman, or curl) to call our API

      1. Select API Explorer

      2. Set Namespace to x_oklt_cookdown_sc

      3. Send the request

    3. Verify that the status is 200 and that the response body contains your alert/incident in the body:

    4. If it does not, repeat the steps using an admin account to check for permissions issues

    5. If the admin account returns the details as expected, verify that the service account has the correct roles assigned

    6. If the roles are already correct, or the admin account does not return the expected data it is probably worth looking at ACL debugging or getting in contact with our support team

Back to top

Custom Script Issues

Back to top

Logging

Back to top

Specific Issues

Alerts in ServiceNow Show a Different Time/Timezone to the Alert Itself in the SCOM Console

Problem

The timestamps on SCOM alerts in ServiceNow don't match those on the alert in SCOM itself, making it hard to understand when an alert was truly raised from within ServiceNow.

Cause

In ServiceNow, the time of each alert is displayed in the timezone that you (the user logging into ServiceNow to view the alert) as.

Solution

Change the timezone of your ServiceNow user account in your profile to the correct one:

  1. Navigate to your Profile

  2. Change the Timezone

Back to top

Creation Rule Users See Strange UI Behavior

Problem

A non-admin user of ServiceNow is reporting strange UI behavior when using creation rules. This typically includes the Configuration Item picker and script-able elements not displaying as expected.

Solution

This is generally caused by the user not having access to certain tables within ServiceNow. Typically Configuration Items. Granting access to these elements resolves the issue (typically through the ITIL role). If you do not require this functionality Alert Sync will continue to operate as normal without them and the role can be left off if desired.

Further information on Connection Center and any related roles can be found here.

Back to top

When Using a Caller That Is Not the Service Account, Alert Closed Incident State Will Not Resolve or Close an Incident

Problem

You have set your creation rule to use a caller other than your service account and the ‘Alert Closed Incident State' from the 'Advanced' tab is set to Resolved, Closed, or Cancelled. When a SCOM alert is closed the Incident remains on its previous status and does not get moved to the desired state.

Cause

Cookdown and ServiceNow both adhere to ITIL best practices by default. They allow a caller to resolve or close their own Incidents but require resolution/closure notes and codes when resolving Incidents for another caller. If at the time of SCOM alert resolution the Incident is linked to a different caller (set either by the creation rule or through other means) by default the Incident will not close. You may also see this behavior using the service account as the caller in a customized instance.

You can verify this is the case by impersonating the service account and updating an alert manually. You will receive an error similar to the following:

Data Policy Exception: The following fields are mandatory: Resolution code, Close notes

Solution

Use the service account as the caller where possible

Workaround

  1. Rather than using ‘Alert Closed Incident State’ in your creation rules use 'Advanced Incident/Task Updates' to script the resolution of the ticket, setting the required fields as appropriate.

  2. In ServiceNow create or adjust your ‘Data Policy' and 'Data Policy Rules’ to exempt Incidents raised by creation rules from needing to fill in these fields.
    Specifically how you approach this would be down to your organizational requirements, however, a suggestion might be to exempt any Incident created/updated by your service account, with a certain contact type, or a combination of both.

Back to top

Creation Rule Fails to Create Task, Catalog Task, and Service Task

Problem

When a new SCOM alert comes in, a creation rule set to create a Task, Catalog Task, or Service Task fails to raise a task of this type

Cause

By default, ServiceNow does not provide users or the ITIL role with access to these tables. You may find that there are no ACL entries for these roles or you may find that there are not any ACL entries for those tables at all…

Solution

Provide the Service account with appropriate access to these tables to use these task types. For example, you could provide the ‘catalog_admin' role to the service account to provide access to create Catalog Tasks. However, rather than doing this, we strongly recommend that you create your own ACL to provide access to the required table via the 'x_oklt_cookdown_sc.scom_sync_service’ role or another role of your choice.

Back to top