How to get Service Provider back in Google Analytics

William Julian-Vicary

Strategy & Innovation Director

Difficulty: 5/5

Requirements: Google Tag Manager & Google Analytics with 1 spare custom dimension

If you're seeing Service Provider (not set) in your Analytics reports - we share your pain. Unfortunately, over the past few months, we’ve heard chatter that Google were planning to drop two relatively unknown but important dimensions: Service Provider and Network Domain. It seems that as of Feb 4th 2020, these dimensions are no longer collecting data.

A couple of use-cases for having these dimensions:

  • Bot Filtering
  • Direct Traffic Analysis

There are more but these are the two we use them for on a regular basis.

Filtering Bot Traffic using an IP Lookup Service

This is the focus of this post - get Service Provider back online ASAP. In the same way Google did it, we can just run a lookup against a user’s IP address and determine their service provider.

To do this we’re going to use an IP Lookup Service which has API access. We’ve built a script which can run in GTM to do all the legwork and provide GA with the ISP info as a new custom dimension.

There are a lot of lookup services available which can help us out here. We’ve chosen one which is free, allows HTTPS requests (not a lot of free ones offer this) and isn’t hugely expensive when you decide you need to pay for more lookups. It also doesn’t require an API key or any authentication, you simply call the JSON API url in JS to get the lookup info. The free service has a request limit of 20 lookups per minute. The paid version is unlimited and costs EUR 20 a month which isn't a bad compromise if you are in dire need of ISP data. We have built this guide to throttle requests per session to help with this limit. If you have a lot of active users on your site at any given time, some of these requests will be ignored and no ISP info will be given.

If you run this JSON request url in your browser, it’ll return your own lookup info:

https://extreme-ip-lookup.com/json/

Example Output

{
   "businessName" : "",
   "businessWebsite" : "",
   "city" : "Sheffield",
   "continent" : "Europe",
   "country" : "United Kingdom",
   "countryCode" : "GB",
   "ipName" : "host12-345-67-890.in-addr.btopenworld.com",
   "ipType" : "",
   "isp" : "British Telecommunications PLC",
   "lat" : "53.38297",
   "lon" : "-1.4659",
   "org" : "British Telecommunications PLC",
   "query" : "12.345.67.890",
   "region" : "England",
   "status" : "success"
}

This info is perfect for our needs and also gives us some extra data on the house. For now, let’s just focus on grabbing the key pair labelled “isp”

 

We’ve done the hard work of fetching, parsing and providing this data as a dataLayer push below:

IP Lookup Code

<script>
  (function() {
    
    // Session cookie to block future attempts to lookup the ISP - only attempt once per session
    try {
        ("gtm_isp_lookup", true, "", "/", );
    } catch (e) {

    }

    var endpoint = 'https://extreme-ip-lookup.com/json/';
    var xhr = new XMLHttpRequest();
    
    xhr.onreadystatechange = function () {
      if (this.readyState == 4 && this.status == 200) {
        var response = JSON.parse(this.responseText);
        if (response.status !== 'success') {
          return
        }
    
        try {
          window.dataLayer = window.dataLayer || [];
          window.dataLayer.push({
            'event': 'isp-lookup',
            'isp-lookup': {
              'lat': response.lat,
              'lon': response.lon,
              'isp': response.isp,
              'org': response.org,
              'continent': response.continent,
              'country': response.country,
              'region': response.region,
              'city': response.city,
              'businessName': response.businessName,
              'businessWebsite': response.businessWebsite
            }
          });
        } catch (e) {}
      }
    };
    xhr.open('GET', endpoint, true);
    xhr.send();

  })();
</script>

To walk through this code:

  1. Save a first-party cookie for the duration of the session to prevent spamming the lookup service. This is a simple flag to tell GTM to not keep re running the code
  2. Open the connection to the service
  3. Check it’s returning a good result
  4. Parse the JSON object into a variable
  5. Send this variable information to the dataLayer as an event with all details of the lookup where available
  6. Close the lookup connection

With this information sent out as a dataLayer event, we can capture it using a Custom Event trigger and pass this information to Google Analytics as a non-interactive Event with a custom dimension set for the ISP.

This code relies on other variables and triggers we will run through below.

Note, it won’t run outside of GTM without modification.

New GTM Variable: 1P Cookie - ISP

Type: First Party Cookie

Value: gtm_isp_lookup

New GTM Variable: URL - Hostname no WWW

Type: URL

Value: Host Name

Tick: Strip “www.”

New GTM Variable: JS - Set Cookie

Type: Custom JavaScript Code

Value: (code snippet below)

function() {
  return function(name, value, ms, path, domain) {
    if (!name || !value) {
      return;
    }
    var d;
    var cpath = path ? '; path=' + path : '';
    var cdomain = domain ? '; domain=' + domain : '';
    var expires = '';
    if (ms) {
      d = new Date();
      d.setTime(d.getTime() + ms);
      expires = '; expires=' + d.toUTCString();
    }
    document.cookie = name + "=" + value + expires + cpath + cdomain;
  }
}

Kudos to Simo for this one. It’s a nice little function saved as a JS variable which we can use in other parts of GTM as a function to process cookie saving.

New GTM Tag: SCRIPT - ISP Lookup

Type: Custom HTML

Simply paste in the script we shared above

New GTM Trigger: GTM Load - ISP Not Set

Type: Window Loaded

Condition: 1P Cookie - ISP does not equal true

Save the trigger to the tag and save the tag. This trigger prevents us from hitting the lookup service per hit and instead will only fire the code when the First Party Cookie isn’t set. As the cookie is set to Session expiry, we are throttling the lookup attempts per user. Also, doing an ISP lookup per hit would be a bit overkill considering the likelihood of it changing is slim.

New Custom Dimension in Google Analytics

Create a new Custom Dimension called ISP. Head over to GA > Admin > Custom Definitions > Custom Dimensions

Click “+ New Custom Dimension

Name it ISP and set Scope to Session

Once saved, you’ll be able to get the Index number and apply it to a new Event in Google Tag Manager later on. Make a note of it for now.

New GTM Variable: EDLV - ISP - ISP

We need to grab the Service Provider data and store it as a Data Layer Variable in GTM. This will be used later to send it to Google Analytics. We will set a Default Value on this variable to allow us to count the number of sessions recording no ISP information.

Type: Data Layer Variable

Value: isp-lookup.isp

Set Default Value: (not set)

Format Value: Change Case to Lowercase

New Google Analytics Event: GA - Event - ISP Lookup

This Event is acting as a transport bus so we can deliver the new ISP info to GA for the Session. You’ll need to adjust the custom dimension index to match a newly created custom dimension in Google Analytics. Ours is 99 for demonstration purposes.

New Trigger: CE - ISP Lookup Completed

The code we’ve written fires a dataLayer event called “isp-lookup” - when this happens, it means the ISP information was searched for. This will be how we fire our GA Event to pass any ISP information found over to Google Analytics.

Add the trigger to the new GA Event and save it.

With all of these pieces in place, and deployed, you will start to see new ISP data populating under your Custom Dimension. Side by side, the data is similar and will aid in filtering out any unwanted bot traffic from your Analytics production views.

For quick analysis, we have compared the data side by side in Data Studio:

If you know the traffic you’re looking to filter, you can build a new view filter in GA and start blocking bot traffic right away.

As a quick example, we could do something like the below screenshot the same way we would using the now deprecated dimension Internet Service Provider.

 

If you're feeling particularly lazy, here's the GTM Container Export I used for this demonstration. Be sure to change the UA ID for the GA Event.

GTM Container Download

Note: this is not 100% plug and play, you should always test imported containers you haven't built for configuration updates, errors and malicious code usage.

Data Privacy Considerations

  • When you utilise a 3rd party vendor such as Extreme IP, they become a Data Processor as they take the user's IP Address to identify Internet Service Provider information.
  • You may want to consider this update as part of your privacy policy if you are looking to conform with regulations such as GDPR or CCPA which require explicit consent to process personally identifiable information (PII)
  • Our method does not store user IP addresses in any format, raw or hashed. We only store the Service Provider information as a Custom Dimension.

Related Posts

Extracting Query Parameters in Google Analytics - Free Tool

Auditing your Google Analytics account for query parameters can be tedious but helps keep page reports clear of unwanted row duplication. We talk about the effects of collecting unwanted queries on your reporting and how to fix it. We’ve also built a free tool to make the process easier.


4 months ago

Tracking HubSpot Forms with Google Tag Manager

When a form has been successfully submitted to HubSpot, HubSpot propagates an event that we can listen to and use to trigger marketing tags within Google Tag Manager. Below we create a Custom HTML Tag that listens for a form success, trigger a dataLayer event which can then be used to trigger a marketing tag to push this data into Google Analytics.


5 months ago

Measuring Content Engagement in Google Analytics - Part 3: Social Site Exits

Track when users leave your site to visit your social media pages. Understand when your content is driving positive move to your social assets.


5 months ago

Measuring Content Engagement in Google Analytics - Part 2: Social Share Click Tracking

The use of social sharing widgets is commonplace. But who’s actually tracking interactions with these? Let’s make a count of how many people are clicking these buttons and record a social share rate to see how many readers amplify content on social media.


7 months ago

DoubleClick Floodlight Tags and Google Tag Manager - Lean up Your Containers

Anybody who uses DoubleClick Floodlight knows the slow process of implementing a new tag for each activity for a campaign interaction or conversion. If you’re using Google Tag Manager to add a tag, trigger and maybe some new variables for every new Floodlight – it won’t take long for your container to overfill with almost identical tags. While there’s nothing wrong with this style of implementation, there is another way.


8 months ago

Measuring Content Engagement in Google Analytics - Part 1: Article Completion Rate

It’s time to ditch bounce rates and average time on page in place of meaningful and measurable insights. Enhance how you measure content engagement with these advanced tracking methods to gain real insights into your content and its value against your business objectives.


8 months ago

Drag