Using Google Analytics Segments to Evaluate Site Search Relevancy

Evaluating the success of a new site search engine or an update to an existing search engine can be difficult. Thankfully, Google Analytics makes this task infinitely easier – if you know how to navigate it's confusing new dashboards.

Is My Site Search a Success?

If you're looking to understand how the implementation of a site search engine has affected conversion rates among the visitors who used the search box during a single visit, it is important to be looking at all sessions where a site search took place – not just the number of users that used site search. You also want to exclude bounces from this segment and traffic that had a search results page as the landing page. 

However, a little while ago, Google changed their default segment for visits in which site search was used into a new segment called "Performed Site Search". By doing this, they changed the scope from session-level to user-level. Previously, the session-based segment was called "Visits with Site Search" and looked like this:

Now, with the new default segment called "Performed Site Search", a user who has, at any point in time, performed a site search will always be flagged up and included in the segment "Performed Site Search" – even if they don't use site search in subsequent sessions. Put simply, this segment describes all users that have ever used site search and not exclusively the sessions with site-search. 

This has the effect that all sessions, regardless of whether a site search was performed of not, will be included when the "Performed site search" segment is chosen.

As shown in the example below, 80% of the sessions (i.e. visits) that were flagged as "Performed Site Search" didn't actually involve a site search. 

What's the Fix?

This is easy to remedy. You just need to change the segment to a session-level segment. This changes the segment to look at sessions where a site search was performed, rather than sessions by a user who has ever performed a site search. 

It's a bit like the phrase: not all digits are thumbs, but all thumbs are digits. But in this case it would be: not all "Performed Site Search" are sessions with site-search, but all sessions with site search are "Performed Site Search".   

There are two ways to fix this so you can truly evaluate the performance of your search engine: 

1. There is a condition you can use when creating a new segment where you look to set "Site Search Status" = "Visits with site search" in the segment conditions tab.
2. You could also copy the default segment "Performed Site Search" and change the scope from user to session.

We also recommend adding an additional condition. Set "Bounces" = 0, to the session-based segment. This will make sure you remove any bounced visitors. 

This is a good idea because visitors who bounce from a search results page have always been directed there from ads or direct links, and the relevancy of those sources are outside of the control of the search engine and may dilute the data.

When evaluating the relevancy of a search engine and that relevancy's impact on conversion rates during a single session, it is misleading to include bounces since those bouncing visitors did not actually perform a site search in the true sense of the word (i.e. they didn't come to your site, navigate to the search box, input a search query and hit "enter" to be directed to a search results page). 

If set up correctly, you should see something like this when comparing the Google default "Performed Site Search" to your new sessions with site search segment (excluding bounces). 

Lastly, because the relevancy of Loop54 search results are smart and improve over time by learning from visitor behaviour, many of our customers use a Loop54 search results page as a landing page for their social media posts, email marketing and PPC campaigns.

Now, there isn't anything inherently wrong with this, but since it tends to be done sporadically and does not reflect an authentic site search behaviour, it is important to segment out this traffic from the overall "Visits with Site Search" segment.

In fact, we advise you to create a separate segment completely to evaluate these landing pages (i.e. set segment condition to "landing page" contains "[url string for your search result pages]".

How Should it Look in the End?

If you've followed these steps correctly, you should end up with one segment for evaluating site search engine performance that looks like this:

• Site search status contains "visits with site search" AND
• Bounces = 0 AND
• Landing page does not contain "search" [or whatever url string identifies a search results page on your site]

Once you have your segment set up properly, you can either use it to evaluate any A/B tests you've set up or use it to conduct an evaluation without doing an A/B test. There's more information on this in our blog on how to evaluate your site search without doing an A/B test.

We hope this article helps you utilise Google Analytics more effectively for evaluating the performance of your search engine. For more information about how Google can help out your site, see our blog on Google Posts. 

We regularly conduct Google Analytics evaluations for our customers, helping them measure the success of their engines and troubleshoot potential issues, so even if you're struggling, we'll be happy to help!