How to: Use Refiners with the Search Web Service (FAST Search for SharePoint)
Posted
Wednesday, December 28, 2011 12:50 PM
by
CoreyRoth
Yesterday, I talked about how we can use FQL with the Search Web Service. As promised, I am continuing the series about the search web service to talk about how we can take advantage of refiners. As you may know, FAST Search for SharePoint gives you exact counts in refinement so why not take advantage of them in your custom search application. We’ll take a look at how to specify which refiners are returned with your search results. After that we’ll take a look at how to use those refiners in a query.
Returning refiners with your results is also quite easy. All we have to do is add a few elements to the input XML document of the Query or QueryEx method. Before you add these elements though, you need to know what you want to refine on. You can refine on any managed property configured to allow refinement (or deep refinement) on the FAST Search Administration page. We’ll work with some out of the box properties. You can look up the names for these by examining the Refinement Panel. In this post, I talk about the Refinement Panel in SharePoint search but in FAST, the Refinement Panel has a different XML file (and managed property names). Take a look at the XML below and take note of the MappedProperty attribute. These are out-of-the-box managed properties that we can use including format (file extension), author, sitename, and write (modified date).
<?xml version="1.0" encoding="utf-8"?>
<FilterCategories>
<Category Title="Result Type" Description="The file extension of the item" Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator" MetadataThreshold="1" NumberOfFiltersToDisplay="4" MaxNumberOfFilters="20" ShowMoreLink="True" MappedProperty="format" MoreLinkText="show more" LessLinkText="show fewer" ShowCounts="Count" />
<Category Title="Site" Description="Which site this document is from" Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator" MetadataThreshold="1" NumberOfFiltersToDisplay="4" MaxNumberOfFilters="20" ShowMoreLink="True" MappedProperty="sitename" MoreLinkText="show more" LessLinkText="show fewer" ShowCounts="Count" />
<Category Title="Author" Description="Use this filter to restrict results authored by a specific author" Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator" MetadataThreshold="1" NumberOfFiltersToDisplay="4" MaxNumberOfFilters="20" ShowMoreLink="True" MappedProperty="author" MoreLinkText="show more" LessLinkText="show fewer" ShowCounts="Count" />
<Category Title="Modified Date" Description="When the item was last updated" Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator" MetadataThreshold="1" NumberOfFiltersToDisplay="6" SortBy="Custom" ShowMoreLink="False" MappedProperty="write" ShowCounts="Count" >
<CustomFilters MappingType="RangeMapping" DataType="Date" ValueReference="Relative" ShowAllInMore="False">
<CustomFilter CustomValue="Past 24 Hours">
<OriginalValue>-1..</OriginalValue>
</CustomFilter>
<CustomFilter CustomValue="Past Week">
<OriginalValue>-7..</OriginalValue>
</CustomFilter>
<CustomFilter CustomValue="Past Month">
<OriginalValue>-30..</OriginalValue>
</CustomFilter>
<CustomFilter CustomValue="Past Six Months">
<OriginalValue>-183..</OriginalValue>
</CustomFilter>
<CustomFilter CustomValue="Past Year">
<OriginalValue>-365..</OriginalValue>
</CustomFilter>
<CustomFilter CustomValue="Earlier">
<OriginalValue>..-365</OriginalValue>
</CustomFilter>
</CustomFilters>
</Category>
<Category Title="Company" Description="Use this filter to restrict results by company" Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator" MetadataThreshold="1" NumberOfFiltersToDisplay="4" MaxNumberOfFilters="20" ShowMoreLink="True" MappedProperty="companies" MoreLinkText="show more" LessLinkText="show fewer" ShowCounts="Count" />
<Category Title="Managed Metadata Columns" Description="Managed metadata of the documents" Type="Microsoft.Office.Server.Search.WebControls.TaxonomyFilterGenerator" MetadataThreshold="3" NumberOfFiltersToDisplay="3" MaxNumberOfFilters="20" ShowMoreLink="True" MappedProperty="owsmetadatafacetinfo" MoreLinkText="show more" LessLinkText="show fewer" />
<Category Title="Tags" Description="All managed metadata of the documents and social tags" Type="Microsoft.Office.Server.Search.WebControls.TaxonomyFilterGenerator" MetadataThreshold="3" NumberOfFiltersToDisplay="3" MaxNumberOfFilters="20" ShowMoreLink="True" MappedProperty="owsmetadatafacetinfo" MoreLinkText="show more" LessLinkText="show fewer" ShowCounts="Count" />
</FilterCategories>
We’ll take a look at format and author first. The managed property for managed metadata fields, owsmetadatafacetinfo, requires a bit more explaining. To add Refinement to your search results, you just need to add the IncludeRefinementResults element somewhere inside your Query element. You then specify the name of the managed properties that you want in individual Refiner elements. Here is what the raw XML looks like.
<IncludeRefinementResults>
<Refiners>
<Refiner>author</Refiner>
<Refiner>write</Refiner>
<Refiner>sitename</Refiner>
<Refiner>format</Refiner>
<Refiner>owsmetadatafacetinfo</Refiner>
</Refiners>
</IncludeRefinementResults>
To use it in the code, I am using a StringBuilder like before. Here is what the whole string looks like.
queryXml.Append("<QueryPacket xmlns=\"urn:Microsoft.Search.Query\" Revision=\"1000\">");
queryXml.Append("<Query domain=\"QDomain\">");
queryXml.Append("<SupportedFormats>");
queryXml.Append("<Format>");
queryXml.Append("urn:Microsoft.Search.Response.Document.Document");
queryXml.Append("</Format>");
queryXml.Append("</SupportedFormats>");
queryXml.Append("<Range>");
queryXml.Append("<Count>50</Count>");
queryXml.Append("</Range>");
queryXml.Append("<Context>");
queryXml.Append("<QueryText language=\"en-US\" type=\"FQL\">");
queryXml.Append(searchQuery);
queryXml.Append("</QueryText>");
queryXml.Append("</Context>");
queryXml.Append("<IncludeRefinementResults>");
queryXml.Append("<Refiners>");
queryXml.Append("<Refiner>author</Refiner>");
queryXml.Append("<Refiner>write</Refiner>");
queryXml.Append("<Refiner>sitename</Refiner>");
queryXml.Append("<Refiner>format</Refiner>");
queryXml.Append("<Refiner>owsmetadatafacetinfo</Refiner>");
queryXml.Append("</Refiners>");
queryXml.Append("</IncludeRefinementResults>");
queryXml.Append("</Query>");
queryXml.Append("</QueryPacket>");
I added several refiners in my case which we will talk about. The rest of the code to execute the query is exactly the same from yesterday’s post. Let’s look at a snippet of the results. For each refiner returns you will get a RefinementResults element returned along with your regular RelevantResults elements that you may be accustomed to by now. Note, that the RefinementResults elements are not grouped in any sort of parent element like the RelevantResults are. Here is a snippet.
<RefinementResults diffgr:id="RefinementResults19" msdata:rowOrder="18">
<RefinerName>format</RefinerName>
<RefinementName>Microsoft Word</RefinementName>
<RefinementValue>^"Microsoft Word"$</RefinementValue>
<RefinementToken>AQ5NaWNyb3NvZnQgV29yZAZmb3JtYXQBAl4iAiIk</RefinementToken>
<RefinementCount>28</RefinementCount>
</RefinementResults>
<RefinementResults diffgr:id="RefinementResults20" msdata:rowOrder="19">
<RefinerName>format</RefinerName>
<RefinementName>Web Page</RefinementName>
<RefinementValue>^"Web Page"$</RefinementValue>
<RefinementToken>AQhXZWIgUGFnZQZmb3JtYXQBAl4iAiIk</RefinementToken>
<RefinementCount>27</RefinementCount>
</RefinementResults>
If you were to write custom code to display these, I recommend using LINQ to XML and grouping elements that have matching RefinerName values. In the case above, format is allowing us to refine by file extension. The RefinementName element contains (usually) a human-readable value to display for the refiner. The exact count of results can be found in RefinementCount. Lastly, I will point out the RefinentToken. This allows us to issue a subsequent query using that refiner. We’ll talk about that shortly after I cover some more details on the results.
When refining by date managed properties (such as write), things get a little more complicated. Take a look at the refinement results, it gave me.
<RefinementResults diffgr:id="RefinementResults16" msdata:rowOrder="15">
<RefinerName>write</RefinerName>
<RefinementName>From 2010-11-11T07:18:13.4998560Z to 2010-11-11T18:49:08.4997920Z</RefinementName>
<RefinementValue>[2010-11-11T07:18:13.4998560Z;2010-11-11T18:49:08.4997919Z]</RefinementValue>
<RefinementToken>AREERnJvbSAyMDEwLTExLTExVDA3OjE4OjEzLjQ5OTg1Nj…==</RefinementToken>
<RefinementCount>27</RefinementCount>
</RefinementResults>
<RefinementResults diffgr:id="RefinementResults17" msdata:rowOrder="16">
<RefinerName>write</RefinerName>
<RefinementName>From 2010-11-11T18:49:08.4997920Z to 2011-12-27T17:12:11.4998400Z</RefinementName>
<RefinementValue>[2010-11-11T18:49:08.4997920Z;2011-12-27T17:12:11.4998399Z]</RefinementValue>
<RefinementToken>AREERnJvbSAyMDEwLTExLTExVDE4OjQ5OjA4LjQ5OTc5MjBaIH…==</RefinementToken>
<RefinementCount>19</RefinementCount>
</RefinementResults>
<RefinementResults diffgr:id="RefinementResults18" msdata:rowOrder="17">
<RefinerName>write</RefinerName>
<RefinementName>2011-12-27T17:12:11.4998400Z or later</RefinementName>
<RefinementValue>[2011-12-27T17:12:11.4998400Z;]</RefinementValue>
<RefinementToken>ARUCMjAxMS0xMi0yN1QxNzoxMjoxMS40OTk4NDAwWiBvciBsYXRlc…=</RefinementToken>
<RefinementCount>22</RefinementCount>
</RefinementResults>
It gives you a few date ranges by default. You have to carefully examine the values of RefinementName to figure it out, but effectively it comes down to documents from yesterday, documents in the last 6 weeks or so, and anything before that. The refinement documentation on MSDN, seems to indicate that you can specify exact dates for it to refine on, but I haven’t fully explored it.
Finally, we’ll talk about how Managed Metadata results get returned. These come from the managed property, owsmetadatafacetinfo.
<RefinementResults diffgr:id="RefinementResults1" msdata:rowOrder="0">
<RefinerName>owsmetadatafacetinfo</RefinerName>
<RefinementName>CompanyName|CompanyName|SIY8F/QVWEiQVLpg6IwnCg==|SL8RuOhIpUekTSlmeLc0AQ==|#04cbd472-7446-45cd-87c3-57ea2e142888|Contoso;#</RefinementName>
<RefinementValue>^CompanyName|CompanyName|SIY8F/QVWEiQVLpg6IwnCg==|SL8RuOhIpUekTSlmeLc0AQ==|#04cbd472-7446-45cd-87c3-57ea2e142888|Contoso;#$</RefinementValue>
<RefinementToken>ARkHQ29tcGFueU5hbWV8Q29tcGFueU5hbWV8U0lZOEYvUVZXRWlRVkxwZzZJd25DZz09fFNMOFJ1T2hJcFVla1RTb…==</RefinementToken>
<RefinementCount>19</RefinementCount>
</RefinementResults>
<RefinementResults diffgr:id="RefinementResults2" msdata:rowOrder="1">
<RefinerName>owsmetadatafacetinfo</RefinerName>
<RefinementName>ProductLine|ProductLine|SIY8F/QVWEiQVLpg6IwnCg==|HQGuvm9zUUGAE1lO7yojyA==|#5bff12c1-ed32-4e0e-aadb-f9bbdd436432|Office;#</RefinementName>
<RefinementValue>^ProductLine|ProductLine|SIY8F/QVWEiQVLpg6IwnCg==|HQGuvm9zUUGAE1lO7yojyA==|#5bff12c1-ed32-4e0e-aadb-f9bbdd436432|Office;#$</RefinementValue>
<RefinementToken>ARgHUHJvZHVjdExpbmV8UHJvZHVjdExpbmV8U0lZOEYvUVZXRWlRVkxwZzZJd25DZz09fEhRR3V2bTl6VVVHQUUxbE83e…Ek</RefinementToken>
<RefinementCount>7</RefinementCount>
</RefinementResults>
In this case, I have two managed metadata properties defined: CompanyName and ProductLine. Actually, these names map back to underlying site columns. In my case CompanyName is mapped to the built-in managed property Company. The RefinementName element gives you a ton of date that you will have to parse through including the name of the site column, tokens, the GUID of the term and ultimately the text value. Have fun parsing that. :)
Now, that we have our refiners, we can use the RefinementToken to refine a subsequent query. To do this, we just add a RefinementFilters element inside our Query element. We include a RefinementFilter element with the token of each refiner we are using. For example, to filter by Word documents from our previous results, we’ll add the values like this.
<RefinementFilters>
<RefinementFilter>AQ5NaWNyb3NvZnQgV29yZAZmb3JtYXQBAl4iAiIk</RefinementFilter>
</RefinementFilters>
You’ll get another set of results back like you did before only the results are now refined. You can include multiple RefinementFilter elements if you want as well. I know that’s a lot of XML for one post, but hopefully it’s useful.