Corey Roth [MVP]

Last week, I showed you how to use the QueryManager class to query SharePoint Enterprise Search.  In that post, I mentioned that we can actually use the QueryManger class to harness the true power of federated search and query multiple locations at once.  As you will see today, it’s pretty easy to do, but you might have to do some work with the XML you get back.  We’ll start with the code from last week, and then add the search results from the included Bing.com Internet Search Results provider.  To add another location, we just add another location to the LocationList object.

Remember that the name of the Location comes from the Location Name field on the federated location.  Refer to the table in the QueryManager post for more information.   Using the code from last week, we put all this together and it looks like this:

As you can see, it’s pretty simple to add another location.  What do the search results look like though?  Well, the results may not be what you expected.  It just appends the second location’s results to the XML document.  What is bad is that the schema isn’t the same. Since Bing results are coming in via an OpenSearch provider, it just gives you the raw RSS feed of the results.  I won’t post the entire XML document here, but you should get the idea of what I mean.

This may not be ideal, but if you need the data, you can get it.  You may just have to work with the XML a bit.  Luckily LINQ to XML makes that pretty easy.  What about if you add a third location?  Well it just appends it right onto the end.  I added another SharePoint location and it simple adding Result elements right onto the document.  What I don’t like about it is that there is nothing in the Result element that tells you what location it came from. 

You know reading that copyright message on the Bing results makes me wonder.  It specifically says that the results can only be used with an RSS aggregator for personal, non-commercial use.  I would venture to say most uses of SharePoint are commercial.  Does this mean we are not supposed to use Bing results there?  Perhaps that is a question for the forums.

Even if you have to work with the XML a bit to get the results you need, I think this is very powerful.  It’s nice that you can get results from more than one location so easily.  I’ve attached an XML file to this post with the complete result set from three locations if you want to examine it further.

One request I have seen a lot from people since MOSS 2007 is the desire to add a link to the subfolder of a document on the search results page.  This is useful when the user wants to see what else is in that folder or if they want to perform other operations on that document (such as viewing properties or compliance details).  In SharePoint 2007, I wrote the Document Link Handler to help with this.  It simply looked up the details on the item and redirected the user to the appropriate page.  In SharePoint 2010, this is not necessary any more since they added the new sitename column in the search results XML document.  Today we will customize the look of the search results page.  All we have to do is use the new sitename column in the XSL that displays the results.

There are two ways to go about this.  If you only want this new link to show up on certain results pages, you can edit the CoreResultsWebPart in a given search center.  If you want the link to be global and to appear on all search centers, you can edit the Local Search Results federated location.  Today we’ll demonstrate modifying the federated location, but if you want to just change a specific page, edit it, and then edit the Core Results Web Part.  You will then click the XSL editor button and make the same changes as we are about to do in the federated location. 

To edit the federated location go to your Search Service Application –> Federated Locations, then edit Local Search Results.  Expand the Display Information and scroll down to the Core Search Results Display Metadata section.

Ensure Use Default Formatting is not checked and click the button next to the XSL textbox.  I found that the best place to add the link is near the location where the View in Browser link normally appears.  The place in the file is more than half way down.  If you see the ViewInBrowser link you know you are near the right spot.  Here is what it looks like.

Now, what do we add there?  It’s pretty simple.  We use XSL to create a link and pass the value of the sitename element.  However, I recommend that you only show the link for documents.  The sitename element is always populated but you might find that it has less relevance on things like non-SharePoint sites or for list items.  It’s up to you if you want to include it or not though.  Here is the code that you will want to add.

One thing to note is that in SharePoint 2007, you used to compare isdocument to a value of 1.  That doesn’t work anymore.  You have to compare it to True or False.  That’s really all you have to do.  Here is what it looks like together.

Once you have the change complete, click OK and then save your changes to the federated location.  It usually takes about a minute for your changes to show up when you do search.  So wait a bit and then go to your search center and give it a try.  Here is what the link looks like on my results screen.

If the link doesn’t show up for you, try another search term.  If you still have issues, try removing the isocument xsl:test element and see if the link shows up then.  This link is pretty easy to add, so give it a try today.  I think the link to the document’s folder is really handy and I think your users will appreciate it.

Last week, I talked about the KeywordQuery class and how to use it in SharePoint 2010.  I mentioned that a new way to query also existed using the QueryManager class.  This class is extremely powerful and can allow us to issue federated queries to multiple locations at one time.  For example, you could make one call and get results back simultaneously from Enterprise Search, FAST, and an OpenSearch provider like Bing.  You can still use the KeywordQuery class but I think the best practice is likely to be the QueryManager class because of the added flexibility.  Another thing I like about the QueryManager class is that the results are returned as XML instead of a ResutlsTableCollection.  Now, in reality if you are just querying one source such as Local Search Results, QueryManager might add a little overhead since it turns around and calls the KeywordQuery class, but I think the the benefits that QueryManager brings are worth it.

Before we begin, let’s take a look at our Federated Locations in the Search Service Application.  In a typical out-of-the-box SharePoint 2010 install, you will have five Federated Locations.  We can query any of these locations using the QueryManager class.  However, we have to know the proper name to pass to the QueryManager to specify the location.  We’ll talk about that here in a bit.  Here is what your federated locations might look like.  In my case, I have an extra federated location that I created myself that queries search on DotNetMafia.com.

For our first example, we will just issue a simple query against the Local Search Results location.  Our first couple of lines are just like the ones we used with the KeywordQuery claass.  We need a reference to the Search Service Application.  In my case, I have named my application Search Service Application.  Change the name of the string to match the name of your service application.

We now need to create an instance of the QueryManager object as well as an object to keep track of the locations we are searching (LocationList).  The QueryManager is kind of an interesting object because it really is just a LocationList with a few extra properties.  There are very few settings you can specify on it other than the query.   Both of these just use default constructors, we pass in the proxy once we start creating new Location objects.

Now, we want to create an instance of a Location object which takes the internal name of the location of the federated location as well as our SearchServiceApplicationProxy.  The example in the SDK had a heap of code, that loops through all of the locations on the server.  I found that it was kind of overkill and can simply be replaced with a single line of code to create the Location object.

As you can see here, I specified LocalSearchIndex for the name of my Location.  You might be wondering where I got that value.  This is the internal name of the federated location (not to be confused with the display name).  You can get this value by looking at the details of the location.

Here is a table with the internal names of all of the out-of-the-box federated locations for future reference.

As you can see there is not a lot of consistency between the internal names so I thought this table would be useful.  Remember, we can query FAST using the same API as we do Enterprise Search.  We just specify FASTSearch as the location.  We’ll talk about that more in a future post as well.  Now, we need to add the location to the LocationList.

Once we have a location set, we can actually specify our query.  We’ll stick with my usual example of querying for accounting documents.

Now we just need to add our LocationList to our QueryManager and we’re ready to query.  We also have to pass it to the IsTriggered method as well.  I’m not fully sure of the reason behind this, but I think if you don’t set it, the location will be added but not executed.

Now we execute the query using the GetResults method.  It returns an XmlDocument class.  I am kind of surprised this isn’t an XDocument, but I guess it’s easy enough to get from one to the other.

If you have everything configured successfully, you should get some XML with results.  Let’s take a look at what one of the results looks like.  I make use of the File object’s WriteAll method to quickly dump the results into an XML file so I can look at it.  That was one of my first blog posts there more than five years ago. :-)  This way I can open it with Visual Studio and use the Format Document (Ctrl+K, Ctrl+D) command to make it readable.  Here is what one of the results looks like.

This is all of the data you get back without specifying any custom managed properties.  If you are familiar with querying Enterprise Search from MOSS 2007, you will know that there is a lot more data returned.  I’ll point out some of the interesting elements.  When a document has multiple authors, we see them all in author element delimited by a semicolor (;).  Each author is also available separately in the author_multival element.  Of course, we have the usual things such as a URL, the size, modification date (write), whether it is a document or not, and it’s content class.  The sitename element is particularly interesting.  It actually gives us the folder that the file exists in.  You don’t know how much I wanted that feature in MOSS 2007.  That is why I wrote the document link handler to help get the folder name of documents.  If you use Excel Services or Office Web Apps, the serverredirectedurl will provide the path to open the document using Office Web Apps.

I would be a bad blogger if I didn’t tell you how to specify your own managed properties.  Unfortunately, this works exactly like it did in SharePoint 2007.  If you specify one managed property, you lose all of the defaults.  You’ll see what I mean here in a minute.  You specify your managed properties on the Location object using the RequestedProperties StringCollection.  Just set the property on the Location before adding it to the LocationList.

Since I didn’t specify any other managed properties, the only thing I get back is the Id and the DocumentType managed properties.  See below.

Not very useful.  So instead, you have to pass the values of all of the fields you want.  I have found that most of the elements you see in the first XML document, you can pass to the RequestedProperties parameter but not all of them.  For example, to get the value of the URL, you actually specify the manager property path, and it gives you both url and urlEncoded.  Here is what the line might look to get all of your managed properties and the new one. 

How did I know all of those properties?  Well the URL thing I remember from working with the KeywordQuery class in the past.  The rest, I just got from the CoreResutlsWebPart.  You can also look at the Local Search Results federated location.

I was planning on showing the code to do multiple locations too, but this post is already getting long, so I think I will include it in the next post.  It’s cool enough to deserve its own post. :-)  I will show you two other properties that are important to know about.  These properties allow us to do paging.  On the Location object we can set the ItemsPerPage and StartItem property to set the number of items per page and what item to start on.  For example, to start on item #21 and show 20 items, we would add the following before the Location is added to the LocationList.

When you get results back, the XML contains the total number of results as well as the number of results returned.  In my case I only had two results in my set since there were only 22 items in the entire set.

I assume these number are estimates in Enterprise Search, but I could be wrong.  I need to test more.  If you know for sure, please leave a comment. 

One more thing I will mention is that exceptions are returned by the Location object.  So if you don’t get results back, check Location.Exception to see what the details are.  It won’t actually throw an exception.  It will just return null.

I know some people (including myself) like to see all of the code together, so here it is.  This is the complete example with our DocumentType managed property included with paging.

The QueryManager class is very powerful for interacting with Enterprise Search.  I hope you found this post useful.  In my next post, we’ll query multiple locations at a time.

By now, I’m sure you know that there have been a ton of changes and improvements in SharePoint 2010 Enterprise Search.  The underlying architecture of Enterprise Search has been ripped out of the old SSP model and is now based on Service Applications.  Although, Microsoft abstracted a lot of this away for us so that our old code still works, it’s worth noting and being aware of the changes.  When it came to querying search programmatically in SharePoint 2007, we had a choice of using the API, Web Services, or using the RSS feed.  When using the search API, we typically used the KeywordQuery or FullTextSqlQuery classes.  To this day, those two posts are still in the top 20 on DotNetMafia.com.  This tells me that people must be pretty interested in querying search using their own code.

Today I am going to talk about the KeywordQuery class in SharePoint 2010.  Your code from 2007 will still probably work, but I thought I would tell you about some of the changes.  This also sets the ground for a series of future posts that are coming about querying Enterprise Search.  We’ll talk about the federated query model using the QueryManager class as well as how to use FQL to do advanced queries in FAST.  Today though, we’ll stick to the KeywordQuery class.

In SharePoint 2007, we often got a reference to the KeywordQuery class by passing the URL of a site collection to the constructor.  We can still do that, however, I think the new best practice will be to pass a reference to the SearchServiceApplicationProxy.  The trick of course is getting that reference.  First, you need to determine the name of your Search Service Application.  For a typical Enterprise Search installation it is called Search Service Application.  However, it can be called anything depending on how you configured SharePoint.  For FAST, it might be called something like FAST Content SSA.  Go to Central Administration –> Service Applications and take a look.

The proxy will usually have the same name as the Service Application, so in my case here the name of my proxy is Search Service Application.  Now we just can’t get a reference to the SearchServiceApplicationProxy directly.  We have to go through the SearchQueryAndSiteSettingsServiceProxy class first.  According to the SDK, the function of going through this service is to ensure queries are load balanced.  Here is how you get a reference to the query and settings proxy.  It also assumes this code is executing on one of the servers in the farm.

Now that we have a reference to the settings proxy, we can get a reference to the SearchServiceApplicationProxy with the name of the proxy that we saw above.  Change the name to match whatever yours is called.

Now you can pass this proxy to the constructor of the KeywordQuery.

The rest is pretty much the same.  There is one new parameter that you may want to consider setting when you have multiple search providers (i.e.: FAST for documents and SharePoint Enterprise Search for People).  This parameter is ResultsProvider.  It’s an enum with a value of Default, FASTSearch, and SharePointSearch.  However, I believe you can use this to switch between FAST and SharePoint Search when you don’t specify the SearchServiceApplicationProxy (i.e.: you used the site collection URL).  So for example when you had FAST installed, if you wanted to query People, you might set it to SharePointSearch.  The examples I have seen so far leave this to default.   Here is what the rest looks like.

I can then use the data visualizer to see my results.  There are a few new managed properties that you get by default in the search results.  I’ll talk about these more when we look at using the QueryManager in an upcoming post.

There are a lot of new properties on the KeywordQuery class and I have only begun to explore them, but here are some of the ones I’ve looked at so far.  The first is EnableFQL.  This allows you to submit queries using FAST Query Language.  That’s a whole series of posts by itself.  Just know that you can submit FQL queries using the KeywordQuery class.  Two other interesting properties are EnableNicknames and EnablePhonetic.  This allows you to turn off the cool people search features that are so great at finding peoples names phonetically.  I’ll talk about more options with the KeywordQuery class in the future.  Anyhow, I hope this gets you started using it in SharePoint 2010.

Two weeks ago, I posted about how to create scopes and scope rules using PowerShell.  At the end of the post I mentioned that there was no rule type for creating scope rules by content type.  I knew there had to be a way to do this, so I created a scope rule using the UI and then I used the Get-SPEnterpriseSearchQueryScopeRule cmdlet to return all the scope rules on my content source.

From this result, the answer was immediately obvious to me.  ContentSource is a managed property and we already know how to create scope rules like that using the PropertyQuery rule type.  Remember how we can query by content source?  Let’s give it a try and see what happens.  I am going to create a new scope and have it use my BCS content source called Products.  Here is what the syntax looks like.

I’ll only mention the relevant parameters today.  If you are curious about the other ones, see the previous post where I go into detail on each one.  On the New-SPEnterpriseSearchQueryScopeRule method, we set the RuleType parameter to ContentSource.  Then, we set the PropertyValue to the name of the content source.  In my case, I set it to Products.  Executing the script above, I get the following result.

It appears to have worked, but let’s verify it in the UI.

Excellent.  The new scope was created with the content source rule and it has 43 items included.  Now, I can pretty much script all of my search settings in PowerShell.  Once you have your scopes created, if you want to create scope display groups be sure and check out the SharePoint PowerShell Community Toolkit.

I’m excited to see that we’ve already got some community contribution to the SharePoint PowerShell Community Toolkit.  Pete Cuttriss provided us with two new methods useful for working with scope display groups.  I also have the source code checked into the CodePlex TFS server now.  Source control is a good thing. :-)  Version 1.0.1.1 was released last night with the following new cmdlets.

• Get-SPEnterpriseSearchQueryScopeDisplayGroup – Returns a scope display group
• Remove-SPEnterpriseSearchQueryScopeDisplayGroup – Removes a scope display group
• Remove-SPEnterpriseSearchQueryScopeDisplayGroupScope – Removes a scope from a scope display group

If you try out any of our cmdlets, be sure sand let use know how they work for you.  As always we welcome your contributions.  Thanks and happy PowerShelling!

Release v1.0.1.1

Follow me on twitter.

This weekend, I got inspired to create what I think could be a great new CodePlex community Project.  Last week, I saw a post in the forums as well a comment on the blog asking if specific PowerShell commands existed in regards to Enterprise Search.  In both cases, I could not find existing commands so I thought why not create these commands.  Then I thought that I should go ahead and provide a framework for other people to implement cmdlets to add to the ones I created.  I took a look out on CodePlex to see if anything existed already and from what I could tell, there is nothing like this yet, so here we are. I am excited to announce the SharePoint PowerShell Community Toolkit on CodePlex.  SharePoint already has 500+ PowerShell commands, but the community always wants to add more. :-)  I thought this would be a great way to do it.  I started out with cmdlets to do a few things in Enterprise Search since I knew there were requests from people in the community out there already. 

Here are the commands I have implemented.

• New-SPEnterpriseSearchQueryScopeDisplayGroup – creates a new scope display group
• New-SPEnterpriseSearchQueryScopeDisplayGroupScope – adds a scope to an existing scope display group
• Reset-SPEnterpriseSearchIndex – resets an Enterprise Search content index
• Start-SPEnterpriseSearchQueryScopesCompilation – starts Enterprise Search scope compilation

Ok, so I know that is not that many commands yet, but how many did you expect me to write over the weekend? :-)  I have a ton of ideas though, so you can expect me to add more soon.  If you want to start using the commands now in your environment, download the distributable from CodePlex and use the install.bat file to copy the assembly to the GAC.  You can then register the snapin inside PowerShell.  More detailed installation instructions are on CodePlex.

If there is a PowerShell collaboration project out there already and somehow I missed it, let me know and I’ll be happy to add my contributions. :-)  So how can you help?  Create cmdlets and send them in.  I’m looking for people to help contribute to the project.  Not sure how to create a cmdlet yet?  No problem!  Check this post out to help you get started.  It’s easier than you think. :-)

SharePoint PowerShell Community Toolkit on CodePlex

I noticed this little issue after doing my last SharePoint 2010 upgrade.  The client was a fan of the Tag Cloud Web Part on his My Site and wanted to bring that functionality onto one of his team sites.  I figured this would be no problem, so we went to the Team Site, clicked Add New Web Part, looked under Social Collaboration only to find the Tag Cloud missing from the list.  In fact, Tag Cloud was not the only thing missing.  Contact Details, Note Board, and Organization Browser were also missing.  I checked the web part gallery and there was no sign of the Tag Cloud present.  I verified that the SharePoint Standard and Enterprise features are activated, but still no sign of it.

At this point, I decide to do some digging.  I wanted to find out which feature installed this web part.  After looking at a working SharePoint 2010 server, I determined that the web part comes from the file TagCloud.dwp (a version 2 web part schema on a brand new web part?  Really?).  I head to the 14 hive and search the features folder and determine that, this web part is deployed by the PortalLayouts and MySiteLayouts hidden features.  Then it all makes since (well sort of).  Since I upgraded this site from MOSS 2007, the new version of PortalLayouts has not activated on my site collection. 

At this point, I had a decision to make.  I could force the PortalLayouts feature to activate again or I could leave well enough alone and just manually upload the .dwp file into the Web Parts gallery.  I went with the latter so I grabbed TagCloud.dwp from 14\TEMPLATE\FEATURES\PortalLayouts\DWP and uploaded it to the web part gallery and the web part worked just fine.  I wasn’t sure what the other unintended results might be on the site collection had I force activated the PortalLayouts feature again so this seemed like the right decision.  If you are interested in the other web parts, you can upload contactwp.dwp, profilebrowser.dwp, and socialcomment.dwp to your gallery as well.  There may be other web parts missing too, but I think this is it.

I’ve been customizing rendering templates a lot lately using a wide variety of scripting technologies such as JavaScript, jQuery, and SPServices.  One requirement I had with all of these rendering templates was to call a custom web service to populate a drop down list.  This is relatively easy to do and there are plenty of resources out there on how to deploy web services with SharePoint.  I thought I would make a point today though to talk about this issue since I seem to forget about it every time. :-)  The issue is that you build and test your pages and they work great as an administrator. However, once an end user tries the page, the script on the page doesn’t work.  With any scripting issue, I bust out FireBug in Firefox and take a look at the page.  It shows the web service call which works great for the administrator but when the user tries it, the result is 403 forbidden.

Obviously, I need to grant permissions, but where?  Since the end user is on the SharePoint site, the identity that the JavaScript is calling with is that of the user.  This of course assumes, you are using NTLM or Kerberos and not doing forms authentication.  The place we have to grant permissions may not make sense to you at all, but the general consensus is that you grant Read permission to the Authenticated Users group on the bin folder of your web application.  This is the case at least when you are deploying your code to the bin folder (and using Code Access Security).  I don’t believe that web services with binaries deployed to the GAC should have this issue but I could be wrong.  Here is what the permissions look like on your bin folder.

Any how, I hope this helps should you run into this issue.  I don’t foresee any potential security issues by granting this group access to the bin folder, but if you think there are any associated risks, I would like to hear about them.

On a recent SharePoint 2010 migration I completed, this was one of the first questions I was asked.  After seeing versions of SharePoint all these months, it never occurred to me that the dropdown list wasn’t there any more until someone pointed it out.

Luckily, it’s easy enough to change this and I found some new settings along the way.  This box uses the built-in (not supposed to be customized) OSSSearchResults.aspx.  However, to use scopes, we need to create a Search Center.  To do this, we can go to Site Settings –> Search Settings to make some configuration changes.  This page was here in 2007, but I was pleased to see some new settings here in 2010 that I had not noticed yet.  We set the path to our search center just as we did before by specifying the Pages library of the search center.

There is a new setting to specify what the site collection search dropdown displays. Now there is a boat load of options for you to pick from.  The default is Do not show scopes dropdown, and default to contextual scope.  Here are all of the options you have to pick from.

The options are fairly self-explanatory with maybe the exception of default to ‘s’ URL parameter.  I’ve talked about this in previous posts, but what this does is tells the drop down list to default to the scope that was passed via the ‘s’ query string parameter.   This means that when you perform a search and are looking at the results page, the drop down will show the scope you just searched for.  As for default to contextual scope, this refers to the This Site or This List search options that you might be used to seeing from SharePoint 2007.  What option you pick here is largely up to you.  I usually end up going with Show and default to contextual scope.  However, I think Show, do not include contextual scopes, and default to ‘s’ URL parameter can also be a good choice at times.

We have one more setting that I want to mention on this page called Site Collection Search Results Page.  This allows you to override the URL used by contextual search.  In SharePoint 2007, all contextual searches went to the application page /_layouts/OSSSearchResults.aspx.  If you ever wanted to customize this page, you had to modify a built-in page which of course is unsupported.  Another side effect was that it changed the look of that page on every site on that server.  This new setting is great because it means we can change the contextual search page on a per site collection basis and simply upload a new application page without affecting the original page.

To configure what scopes show up in the dropdown on the home page, you need to modify the Search Dropdown scope display group.  Modify this group as you see fit and then you should see it on the home page.  In this case, I added my file share scope to be displayed on all of my pages.

This is what my pages look like with the dropdown.

So I can kind of see why they changed the default option for this.  The interface just looks cleaner without the dropdown box.  However, I think a lot of users are used to seeing it there, so this shows you how to add it back in the event that you ever need to.