Corey Roth [MVP] : SharePoint 2010, Enterprise Search, SharePoint, People Search

Are you getting the most out of SharePoint Search?

CoreyRoth — Mon, 03 Jan 2011 19:13:33 GMT

As we start out the new year, I thought I would start out with a post talking about one of my favorite SharePoint topics, Enterprise Search. Over the years, I’ve seen a variety of search configurations. Some companies have great implementations and other companies really struggle. For those of you that struggle with search, hopefully this post will get you thinking in the right direction to get search to meet your organization’s needs. We’re going to look at how you should configure your search environment, what types of content should you index, as well as how you can take advantage of search outside of SharePoint. This post won’t be the “silver bullet” in fixing search in your company, but maybe it will help you justify a search project or two.

Improving Search Results

When interviewing clients, the number one complaint about search is usually summed up by a statement like this: “I can’t find anything!”. Whether you are using SharePoint Foundation with Search Server Express, SharePoint Server 2010, or FAST Search for SharePoint, the quality of the results you receive from search is only as good as the input you give it. What I mean by this is that you can’t just install SharePoint, turn on search, and call it good. Out-of-the-box, SharePoint gives you a very functional search engine, but you need to do some work to really enable the full power it provides. This all starts with proper planning. If you don’t have a governance plan in place, you should probably start here. As you may know this living document specifies how people use your SharePoint environment and what they can and can not do. I’ve made it a habit to start including a topic in on search in the governance plans that I write as it is critical to the adoption and success of SharePoint at a company. Of course, I could write an entire series of blog posts talking about governance but there is already plenty of material out there for you to leverage if you are interested in learning more. If you don’t have a governance plan in place already, it really can be a big project, but it’s worth the time and effort.

Is your SharePoint environment really nothing more than a glorified file share? Unfortunately, a lot of times, companies implement SharePoint, but they don’t create an ECM plan which means no metadata gets assigned to documents. In reality, this is the primary cause of why users complain that they cannot find documents. Metadata, also known as data that describes data, provides additional information about a document. Out-of-the-box, some basic metadata already exists on your default Document content type such as title, author, and date. However, this only provides limited information for search. Search really starts to light up when you get the user to add additional information about a document. This could include things like Department, Division, Document Type (i.e.: Contract, Schematic, or Invoice), Invoice Number, Category. Why would you want to add this extra information to a document? So that your users can query and refine search results with the values from these columns. Let’s discuss some examples.

If we added a site column to our documents called Document Type, I could then execute a query to return all documents that were contracts. In this case, it could return documents that were truly contracts and not any document it happens to find with the word contract in it somewhere such as a PowerPoint presentation. Do you understand the difference? It allows us to return the exact type of documents the users desire. We could then take this a step further and say show me all contracts in the Gulf Coast division. Or maybe if we had a Contract Date column on our documents, we could say show me all contracts for the year 2009.

Let’s look at another example. Say you have a backend system with product information. It’s a complex database and each product has its own unique id. For each product you have multiple documents such as design schematics, technical documents, and marketing materials. In a system without metadata your best bet would be to search by the product name. You would probably get the results you want, but you would probably get a bunch of extra results too. However, if your users stored that Product Id on each document, you could then run queries and get all of the relevant documents for that specific document. Then after the user completed the query by Product Id, he or she could further refine those results by Document Type or Department perhaps.

Think about your SharePoint environment. Can you query search like this? If you can, that’s great! You can start looking at what else SharePoint Search can do for you. If you can’t, then you should start thinking about how you can get there. If you already have some metadata in place on your documents, but just can’t do queries like this, then you are really close! It’s just a matter of configuring search to use those site columns to create a great experience. We’ll talk about that more shortly. However, if you are like the bulk of many SharePoint environments, you have some work to do. I mentioned an ECM plan earlier. In this document, you plan out the many content types and site columns (metadata) that you will need for your documents. The user will see these additional fields after he or she uploads a new document. Depending on the size of your company, this can take quite some time to develop and implement. This usually consists of a series of meetings with the various departments across your organization to gather requirements on what each one needs. Many times, companies do this incrementally, just tackling a few departments at a time and then moving onto the next. Once you finish gathering requirements, you implement new site columns and content types in SharePoint. It may seem like a long process, but it will pay off in the end.

“But, wait! My users will complain if they have to fill out additional information every time they upload a new document.” Probably, true. However, do they complain more that they can’t find the documents they are looking for? Justify the additional effort when uploading with the amount of effort your users will save when trying to find the documents later. If your users can find the document they need with just 2 – 3 clicks instead of 20, you are already saving time. You can also make use of SharePoint 2010 features such as the Content Organizer and Default Column Values which can help prepopulate values in the form for the user. For more advanced scenarios or if you are still using SharePoint 2007, you can write event receivers which can automatically set values for users as well.

Along with your own custom metadata fields, I also recommend adding Enterprise Keywords and the Ratings field to your document libraries if you are using SharePoint 2010. The Enterprise Keywords field allows your users to “tag” documents when they upload them. The Managed Metadata service keeps track of commonly used tags and prompts the user with existing terms as he or she types it in. These tags also populate the Tag Cloud Web Part and can refine search results. The Ratings field allows users to rate documents with 0 – 5 stars. As more people rate documents, it allows other users to quickly see which documents people find more valuable.

When you have a Governance and ECM plan in place, you can move on to developing a plan for Search. Your Search plan might overlap some with your Governance plan, but will also include which site columns you want to use with Enterprise Search. When Search crawls your content for the first time, it creates something called a Crawled Property which maps to your site columns. In the case of SharePoint content, your site column will get prefixed with ows_ and the spaces in your column name will get encoded. For example, a site column called, Product Id, would become ows_Product_x0020_Id. In my article on Naming Conventions, I mention this is one reason why I am not a fan of spaces in the names of site columns. Whether you want to query by, refine by, or see your site column in search results, you will need to map it to what SharePoint calls a Managed Property. You can map one or more crawled properties to a given managed property to make it available to search. In your Search Plan, you should document all of the site columns that you plan to map to managed properties so that you can go back and reference it in the future or recreate the mappings in another environment. Typically, most people create their managed properties in the Service Application UI. However, you can also create them using PowerShell. When you finish creating your managed properties, you must crawl again to make them active.

With your managed properties in place, you then need to decide how you want to use them. Typically this involves customizing the web parts inside the Search Center. You can do many customizations easily without having to write any code. By editing the results.aspx page, you can customize how search results look as well as your refinement options. If you want to display additional metadata in your search results such as Document Type or Invoice Number, you can edit the XSLT inside the CoreResultsWebPart. If you want to let users refine search results based upon your own metadata (i.e.: department or document type), then take a look at modifying the RefinementWebPart. Lastly, many times users want to customize the query before they ever execute it. You can customize the existing Advanced Search Web Part, but it’s quite ugly and not very flexible. If you want to write a little bit of code, then you can make your own user control or Silverlight application to do the same thing. Read up on the customizations that you can do and then document what you plan to implement in your Search plan.

Indexing Content outside of SharePoint

In my experience, only a small percentage of organizations know that Enterprise Search can index content outside of SharePoint. Some know that you can index file shares. However, did you know that SharePoint can index data from custom applications using SQL or WCF services? It can also crawl your public facing web site, lotus notes, and Exchange public folders. If that’s not enough, you can even write custom code to index whatever system you want. Do you have an old mainframe or AS/400 system that has data you want to show in SharePoint? Well if you can write .NET code to get to the system, you can expose its data in SharePoint through the use of Business Connectivity Services. I’ve met some CIOs, that when learning about this feature, want to index every system in their organization. I think that’s great, but you need to plan appropriately or you will overwhelm your users with search results. The last thing your user wants to see is some file that is 10 years old on your file share when they are looking for this year’s holiday schedule.

One way to allow a user to restrict a query to a subset of the index is with the use of Scopes. Scopes simply allow you to create a set of rules that include and exclude content. Out-of-the-box, SharePoint comes with two scopes All Sites and People. The All Sites scope returns everything in the content index (minus People). When you start adding content sources to index file shares or custom applications, you will want to create a scope or two that allows the user to restrict results to just those areas. For example, this would allow the user to query just SharePoint content or just File Share content. You can also create more advanced scopes using managed properties such as creating a scope that returned only Invoices. Once you create scopes, you can display them in the Search Center as well as in a drop down list on your master page. This gives the user more flexibility on how they search.

In large organizations, finding the right person to ask a question can be challenging. In SharePoint 2007, we gained People Search which allowed us to index information from Active Directory or an ERP system (through the BDC) about the people in an organization. SharePoint 2010 extends this by adding a wealth of social features. It also gives us phonetic searching which makes it easier for users to find people with difficult names to spell. Although, People Search can infer a lot of information from Active Directory and other source systems, it is only as good as the information you provide it. People also have metadata (in their profile) and you should work to have it as complete as possible. Prepopulate what you can for the user with User Profile Synchronization then encourage the people in your organization to complete the rest of their profiles manually especially the Ask Me About section. This section allows other users to know what that user is responsible for (I.e.: Accounts Payable, Benefits, or Employee Recruiting).

I also recommend making sure you have the Manager field set in People Search. By default, People Search gets this from the Manager property in Active Directory. Setting this allows SharePoint to determine your organization chart. If your organization already populates this field, then you can take advantage of the organization chart feature in SharePoint right away.

Before moving on, I recommend taking a look at your existing web applications in your organization along with any ones currently in development. SharePoint can expose the data of Many web applications whether developed in ASP.NET or another language. Before SharePoint, creating a search experience for these applications usually involved a bit of code with the use of things such as T-SQL LIKE statements. Using Business Connectivity Services (BCS), SharePoint can actually crawl your custom application by directly accessing the database tables, through WCF services, or the new BDC Entity Model. Many times you can do this indexing without having to make any changes to your existing application. By using SharePoint Designer 2010, you can connect to your SQL database or WCF services and choose the entities you want to bring into SharePoint as External Content Types. With the External Content Type, you can create a Content Source to index that application.

At this point, you might think “How do the links from the results page go to my custom application?”. This depends on your application. In your External Content Type, you can define a default action which SharePoint uses on the results page when the user clicks on a link. SharePoint can use the data it indexes to help you build a URL (think String.Format()). For example, let’s assume we are indexing a custom ASP.NET application which has product information. The External Content Type contains fields such as Name, ProductId, Description, and Price which comes from a database table. Your custom application is hosted on a server called server and it has a page which allowed the user to edit each product named ProductEdit.aspx. That page takes a query string parameter called Id representing the ProductId in the database table. SharePoint can construct this URL given the information it has to link the user directly to that product editing page. See the picture below for an example.

Federating Search

Sometimes you may have too much content outside of SharePoint or maybe you have a system that already has a capable search engine (i.e.: an ERP or another document management system). This calls for federated search. As another example, you might use a federated search to display results from your public facing web site or from a public search engine like Bing. You don’t want to completely index their data, but you would still like to see results from those external systems when searching from SharePoint. The Federated Search feature allows you to display results from any search engine supporting the OpenSearch 1.1 protocol alongside your local search results. If you are not familiar with OpenSearch, the results come back as an RSS feed. Even if your external system doesn’t support OpenSearch, you can write some code to refactor the results as RSS and integrate them easily into SharePoint. Here is an example, where the federated results come from DotNetMafia.com on the right side of the screen.

Exposing SharePoint Search to other applications

We can index data in other applications, but maybe you don’t want to make the user go to SharePoint to see the search results. Luckily SharePoint has an extensive API that allows you to query SharePoint remotely using web services. Calling the Search Web Service is pretty simple. Effectively you construct an XML input document and then you can call the Query method to return results as XML or the QueryEx method to return results as an ADO.NET DataSet. For a complete example, see my post on how to use the Search Web Service. This also is your stepping stone for developing advanced search driven applications. For example, you could call this web service from Silverlight to present the results using a rich UI.

Chances are the army of ASP.NET developers that you employ don’t know that SharePoint can index their applications. Consider this as an option the next time you build a custom application. There is no point in reinventing the wheel when SharePoint can do the heavy lifting for you. It might even shave off a few hours of your next project. Plus, it allows you to search from multiple sources at the same time. Just think you could get results from SharePoint, a file share, and multiple custom applications in one results screen. With the built-in refinement features, you can drill down into the results you want.

Other Considerations

You may be thinking “All of this is great, but I bet I need FAST Search for SharePoint to make it all happen.” That’s not actually the case. Yes, FAST can do a lot of powerful things and allows you to make some great customizations to your search experience. However, most of the things I have talked about today have been around since SharePoint 2007. In fact, with the exception of People Search, you can implement everything in this post today using SharePoint Foundation with Search Server Express. If you use SPF today but don’t use Search Server Express, I highly recommend it. It adds a lot of functionality to out-of-the-box search experience in SPF.

Don’t be afraid to ask for help. This posts covers a lot and sometimes it is nice to have a little guidance along the way when you try to implement. There are many great community resources out there talking about Search including blogs, forums (2007), twitter, and SharePoint Saturdays. I won’t do a shameless plug for my consulting company today, but don’t be afraid to reach out to a firm with some SharePoint consultants to help you develop things like a Governance, ECM, or Search plan. Getting your plan correct before you start will help keep you from having to deal with problems later.

Follow me on twitter.

How to set up People Search in SharePoint 2010

CoreyRoth — Mon, 29 Mar 2010 18:59:25 GMT

Since I’m talking about Enterprise Search tonight at OKCSUG, I thought I would write up a quick post on the steps you need to follow in order to get People Search working in SharePoint 2010. The critical piece of course is getting the User Profile Synchronizations service working. There are plenty of posts about it, so I’m not going to go into the details of how to get that part of it work. SharePointDevWiki and MSDN do a pretty good job of getting you started. The main thing is you want your Forefront Services started and looking something like this.

In Central Administration, on Services on Server, you want your User Profile services to show that they are started.

If your server doesn’t look like what you have above, then you probably have a problem. With Beta 2, if you have made any changes to these services or so much as looked at them, you will need to reinstall SharePoint. Don’t bother trying to get it to work, because it will never happen. Just ask twitter, it’s true. Don’t waste your time, just reinstall now.

If everything is running correctly, you need to set up a profile synchronization connection. This is somewhat similar to the process you used in MOSS 2007, but they updated it a bit to make it a bit more user friendly. For example, you don’t have to specify LDAP queries to get the objects you want any more. Just specify an account on the domain, and then pick the object you want to import using a tree viewer. The SharePointDevWiki article has some good screenshots on setting this up for the beta. However, Harbar by far has the most definitive guide on getting User Profile sync working (follow it exactly). If you have any issues with JavaScript errors on the screens, hit F12 and use IE7 mode and it will probably work. Once you have configured your connection, start a new synchronization job and wait for it to finish.

At this point you have user profiles, but you can’t search them yet. You’re dying to see all of the new features in People Search, so we have to do a crawl to make that happen. At this point, it’s much like MOSS 2007. In fact it uses the same protocol handler, sps3. Hopefully, you know where they moved Search Administration to now that they have gotten rid of the SSP. If not, it’s under Central Administration –>Service Applications –> Search Service Application –> Manage button. Once you are here, go to Content Sources. You should see one content source called Local SharePoint Sites. Depending on how you installed, this may be correct or it may not be. For start addresses, you should see something like http://server-name and sps3://server-name. Here is what mine looks like.

Sometimes, the sps3 address won’t be there, so you need to add it. You can add it to this content source or create another one. It really doesn’t matter. Before we crawl, we need to check some new settings to make sure our crawl account has the necessary permissions. Sometimes, SharePoint will take care of some of this for you other times it won’t. The first thing you need to do is go to Central Administration –> Security –> Specify web application user policy. Here, you need to make sure that your default content access account has Full Read permission. I’ll take this as yet another time to remind you that your crawl account should not have administrator permissions. It should look something like this.

The next thing we need to do is grant this account permissions to the User Profile Service Application. To do that, go to the Service Applications page, click on User Profile Service Application (click next to it to highlight it, don’t click on the link). When its highlighted, click on the Administrators button at the top. This gives you a window that looks like this.

If your default content access account (crawl account) is not listed, add it and then select the checkbox Retrieve People Data for Search Crawlers. If you don’t complete this step, you will likely get an access denied error message when you crawl.

Once it’s created, you need to do a full crawl. If you haven’t created an Enterprise Search Center yet, now is a good time to do so. Once you create it, click on the People tab and search for someone you know. That’s all there really is to it. If you have User Profile sync working, then the rest is easy.

This post has been updated for SharePoint 2010 RTM.

Dude! Where’s my Search Center?

CoreyRoth — Thu, 11 Mar 2010 19:24:23 GMT

You might have seen my last post on People Search and were so excited to try it out only to find that you only have a Basic Search Center template available or the search center you have already created doesn’t have a People tab. By default, on the New Site menu, you are only able to create a Basic Search Center. This site template is functional but it doesn’t give you the ability to do a people search. This effectively corresponds to the Search Center site template in MOSS 2007.

There are two new search center templates available though Enterprise Search Center and FAST Search Center. However, to see those, you need to activate the Enterprise Features on your site collection.

Once you activate them, you’ll see the two new site templates available on the New Site menu. Don’t you just love that new Silverlight app to pick new sites?

The Enterprise Search Center is the one you want (unless you actually got FAST working). This site template corresponds to the old publishing site template Search Center with Tabs back in MOSS 2007. You also had to do the same thing in the last version of SharePoint as well.

A quick look at phonetic People Search in SharePoint 2010

CoreyRoth — Thu, 11 Mar 2010 17:38:25 GMT

Maybe you hadn’t noticed, but talking about Search is my favorite topic in SharePoint (with ECM closely following it). :) This is why I am so excited about the new features we are seeing in Enterprise Search. Wildcards and built in refinement were enough to bring a tear to my eye, but People Search is what I think a lot of people will be impressed by. You see People Search is the gateway to many of the new social aspects in SharePoint 2010. Your old stuffy CIOs are going to absolutely hate it (since they were probably the ones that blocked twitter and facebook at your organization). The new hip CIOs that find value in social media are going to totally embrace it and organizational efficiency and the sense of community is really going to go up. Not to tangent on the whole social thing though, let’s just take a look today at how it’s easier to find people in SharePoint 2010.

Let’s consider my great new fictitious company. It’s not a big company but there are a number of employees. Ok, so my fake company isn’t as complete as the one we have seen at the demos at SPC or anything, but hopefully you get the point. Consider the following users stored in Active Directory.

To work with People Search, you must first have successfully configured user profile synchronization, set up a connection, and then done a full crawl on your Local SharePoint Sites content source. If your user profile synchronization service isn’t working, do yourself a favor and just reinstall because you will never get it to work. The content source still uses the sps3 protocol to crawl user profiles. If no people are returned when you search make sure you have a content source setup using that protocol handler. I’ll probably write another post on how to set this up pretty soon.

To get started with People Search we start with an Enterprise Search center and click on the people link which gives us a plain search box like the one below.

Let’s say that I need to lookup the phone number for my IT Director, Michael Adams. I can of course type his full name in like this and get a result (this is what we basically had in SharePoint 2007).

We get a result as expected, but you know, I really don’t call him Michael, I call him Mike. Shouldn’t that return me a result too?

Absolutely! In SharePoint 2007, if you asked for that people would look at you like you wanted the impossible. Now it works beautifully. Clicking on the user’s name takes you to the profile page where all of the cool social stuff happens. I could do several posts on it alone, but here is a quick look at what it looks like. It allows you to see the org chart along with recent activities along with a corporate style facebook. Very cool.

What else can this thing do though? Say we want to look up our Team Lead, Richard Jackson. He doesn’t go by Richard though. He goes by *** (no jokes please). Will that yield a result?

How cool is that. The search engine seems to understand nicknames. So this really isn’t necessarily happening phonetically, it just has a nice thesaurus in it now, but at SPC, we did see search examples where things were spelled phonetically and it worked. I haven’t had a ton of luck getting that to work. I also saw it work with non-English names. Hopefully I can demo that stuff too pretty soon, but I still thought these things were worth showing off.

Let’s do one more example though. Let’s say I know the last name of the person I need to contact is Williams. We’ll pretend I am lazy and I only type in Willi in the search box. Will that give me a result?

The answer this time is actually no. People Search does support wildcards now, but it is not on by default. Change the query to Willi* and we’re in business.

It returns matches in both the first and last name. Very cool. I’d love to be able to inherit from the PeopleCoreResultsWebPart to add an option to allow wildcards all the time, but Harneet Sidhana [MSFT] does not think we should be able to. Apparently the web part is perfect and no one would possibly ever want to change it. Ok, so the web part is pretty cool, but I would like to make changes and that’s just not going to be possible. :-(

Stepping aside from that tangent, I have to mention the Organization Browser. It’s a cool use of Silverlight that makes it very easily to browser the organization.

This picture doesn’t do it justice. It certainly would be a lot cooler if I had pictures uploaded for all of my users. From all of this, hopefully you can see that it should be a lot easier for end users to find people in your organization. I’m really excited about these new features.