Corey Roth [MVP]

This is something that I have been trying to figure out for quite some time.  I’ve seen numerous people ask in the forums and no one seemed to have a conclusive answer (at least the last time I checked).  The issue is simple.  You want to index a regular non-SharePoint web site.  Usually, it’s your company’s public-facing web site.  That site has common navigation on every page with terms such as Contact, Locations, Privacy Policy, About, etc. that you don’t want to be indexed.  If it is indexed, every time a user types in contact, they end up having every page on the site returned in the search results.  When I was at FAST University, I had a chance to ask Leonardo Souza about this issue and he told me the secret.  Let’s take a look at my example site so you can see what I mean.

Here’s the home page.  As you can see, I spent hours working on the branding for it. :)  The Contact Us and Privacy Policy links are considered the navigation and are repeated on each page.

The Contact Us page looks similar with the same navigation.

Lastly, the Privacy Policy page has the same navigation as well.

We want to exclude the contact us and privacy policy links in the navigation from our search results.  How do we do that?  It’s pretty simple actually.  Just put the content that you do not want indexed in a div tag with a class of noindex.   Let’s look at the complete HTML of the home page.

You can see that the Contact Us and Privacy Policy links are inside <div class=”noindex”>.  You might have noticed that the body of the page also has links to these two pages.  I had to include these so that those pages would get indexed.  Since the common navigation is excluded there was no way for the crawler to follow those links.  This is something you will  want toconsider when you are designing master pages because you will need to have at least one link to each page on the site somewhere.

Since I learned about this in the context of FAST Search for SharePoint, I decided to look at it first.  The first thing I will do is show you the results of the entire content source.  That way you will believe me that all of the pages are in the index. :)  I do this with the ContentSource keyword as I have mentioned in my handy keywords post.

The search results shows the four pages from site.  Now let’s verify that the noindex class worked.  Searching for the word contact yields a single result.

Searching for privacy policy also yields a single result.

The noindex class works great with FAST Search for SharePoint.  At this point though, I wondered would this also work with Enterprise Search in SharePoint 2010?  I decided to give it a try and sure enough it works there too.

Will this also work in SharePoint 2007?  I haven’t had time to try it yet.  If you have tried it before, please leave a comment and let us know.  Maybe you already knew about this technique, but I think there are plenty of people who don’t so I hope this post helps.  I highly recommend making use of the noindex attribute any time you want to index a non-SharePoint site, such as your public-facing company web site.  By excluding redundant sections of the page, you make your search results much more usable.

One thing I find annoying after attaching a SharePoint 2007 content database is that several of the new cool SharePoint 2010 web parts are simply not present on your upgraded site.  This includes the new social web parts such as the Tag Cloud, Organization Browser, and Note Board.  On an upgraded site, you may find that your Social Collaboration category looks like this:

I have talked about this in the past, but I’ve changed how I resolve the issue so I thought I would share a quick update.  Before we begin, it’s a good idea to understand what has happened.   In SharePoint 2007, you had site templates and they likely activated a feature called PortalLayouts.  This feature still exists in SharePoint 2010, but it has changed.  Specifically the file that we are interested in, webpartdwpfiles.xml, activates different web parts now.

Excuse me for the large display of XML here, but I think seeing it will make more sense.  In SharePoint 2007, webpartdwpfiles.xml contained the following XML.  Each File element installs a specific web part into the web parts gallery.

Here is what the file looks like in SharePoint 2010.

If you look at some of the differences, you’ll notice that several web parts have been added in 2010, but several have also been removed.  Web Parts that have been added include:

• Tag Cloud (TagCloud.dwp)
• Note Board (SocialComment.dwp)
• Organization Browser (ProfileBrowser.dwp)
• Dual Chinese Search (DualChineseSearch.dwp)
• Refinement Panel (Refinement.dwp)
• People Refinement Panel (PeopleRefinement.dwp)

Web Parts that have been removed in SharePoint 2010 include:

• This Week in Pictures (ThisWeekInPictures.dwp)
• Top Sites (TopSitesWebPart.webpart)
• I need to… (TaskAndTools.webpart)

When you do an upgrade, you’ll still have your removed web parts in the gallery.  However, if you are on a new installation and want one of those old web parts, you’ll have to do a little work.  From what I have read, you can export the .webpart or .dwp file of the web part and upload it to the solution gallery manually on your new server. It appears the underlying code is still present in the assemblies, just the web parts definitions have been removed from the web part gallery.

However, we’re interested in adding the new SharePoint 2010 web parts to the site collection’s web part gallery.  You can manually upload them if you so choose.  However, I’m not really a fan of that solution, but it will work in a pinch.  Instead my solution now is to activate the PortalLayouts feature again.  The feature is hidden, so your best bet to do this is with PowerShell.  We do this with the Enable-SPFeature cmdlet.  Since the feature was technically already activated in SharePoint 2007, we have to use the –force parameter.  Here is the complete command for a root site located at http://sp2010. 

Enable-SPFeature –Identity PortalLayouts –url http://sp2010 –force

If the command worked successfully, you should receive no errors and another command prompt.  At this point, you can add your new web parts to an existing page.  You should see three new ones in the Social section as shown.

I have not encountered any issues by reactivating this feature yet.  It works for me and hopefully it works for you too.  If you are missing web parts, try it out and see how it works for you.

For some reason, I always forget to set permissions on the database in SQL Server when I am doing a database attach upgrade in SharePoint 2010.  I figure if I forget it a lot maybe other people do, so I thought I would post a quick tip on it.  Before even trying Test-SPContentDatabase or Mount-SPContentDatabase, make sure you open SQL Server and set the right permissions.  You need to edit the permissions on the account that you are running the PowerShell commands with and make this account a db_owner on the content database.  If you don’t, you will get an error stating that you don’t have the required permissions and you will look silly in front of your client.  :-)  It’s not a big deal to resolve, but I figure if I write this maybe I won’t forget to set it next time I do an upgrade.

If you have ever installed FAST Search, you know memory is a commodity.  When you are looking at an environment with Active Directory, SQL Server, SharePoint Server, and FAST installed on top of it, your virtual server is consuming a lot of memory.  I picked up this quick tip from FAST University that is recommended to release a little bit of memory.  It’s no silver bullet and you might not ever even notice a difference, but I thought I would share it any way.

By default, a FAST Search server has four Document Processor services running.  You don’t need four in your development environment, so the recommendation is to take this down to one.  You can verify how many you have first by tying the following at a command prompt.

nctrl status

Here you will see all of the FAST processes.  Notice that there are four Document Processor modules listed.

We only want one Document Processor, so we simply issue the command below to remove a document processor.

nctrl remove procserver

Issue it three times like you see below to remove three document processors.

When you check nctrl status again, you will see only one Document Processor listed now.

How much memory might you reclaim?  Well not a whole lot but every little bit helps.  I think it will help some on your CPU utilization too when you are indexing (although it may take longer with only one processor running).  Here is what my utilization looked like before and after.

Anyhow, give it a try on your own VM if you want to see if it helps.  Why is the nctrl command say procserver?  Document Processors were called Processor Servers in FAST ESB from my understanding.

In spite of the weather, I still had some people show up to my updated SharePoint Search talk, Getting the most out of SharePoint Search.  This talk relates closely to my recent blog post Are you getting the most out of SharePoint Search.  The talk focused on understanding the basics of search and how you can improve the search experience for your users.  I reiterated the theme that Search is only as good as the information you provide it.  I also gave the attendees of a choice of seeing the demo with Search Server Express, SharePoint Server, or FAST Search for SharePoint.  They ended up choosing FAST and I spent a good amount of time showing off the new features.  This is the first time I have dared giving a demo using a FAST VM and it worked fairly well.  I only had queries time out  and fail on me twice. :)  I also showed some of the basics of using FAST Query Language (FQL) with a code demo showing how you can dynamically affect ranking.  I hope everyone enjoyed the talk.  For those that didn’t attend because of the snow you really missed out.  We didn’t even have any snow accumulation the entire night. :)  You can find my slides from the talk on SlideShare.

Getting the most out of SharePoint Search

I was looking at the analytics of the site the other day and I noticed that my post, Adding and Deploying Solutions with PowerShell, currently has the most traffic on the site.  I had always intended to write a follow up post on this, so here it is.  Today I’m going to talk about how to enable and disable features using PowerShell.  It’s great that you know how to deploy solutions now with PowerShell, but now you want to activate your features.

In SharePoint 2007 to activate a feature from the command line you might have used a statement like the one below.  I’ll use the Reporting feature on my server named sp2010 as an example.  The value specified in the name parameter refers to the actual folder name of the feature that is in your SharePoint Root folder (14 hive).

stsadm.exe –o activatefeature –name Reporting –url http://sp2010

To get started with PowerShell, run the SharePoint 2010 Management Console located in your Microsoft SharePoint 2010 Products folder on your start menu.  This automatically loads the Microsoft.SharePoint.PowerShell snapin so that we can execute SharePoint commands.    You might have noticed earlier that I said we will talk about enabling and disabling features.  The words enable and disable are key here because PowerShell uses those verbs to activate and deactivate features.  To issue the same command above but with PowerShell, we use the Enable-SPFeature command.  The rest of the syntax is pretty similar as you can see below.  Just use –Identity to specify the name of the feature.  You can even pass the –Force command like you could with stsadm.exe.

Enable-SPFeature –Identity Reporting –url http://sp2010

If your command worked successfully, you will see nothing and just get a blank prompt back.

If you would like to get some feedback from the command, you can have it return the SPFeature object back to you by specifying the –PassThru parameter.

We can confirm that the feature did in fact activate successfully using the SharePoint UI.

Can we confirm that the feature is enabled with PowerShell?  Yes, we can by using Get-SPFeature.  Now this cmdlet behaves differently depending upon the parameters passed through it.  For example, if you type the following, it lists every feature on the SharePoint farm along with its Id and Scope.

Get-SPFeature

However, if you want to know which features are enabled, you can pass it a URL for a given scope (i.e.: –Web, –Site, –WebApplication, and –Farm).  So to get a list of all Site Collection scoped features, we would use the –Site parameter.

Get-SPFeature –Site http://sp2010

You can still specify a specific feature or even use Where-Object to query the list as well.

Get-SPFeature –Identity Reporting –Site http://sp2010

If you get something like you see in the screenshot above, you know the feature has been activated.  When the feature hasn’t been activated, you will get a lovely red error message.

The Enable-SPFeature command can also take a Pipe Bind from Get-SPFeature.  This means that you could even activate multiple features at once this way.  I’ll include an example of this in the future.

At some point, you will want to disable that feature you activated.  As you probably guessed, the cmdlet you want is Disable-SPFeature.  You use the same parameters that you use with Enable-SPFeature.  When you execute the command, it prompts for confirmation as you can see in the screenshot.

Disable-SPFeature –Identity Reporting –url http://sp2010

If you want to disable confirmation, you can pass the –confirm:$false parameter.

Disable-SPFeature –Identity Reporting –url http://sp2010 –Confirm:$false

Setting the confirm parameter eliminates the confirmation prompt and you can use it with other PowerShell cmdlets as well.  That is all that is involved in activating and deactivating features with PowerShell.  It’s pretty easy.   Hopefully, you can incorporate these commands in your deployment scripts.

As we start out the new year, I thought I would start out with a post talking about one of my favorite SharePoint topics, Enterprise Search.  Over the years, I’ve seen a variety of search configurations.  Some companies have great implementations and other companies really struggle.  For those of you that struggle with search, hopefully this post will get you thinking in the right direction to get search to meet your organization’s needs.  We’re going to look at how you should configure your search environment, what types of content should you index, as well as how you can take advantage of search outside of SharePoint.  This post won’t be the “silver bullet” in fixing search in your company, but maybe it will help you justify a search project or two.

Improving Search Results

When interviewing clients, the number one complaint about search is usually summed up by a statement like this: “I can’t find anything!”.  Whether you are using SharePoint Foundation with Search Server Express, SharePoint Server 2010, or FAST Search for SharePoint, the quality of the results you receive from search is only as good as the input you give it.  What I mean by this is that you can’t just install SharePoint, turn on search, and call it good.  Out-of-the-box, SharePoint gives you a very functional search engine, but you need to do some work to really enable the full power it provides.  This all starts with proper planning.  If you don’t have a governance plan in place, you should probably start here.  As you may know this living document specifies how people use your SharePoint environment and what they can and can not do.  I’ve made it a habit to start including a topic in on search in the governance plans that I write as it is critical to the adoption and success of SharePoint at a company.  Of course, I could write an entire series of blog posts talking about governance but there is already plenty of material out there for you to leverage if you are interested in learning more.  If you don’t have a governance plan in place already, it really can be a big project, but it’s worth the time and effort.

Is your SharePoint environment really nothing more than a glorified file share?  Unfortunately, a lot of times, companies implement SharePoint, but they don’t create an ECM plan which means no metadata gets assigned to documents.  In reality, this is the primary cause of why users complain that they cannot find documents.  Metadata, also known as data that describes data, provides additional information about a document.  Out-of-the-box, some basic metadata already exists on your default Document content type such as title, author, and date.  However, this only provides limited information for search.  Search really starts to light up when you get the user to add additional information about a document.  This could include things like Department, Division, Document Type (i.e.: Contract, Schematic, or Invoice), Invoice Number, Category.  Why would you want to add this extra information to a document?  So that your users can query and refine search results with the values from these columns.   Let’s discuss some examples.

If we added a site column to our documents called Document Type, I could then execute a query to return all documents that were contracts.  In this case, it could return documents that were truly contracts and not any document it happens to find with the word contract in it somewhere such as a PowerPoint presentation.  Do you understand the difference?  It allows us to return the exact type of documents the users desire.  We could then take this a step further and say show me all contracts in the Gulf Coast division.  Or maybe if we had a Contract Date column on our documents, we could say show me all contracts for the year 2009. 

Let’s look at another example.  Say you have a backend system with product information.  It’s a complex database and each product has its own unique id.  For each product you have multiple documents such as design schematics, technical documents, and marketing materials.  In a system without metadata your best bet would be to search by the product name.  You would probably get the results you want, but you would probably get a bunch of extra results too.  However, if your users stored that Product Id on each document, you could then run queries and get all of the relevant documents for that specific document.  Then after the user completed the query by Product Id, he or she could further refine those results by Document Type or Department perhaps.

Think about your SharePoint environment.  Can you query search like this?  If you can, that’s great!  You can start looking at what else SharePoint Search can do for you.  If you can’t, then you should start thinking about how you can get there.  If you already have some metadata in place on your documents, but just can’t do queries like this, then you are really close!  It’s just a matter of configuring search to use those site columns to create a great experience.  We’ll talk about that more shortly.  However, if you are like the bulk of many SharePoint environments, you have some work to do.  I mentioned an ECM plan earlier.  In this document, you plan out the many content types and site columns (metadata) that you will need for your documents.  The user will see these additional fields after he or she uploads a new document.  Depending on the size of your company, this can take quite some time to develop and implement.  This usually consists of a series of meetings with the various departments across your organization to gather requirements on what each one needs.  Many times, companies do this incrementally, just tackling a few departments at a time and then moving onto the next.  Once you finish gathering requirements, you implement new site columns and content types in SharePoint.  It may seem like a long process, but it will pay off in the end.

“But, wait!  My users will complain if they have to fill out additional information every time they upload a new document.”  Probably, true.  However, do they complain more that they can’t find the documents they are looking for?  Justify the additional effort when uploading with the amount of effort your users will save when trying to find the documents later.  If your users can find the document they need with just 2 – 3 clicks instead of 20, you are already saving time.  You can also make use of SharePoint 2010 features such as the Content Organizer and Default Column Values which can help prepopulate values in the form for the user.  For more advanced scenarios or if you are still using SharePoint 2007, you can write event receivers which can automatically set values for users as well.

Along with your own custom metadata fields, I also recommend adding Enterprise Keywords and the Ratings field to your document libraries if you are using SharePoint 2010.  The Enterprise Keywords field allows your users to “tag” documents when they upload them.  The Managed Metadata service keeps track of commonly used tags and prompts the user with existing terms as he or she types it in.  These tags also populate the Tag Cloud Web Part and can refine search results.  The Ratings field allows users to rate documents with 0 – 5 stars.  As more people rate documents, it allows other users to quickly see which documents people find more valuable.

When you have a Governance and ECM plan in place, you can move on to developing a plan for Search.  Your Search plan might overlap some with your Governance plan, but will also include which site columns you want to use with Enterprise Search.  When Search crawls your content for the first time, it creates something called a Crawled Property which maps to your site columns.  In the case of SharePoint content, your site column will get prefixed with ows_ and the spaces in your column name will get encoded.  For example, a site column called, Product Id, would become ows_Product_x0020_Id.  In my article on Naming Conventions, I mention this is one reason why I am not a fan of spaces in the names of site columns.  Whether you want to query by, refine by, or see your site column in search results, you will need to map it to what SharePoint calls a Managed Property.  You can map one or more crawled properties to a given managed property to make it available to search.  In your Search Plan, you should document all of the site columns that you plan to map to managed properties so that you can go back and reference it in the future or recreate the mappings in another environment.  Typically, most people create their managed properties in the Service Application UI.  However, you can also create them using PowerShell.  When you finish creating your managed properties, you must crawl again to make them active.

With your managed properties in place, you then need to decide how you want to use them.  Typically this involves customizing the web parts inside the Search Center.  You can do many customizations easily without having to write any code.  By editing the results.aspx page, you can customize how search results look as well as your refinement options.  If you want to display additional metadata in your search results such as Document Type or Invoice Number, you can edit the XSLT inside the CoreResultsWebPart.  If you want to let users refine search results based upon your own metadata (i.e.: department or document type), then take a look at modifying the RefinementWebPart.  Lastly, many times users want to customize the query before they ever execute it.  You can customize the existing Advanced Search Web Part, but it’s quite ugly and not very flexible.  If you want to write a little bit of code, then you can make your own user control or Silverlight application to do the same thing.  Read up on the customizations that you can do and then document what you plan to implement in your Search plan.

Indexing Content outside of SharePoint

In my experience, only a small percentage of organizations know that Enterprise Search can index content outside of SharePoint.  Some know that you can index file shares.  However, did you know that SharePoint can index data from custom applications using SQL or WCF services?  It can also crawl your public facing web site, lotus notes, and Exchange public folders.  If that’s not enough, you can even write custom code to index whatever system you want.  Do you have an old mainframe or AS/400 system that has data you want to show in SharePoint?  Well if you can write .NET code to get to the system, you can expose its data in SharePoint through the use of Business Connectivity Services.  I’ve met some CIOs, that when learning about this feature, want to index every system in their organization.  I think that’s great, but you need to plan appropriately or you will overwhelm your users with search results.  The last thing your user wants to see is some file that is 10 years old on your file share when they are looking for this year’s holiday schedule.

One way to allow a user to restrict a query to a subset of the index is with the use of Scopes.  Scopes simply allow you to create a set of rules that include and exclude content.  Out-of-the-box, SharePoint comes with two scopes All Sites and People.  The All Sites scope returns everything in the content index (minus People).  When you start adding content sources to index file shares or custom applications, you will want to create a scope or two that allows the user to restrict results to just those areas.  For example, this would allow the user to query just SharePoint content or just File Share content.  You can also create more advanced scopes using managed properties such as creating a scope that returned only Invoices.  Once you create scopes, you can display them in the Search Center as well as in a drop down list on your master page.  This gives the user more flexibility on how they search.

In large organizations, finding the right person to ask a question can be challenging.  In SharePoint 2007, we gained People Search which allowed us to index information from Active Directory or an ERP system (through the BDC) about the people in an organization.  SharePoint 2010 extends this by adding a wealth of social features.  It also gives us phonetic searching which makes it easier for users to find people with difficult names to spell.  Although, People Search can infer a lot of information from Active Directory and other source systems, it is only as good as the information you provide it.  People also have metadata (in their profile) and you should work to have it as complete as possible.  Prepopulate what you can for the user with User Profile Synchronization then encourage the people in your organization to complete the rest of their profiles manually especially the Ask Me About section.  This section allows other users to know what that user is responsible for (I.e.: Accounts Payable, Benefits, or Employee Recruiting).

I also recommend making sure you have the Manager field set in People Search.  By default, People Search gets this from the Manager property in Active Directory.  Setting this allows SharePoint to determine your organization chart.  If your organization already populates this field, then you can take advantage of the organization chart feature in SharePoint right away.

Before moving on, I recommend taking a look at your existing web applications in your organization along with any ones currently in development.  SharePoint can expose the data of Many web applications whether developed in ASP.NET or another language.  Before SharePoint, creating a search experience for these applications usually involved a bit of code with the use of things such as T-SQL LIKE statements.  Using Business Connectivity Services (BCS), SharePoint can actually crawl your custom application by directly accessing the database tables, through WCF services, or the new BDC Entity Model.  Many times you can do this indexing without having to make any changes to your existing application.  By using SharePoint Designer 2010, you can connect to your SQL database or WCF services and choose the entities you want to bring into SharePoint as External Content Types.  With the External Content Type, you can create a Content Source to index that application. 

At this point, you might think “How do the links from the results page go to my custom application?”.  This depends on your application.  In your External Content Type, you can define a default action which SharePoint uses on the results page when the user clicks on a link.  SharePoint can use the data it indexes to help you build a URL (think String.Format()).  For example, let’s assume we are indexing a custom ASP.NET application which has product information.  The External Content Type contains fields such as Name, ProductId, Description, and Price which comes from a database table.  Your custom application is hosted on a server called server and it has a page which allowed the user to edit each product named ProductEdit.aspx.  That page takes a query string parameter called Id representing the ProductId in the database table.  SharePoint can construct this URL given the information it has to link the user directly to that product editing page.  See the picture below for an example.

Federating Search

Sometimes you may have too much content outside of SharePoint or maybe you have a system that already has a capable search engine (i.e.: an ERP or another document management system).  This calls for federated search. As another example, you might use a federated search to display results from your public facing web site or from a public search engine like Bing.  You don’t want to completely index their data, but you would still like to see results from those external systems when searching from SharePoint.  The Federated Search feature allows you to display results from any search engine supporting the OpenSearch 1.1 protocol alongside your local search results.  If you are not familiar with OpenSearch, the results come back as an RSS feed.  Even if your external system doesn’t support OpenSearch, you can write some code to refactor the results as RSS and integrate them easily into SharePoint.  Here is an example, where the federated results come from DotNetMafia.com on the right side of the screen.

Exposing SharePoint Search to other applications

We can index data in other applications, but maybe you don’t want to make the user go to SharePoint to see  the search results.  Luckily SharePoint has an extensive API that allows you to query SharePoint remotely using web services.  Calling the Search Web Service is pretty simple.  Effectively you construct an XML input document and then you can call the Query method to return results as XML or the QueryEx method to return results as an ADO.NET DataSet.  For a complete example, see my post on how to use the Search Web Service.  This also is your stepping stone for developing advanced search driven applications.  For example, you could call this web service from Silverlight to present the results using a rich UI.

Chances are the army of ASP.NET developers that you employ don’t know that SharePoint can index their applications.  Consider this as an option the next time you build a custom application.  There is no point in reinventing the wheel when SharePoint can do the heavy lifting for you.   It might even shave off a few hours of your next project.  Plus, it allows you to search from multiple sources at the same time.  Just think you could get results from SharePoint, a file share, and multiple custom applications in one results screen.  With the built-in refinement features, you can drill down into the results you want.

Other Considerations

You may be thinking “All of this is great, but I bet I need FAST Search for SharePoint to make it all happen.”  That’s not actually the case.  Yes, FAST can do a lot of powerful things and allows you to make some great customizations to your search experience.  However, most of the things I have talked about today have been around since SharePoint 2007.  In fact, with the exception of People Search, you can implement everything in this post today using SharePoint Foundation with Search Server Express.  If you use SPF today but don’t use Search Server Express, I highly recommend it.  It adds a lot of functionality to out-of-the-box search experience in SPF.

Don’t be afraid to ask for help.  This posts covers a lot and sometimes it is nice to have a little guidance along the way when you try to implement.  There are many great community resources out there talking about Search including blogs, forums (2007), twitter, and SharePoint Saturdays.  I won’t do a shameless plug for my consulting company today, but don’t be afraid to reach out to a firm with some SharePoint consultants to help you develop things like a Governance, ECM, or Search plan.  Getting your plan correct before you start will help keep you from having to deal with problems later.

Follow me on twitter.