Lately, I have really put a lot of thought into this concept I think of as Personal Governance. I am trying to set an informal policy for myself on where I put my documents. This includes all types of documents such as things I am working on at work as well as personal things like receipts and photos. Microsoft offers both SkyDrive and SkyDrive Pro (at least that’s what they are currently called) to help us with these tasks but there are decisions to be made. When it comes to where a file should be stored, the answers to some of these questions is simple. Business documents go in SkyDrive Pro and things like photos go in SkyDrive. This is basically how Microsoft describes it. However, I see a case where there is an interim period for some of my documents that start in SkyDrive and get moved over to SkyDrive Pro or perhaps a team site. Largely this depends on whether I need to share the file with others in my organization. As I am trying to set up a policy for myself, I ask myself these types of questions. Admittedly, I don’t plan on drafting up a sixty page document on self-governance, but I still want to solidify a plan in my head.
I have become heavily addicted to SkyDrive and not a day goes by that I am not connected to it in some way whether that’s my PC, my Surface RT, or my Windows Phone. I was grandfathered into the service back when it offered 25 GB for free and I have had to purchase additional storage because I store so much out there. This is largely because I have every photo I have ever taken on my SkyDrive since I got my first digital camera back in 2000. I figure SkyDrive is a pretty safe place to store things like this and things are accessible on any of my devices. However, since my organization is not fully on SharePoint 2013 yet, I find myself storing a lot of work documents on my SkyDrive because I want to be able to access them on my Surface RT (running 8.1) when I am offline. The nice thing about Office is it can pull files from SharePoint or SkyDrive and it is seamless. That makes things convenient. However, this leads me to having two copies of the same file and that drives me nuts because then I have to think about where the latest version of the document is.
If you haven’t noticed yet SkyDrive and SharePoint have a lot of similar features. They both have things like sharing, Office Web Apps, and version history. I rue the day if they ever add content types and site columns to SkyDrive because I know the compulsive nature in me would be forced to sit down and classify every single document I have. :) Since there are so many similarities I tend to apply similar practices to SkyDrive that I do in SharePoint. I regularly waste time reorganizing things needlessly.
You might be guessing that I have a lot of junk in my SkyDrive and I would say that’s probably correct. I have receipts from expense reports more than seven years old. This brings me to a future topic in how do you enforce a Personal Retention Policy? For companies, this is more about compliance and avoiding legal issues. For an individual, this is more about cleaning up your junk. SkyDrive lets me query by recent documents through the web interface, but figuring out which documents are old and need to go away is a bit trickier. Your best bet is likely by syncing everything and then using traditional file system or PowerShell commands. That’s not exactly ideal especially for an end user. I wouldn’t be surprised if Microsoft adds tools to the interface some day to help you identify old files. Admittedly if there was a disposition job that ran on my files in SkyDrive like in SharePoint, I would be a fan. :) After all it’s in their best interest to have you clean up your files as well, right? What do you think, should individuals enforce a retention policy on themselves? How do you handle it with the files you have? Am I just a big nerd for thinking of this kind of stuff? Probably. :) Leave your comments below.
When it comes to SharePoint search, managed properties have always had a special place in my heart. After all they are what lets you do all of the cool stuff in SharePoint search when it comes to querying, refining, and displaying information. I’ve been meaning to write a post on them in SharePoint 2013 for a while now. A friend asked me recently some more details how they worked so I thought I would dig into some details today. Managed properties have evolved over the year and now in SharePoint 2013, we have more granularity than ever when working with them. I’ll explain what that means shortly. Now this part of search is collectively referred to as the Search Schema.
In SharePoint 2007, we simply mapped managed properties, we could then query on them and display them in search results. SharePoint 2010 worked roughly the same, but when you added FAST Search for SharePoint, you could not configure which properties could be used for refinement and sorting. SharePoint Online came out and search was there but we couldn’t actually configure anything. Fast forward to today with SharePoint 2013 and the new version of SPO and we have all the flexibility F4SP and then some. We can now configure managed properties and specify exactly how we want to use them.
Before, we get started with managed properties, let’s take a quick look at their counter part, crawled properties. A crawled property is created for every site column (or database column if using BCS) the crawler finds while indexing. Remember though that there needs to be data in that site column in at least one list item before SharePoint will add it to the index. We can’t do anything with crawled properties until we map them to a managed property (I’ll explain how below). SharePoint 2013 typically does this automatically for you now, although I have seen times when it doesn’t work. Once we do have a mapping though, we can query on them, display them, sort on them, etc. You may also want to create your own mapping so that you have a nicer more human-readable name to work with. The crawler only creates crawled properties and property mappings when you do a full crawl though. So if you want to create your own custom managed property, you have to crawl once to get the crawled property and again after you create the managed property and do a mapping. Another thing to note is that SharePoint site columns will always be found in the SharePoint category of the list and they will also be prefixed with the letters ows_. If your site column had spaces in it, these will be encoded as _x0020_. These can be unsightly so all the more reason to leave spaces out of your site column names.
Another change we have with SharePoint 2013 is the ability to also define the search schema at the tenant (think SPO) and site collection. Effectively what this does is allow the administrator at the appropriate level to override the property mappings for their area. This is an important concept and is worthy of it’s own blog post though because you can do some interesting things with it. Therefore, today, I’ll be focusing on managed properties at the service application level.
For our example here today, let’s assume we have crawled our SharePoint site and we have a document library with some invoices in it. In this library we have some site columns for tracking the document type, submission date, and an invoice number. We’ll also use a few other examples as well. At this point, I have already done a full crawl so that we have crawled properties for each of them. Here is what my library looks like.
When we are done, we’ll be able to query these items in search by any of these attributes.
To begin working with managed properties at the SA level, you’ll need to go to your Search Service Application and then choose the Search Schema link (it wasn’t called that in SharePoint 2010). The page has been updated a bit and it allows us to see the key attributes of each managed property at a glance. We’ll be talking about each of these attributes in detail.
To create a new managed property, click the New Managed Property button. We’re going to look at each of the relevant settings. We first need to name it and choose it’s type. If you have a site column of a non-text type (i.e.: number or date) this is important. You won’t be able to map it if the type doesn’t match up and you can’t take advantage of the type of queries you would (such as show me all of the invoices submitted in July). I am going to start with our InvoiceNumber site column. Even though it’s technically a number, the assumption is that invoice numbers could have some letters in them as well so I have configured it with a type of Text.
I am going to jump straight to the mapping screen as it is the most relevant. Based on the type of managed property you select, the crawled properties that you can pick will vary to match the type. You can map one or more crawled properties to your managed property here. This allows you to map properties from different locations with different names and query them all simultaneously. For example, if one lists calls the site column, Invoice Number and another list calls it InvoiceNo, you can map it to one managed property called InvoiceNumber. Obviously my managed property name is correct because there aren’t any spaces and abbreviations. :) In our example below, we have mapped out SubmissionDate site column. Remember, that SharePoint site columns are always prefixed with ows_, so the site column shows as ows_SubmissionDate.
Now, we are going to go through the various attributes you can set for managed properties. With the site columns I had added to my document library, all of them have special characteristics that we can highlight. The MSDN post does a good job covering this but today we’ll be discussing each attribute in detail and show you them in action. Note, that none of these options are selected by default so you will need to make some decisions here. Hopefully, this information will help you with that.
Searchable
The Searchable attribute specifies that the contents can be found in the full-text index. Think of this as the default behavior that you got with previous versions of SharePoint. What this equates to is if you type something in the search box anything matching in this column will return a result. For example, if I search for GL, I’ll get the following results. This had a hit in the DocumentType column, but that it also hit on the page from the document library as well because it found the text in the page. What’s nice is that you can disable this behavior now so if you don’t want values coming back from a column in the general search results that is possible.
Advanced Searchable Settings
This section lets you tweak the weight of the managed property. It should be pretty rare that any of you all actually need to changes these settings, so I would advise that you leave them alone.
Queryable
This property is useful for when you want to query on a specific column for a value (as opposed to the full-text index). This is great for when you want to search for something specifically in a column such as document type or department. In our example above with the Searchable property, we got noise in our search results because the page containing the document results also came back with the matching document. With the Queryable property, we can eliminate that noise by only returning documents that have a match on the column we are interested in.
In this example above, I am using the managed property that the crawler created for me. It takes the name of your site column and prefixes it with owstaxid (in the case of managed metadata columns).
Retrievable
When you want to actually see your search results somewhere, this is the attribute to use. Back in previous versions of SharePoint, you would then go and edit the XSLT of the CoreResultsWebPart so you could display them in your search results. Now, you can use any property marked as Retrievable in a display template. If you have read up on display templates already you know that you can use them to customize how search results look based on various criteria. I won’t cover how to build a display template today as this blog post is already long enough but there are plenty of examples out there. In the example below, I added the InvoiceNumber managed property to the template so that the user could see it when they got their search results.
Allow multiple values
This setting allows search to be aware that it may get multiple values stored for the given property. The example the documentation gives with Author is probably the best. Search will keep track and show when multiple authors have edited a document. In my experience, I haven’t used this attribute a ton but it can come in handy. You can only use this setting at the Service Application level as well.
Refinable
Query refinement is one of the more exciting things on the search page. It gives users the ability to drill down into search results after they make a query. Effectively, it makes it easier for them to find what they are looking for. Refinement is not enabled by default. When you turn it on, you have two choices for Yes (active and latent). Yes – active is what you want. The latent setting basically says you want to turn it on in the near future but not quite yet. You don’t have to perform a full crawl when switching from latent to active. However, you still have to do a full crawl to begin with. I doubt you will ever use the latent setting. One other thing to note about this attribute is that you must also mark the property as Queryable if you turn on refinement.
To make use of a refinable property, edit your results page and then edit the refinement panel on the left. Click the Choose Refiners button and you will see a list of refiners. In this case, I have picked my site column, owstaxidDocumentType (remember that the crawler created the property for me automatically). Select it and you will get a preview of the results based on your current query on the results page.
When we save our changes and execute the query again, we’ll get our new updated refinement panel. Note the Document Type refiner on the left that I added to the search results page using our managed property marked as refinable.
Sortable
Sorting has always been a big deal and highly requested feature in previous versions of SharePoint. You really couldn’t sort by many criteria at all without writing custom code. In SharePoint 2010, FAST Search for SharePoint added sorting capabilities but many people didn’t care to pay for it. Well, now sorting is a first class feature in SharePoint 2013. You can mark managed properties as sortable now and then use them in the search results web part. Note, that you can only do this for properties at the Service Application level. In my example, I am going to make the SubmissionDate property sortable.
To add your custom sort order, it is well, less than user friendly. You have to understand a bit of JSON. I have a relevant post on how to sort by query string in 2013 that will help us here. Start by editing your search results page and then the Search Results web part. Open the Settings section and you will see the sorting settings. The Show sort dropdown is not checked by default so check it.
That nasty JSON string is what we need to deal with. Here is what the text actually looks like.
[{"name":"Relevance","sorts":[]},{"name":"Date(Newest)","sorts":[{"p":"Write","d":1}]},{"name":"Date(Oldest)","sorts":[{"p":"Write","d":0}]},{"name":"Lifetime Views","sorts":[{"p":"ViewsLifeTime","d":1}]},{"name":"Recent Views","sorts":[{"p":"ViewsRecent","d":1}]}]
We need to specify the name and then give it the managed property to sort on. Effectively there is an embedded object here, so what we need to do is add our sort string in the right sport. The d parameter is the direction (0 for ascending, 1 for descending). Our goal is to get SubmissionDate in that list in descending order. Let’s construct my new sort order string first. Then we just need to add it to the larger string.
{"name":"Submission Date(Newest)","sorts":[{"p":"SubmissionDate","d":1}]}
The entire string together would be as follows:
[{"name":"Relevance","sorts":[]},{"name":"Date(Newest)","sorts":[{"p":"Write","d":1}]},{"name":"Date(Oldest)","sorts":[{"p":"Write","d":0}]},{"name":"Lifetime Views","sorts":[{"p":"ViewsLifeTime","d":1}]},{"name":"Recent Views","sorts":[{"p":"ViewsRecent","d":1}]},{"name":"Submission Date(Newest)","sorts":[{"p":"SubmissionDate","d":1}]}]
Once we save our settings, we’ll now see the drop down list in our search results.
Safe
I haven’t had an opportunity to use this attribute yet but I am pretty excited about it. It allows you to mark managed properties as not safe for anonymous users to see. If you are surfacing some semi-sensitive information in your search index you can use this setting to help safeguard the data even further when exposing search on the Internet. On top of that you can use to implement scenarios where anonymous users see some properties in search results but can’t see others. It’s a pretty interesting attribute if you do a lot of public facing web sites.
Alias
Aliases are somewhat interesting. It allows you to give a friendlier name to an existing managed property. I think the sole purpose of them being there is to rename managed properties that were automatically created such as those for managed metadata fields. I personally can’t stand using managed properties prefixed with owstaxid. It just doesn’t look pretty at all. I can make an alias for a more human-readable name to give the property a better name. Otherwise, I would just end up creating another managed property which just puts more clutter in the index. In our example, I create an alias called DocumentType so that I can query on it like you see below.
Token Normalization
Token Normalization allows you to turn off case-insensitivity. This is not something you would want very often, but I am sure there are use cases.
Complete Matching
Complete Matching is useful when you want to allow users to search for IDs such as product numbers or invoice numbers in our case. This disables partial matching and ensures that you only get the exact search result you are looking for. In the example below, we only get a single result even though there are other invoice numbers that have the same prefix.
Company Extraction
Company Extraction is kind of a funky feature that came over from FAST Search for SharePoint. Basically, SharePoint has a dictionary of company names that it will recognize in your documents and allow you to refine on them. This might be useful if you have a site column tracking vendors or something like that but I don’t really see too many people using it. If you have an example where you are successful with it, I’d love to hear about it so leave a comment below.
Custom Entity Extraction
I wrote about Custom Entity Extraction in a previous blog post. This gives you the ability to seed search with a dictionary of terms and then look for that content and use it for refinement. For more details, be sure and check out my post on that because it is one of the most useful features in search in SharePoint 2013.
Conclusion
This wraps up my overview of managed properties and the search schema. Just remember any time you adjust these settings, you will need to do a full crawl. IN a future post, we’ll talk about working with managed properties at the site collection level as well as from the tenant level in SharePoint Online.