Post by: Oliver Wirkus
Posted on: 3/28/2017 5:02:00 PM
Categories: Office 365;SharePoint
Description: How to overcome the list view threshold in SharePoint online by using indexed columns.
you add more than 5000 items to a list or library in SharePoint online, you are
faced with this error message Unfortunately, this list view threshold can't be changed in SharePoint online, but there are options to overcome this limitation. If you use Bing (or Google) to look for options, you will probably find this article published by Microsoft. It highlights some options to manage large lists (with more than 5000 items) in SharePoint online. One of those options is to create a view using an indexed column as a filter. Because I used this option in one of my recent projects to overcome the list view threshold in SharePoint online, I created this blog post to share my findings. SharePoint online saves all the data in a SQL Server database. One option to improve the performance and to reduce the CPU load while retrieving items is to use indexed columns. I guess Microsoft suggested to use views with indexed columns to overcome the list view threshold because of that reason. Let's have a look on how to create an indexed column first before I continue with setting up a view. To create an indexed column, you need to navigate to the list settings first. On the settings page look for the list of columns. At the bottom of the list of columns, you'll find the link 'Indexed Columns'. If you click on this link, you'll get to the list of indexed columns. Here new indexed columns can be created, but keep in mind the number of indexed columns is limited to 20 per list. In addition, not all of the existing columns can be used as indexed columns. The following table is providing detailsSupported Column TypesUnsupported Column TypesSingle line of textMultiple lines of textNumberHyperlink/PictureCurrencyAny custom field typeDateTimeCalculated fieldChoice filed (single value only)Multi-value choice fieldLookup (single value only)Multi-value lookup fieldPerson/Group (single value only)External dataManaged Metadata Here is one thing I would like to note from my personal experience, creating an indexed column based on a lookup field does not help in terms of overcoming the list view threshold. My recommendation is to replace Choice fields with managed metadata fields which will also enhance the maintainability of the data. After an indexed column has been created, it can't be used right away. It usually takes some time for SharePoint online to propagate the changes to the underlying SQL server database. If you are creating a new indexed column because you want to fix an issue with a list that has already hit the 5000 items list view threshold, then this won't work! As creating an indexed column requires SharePoint online to 'touch' every single item in the list, the list view threshold will be hit again while the index gets created, which will stop the process internally. You'll end up thinking the index has been created, but it hasn't. If you need to create an indexed column for a list that has already hit the list view threshold, you need to create an empty list first, create all the indexed columns and when SharePoint online is done with that, copy the items to the list. This is the only way I know to create an indexed column for a list that has more than 5000 items. Let's continue with creating a basic view. For this blog post, I'm creating a basic view that is using the indexed ID column to create a filter as this column is an indexed column by default. The following screenshot shows that a view based on an indexed column is able to show more than 5000 items. Let's continue with adding a sort to the view I have just created. The following screenshot show that I just added the 'End Date' column as a sorting parameter to the view. I want the view to display the items with the most recent end date first. But if I use this view, SharePoint online displays the odd error message again. Why, the view worked before, didn't it? The answer is easy! Yes, the view worked perfectly as long as it was just using the indexed column. Things changed when I added an additional column to sort the list items. This will add an additional column to the internal query SharePoint online is using to retrieve items. As this additional column is not an indexed column yet, we are hitting the 5000 item list view threshold again. Lessons learned if you want to use indexed columns to overcome the 5000 items list view threshold, you need to keep the following in mind You can only have 20 indexed columns per list. Plan thoughtfully! You can't create an indexed column if the list is already hitting the list view threshold. If you use additional sorting in a view, the sort column also needs to be an indexed column. Review the list items and check if some can be deleted or archived to another list. Although I understand why Microsoft implemented this list view threshold in SharePoint online many years ago, I don't think it is really needed anymore. Today's hardware and modern data centers should be robust enough to allow tenants to have lists with more than 5000 items. I wouldn't be too surprised if we would see changes in terms of the list view threshold in the near future. Update Unfortunately there is another limitation I wasn't aware of. This is what I found out "When you create a filtered view, make sure the first indexed column in the filter expression does not exceed the List View Threshold. SharePoint selects the first indexed column in a query. Other columns you specify in the view filter may or may not be indexed, but the view does not use those indexes, even if the result of the filtered view returns less than the List View Threshold." (https//support.office.com/en-us/article/Manage-large-lists-and-libraries-in-SharePoint-b8588dae-9387-48c2-9248-c24122f07c59). To me this means that even with indexed columns there is no viable way how to overcome the list view threshold in SharePoint online! Here is the PowerShell script I used to create list items automatically (I used an additional number field 'LVTNumber') # Add references to SharePoint client assemblies and authenticate to Office 365 site - required for CSOM
Add-Type -Path “C\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll”
Add-Type -Path “C\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll”
Add-Type -Path “C\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.WorkflowServices.dll”
# Specify tenant admin and site URL (replace '...' with your settings)
$SiteUrl = "..."
$ListName = "..."
$UserName = "..."
$SecurePassword = ConvertTo-SecureString "..." -AsPlainText -Force
# Bind to site collection
$ClientContext = New-Object Microsoft.SharePoint.Client.ClientContext($SiteUrl)
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName, $SecurePassword)
$ClientContext.Credentials = $credentials
# Get the list
$List = $ClientContext.Web.Lists.GetByTitle($ListName)
# Loop to create list items
for ($i=1; $i -le 10000; $i++)
$ListItemCreationInformation = New-Object Microsoft.SharePoint.Client.ListItemCreationInformation
$NewListItem = $List.AddItem($ListItemCreationInformation)
$NewListItem["Title"] = "LVTTest_$($i)"
$NewListItem["LVTNumber"] = $i;
write-host "Item created LVTTest_$($i)"
write-host "Script finished!" Post by: Oliver Wirkus
Posted on: 3/24/2017 4:43:00 PM
Categories: Business;Office 365;SharePoint
Description: This blog post describes how to use calendar overlays in SharePoint
Calendars are very popular in SharePoint. They get added to many sites and subsites in various SharePoint portals, especially to team and project sites. Although calendars are widely used in SharePoint, not many site owners spend time planning how to use them. I often see subsites with a single calendar crammed with many appointments of different kinds. My usual recommendation is to use one calendar for each type of appointment, but when suggesting this improvement, many site owners are afraid it may affect usability. This is where calendar overlays come into play. What are calendar overlays in SharePoint? In a nutshell, with calendar overlays, you can merge multiple calendar views. Those additional views can come from different calendars or from just one calendar. Each overlaid calendar can be displayed in a different color. How can overlaid calendars be used in SharePoint? I see two typical use cases for calendar overlays in SharePoint. The first one is to aggregate appointments from multiple calendars into a single view. Here is an example a marketing team uses one calendar for vacations, another for stat holidays and a third calendar for managing events. With overlaid calendars, the event planning will become much easier as holidays and vacations are superimposed on the events calendar, even though they are managed in different calendars. The second use case is the popular color coding with different types of events being displayed using different colors. This can be achieved by creating a view for each type of appointment, even if only one calendar is used. Overlaying these views will result in a single calendar view displaying different types of appointments in different colors. How are calendar overlays created? Let me show you this by using an example you can recreate in your environment easily. In my demo, I have a subsite with three calendars, a primary calendar, a calendar listing stat holidays and a calendar for vacation requests. I would like to superimpose the calendar with the stat holidays and the vacation calendar on the main calendar. Both calendars are providing a calendar view called 'All Items'. Before I continue with some screenshots, I should mention that calendar overlays only work with calendar views. List views cannot be overlaid. In my example, I have added Easter Sunday to the stat holidays calendar and a vacation request to the vacation calendar. To create the overlays, I turn to the primary calendar, open the ribbon and select 'Calendar Overlays' The following dialogue shows up, and I click on the ‘New Calendar’ link I create a new calendar overlay for vacations as
shown in the next screenshot (don’t
forget to click on ‘Resolve’ to be able to select an additional calendar list) I repeat this step with the stat holidays calendar. After I have done that, the list of overlaid calendars looks like this Let's switch back to the primary calendar and see what the overlaid calendars look like Here you can see that the view from the vacation calendar (red) is added to the primary calendar and the view from the stat holiday calendar shows up in green. Keep in mind that both appointments are NOT part of the primary calendar. They are still managed in their own calendars. On the left side, SharePoint shows a legend explaining which calendar is displayed and in what color. To navigate to one of the overlaid calendars, click on the calendar name in the left navigation. As you can see, it just takes a little additional configuration, to create calendar overlays which improve usability and maintainability. That's why calendar overlays are my hidden gems in SharePoint.Post by: Oliver Wirkus
Posted on: 3/20/2017 6:23:00 PM
Categories: Business;Office 365;SharePoint
Description: Migrating content and configuration from SharePoint 2007 to SharePoint online can be done, but there are some important things to consider. This blog post provides my recommendations and outlines important topics which need to be considered before starting the migration.
Although Microsoft released SharePoint 2016 many months ago, there are still some companies out there which are using older SharePoint versions (like SharePoint 2007 or SharePoint 2010). A SharePoint migration from an older version of SharePoint (like SharePoint 2010 or SharePoint 2007) to an up-to-date version of SharePoint (like SharePoint 2016 or SharePoint online) is not an extraordinary process. If content from an older version of SharePoint is to be migrated to SharePoint online, there are some things that should be considered, before starting the migration. 1. Pre-Migration Assessment Although a pre-migration assessment usually is an important part of the pre-migration planning, it becomes more important, if content from an older version of SharePoint needs to be migrated. A pre-migration assessment is helpful to find out if the selected migration tool can migrate content and settings faultlessly. Usually, the log files of a pre-migration assessment provide valuable hints which list, library, or web part might cause problems during the migration. I recommend doing a pre-migration assessment as it is needed to plan the migration process thoroughly. 2. Review site and subsite structure With SharePoint 2007 it was common to create the site structure similar to the structure of the organization. In other words the site structure in SharePoint was reflecting the organization's structure one to one. At that time this was the usual approach, but today a site structure does not reflect the structure of the organization anymore. Instead, structures are reflecting internal procedures or are focusing on department-comprehensive projects. In other words today's site structures are user-centric and are based on processes and internal collaboration rather than copying the internal structure of an enterprise. That's the reason why the site structure of an older SharePoint farm needs to be reviewed very thoroughly. In most cases, it makes sense to create a kind of mapping table for sites and subsites. This mapping table lists all the sites and subsites in the old portal and describes how this translates to the new site structure in SharePoint online. 3. Review existing business applications Although SharePoint 2007 was not used as a platform for integrated business applications like newer versions of SharePoint, many companies created what I like to call 'Mini Applications'. Those mini applications are not based on custom code, but on joined lists with lookup fields, content types, and sometimes calculated fields. Some enterprises created very complex structures by just using multiple generic lists which are joined by lookup fields. The problem with that kind of applications is, that they have been used for many years and that employees have become accustomed to them. While these applications can be migrated to SharePoint without major issues, it makes sense to check, if these applications can be updated or modernized before migrating them. Sometimes a SharePoint-hosted add-in (which wasn't available to previous versions of SharePoint) can be used to replace some functionality and to improve the usability. If a mini application is frequently used, it can be beneficial to rethink its internal structure and use-case to see if it can be replaced by a SharePoint-hosted add-in or a Provider-hosted add-in. 4. Review branding If you compare the out-of-the-box branding of SharePoint online with the out-of-the-box branding of SharePoint 2007, you'll notice a huge difference - and that's true for the user interface as well. If a company has invested a significant effort in creating a corporate SharePoint branding and wishes to migrate this, it will be problematic. Due to the massive changes in SharePoint online compared to SharePoint 2007, an old SharePoint 2007 branding can't be migrated to SharePoint online completely. My recommendation is to take the old SharePoint 2007 branding as a basis and create a new corporate branding for SharePoint online which is adapting the modern user interface of SharePoint online in a beneficial way but still looks similar to the old branding. 5. Review the size of lists and libraries At the times when SharePoint 2007 was popular, there was no such thing as a 5000-items limit like it exists in SharePoint online today. As a result many enterprises created generic lists which contained over 5000 items, which resulted in a decreased performance, but still worked. While these large lists can be migrated to SharePoint online, problems will occur as soon as these lists have been migrated to SharePoint online. If a view is returning more than 5000 items, SharePoint online is displaying an error message. If there are lists with several thousands of items in the old SharePoint farm, I highly recommend reviewing those lists before migrating them. Sometimes the problems can be solved by creating additional filtered views which are based on indexed columns. Sometimes it makes more sense to split up those large lists into multiple lists based on particular topics or to create annual lists. 6. Plan timelines thoroughly Migrations are usually done with the help of tools (like ShareGate or Metalogix). If these tools are used to migrate content from SharePoint on-premises to SharePoint online, these tools need to utilize the Client-side API which is significantly slower compared to the popular and powerful SharePoint Server-side API. The Client-side API is not the only factor which affects the time needed to migrate content and settings to SharePoint online. The varying performance of Office 365 and the throughput rate of the internet connection used during the migration can also affect the duration of the migration. To get some basic time estimates, a look at the reports of the pre-migration assessment can be helpful as they include information about the amount of data and throughput rates. 7. Prepare user training As I have mentioned earlier, the user interface of SharePoint online has been changed dramatically compared to SharePoint 2007. Although the basic functionality (like adding items to a list or updating the metadata of a document) still exists in SharePoint online, the way that functionality is exposed through the user interface has been changed considerably. Without an aligned user training, the user acceptance of the new SharePoint online portal will suffer significantly simply because the employees feel confused or overwhelmed by the new user-interface.