Posted on:
Categories: Office 365;SharePoint
Description:

​If you add more than 5000 items to a list or library in SharePoint online, you are faced with this error message:

 

Unfortunately, this list view threshold can't be changed in SharePoint online, but there are options to overcome this limitation.

If you use Bing (or Google) to look for options, you will probably find this article published by Microsoft. It highlights some options to manage large lists (with more than 5000 items) in SharePoint online. One of those options is to create a view using an indexed column as a filter. Because I used this option in one of my recent projects to overcome the list view threshold in SharePoint online, I created this blog post to share my findings.

SharePoint online saves all the data in a SQL Server database. One option to improve the performance and to reduce the CPU load while retrieving items is to use indexed columns. I guess Microsoft suggested to use views with indexed columns to overcome the list view threshold because of that reason.

Let's have a look on how to create an indexed column first before I continue with setting up a view.

To create an indexed column, you need to navigate to the list settings first. On the settings page look for the list of columns. At the bottom of the list of columns, you'll find the link 'Indexed Columns'.

 

If you click on this link, you'll get to the list of indexed columns. Here new indexed columns can be created, but keep in mind the number of indexed columns is limited to 20 per list.

 

In addition, not all of the existing columns can be used as indexed columns. The following table is providing details:

Supported Column TypesUnsupported Column Types
Single line of textMultiple lines of text
NumberHyperlink/Picture
CurrencyAny custom field type
DateTimeCalculated field
Choice filed (single value only)Multi-value choice field
Lookup (single value only)Multi-value lookup field
Person/Group (single value only)External data
Managed Metadata 

 

Here is one thing I would like to note: from my personal experience, creating an indexed column based on a lookup field does not help in terms of overcoming the list view threshold. My recommendation is to replace Choice fields with managed metadata fields which will also enhance the maintainability of the data.

After an indexed column has been created, it can't be used right away. It usually takes some time for SharePoint online to propagate the changes to the underlying SQL server database. If you are creating a new indexed column because you want to fix an issue with a list that has already hit the 5000 items list view threshold, then this won't work! As creating an indexed column requires SharePoint online to 'touch' every single item in the list, the list view threshold will be hit again while the index gets created, which will stop the process internally. You'll end up thinking the index has been created, but it hasn't. If you need to create an indexed column for a list that has already hit the list view threshold, you need to create an empty list first, create all the indexed columns and when SharePoint online is done with that, copy the items to the list. This is the only way I know to create an indexed column for a list that has more than 5000 items.

Let's continue with creating a basic view. For this blog post, I'm creating a basic view that is using the indexed ID column to create a filter as this column is an indexed column by default.

The following screenshot shows that a view based on an indexed column is able to show more than 5000 items.

 

Let's continue with adding a sort to the view I have just created. The following screenshot show that I just added the 'End Date' column as a sorting parameter to the view. I want the view to display the items with the most recent end date first.

 

But if I use this view, SharePoint online displays the odd error message again. Why, the view worked before, didn't it?

The answer is easy! Yes, the view worked perfectly as long as it was just using the indexed column. Things changed when I added an additional column to sort the list items. This will add an additional column to the internal query SharePoint online is using to retrieve items. As this additional column is not an indexed column yet, we are hitting the 5000 item list view threshold again.

Lessons learned: if you want to use indexed columns to overcome the 5000 items list view threshold, you need to keep the following in mind:

  1. You can only have 20 indexed columns per list. Plan thoughtfully!

  2. You can't create an indexed column if the list is already hitting the list view threshold.

  3. If you use additional sorting in a view, the sort column also needs to be an indexed column.

  4. Review the list items and check if some can be deleted or archived to another list.

Although I understand why Microsoft implemented this list view threshold in SharePoint online many years ago, I don't think it is really needed anymore. Today's hardware and modern data centers should be robust enough to allow tenants to have lists with more than 5000 items. I wouldn't be too surprised if we would see changes in terms of the list view threshold in the near future.

 

Update: Unfortunately there is another limitation I wasn't aware of. This is what I found out: "When you create a filtered view, make sure the first indexed column in the filter expression does not exceed the List View Threshold. SharePoint selects the first indexed column in a query. Other columns you specify in the view filter may or may not be indexed, but the view does not use those indexes, even if the result of the filtered view returns less than the List View Threshold." (https://support.office.com/en-us/article/Manage-large-lists-and-libraries-in-SharePoint-b8588dae-9387-48c2-9248-c24122f07c59). To me this means that even with indexed columns there is no viable way how to overcome the list view threshold in SharePoint online!

 

Here is the PowerShell script I used to create list items automatically (I used an additional number field 'LVTNumber'): 

# Add references to SharePoint client assemblies and authenticate to Office 365 site - required for CSOM
Add-Type -Path “C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll”
Add-Type -Path “C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll”
Add-Type -Path “C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.WorkflowServices.dll”

# Specify tenant admin and site URL (replace '...' with your settings)
$SiteUrl = "..."
$ListName = "..."
$UserName = "..."
$SecurePassword = ConvertTo-SecureString "..." -AsPlainText -Force

# Bind to site collection
$ClientContext = New-Object Microsoft.SharePoint.Client.ClientContext($SiteUrl)
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName, $SecurePassword)
$ClientContext.Credentials = $credentials
$ClientContext.ExecuteQuery()

# Get the list
$List = $ClientContext.Web.Lists.GetByTitle($ListName)
$ClientContext.Load($List)
$ClientContext.ExecuteQuery()

# Loop to create list items
for ($i=1; $i -le 10000; $i++)
{
  $ListItemCreationInformation = New-Object Microsoft.SharePoint.Client.ListItemCreationInformation
  $NewListItem = $List.AddItem($ListItemCreationInformation)
  $NewListItem["Title"] = "LVTTest_$($i)"
  $NewListItem["LVTNumber"] = $i;
  $NewListItem.Update()
  $ClientContext.ExecuteQuery()

  write-host "Item created: LVTTest_$($i)"
}
write-host "Script finished!"