Sharepoint 2010: Programatically adding documents to large library

0
votes

We have a system where documents are uploaded to a SharePoint library using a web interface. I'm accomplishing this using the client API. Specifically I use SaveBinaryDirect to upload the document, after which a CAML query is used to retrieve the document so that the Metadata columns can be set. The CAML query retrieves the document based on the FileLeafRef field, which is populated with the file name of the document which was uploaded.

The problem is that SharePoint imposes a maximum limit of 5000 items returned from any query so once the document library grows to 5000 items, the CAML query will no longer work. (Even though it only ever returns the single document, it apparently requires a scan of the entire library)

I have already investigated indexing columns but apparently you can't index FileLeafRef, and it seems to be the only field which is populated with the document name immediately upon upload. I have postponed the problem by increasing the item limit from 5000 to 10000, but we're about to run up against this new limit, and people are starting to report intermittent performance problems.

Is there any way to upload documents via the Client API and set their Metadata without running up against the maximum item limit?

2
sharepointsharepoint-2010
You could try and see if there are any performance benefits from using a connector instead. In example bendsoft.com/net-sharepoint-connector. This kind of tools have been built to handle situations like yours. - Eric Herlitz
And, if third party software is not an option please consider to post some of your caml so we can see what kind of selections you are doing. - Eric Herlitz
Interesting software. Do you know if is capable of circumventing the mentioned item limits? - Trent
It should handle that as well, either by stating LIMIT in the selections or by setting the DefaultLimit parameter in the connectionstring, bendsoft.com/documentation/camelot-net-connector/latest/… - Eric Herlitz

2 Answers

0
votes

Maybe you could create another field (a text field) that would be a simple copy of the FileLeafRef, and then index this new field?

0
votes

Solved the problem by not relying on CAML to locate the file. Instead I'm using GetFileByServerRelativeUrl, which isn't subject to the item limit. The file object has a method called ListItemAllFields, which allows you to query and set the metadata.

Should there be a need to return items with criteria other than file name, the above method will allow the metadata to be set initially, making it possible to store and index whatever criteria is necessary to satisfy the query.

If there was a requirement to query a large list based on criteria that is not indexed, or cannot be indexed, then it appears the solution involves using ContentIterator objects to spit out the list in small batches. But this doesn't seem to be available for the Client API, so I'm at a loss as to how you're supposed to work with large lists programmatically.