solr returns duplicate records when paginating using edismax

Question

I am using solr 3.5 and facing a weird behavior when paginating. Some identical records / documents are being returned by solr when paginating; Example, I get a document in page one and I am getting the same document in the third page. After troubleshooting I realized that Solr behaves like this when using edismax query parser. Pagination works well when using the default parser. Any ideas why??

below are the decoded queries for each paginated page

page 1

spellcheck=true&spellcheck.onlyMorePopular=false&spellcheck.extendedResults=true&spellcheck.count=1&spellcheck.collate=true&q=coffee AND countryId:1&defType=edismax&start=0&rows=10&fl=outletId,outletPhone1,outletPhone2,outletPhone3,outletUrl,outletGpsLatitude,outletGpsLongitude,outletAddress,outletPrimaryName,outletSecondaryName,brateCount,brateit_review,views,outletAvarageRating,outletRatingId,outletRatingValue,poBox,locatedInOutletId,locatedInOutletPrimaryName,locatedInOutletSecondaryName,locatedInName,floor,addressPlaceId,addressPlaceName_en,streetId,streetName_en,areaId,areaName_en,cityId,cityName_en,countryId,countryCode,countryName_en,plsId,plsName_en,categoryId,categoryName_en,ratingId,ratingName_en,primaryCategoryId,primaryCategoryName_en,primaryIndustryId,primaryIndustryName_en&facet=true&facet.limit=-1&facet.sort=count&facet.mincount=1&facet.field=facetCityName_en&facet.field=facetPoolCategory_en&facet.field=facetAreaName_en&sort=score desc, outletAvarageRating desc,brateCount desc,views desc,random_1347951132109 desc

page 2

spellcheck=true&spellcheck.onlyMorePopular=false&spellcheck.extendedResults=true&spellcheck.count=1&spellcheck.collate=true&q=coffee AND countryId:1&defType=edismax&start=10&rows=10&fl=outletId,outletPhone1,outletPhone2,outletPhone3,outletUrl,outletGpsLatitude,outletGpsLongitude,outletAddress,outletPrimaryName,outletSecondaryName,brateCount,brateit_review,views,outletAvarageRating,outletRatingId,outletRatingValue,poBox,locatedInOutletId,locatedInOutletPrimaryName,locatedInOutletSecondaryName,locatedInName,floor,addressPlaceId,addressPlaceName_en,streetId,streetName_en,areaId,areaName_en,cityId,cityName_en,countryId,countryCode,countryName_en,plsId,plsName_en,categoryId,categoryName_en,ratingId,ratingName_en,primaryCategoryId,primaryCategoryName_en,primaryIndustryId,primaryIndustryName_en&facet=true&facet.limit=-1&facet.sort=count&facet.mincount=1&facet.field=facetCityName_en&facet.field=facetPoolCategory_en&facet.field=facetAreaName_en&sort=score desc, outletAvarageRating desc,brateCount desc,views desc,random_1347951219828 desc

page 3

spellcheck=true&spellcheck.onlyMorePopular=false&spellcheck.extendedResults=true&spellcheck.count=1&spellcheck.collate=true&q=coffee AND countryId:1&defType=edismax&start=20&rows=10&fl=outletId,outletPhone1,outletPhone2,outletPhone3,outletUrl,outletGpsLatitude,outletGpsLongitude,outletAddress,outletPrimaryName,outletSecondaryName,brateCount,brateit_review,views,outletAvarageRating,outletRatingId,outletRatingValue,poBox,locatedInOutletId,locatedInOutletPrimaryName,locatedInOutletSecondaryName,locatedInName,floor,addressPlaceId,addressPlaceName_en,streetId,streetName_en,areaId,areaName_en,cityId,cityName_en,countryId,countryCode,countryName_en,plsId,plsName_en,categoryId,categoryName_en,ratingId,ratingName_en,primaryCategoryId,primaryCategoryName_en,primaryIndustryId,primaryIndustryName_en&facet=true&facet.limit=-1&facet.sort=count&facet.mincount=1&facet.field=facetCityName_en&facet.field=facetPoolCategory_en&facet.field=facetAreaName_en&sort=score desc, outletAvarageRating desc,brateCount desc,views desc,random_1347951286359 desc

thanks for the reply. i added my solr query requests above. the schema is a bit big though; should i post it as well? — Sameeh Harfoush
Could you show us what is the response that you get and what you would expect instead? — javanna

Sameeh Harfoush Sameeh Harfoush · Accepted Answer · 2013-05-06T09:43:42

the problem was with my sorting implementation. i was sorting by random field on each page causing the document to keep shifting between pages.

String randomField = random_"+ (new Date()).getTime();

solr returns duplicate records when paginating using edismax

1 Answers