We have a problem while rebuilding the Sphinx index for one of our MySQL databases. Rotating and rebuilding the index fails repeatedly, and the system continues to use the old index. The database size has doubled in the last months and it contains now tables with more than 10 million rows. During rebuilding of the Sphinx index with
indexer --config our_config.conf --rotate --all
(generated by the UltraSphinx plugin used here), we get the following error in the Sphinx daemon log file searchd.log
WARNING: rotating index 'main': preread failed: failed to open
~/etc/sphinx//sphinx_index_main.new.spi: No such file or directory;
using old index
WARNING: rotating index 'main': preread failure reported
The error messages occurred in increasing frequency as the database grew larger. Full indexing takes now about 1 1/2 hours, and the index files together are larger than one GB. We updated the Sphinx Version from 0.9.8.1 to 2.0.4 to solve this problem, but it still exists. The server has about 4 GB memory, and the configuration file begins with
indexer {
mem_limit = 1024M
}
searchd {
read_timeout = 5
max_children = 300
log = ~/etc/sphinx/log/searchd.log
port = 3313
max_matches = 100000
query_log = ~/etc/sphinx/log/query.log
seamless_rotate = 1
pid_file = ~/etc/sphinx/log/searchd.pid
address = 127.0.0.1
}
..
Has anyone an idea how the problem can be solved? I guess there seems to be some kind of index size limit which has to do with the 2 GB limit, wich is the maximal value for a signed 32 bit integer and maximal value for file sizes on certain file systems. The ext3 file system we use on our Ubuntu/Debian systems allows up to 16 GB for files, but certain libraries may limit the file size in practice to 2 GB (see here and here).
