Fast open thousands of files in linux/c

5
votes

I am doing SDCARD scanner which scans each folder and image file.

I am using C code & using opendir, readdir, fopen APIs to enumerate directory and file open. i have dirent, DIR variables.

Problem is fopen() is taking too much time (300 sec for 10000 files) while director traversing is taking ~25 seconds.

Is there any API which allows me to speed up file open operation, using handle, dir_ino or similar so that i can open file using directory handle or similar.

So far i have looked n tried to use dirent->dir_ino, DIR* but no luck.

I m seeking for low-level api which takes lesser time than fopen.

edit will fts and ftw apis be usefull? they seems to be related to traversing directory only...any other hack or method?

1
androidcfile-io
Before I edit it out, why the Android tag? – Simon
Speed limitations don't come from the API but from the SD card itself. – mouviciel
@Simon I guess, because it's on an Android system. – Olaf Dietsche
@mouviciel: right, but fopen takes a filename, which means it will re-visit the directory listing that the program has just been looking in (potentially several levels deep, therefore scanning several directories). That may or may not cost something significant, you'd sort of hope that either the filesystem or the block device will avoid unnecessary disk (card) activity by caching directory listings. If you're really lucky, Android provides some guarantees that it will. – Steve Jessop
@SteveJessop okey dokey, just checking. – Simon

1 Answers

1
votes

The only "low level" function is open 

int fd = open(dirent->d_name, O_RDONLY);

Though, fopen shouldn't be a lot slower.