MATLAB seems to use coordinate list (COO) format for its sparse matrices. In MATLAB's COO format, the elements are sorted by column indices first then raw indices among elements of the same column indices. Thus, every time you specify an element with its row index, MATLAB has to search the entire indices of the sparse matrix to get the elements with the right row indices.
It would be significantly faster if you can load your original matrix as its transpose, then access the columns of the sparse matrix, i.e.
new_data = data_transpose_of_original(:,index)
However, don't transpose your original matrix inside MATLAB, as it would take longer.
Or you can create your own sparse matrix format. Compressed sparse row (CSR) format is very fast for the row-wise access.
In conclusion, if you can change the format of your sparse matrix outside the MATLAB (take transpose of the original matrix, or make that into CSR format and create a MATLAB function to load/manipulate it.). Then you can access the element faster. Otherwise I cant think of any speedup.
