EDITED: Modified the whole answer after more details were given (code example and a better description of the requirements)
The best way to avoid the errors due to an auth token expired is to do the load quicker (5 hours for 16k is way too much time). This can be achieved in many ways, but after reviewing the code provided I created two nodejs scripts that can fulfill your needs of creating a "reversal linked list" without taking more than 1 minute to load the whole set of 16k records.
Explanation: Creating a record and reading it in the same transaction is an slow approach, it's better to create the records and then query the full data set and update accordingly. Also it's highly recommended to use batches for the writing and updating operations because the server side library does parallelized individual writes, more information about this can be found in the next link: Batched writes:
For bulk data entry, use a server client library with parallelized individual writes. Batched writes perform better than serialized writes but not better than parallel writes. You should use a server client library for bulk data operations and not a mobile/web SDK.
Script: The first part is to load the whole data set, I used the dummy load provided on your code to exemplify this, however you can replace the records variable with the loaded rows from your csv file. Also please note that I use a numeric index to fill the previous field, this is important to keep tracking on the order that we loaded the records.
batch_writing.js
const {Firestore} = require('@google-cloud/firestore');
const firestore = new Firestore();
async function batch_writting() {
var records = [];
let writeBatch = firestore.batch();
for(var i = 0; i < 17000; i++) {
records.push(`Filler: ${i}`);
}
let index = 0;
for (const record of records) {
let documentRef = firestore.collection('testing').doc();
let data = {
name: record,
previous: index,
created: Firestore.Timestamp.now()
}
writeBatch.create(documentRef,data);
if((index+1) % 500 === 0){
writeBatch.commit().then(() => {
console.log('Successfully executed batch.');
}).catch(e => console.error(e));
writeBatch = firestore.batch();
}
index++;
}
writeBatch.commit().then(() => {
console.log('Successfully executed batch.');
}).catch(e => console.error(e));
}
batch_writting().catch(e => console.error(e));
After the writing part is done and the previous field is populated with an incrementing index, we can then proceed to query the full recently created dataset and proceed to update the previous key with the correct Document ID from the previous row loaded. This is done by ordering the query by the previous field, that was populated with in incremental index. Please note that the first record will have a numeric value of 0 due to it not having a previous loaded row.
update_writing.js
const {Firestore} = require('@google-cloud/firestore');
const firestore = new Firestore();
async function batch_update(){
let query = firestore.collection('testing');
query.orderBy('previous', 'asc').get().then(querySnapshot => {
let writeBatch = firestore.batch();
let previousID = 0;
let index = 0;
querySnapshot.forEach(documentSnapshot => {
writeBatch.update(documentSnapshot.ref,{previous: previousID})
previousID = documentSnapshot.id;
if((index+1) % 500 === 0){
writeBatch.commit().then(() => {
console.log('Successfully executed batch.');
}).catch(e => console.error(e));
writeBatch = firestore.batch();
}
index++;
});
writeBatch.commit().then(() => {
console.log('Successfully executed batch.');
}).catch(e => console.error(e));
});
}
batch_update().catch(e => console.error(e));
If you need to query more fields or another collections, feel free to change the scripts, but remember to avoid querying an specific field and favor the queries that gather the full data set. Also if you need additional writes favor the use of batch writings.
