TypeORM bulk create with relations

Question

Is there a way to insert large amount of datas without blowing the JS heap memory ? I have a model which is Email as follow :

@Entity("email")
export class Email extends BaseEntity {

    @PrimaryGeneratedColumn()
    public id: number;

    @ManyToOne((type) => Category, (cat) => cat.category, {nullable: false, cascade: ['insert']})
    public category: Category;

    @Column({type: "text", name: "email"})
    public email: string;

}

and Category :

@Entity("category")
export class Category extends BaseEntity {

    @PrimaryGeneratedColumn()
    public id: number;

    @Column({type: "text", name: "category"})
    public category: string;

    @OneToMany((type) => Email, (email) => email.category, {nullable: true})
    public emails: Email[];

}

First problem I had is when I try to save {email: '[email protected]', category: 'default'} it says that Category must be an ID, but the thing is I want to add emails and create the category if it doesnt exist or asign the ID to the email if it exists. I did the following code :

 public async bulkCreate(emails: Email[]): Promise<any> {
        try {
            const emailRepo = await getRepository(Email);
            const categoryRepo = await getRepository(Category);
            await Promise.all(emails.map(async (mail) => {
                const cat = await categoryRepo.findOne({where: {category: mail.category}});
                if (cat) {
                    // @ts-ignore
                    mail.category = cat.id;
                } else {
                    const newCat = await categoryRepo.save(Object.assign(new Category(), mail));
                    // @ts-ignore
                    mail.category = newCat.id;
                }
                await emailRepo.save(mail);
            }));
        } catch (e) {
            console.log(e);
            throw new Error(e);
        }
    }

Worked for a few emails, but when I try to add even only 1,000 memory goes up to Like 4Gig and just crash.

What should I do? I'd like to add more than 1,000 emails at once.

Adams Adams · Accepted Answer · 2020-05-01T23:12:59

I know it's little bit late, but solution for this use Bluebird Promise.map so you can define concurrency. instead executing in one run.

TypeORM bulk create with relations

1 Answers