in your project directory, add a folder called templates, same level as your apps such as users, blogs, etc. Inside of it, create another folder for two_factor. Inside of this folder, put your modified base.html.
If somehow you get error message, in your settings.py under TEMPLATES = [. Replace
'DIRS': [],
with
'DIRS': [os.path.join(BASE_DIR, 'templates')], # in django2
or
'DIRS': [BASE_DIR.joinpath('templates')],# in django3
Here is Doc about templates
Update after op's own answer
If you want to have two_factor/templates/two_factor/core/login.html use your own base.html, you should follow what I mentioned above, you will have templates/two_factor/core/login.html. Note that templates is in the same directory level as your other apps. In the templates folder, generate two_factor folder, then inside of it, generate core folder, then inside of that generate login.html. You can now modify it whichever you want to.
The reason that you want to override but not change things in the package directory is because when you update your package you have to make the changes again manually.
The reason that the approach above works is since we put 'DIRS': [os.path.join(BASE_DIR, 'templates')] in settings, Django will look for this folder first for templates. The detailed application of this method is here Overriding from the project’s templates directory