I have certain phrases in my corpus that I need spacy to disregard (with the hopes of avoiding overfitting). They're fairly simple regex formulas that I can also write in the format of Spacy's rule based matching like here.
I'd like the matches to be marked as stop words in my model before going on to the NER and TextCat pipes. I see how to write the matcher, but I'm not sure how to incorporate it into my model? Do I just add it as a pipe?
Thanks!
Clever thought. Adding a rule matcher in pipeline should be fairly easy. Custom components are just functions with take a doc object as a parameter and return a possibly modified doc object. So basically, you would do something like: def my_component(doc): # matcher work goes here return doc
Keep in mind however that the is_stop attribute of Token is not writable, which just means that you won't be able to change it. You can of course set a custom Token extension but this would by no means be taken into consideration for NER labels predictions.
A simpler approach to the problem would be to disregard those words before creating the doc object. If you say that the expressions are just regular expressions, then Matcher would not do any further good to you.
Hope it helps :)
