Spacy - NLTK: Language detection

Question

I am currently working on a project dealing with a bunch of social media posts. Some of these posts are in English and some in Spanish.

My current code runs quite smoothly. However, I am asking myself does Spacy/NLTK automatically detect which language stemmer/stopwords/etc. it has to use for each post (depending on whether it is an English or Spanish post)? At the moment, I am just parsing each post to a stemmer without explicitly specifying the language.

This is a snippet of my current script:

import re
import pandas as pd
!pip install pyphen
import pyphen
!pip install spacy
import spacy
!pip install nltk
import nltk
from nltk import SnowballStemmer
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
!pip install spacy-langdetect
from spacy_langdetect import LanguageDetector
!pip install textblob
from textblob import TextBlob

# Download Stopwords
nltk.download('stopwords')
stop_words_eng = set(stopwords.words('english'))
stop_words_es = set(stopwords.words('spanish'))

# Import Stemmer
p_stemmer = PorterStemmer()    
#Snowball (Porter2): Nearly universally regarded as an improvement over porter, and for good reason. 
snowball_stemmer = SnowballStemmer("english")
dic = pyphen.Pyphen(lang='en')

# Load Data
data = pd.read_csv("mergerfile.csv", error_bad_lines=False)
pd.set_option('display.max_columns', None)
posts = data.loc[data["ad_creative"] != "NONE"]

# Functions
def get_number_of_sentences(text):
    sentences = [sent.string.strip() for sent in text.sents]

    return len(sentences)


def get_average_sentence_length(text):
    number_of_sentences = get_number_of_sentences(text)
    tokens = [token.text for token in text]

    return len(tokens) / number_of_sentences


def get_token_length(text):
    tokens = [token.text for token in text]

    return len(tokens)

def text_analyzer(data_frame):
    content = []
    label = []
    avg_sentence_length = []
    number_sentences = []
    number_words = []

   for string in data_frame:
       string.join("")

       if len(string) <= 4:
           print(string)
           print("filtered")
           content.append(string)
           avg_sentence_length.append("filtered")
           number_sentences.append("filtered")
           number_words.append("filtered")

       else:
           # print list
           print(string)
           content.append(string)
           ##Average Sentence Lenght
           result = get_average_sentence_length(nlp(string))
           avg_sentence_length.append(result)
           print("avg sentence length:", result)

           ##Number of Sentences
           result = get_number_of_sentences(nlp(string))
           number_sentences.append(result)
           print("#sentences:", result)

           ##Number of words
           result = get_token_length(nlp(string))
           number_words.append(result)
           print("#Words", result)

content, avg_sentence_length, number_sentences, number_words = text_analyzer(
    data["posts"])

thorntonc thorntonc · Accepted Answer · 2020-08-07T03:08:35

Short answer is no, neither NLTK nor SpaCy will automatically determine the language and apply appropriate algorithms to a text.

SpaCy has separate language models with their own methods, part-of-speech and dependency tagsets. It also has a set of stopwords for each available language.

NLTK is more modular; for stemming there is RSLPStemmer (Portuguese), ISRIStemmer (Arabic), and SnowballStemmer (Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish).

When you determine the language of a post through spacy_langdetect, the next thing you should do is explicitly instruct to use the appropriate SpaCy language model or NLTK module.

Spacy - NLTK: Language detection

2 Answers

Use GoogleTrans Library for this

This Returns