Complete Python Topics for Data Analysis: https://t.me/sqlspecialist/548
Natural Language Processing (NLP)
Natural Language Processing involves working with human language data, enabling computers to understand, interpret, and generate human-like text.
1. Text Preprocessing:
- Tokenization:
- Break text into words or phrases (tokens).
from nltk.tokenize import word_tokenize
text = "Natural Language Processing is fascinating!"
tokens = word_tokenize(text)
- Stopword Removal:
- Eliminate common words (stopwords) that often don't contribute much meaning.
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
filtered_tokens = [word for word in tokens if word.lower() not in stop_words]
2. Text Analysis:
- Frequency Analysis:
- Analyze the frequency of words in a text.
from nltk.probability import FreqDist
freq_dist = FreqDist(filtered_tokens)
- Word Clouds:
- Visualize word frequency using a word cloud.
from wordcloud import WordCloud
import matplotlib.pyplot as plt
wordcloud = WordCloud().generate_from_frequencies(freq_dist)
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()
3. Sentiment Analysis:
- VADER Sentiment Analysis:
- Assess the sentiment (positive, negative, neutral) of a piece of text.
from nltk.sentiment import SentimentIntensityAnalyzer
analyzer = SentimentIntensityAnalyzer()
sentiment_score = analyzer.polarity_scores("I love NLP!")
4. Named Entity Recognition (NER):
- Spacy for NER:
- Identify entities (names, locations, organizations) in text.
import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp("Apple Inc. is headquartered in Cupertino.")
for ent in doc.ents:
print(ent.text, ent.label_)
5. Topic Modeling:
- Latent Dirichlet Allocation (LDA):
- Identify topics within a collection of text documents.
from gensim import corpora, models
dictionary = corpora.Dictionary(documents)
corpus = [dictionary.doc2bow(text) for text in documents]
lda_model = models.LdaModel(corpus, num_topics=3, id2word=dictionary)
NLP is a vast field with applications ranging from chatbots to sentiment analysis.
Share with credits: https://t.me/sqlspecialist
Hope it helps :)