AI-Master-Book
  • about AI-Master-Book
  • AI Master Book
    • 이상치 탐지 with Python
    • 베이지안 뉴럴네트워크 (BNN) with Python
    • 그래프 뉴럴네트워크 (GNN) with Python
    • 데이터 마케팅 분석 with Python
  • LLM MASTER BOOK
    • OpenAI API 쿡북 with Python
    • 기초부터 심화까지 RAG 쿡북 with Python
  • LLMs
    • OpenAI API
      • 1️⃣ChatCompletion
      • 2️⃣DALL-E
      • 3️⃣Text to Speech
      • 4️⃣Video to Transcripts
      • 5️⃣Assistants API
      • 6️⃣Prompt Engineering
      • 7️⃣OpenAI New GPT-4o
    • LangChain
      • LangChain Basic
        • 1️⃣Basic Modules
        • 2️⃣Model I/O
        • 3️⃣Prompts
        • 4️⃣Chains
        • 5️⃣Agents
        • 6️⃣Tools
        • 7️⃣Memory
      • LangChain Intermediate
        • 1️⃣OpenAI LLM
        • 2️⃣Prompt Template
        • 3️⃣Retrieval
        • 4️⃣RAG ChatBot
        • 5️⃣RAG with Gemini
        • 6️⃣New Huggingface-LangChain
        • 7️⃣Huggingface Hub
        • 8️⃣SQL Agent & Chain
        • 9️⃣Expression Language(LCEL)
        • 🔟Llama3-8B with LangChain
      • LangChain Advanced
        • 1️⃣LLM Evaluation
        • 2️⃣RAG Evaluation with RAGAS
        • 3️⃣LangChain with RAGAS
        • 4️⃣RAG Paradigms
        • 5️⃣LangChain: Advance Techniques
        • 6️⃣LangChain with NeMo-Guardrails
        • 7️⃣LangChain vs. LlamaIndex
        • 8️⃣LangChain LCEL vs. LangGraph
    • LlamaIndex
      • LlamaIndex Basic
        • 1️⃣Introduction
        • 2️⃣Customization
        • 3️⃣Data Connectors
        • 4️⃣Documents & Nodes
        • 5️⃣Naive RAG
        • 6️⃣Advanced RAG
        • 7️⃣Llama3-8B with LlamaIndex
        • 8️⃣LlmaPack
      • LlamaIndex Intermediate
        • 1️⃣QueryEngine
        • 2️⃣Agent
        • 3️⃣Evaluation
        • 4️⃣Evaluation-Driven Development
        • 5️⃣Fine-tuning
        • 6️⃣Prompt Compression with LLMLingua
      • LlamaIndex Advanced
        • 1️⃣Agentic RAG: Router Engine
        • 2️⃣Agentic RAG: Tool Calling
        • 3️⃣Building Agent Reasoning Loop
        • 4️⃣Building Multi-document Agent
    • Hugging Face
      • Huggingface Basic
        • 1️⃣Datasets
        • 2️⃣Tokenizer
        • 3️⃣Sentence Embeddings
        • 4️⃣Transformers
        • 5️⃣Sentence Transformers
        • 6️⃣Evaluate
        • 7️⃣Diffusers
      • Huggingface Tasks
        • NLP
          • 1️⃣Sentiment Analysis
          • 2️⃣Zero-shot Classification
          • 3️⃣Aspect-Based Sentiment Analysis
          • 4️⃣Feature Extraction
          • 5️⃣Intent Classification
          • 6️⃣Topic Modeling: BERTopic
          • 7️⃣NER: Token Classification
          • 8️⃣Summarization
          • 9️⃣Translation
          • 🔟Text Generation
        • Audio & Tabular
          • 1️⃣Text-to-Speech: TTS
          • 2️⃣Speech Recognition: Whisper
          • 3️⃣Audio Classification
          • 4️⃣Tabular Qustaion & Answering
        • Vision & Multimodal
          • 1️⃣Image-to-Text
          • 2️⃣Text to Image
          • 3️⃣Image to Image
          • 4️⃣Text or Image-to-Video
          • 5️⃣Depth Estimation
          • 6️⃣Image Classification
          • 7️⃣Object Detection
          • 8️⃣Segmentatio
      • Huggingface Optimization
        • 1️⃣Accelerator
        • 2️⃣Bitsandbytes
        • 3️⃣Flash Attention
        • 4️⃣Quantization
        • 5️⃣Safetensors
        • 6️⃣Optimum-ONNX
        • 7️⃣Optimum-NVIDIA
        • 8️⃣Optimum-Intel
      • Huggingface Fine-tuning
        • 1️⃣Transformer Fine-tuning
        • 2️⃣PEFT Fine-tuning
        • 3️⃣PEFT: Fine-tuning with QLoRA
        • 4️⃣PEFT: Fine-tuning Phi-2 with QLoRA
        • 5️⃣Axoltl Fine-tuning with QLoRA
        • 6️⃣TRL: RLHF Alignment Fine-tuning
        • 7️⃣TRL: DPO Fine-tuning with Phi-3-4k-instruct
        • 8️⃣TRL: ORPO Fine-tuning with Llama3-8B
        • 9️⃣Convert GGUF gemma-2b with llama.cpp
        • 🔟Apple Silicon Fine-tuning Gemma-2B with MLX
        • 🔢LLM Mergekit
    • Agentic LLM
      • Agentic LLM
        • 1️⃣Basic Agentic LLM
        • 2️⃣Multi-agent with CrewAI
        • 3️⃣LangGraph: Multi-agent Basic
        • 4️⃣LangGraph: Agentic RAG with LangChain
        • 5️⃣LangGraph: Agentic RAG with Llama3-8B by Groq
      • Autonomous Agent
        • 1️⃣LLM Autonomous Agent?
        • 2️⃣AutoGPT: Worldcup Winner Search with LangChain
        • 3️⃣BabyAGI: Weather Report with LangChain
        • 4️⃣AutoGen: Writing Blog Post with LangChain
        • 5️⃣LangChain: Autonomous-agent Debates with Tools
        • 6️⃣CAMEL Role-playing Autonomous Cooperative Agents
        • 7️⃣LangChain: Two-player Harry Potter D&D based CAMEL
        • 8️⃣LangChain: Multi-agent Bid for K-Pop Debate
        • 9️⃣LangChain: Multi-agent Authoritarian Speaker Selection
        • 🔟LangChain: Multi-Agent Simulated Environment with PettingZoo
    • Multimodal
      • 1️⃣PaliGemma: Open Vision LLM
      • 2️⃣FLUX.1: Generative Image
    • Building LLM
      • 1️⃣DSPy
      • 2️⃣DSPy RAG
      • 3️⃣DSPy with LangChain
      • 4️⃣Mamba
      • 5️⃣Mamba RAG with LangChain
      • 7️⃣PostgreSQL VectorDB with pgvorco.rs
Powered by GitBook
On this page
  • Topic Modeling: BERTopic
  • BERTopic
  • Dataset
  • Return Result
  • Visualization
  • Topic Frequnt Count
  • Model Save & Load
  • Using Huggingface
  1. LLMs
  2. Hugging Face
  3. Huggingface Tasks
  4. NLP

Topic Modeling: BERTopic

Topic Modeling: BERTopic

토픽 모델링은 텍스트 내에서 추상적인 토픽을 발견하는 데 초점을 맞춘 NLP의 하위 분야입니다. 주요 목표는 대규모 텍스트 코퍼스에서 숨겨진 주제 구조를 발견하여 비정형 텍스트의 대규모 데이터 세트를 더 쉽게 이해하고 정리하는 것입니다.

  • 문서에 존재하는 토픽을 식별함으로써 토픽 모델링을 사용해 유사한 문서를 분류하거나 클러스터링할 수 있습니다.

  • 검색 엔진은 토픽 분포를 기반으로 문서를 색인화하여 이를 활용할 수 있습니다.

  • 또한 토픽을 기반으로 유사한 기사나 논문을 추천하는 데 사용할 수 있습니다.

BERTopic

BERTopic은 BERT 임베딩과 클래스 기반 TF-IDF를 활용하여 밀도가 높은 클러스터를 생성하는 토픽 모델링 기법으로, 토픽 설명에서 중요한 단어를 유지하면서 쉽게 해석할 수 있는 토픽을 생성할 수 있습니다.

최첨단 언어 모델과 새로운 알고리즘 접근 방식을 사용함으로써 기존의 토픽 모델링 접근 방식과 차별화됩니다.

%pip install bertopic

Dataset

!mkdir dataset
!wget https://github.com/sharmaroshan/Twitter-Sentiment-Analysis/raw/master/train_tweet.csv -O ./dataset/tokyo_2020_tweets.csv
--2024-05-19 08:57:58--  https://github.com/sharmaroshan/Twitter-Sentiment-Analysis/raw/master/train_tweet.csv
Resolving github.com (github.com)... 20.200.245.247
Connecting to github.com (github.com)|20.200.245.247|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/sharmaroshan/Twitter-Sentiment-Analysis/master/train_tweet.csv [following]
--2024-05-19 08:57:59--  https://raw.githubusercontent.com/sharmaroshan/Twitter-Sentiment-Analysis/master/train_tweet.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3103165 (3.0M) [text/plain]
Saving to: ‘./dataset/tokyo_2020_tweets.csv’

./dataset/tokyo_202 100%[===================>]   2.96M  --.-KB/s    in 0.03s   

2024-05-19 08:58:00 (103 MB/s) - ‘./dataset/tokyo_2020_tweets.csv’ saved [3103165/3103165]
import pandas as pd 

df = pd.read_csv("dataset/tokyo_2020_tweets.csv", engine='python')
df.head()
id
label
tweet

0

1

0

@user when a father is dysfunctional and is s...

1

2

0

@user @user thanks for #lyft credit i can't us...

2

3

0

bihday your majesty

3

4

0

#model i love u take with u all the time in ...

4

5

0

factsguide: society now #motivation

docs = df[0:10000].tweet.to_list()
docs
[' @user when a father is dysfunctional and is so selfish he drags his kids into his dysfunction.   #run',
 "@user @user thanks for #lyft credit i can't use cause they don't offer wheelchair vans in pdx.    #disapointed #getthanked",
 '  bihday your majesty',
 '#model   i love u take with u all the time in urð\x9f\x93±!!! ð\x9f\x98\x99ð\x9f\x98\x8eð\x9f\x91\x84ð\x9f\x91\x85ð\x9f\x92¦ð\x9f\x92¦ð\x9f\x92¦  ',
 ' factsguide: society now    #motivation',
 '[2/2] huge fan fare and big talking before they leave. chaos and pay disputes when they get there. #allshowandnogo  ',]
from bertopic import BERTopic
model = BERTopic(verbose=True)
model.fit(docs)

topics, probabilities = model.transform(docs)
2024-05-19 09:00:04,747 - BERTopic - Embedding - Transforming documents to embeddings.
Batches: 100%|██████████| 313/313 [00:03<00:00, 82.45it/s] 
2024-05-19 09:00:10,998 - BERTopic - Embedding - Completed ✓
2024-05-19 09:00:11,000 - BERTopic - Dimensionality - Fitting the dimensionality reduction algorithm
2024-05-19 09:00:42,984 - BERTopic - Dimensionality - Completed ✓
2024-05-19 09:00:42,987 - BERTopic - Cluster - Start clustering the reduced embeddings
2024-05-19 09:00:43,454 - BERTopic - Cluster - Completed ✓
2024-05-19 09:00:43,462 - BERTopic - Representation - Extracting topics from clusters using representation models.
2024-05-19 09:00:43,724 - BERTopic - Representation - Completed ✓
Batches: 100%|██████████| 313/313 [00:02<00:00, 108.64it/s]
2024-05-19 09:00:47,024 - BERTopic - Dimensionality - Reducing dimensionality of input embeddings.
2024-05-19 09:00:47,055 - BERTopic - Dimensionality - Completed ✓
2024-05-19 09:00:47,056 - BERTopic - Clustering - Approximating new points with `hdbscan_model`
2024-05-19 09:00:47,437 - BERTopic - Cluster - Completed ✓

Return Result

model.get_topic_freq().head(10)
Topic
Count

1

-1

4119

27

0

346

9

1

303

10

2

198

11

3

166

14

4

155

29

5

144

35

6

142

2

7

131

13

8

125

model.get_topic(2)
[('fathers', 0.09351424852788441),
 ('fathersday', 0.07314833819088623),
 ('dad', 0.058599370842283095),
 ('day', 0.04736091349511204),
 ('father', 0.03951218907262783),
 ('dads', 0.03821332920448442),
 ('daddy', 0.02461901170173862),
 ('all', 0.023692468898816973),
 ('happy', 0.015781514379331344),
 ('man', 0.01359205432211061)]

Visualization

model.visualize_topics()
model.visualize_barchart()
model.visualize_heatmap()

Topic Frequnt Count

model = BERTopic(nr_topics=5) 
model.fit(docs)

topics, probs = model.transform(docs)
model.get_topic_freq().head(10)
Topic
Count

0

0

5517

1

-1

4226

2

1

206

4

2

28

3

3

23

Model Save & Load

model.save("new_model")

model = BERTopic.load("new_model")

Using Huggingface

from datasets import load_dataset

dataset = load_dataset("OpenAssistant/oasst1")
dataset
Downloading readme: 100%|██████████| 10.2k/10.2k [00:00<00:00, 8.87MB/s]
Downloading data: 100%|██████████| 39.5M/39.5M [00:04<00:00, 8.15MB/s]
Downloading data: 100%|██████████| 2.08M/2.08M [00:01<00:00, 1.98MB/s]
Generating train split: 100%|██████████| 84437/84437 [00:00<00:00, 117346.11 examples/s]
Generating validation split: 100%|██████████| 4401/4401 [00:00<00:00, 115404.20 examples/s]





DatasetDict({
    train: Dataset({
        features: ['message_id', 'parent_id', 'user_id', 'created_date', 'text', 'role', 'lang', 'review_count', 'review_result', 'deleted', 'rank', 'synthetic', 'model_name', 'detoxify', 'message_tree_id', 'tree_state', 'emojis', 'labels'],
        num_rows: 84437
    })
    validation: Dataset({
        features: ['message_id', 'parent_id', 'user_id', 'created_date', 'text', 'role', 'lang', 'review_count', 'review_result', 'deleted', 'rank', 'synthetic', 'model_name', 'detoxify', 'message_tree_id', 'tree_state', 'emojis', 'labels'],
        num_rows: 4401
    })
})
topic_model = BERTopic.load("davanstrien/chat_topics")

train_texts = [item['text'] for item in dataset['train']][:10]
topics.json: 100%|██████████| 263k/263k [00:00<00:00, 120MB/s]
config.json: 100%|██████████| 271/271 [00:00<00:00, 726kB/s]
topic_embeddings.safetensors: 100%|██████████| 230k/230k [00:00<00:00, 410kB/s]
modules.json: 100%|██████████| 349/349 [00:00<00:00, 924kB/s]
config_sentence_transformers.json: 100%|██████████| 116/116 [00:00<00:00, 340kB/s]
README.md: 100%|██████████| 10.6k/10.6k [00:00<00:00, 14.4MB/s]
sentence_bert_config.json: 100%|██████████| 53.0/53.0 [00:00<00:00, 145kB/s]
config.json: 100%|██████████| 571/571 [00:00<00:00, 1.69MB/s]
model.safetensors: 100%|██████████| 438M/438M [00:03<00:00, 115MB/s] 
tokenizer_config.json: 100%|██████████| 363/363 [00:00<00:00, 699kB/s]
vocab.txt: 100%|██████████| 232k/232k [00:00<00:00, 621kB/s]
tokenizer.json: 100%|██████████| 466k/466k [00:00<00:00, 832kB/s]
special_tokens_map.json: 100%|██████████| 239/239 [00:00<00:00, 696kB/s]
1_Pooling/config.json: 100%|██████████| 190/190 [00:00<00:00, 393kB/s]
train_topics, _ = topic_model.transform(train_texts)
train_topics
Batches: 100%|██████████| 1/1 [00:00<00:00,  9.16it/s]
2024-05-19 09:07:30,576 - BERTopic - Predicting topic assignments through cosine similarity of topic and document embeddings.





array([10, 10, 19, 10, 10, 10, 10, 10, 10, 52])
topic_model.get_topic_info(10)
Topic
Count
Name
CustomName
Representation
Representative_Docs

0

10

408

10_communism_capitalism_marx_economic

10_communism_capitalism_marx_economic

[communism, capitalism, marx, economic, econom...

NaN

train_texts[0]
'Can you write a short introduction about the relevance of the term "monopsony" in economics? Please use examples related to potential monopsonies in the labour market and cite relevant research.'
PreviousIntent ClassificationNextNER: Token Classification

Last updated 11 months ago

6️⃣