A Curated List of Dataset and Usable Library Resources for NLP in Bahasa Indonesia

NLP Bahasa Indonesia Resources

This repository provides link to useful dataset and another resources for NLP in Bahasa Indonesia.

Dictionary

Sentiment Words

1. (Negative) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/negatif_ta2.txt
2. (Negative) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/negative_add.txt
3. (Negative) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/negative_keyword.txt
4. (Negative) https://github.com/masdevid/ID-OpinionWords/blob/master/negative.txt
5. (Positive) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/positif_ta2.txt
6. (Positive) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/positive_add.txt
7. (Positive) https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/positive_keyword.txt
8. (Positive) https://github.com/masdevid/ID-OpinionWords/blob/master/positive.txt
9. (Score) https://github.com/agusmakmun/SentiStrengthID/blob/master/id_dict/sentimentword.txt
10. (InSet Lexicon) https://github.com/fajri91/InSet [Paper]

Position / Degree Words

1. https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/psuf.txt
2. https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/lldr.txt
3. https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/opos.txt
4. https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/ptit.txt

Root Words

1. https://github.com/agusmakmun/SentiStrengthID/blob/master/id_dict/rootword.txt
2. https://github.com/sastrawi/sastrawi/blob/master/data/kata-dasar.original.txt
3. https://github.com/sastrawi/sastrawi/blob/master/data/kata-dasar.txt
4. https://github.com/prasastoadi/serangkai/blob/master/serangkai/kamus/data/kamus-kata-dasar.csv

I have made the combined root words list from all of the above repositories.

Slang Words

1. https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/kbba.txt
2. https://github.com/agusmakmun/SentiStrengthID/blob/master/id_dict/slangword.txt
3. https://github.com/panggi/pujangga/blob/master/resource/formalization/formalizationDict.txt

I have made the combined slang words dictionary from all of the above repositories.

Stop Words

1. https://github.com/yasirutomo/python-sentianalysis-id/blob/master/data/feature_list/stopwordsID.txt
2. https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/stopword.txt
3. https://github.com/abhimantramb/elang/tree/master/word2vec/utils/stopwords-list

I have made the combined stop words list from all of the above repositories.

Emoticon

1. https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/emoticon.txt
2. https://github.com/jolicode/emoji-search/blob/master/synonyms/cldr-emoji-annotation-synonyms-id.txt
3. https://github.com/agusmakmun/SentiStrengthID/blob/master/id_dict/emoticon.txt

Acronym

1. https://github.com/ramaprakoso/analisis-sentimen/blob/master/kamus/acronym.txt
2. https://github.com/panggi/pujangga/blob/master/resource/sentencedetector/acronym.txt
3. https://id.wiktionary.org/wiki/Lampiran:Daftar_singkatan_dan_akronim_dalam_bahasa_Indonesia#A

Indonesia Region

1. https://github.com/abhimantramb/elang/blob/master/word2vec/utils/indonesian-region.txt
2. https://github.com/edwardsamuel/Wilayah-Administratif-Indonesia/tree/master/csv
3. https://github.com/pentagonal/Indonesia-Postal-Code/tree/master/Csv

Swear Words

1. https://github.com/abhimantramb/elang/blob/master/word2vec/utils/swear-words.txt

Composite Words

1. https://github.com/panggi/pujangga/blob/master/resource/tokenizer/compositewords.txt

Country

1. https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/country.txt

Region Words

1. https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/lpre.txt

Title of Name Words

1. https://github.com/panggi/pujangga/blob/master/resource/netagger/contextualfeature/ppre.txt

Gender by Name

1. https://github.com/seuriously/genderprediction/blob/master/namatraining.txt

Organization Words

1. https://github.com/panggi/pujangga/blob/master/resource/reference/opre.txt

Number Words

1. https://github.com/panggi/pujangga/blob/master/resource/netagger/morphologicalfeature/number.txt

Pre-trained word embedding

1. https://github.com/meisaputri21/Indonesian-Twitter-Emotion-Dataset. [Paper]
2. https://github.com/Kyubyong/wordvectors
3. https://drive.google.com/uc?id=0B5YTktu2dOKKNUY1OWJORlZTcUU&export=download
4. https://github.com/deryrahman/word2vec-bahasa-indonesia
5. https://sites.google.com/site/rmyeid/projects/polyglot

Train Word Embedding by Your Self

1. (FastText). https://structilmy.com/2019/08/membuat-model-word-embedding-fasttext-bahasa-indonesia/
2. (Word2Vec). https://yudiwbs.wordpress.com/2018/03/31/word2vec-wikipedia-bahasa-indonesia-dengan-python-gensim/

Usable Library

1. Pujangga: Indonesian Natural Language Processing REST API. https://github.com/panggi/pujangga
2. Sastrawi Stemmer Bahasa Indonesia. https://github.com/har07/PySastrawi
3. MorphInd: Indonesian Morphological Analyzer. http://septinalarasati.com/morphind/
4. INDRA: Indonesian Resource Grammar. https://github.com/davidmoeljadi/INDRA
5. Typo Checker. https://github.com/mamat-rahmat/checker_id
6. https://bagas.me/spacy-bahasa-indonesia.html
7. https://github.com/yohanesgultom/nlp-experiments
8. https://github.com/yasirutomo/python-sentianalysis-id
9. https://github.com/riochr17/Analisis-Sentimen-ID
10. https://github.com/yusufsyaifudin/indonesia-ner

Topic Analysis

1. (Introduction to LSA & LDA). https://monkeylearn.com/blog/introduction-to-topic-modeling/
2. (Introduction to LDA w/ Code & Tips). https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/
3. (Topic Modeling Methods Comparison Paper). https://thesai.org/Downloads/Volume6No1/Paper_21-A_Survey_of_Topic_Modeling_in_Text_Mining.pdf
4. (Original LDA Paper). http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf
5. (LDA Python Library). https://pypi.org/project/lda/; https://radimrehurek.com/gensim/models/ldamodel.html; https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html
6. (Original CTM Paper). http://people.ee.duke.edu/~lcarin/Blei2005CTM.pdf
7. (CTM Python Library). https://pypi.org/project/tomotopy/; https://github.com/kzhai/PyCTM
8. (Gaussian LDA Paper). https://www.aclweb.org/anthology/P15-1077.pdf
9. (Gaussian LDA Library). https://github.com/rajarshd/Gaussian_LDA
10. (Temporal Topic Modeling Comparison Paper). https://thesai.org/Downloads/Volume6No1/Paper_21-A_Survey_of_Topic_Modeling_in_Text_Mining.pdf
11. (TOT: A Non-Markov Continuous-Time Model of Topical Trends Paper). https://people.cs.umass.edu/~mccallum/papers/tot-kdd06s.pdf
12. (TOT Library). https://github.com/ahmaurya/topics_over_time
13. (Example of LDA in Bahasa Project Code). https://github.com/kirralabs/text-clustering

Translation

Sometimes there is an english word within our text and we have to translate it. We can exploit the english word dictionary provided here and we can use the Google Translate API for Python

Other Resources:

1. https://github.com/irfnrdh/Awesome-Indonesia-NLP