rhetenor/awesome-awesomeness
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update render script and Makefile

Browse Source
This commit is contained in:
Jonas Zeunert
2024-04-22 21:54:39 +02:00
parent 2d63fe63cd
commit 4d0cd768f7
10975 changed files with 47095 additions and 4031084 deletions
Download Patch File Download Diff File Expand all files Collapse all files

216
terminal/nlp
View File

@@ -1,4 +1,4 @@
 awesome-nlp
 awesome-nlp
!Awesome (https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg) (https://github.com/sindresorhus/awesome)
@@ -8,7 +8,8 @@
Read this in English (./README.md), Traditional Chinese (./README-ZH-TW.md)
_Please read the contribution guidelines (contributing.md) before contributing. Please add your favourite NLP resource by raising a pull request (https://github.com/keonkim/awesome-nlp/pulls)_
_Please read the contribution guidelines (contributing.md) before contributing. Please add your favourite NLP resource by raising a pull request 
(https://github.com/keonkim/awesome-nlp/pulls)_
Contents
@@ -56,8 +57,8 @@
Research Summaries and Trends
⟡ NLP-Overview (https://nlpoverview.com/) is an up-to-date overview of deep learning techniques applied to NLP, including theory, implementations, applications, and state-of-the-art results. This is a great Deep
NLP Introduction for researchers.
⟡ NLP-Overview (https://nlpoverview.com/) is an up-to-date overview of deep learning techniques applied to NLP, including theory, implementations, applications, and state-of-the-art results. 
This is a great Deep NLP Introduction for researchers.
⟡ NLP-Progress (https://nlpprogress.com/) tracks the progress in Natural Language Processing, including the datasets and the current state-of-the-art for the most common NLP tasks
⟡ NLP's ImageNet moment has arrived (https://thegradient.pub/nlp-imagenet/)
⟡ ACL 2018 Highlights: Understanding Representation and Evaluation in More Challenging Settings (http://ruder.io/acl-2018-highlights/)
@@ -73,15 +74,16 @@
⟡ The Berkeley NLP Group (http://nlp.cs.berkeley.edu/index.shtml) - Notable contributions include a tool to reconstruct long dead languages, referenced here 
(https://www.bbc.com/news/science-environment-21427896) and by taking corpora from 637 languages currently spoken in Asia and the Pacific and recreating their descendant.
⟡ Language Technologies Institute, Carnegie Mellon University (http://www.cs.cmu.edu/~nasmith/nlp-cl.html) - Notable projects include Avenue Project (http://www.cs.cmu.edu/~avenue/), a syntax driven machine 
translation system for endangered languages like Quechua and Aymara and previously, Noah's Ark (http://www.cs.cmu.edu/~ark/) which created AQMAR (http://www.cs.cmu.edu/~ark/AQMAR/) to improve NLP tools for 
Arabic.
⟡ NLP research group, Columbia University
 (http://www1.cs.columbia.edu/nlp/index.cgi) - Responsible for creating BOLT ( interactive error handling for speech translation systems) and an un-named project to characterize laughter in dialogue.
⟡ The Center or Language and Speech Processing, John Hopkins University (http://clsp.jhu.edu/) - Recently in the news for developing speech recognition software to create a diagnostic test or Parkinson's 
Disease, here (https://www.clsp.jhu.edu/2019/03/27/speech-recognition-software-and-machine-learning-tools-are-being-used-to-create-diagnostic-test-for-parkinsons-disease/#.XNFqrIkzYdU).
⟡ Computational Linguistics and Information Processing Group, University of Maryland (https://wiki.umiacs.umd.edu/clip/index.php/Main_Page) - Notable contributions include Human-Computer Cooperation or 
Word-by-Word Question Answering (http://www.umiacs.umd.edu/~jbg/projects/IIS-1652666) and modeling development of phonetic representations. 
⟡ Language Technologies Institute, Carnegie Mellon University (http://www.cs.cmu.edu/~nasmith/nlp-cl.html) - Notable projects include Avenue Project (http://www.cs.cmu.edu/~avenue/), a syntax
driven machine translation system for endangered languages like Quechua and Aymara and previously, Noah's Ark (http://www.cs.cmu.edu/~ark/) which created AQMAR 
(http://www.cs.cmu.edu/~ark/AQMAR/) to improve NLP tools for Arabic.
⟡ NLP research group, Columbia University (http://www1.cs.columbia.edu/nlp/index.cgi) - Responsible for creating BOLT ( interactive error handling for speech translation systems) and an 
un-named project to characterize laughter in dialogue.
⟡ The Center or Language and Speech Processing, John Hopkins University (http://clsp.jhu.edu/) - Recently in the news for developing speech recognition software to create a diagnostic test or
Parkinson's Disease, here 
(https://www.clsp.jhu.edu/2019/03/27/speech-recognition-software-and-machine-learning-tools-are-being-used-to-create-diagnostic-test-for-parkinsons-disease/#.XNFqrIkzYdU).
⟡ Computational Linguistics and Information Processing Group, University of Maryland (https://wiki.umiacs.umd.edu/clip/index.php/Main_Page) - Notable contributions include Human-Computer 
Cooperation or Word-by-Word Question Answering (http://www.umiacs.umd.edu/~jbg/projects/IIS-1652666) and modeling development of phonetic representations. 
⟡ Penn Natural Language Processing, University of Pennsylvania (https://nlp.cis.upenn.edu/)- Famous for creating the Penn Treebank (https://www.seas.upenn.edu/~pdtb/).
⟡ The Stanford Nautral Language Processing Group (https://nlp.stanford.edu/)- One of the top NLP research labs in the world, notable for creating Stanford CoreNLP 
(https://nlp.stanford.edu/software/corenlp.shtml) and their coreference resolution system (https://nlp.stanford.edu/software/dcoref.shtml)
@@ -94,8 +96,8 @@
General Machine Learning
⟡ Machine Learning 101
 (https://docs.google.com/presentation/d/1kSuQyW5DTnkVaZEjGYCkfOxvzCqGEFzWBy4e9Uedd9k/edit?usp=sharing) from Google's Senior Creative Engineer explains Machine Learning for engineer's and executives alike
⟡ Machine Learning 101 (https://docs.google.com/presentation/d/1kSuQyW5DTnkVaZEjGYCkfOxvzCqGEFzWBy4e9Uedd9k/edit?usp=sharing) from Google's Senior Creative Engineer explains Machine Learning 
for engineer's and executives alike
⟡ AI Playbook (https://aiplaybook.a16z.com/) - a16z AI playbook is a great link to forward to your managers or content for your presentations
⟡ Ruder's Blog (http://ruder.io/#open) by Sebastian Ruder (https://twitter.com/seb_ruder) for commentary on the best of NLP Research
⟡ How To Label Data (https://www.lighttag.io/how-to-label-data/) guide to managing larger linguistic annotation projects
@@ -116,7 +118,8 @@
Blogs and Newsletters
⟡ Deep Learning, NLP, and Representations (https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/)
⟡ The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) (https://jalammar.github.io/illustrated-bert/) and The Illustrated Transformer (https://jalammar.github.io/illustrated-transformer/)
⟡ The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) (https://jalammar.github.io/illustrated-bert/) and The Illustrated Transformer 
(https://jalammar.github.io/illustrated-transformer/)
⟡ Natural Language Processing (https://nlpers.blogspot.com/) by Hal Daumé III
⟡ arXiv: Natural Language Processing (Almost) from Scratch (https://arxiv.org/pdf/1103.0398.pdf)
⟡ Karpathy's The Unreasonable Effectiveness of Recurrent Neural Networks (https://karpathy.github.io/2015/05/21/rnn-effectiveness)
@@ -130,16 +133,16 @@
⟡ Deep Natural Language Processing (https://github.com/oxford-cs-deepnlp-2017/lectures) - Lectures series from Oxford
⟡ Deep Learning for Natural Language Processing (cs224-n) (https://web.stanford.edu/class/cs224n/) - Richard Socher and Christopher Manning's Stanford Course
⟡ Neural Networks for NLP (http://phontron.com/class/nn4nlp2017/) - Carnegie Mellon Language Technology Institute there
⟡ Deep NLP Course
 (https://github.com/yandexdataschool/nlp_course) by Yandex Data School, covering important ideas from text embedding to machine translation including sequence modeling, language models and so on.
⟡ fast.ai Code-First Intro to Natural Language Processing (https://www.fast.ai/2019/07/08/fastai-nlp/) - This covers a blend of traditional NLP topics (including regex, SVD, naive bayes, tokenization) and recent
neural network approaches (including RNNs, seq2seq, GRUs, and the Transformer), as well as addressing urgent ethical issues, such as bias and disinformation. Find the Jupyter Notebooks here 
(https://github.com/fastai/course-nlp)
⟡ Deep NLP Course (https://github.com/yandexdataschool/nlp_course) by Yandex Data School, covering important ideas from text embedding to machine translation including sequence modeling, 
language models and so on.
⟡ fast.ai Code-First Intro to Natural Language Processing (https://www.fast.ai/2019/07/08/fastai-nlp/) - This covers a blend of traditional NLP topics (including regex, SVD, naive bayes, 
tokenization) and recent neural network approaches (including RNNs, seq2seq, GRUs, and the Transformer), as well as addressing urgent ethical issues, such as bias and disinformation. Find the
Jupyter Notebooks here (https://github.com/fastai/course-nlp)
⟡ Machine Learning University - Accelerated Natural Language Processing
 (https://www.youtube.com/playlist?list=PL8P_Z6C4GcuWfAq8Pt6PBYlck4OprHXsw) - Lectures go from introduction to NLP and text processing to Recurrent Neural Networks and Transformers.
Material can be found here (https://github.com/aws-samples/aws-machine-learning-university-accelerated-nlp).
⟡ Applied Natural Language Processing (https://www.youtube.com/playlist?list=PLH-xYrxjfO2WyR3pOAB006CYMhNt4wTqp)- Lecture series from IIT Madras taking from the basics all the way to autoencoders and everything.
The github notebooks for this course are also available here (https://github.com/Ramaseshanr/anlp)
⟡ Applied Natural Language Processing (https://www.youtube.com/playlist?list=PLH-xYrxjfO2WyR3pOAB006CYMhNt4wTqp)- Lecture series from IIT Madras taking from the basics all the way to 
autoencoders and everything. The github notebooks for this course are also available here (https://github.com/Ramaseshanr/anlp)
Books
@@ -171,24 +174,25 @@
⟡  | Back to Top (#contents)
 - sentimental-onix (https://github.com/sloev/sentimental-onix) Sentiment models for spacy using onnx
 - TextAttack (https://github.com/QData/TextAttack) - Adversarial attacks, adversarial training, and data augmentation in NLP
 - TextBlob (http://textblob.readthedocs.org/) - Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of Natural Language Toolkit (NLTK) 
(https://www.nltk.org/) and Pattern (https://github.com/clips/pattern), and plays nicely with both :+1:
 - TextBlob (http://textblob.readthedocs.org/) - Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of Natural Language 
Toolkit (NLTK) (https://www.nltk.org/) and Pattern (https://github.com/clips/pattern), and plays nicely with both :+1:
 - spaCy (https://github.com/explosion/spaCy) - Industrial strength NLP with Python and Cython :+1:
 - Speedster (https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster) - Automatically apply SOTA optimization techniques to achieve the maximum inference speed-up on your hardware
 - Speedster (https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster) - Automatically apply SOTA optimization techniques to achieve the maximum inference speed-up on your 
hardware
- **textacy** (https://github.com/chartbeat-labs/textacy) - Higher level NLP built on spaCy
 - gensim (https://radimrehurek.com/gensim/index.html) - Python library to conduct unsupervised semantic modelling from plain text :+1:
 - scattertext (https://github.com/JasonKessler/scattertext) - Python library to produce d3 visualizations of how language differs between corpora
 - GluonNLP (https://github.com/dmlc/gluon-nlp) - A deep learning toolkit for NLP, built on MXNet/Gluon, for research prototyping and industrial deployment of state-of-the-art models on a wide range of NLP 
tasks.
 - GluonNLP (https://github.com/dmlc/gluon-nlp) - A deep learning toolkit for NLP, built on MXNet/Gluon, for research prototyping and industrial deployment of state-of-the-art models on a 
wide range of NLP tasks.
 - AllenNLP (https://github.com/allenai/allennlp) - An NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.
 - PyTorch-NLP (https://github.com/PetrochukM/PyTorch-NLP) - NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer representations, 
common NLP metrics such as BLEU
 - PyTorch-NLP (https://github.com/PetrochukM/PyTorch-NLP) - NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer 
representations, common NLP metrics such as BLEU
 - Rosetta (https://github.com/columbia-applied-data-science/rosetta) - Text processing tools and wrappers (e.g. Vowpal Wabbit)
 - PyNLPl (https://github.com/proycon/pynlpl) - Python Natural Language Processing Library. General purpose NLP library for Python, handles some specific formats like ARPA language models, Moses phrasetables, 
GIZA++ alignments.
 - PyNLPl (https://github.com/proycon/pynlpl) - Python Natural Language Processing Library. General purpose NLP library for Python, handles some specific formats like ARPA language models, 
Moses phrasetables, GIZA++ alignments.
 - foliapy (https://github.com/proycon/foliapy) - Python library for working with FoLiA (https://proycon.github.io/folia/), an XML format for linguistic annotation.
 - PySS3 (https://github.com/sergioburdisso/pyss3) - Python package that implements a novel white-box machine learning model for text classification, called SS3. Since SS3 has the ability to visually explain 
its rationale, this package also comes with easy-to-use interactive visualizations tools (online demos (http://tworld.io/ss3/)).
 - PySS3 (https://github.com/sergioburdisso/pyss3) - Python package that implements a novel white-box machine learning model for text classification, called SS3. Since SS3 has the ability to
visually explain its rationale, this package also comes with easy-to-use interactive visualizations tools (online demos (http://tworld.io/ss3/)).
 - jPTDP (https://github.com/datquocnguyen/jPTDP) - A toolkit for joint part-of-speech (POS) tagging and dependency parsing. jPTDP provides pre-trained models for 40+ languages.
 - BigARTM (https://github.com/bigartm/bigartm) - a fast library for topic modelling
 - Snips NLU (https://github.com/snipsco/snips-nlu) - A production ready library for intent parsing
@@ -198,13 +202,13 @@
 - Natural Language Toolkit (NLTK) (https://www.nltk.org/) - A library containing a wide variety of NLP functionality, supporting over 50 corpora.
 - NLP Architect (https://github.com/NervanaSystems/nlp-architect) - A library for exploring the state-of-the-art deep learning topologies and techniques for NLP and NLU
 - Flair (https://github.com/zalandoresearch/flair) - A very simple framework for state-of-the-art multilingual NLP built on PyTorch. Includes BERT, ELMo and Flair embeddings.
 - Kashgari (https://github.com/BrikerMan/Kashgari) - Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), part-of-speech tagging 
(PoS) and text classification tasks. Includes BERT and word2vec embedding.
 - Kashgari (https://github.com/BrikerMan/Kashgari) - Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), 
part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding.
 - FARM (https://github.com/deepset-ai/FARM) - Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
 - Haystack (https://github.com/deepset-ai/haystack) - End-to-end Python framework for building natural language search interfaces to data. Leverages Transformers and the State-of-the-Art of NLP. Supports DPR, 
Elasticsearch, HuggingFace’s Modelhub, and much more!
 - Rita DSL (https://github.com/zaibacu/rita-dsl) - a DSL, loosely based on RUTA on Apache UIMA (https://uima.apache.org/ruta.html). Allows to define language patterns (rule-based NLP) which are then translated
into spaCy (https://spacy.io/), or if you prefer less features and lightweight - regex patterns.
 - Haystack (https://github.com/deepset-ai/haystack) - End-to-end Python framework for building natural language search interfaces to data. Leverages Transformers and the State-of-the-Art of
NLP. Supports DPR, Elasticsearch, HuggingFace’s Modelhub, and much more!
 - Rita DSL (https://github.com/zaibacu/rita-dsl) - a DSL, loosely based on RUTA on Apache UIMA (https://uima.apache.org/ruta.html). Allows to define language patterns (rule-based NLP) which
are then translated into spaCy (https://spacy.io/), or if you prefer less features and lightweight - regex patterns.
 - Transformers (https://github.com/huggingface/transformers) - Natural Language Processing for TensorFlow 2.0 and PyTorch.
 - Tokenizers (https://github.com/huggingface/tokenizers) - Tokenizers optimized for Research and Production.
 - fairSeq (https://github.com/pytorch/fairseq) Facebook AI Research implementations of SOTA seq2seq models in Pytorch. 
@@ -218,11 +222,12 @@
- | Back to Top (#contents)
 - InsNet (https://github.com/chncwang/InsNet) - A neural network library for building instance-dependent NLP models with padding-free dynamic batching.
 - MIT Information Extraction Toolkit (https://github.com/mit-nlp/MITIE) - C, C++, and Python tools for named entity recognition and relation extraction
 - CRF++ (https://taku910.github.io/crfpp/) - Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing tasks.
 - CRF++ (https://taku910.github.io/crfpp/) - Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing 
tasks.
 - CRFsuite (http://www.chokkan.org/software/crfsuite/) - CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data.
 - BLLIP Parser (https://github.com/BLLIP/bllip-parser) - BLLIP Natural Language Parser (also known as the Charniak-Johnson parser)
 - colibri-core (https://github.com/proycon/colibri-core) - C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams and skipgrams in a 
quick and memory-efficient way.
 - colibri-core (https://github.com/proycon/colibri-core) - C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams
and skipgrams in a quick and memory-efficient way.
 - ucto (https://github.com/LanguageMachines/ucto) - Unicode-aware regular-expression based tokenizer for various languages. Tool and C++ library. Supports FoLiA format.
 - libfolia (https://github.com/LanguageMachines/libfolia) - C++ library for the FoLiA format (https://proycon.github.io/folia/)
 - frog (https://github.com/LanguageMachines/frog) - Memory-based NLP suite developed for Dutch: PoS tagger, lemmatiser, dependency parser, NER, shallow parser, morphological analyzer.
@@ -239,8 +244,8 @@
 - ReVerb (https://github.com/knowitall/reverb/) Web-Scale Open Information Extraction
 - OpenRegex (https://github.com/knowitall/openregex) An efficient and flexible token-based regular expression language and engine.
 - CogcompNLP (https://github.com/CogComp/cogcomp-nlp) - Core libraries developed in the U of Illinois' Cognitive Computation Group.
 - MALLET (http://mallet.cs.umass.edu/) - MAchine Learning for LanguagE Toolkit - package for statistical natural language processing, document classification, clustering, topic modeling, information 
extraction, and other machine learning applications to text.
 - MALLET (http://mallet.cs.umass.edu/) - MAchine Learning for LanguagE Toolkit - package for statistical natural language processing, document classification, clustering, topic modeling, 
information extraction, and other machine learning applications to text.
 - RDRPOSTagger (https://github.com/datquocnguyen/RDRPOSTagger) - A robust POS tagging toolkit available (in both Java & Python) together with pre-trained models for 40+ languages.
- | Back to Top (#contents)
@@ -253,8 +258,8 @@
 - tm (https://github.com/ispras/tm) - Implementation of topic modeling based on regularized multilingual PLSA (https://en.wikipedia.org/wiki/Probabilistic_latent_semantic_analysis).
 - word2vec-scala (https://github.com/Refefer/word2vec-scala) - Scala interface to word2vec model; includes operations on vectors like word-distance and word-analogy.
 - Epic (https://github.com/dlwh/epic) - Epic is a high performance statistical parser written in Scala, along with a framework for building complex structured prediction models.
 - Spark NLP (https://github.com/JohnSnowLabs/spark-nlp) - Spark NLP is a natural language processing library built on top of Apache Spark ML that provides simple, performant & accurate NLP annotations for 
machine learning pipelines that scale easily in a distributed environment.
 - Spark NLP (https://github.com/JohnSnowLabs/spark-nlp) - Spark NLP is a natural language processing library built on top of Apache Spark ML that provides simple, performant & accurate NLP 
annotations for machine learning pipelines that scale easily in a distributed environment.
- | Back to Top (#contents)
 - text2vec (https://github.com/dselivanov/text2vec) - Fast vectorization, topic modeling, distances and GloVe word embeddings in R.
@@ -304,15 +309,17 @@
- Wit-ai (https://github.com/wit-ai/wit) - Natural Language Interface for apps and devices
- IBM Watson's Natural Language Understanding (https://github.com/watson-developer-cloud/natural-language-understanding-nodejs) - API and Github demo
- Amazon Comprehend (https://aws.amazon.com/comprehend/) - NLP and ML suite covers most common tasks like NER, tagging, and sentiment analysis
- Google Cloud Natural Language API (https://cloud.google.com/natural-language/) - Syntax Analysis, NER, Sentiment Analysis, and Content tagging in atleast 9 languages include English and Chinese (Simplified and
Traditional).
- Google Cloud Natural Language API (https://cloud.google.com/natural-language/) - Syntax Analysis, NER, Sentiment Analysis, and Content tagging in atleast 9 languages include English and 
Chinese (Simplified and Traditional).
- ParallelDots (https://www.paralleldots.com/text-analysis-apis) - High level Text Analysis API Service ranging from Sentiment Analysis to Intent Analysis
- Microsoft Cognitive Service (https://azure.microsoft.com/en-us/services/cognitive-services/text-analytics/)
- TextRazor (https://www.textrazor.com/)
- Rosette (https://www.rosette.com/)
- Textalytic (https://www.textalytic.com) - Natural Language Processing in the Browser with sentiment analysis, named entity extraction, POS tagging, word frequencies, topic modeling, word clouds, and more
- Textalytic (https://www.textalytic.com) - Natural Language Processing in the Browser with sentiment analysis, named entity extraction, POS tagging, word frequencies, topic modeling, word 
clouds, and more
- NLP Cloud (https://nlpcloud.io) - SpaCy NLP models (custom and pre-trained ones) served through a RESTful API for named entity recognition (NER), POS tagging, and more.
- Cloudmersive (https://cloudmersive.com/nlp-api) - Unified and free NLP APIs that perform actions such as speech tagging, text rephrasing, language translation/detection, and sentence parsing
- Cloudmersive (https://cloudmersive.com/nlp-api) - Unified and free NLP APIs that perform actions such as speech tagging, text rephrasing, language translation/detection, and sentence 
parsing
Annotation Tools
@@ -329,14 +336,14 @@
- Label Studio (https://www.heartex.ai/) - Hosted and managed text annotation tool for teams, freemium based, costs $
- Datasaur (https://datasaur.ai/) support various NLP tasks for individual or teams, freemium based
- Konfuzio (https://konfuzio.com/en/) - team-first hosted and on-prem text, image and PDF annotation tool powered by active learning, freemium based, costs $
- UBIAI (https://ubiai.tools/) - Easy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as OCR annotation for 
invoice labeling, costs $
- Shoonya (https://github.com/AI4Bharat/Shoonya-Backend) - Shoonya is free and open source data annotation platform with wide varities of organization and workspace level management system. Shoonya is data 
agnostic, can be used by teams to annotate data with various level of verification stages at scale.
- Annotation Lab (https://www.johnsnowlabs.com/annotation-lab/) - Free End-to-End No-Code platform for text annotation and DL model training/tuning. Out-of-the-box support for Named Entity Recognition, 
Classification, Relation extraction and Assertion Status Spark NLP models. Unlimited support for users, teams, projects, documents. Not FOSS. 
- FLAT (https://github.com/proycon/flat) - FLAT is a web-based linguistic annotation environment based around the FoLiA format (http://proycon.github.io/folia), a rich XML-based format for linguistic annotation.
Free and open source.
- UBIAI (https://ubiai.tools/) - Easy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as 
OCR annotation for invoice labeling, costs $
- Shoonya (https://github.com/AI4Bharat/Shoonya-Backend) - Shoonya is free and open source data annotation platform with wide varities of organization and workspace level management system. 
Shoonya is data agnostic, can be used by teams to annotate data with various level of verification stages at scale.
- Annotation Lab (https://www.johnsnowlabs.com/annotation-lab/) - Free End-to-End No-Code platform for text annotation and DL model training/tuning. Out-of-the-box support for Named Entity 
Recognition, Classification, Relation extraction and Assertion Status Spark NLP models. Unlimited support for users, teams, projects, documents. Not FOSS. 
- FLAT (https://github.com/proycon/flat) - FLAT is a web-based linguistic annotation environment based around the FoLiA format (http://proycon.github.io/folia), a rich XML-based format for 
linguistic annotation. Free and open source.
Techniques
@@ -347,17 +354,18 @@
- Thumb Rule: fastText >> GloVe > word2vec
- word2vec (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) - implementation (https://code.google.com/archive/p/word2vec/) - explainer blog 
(http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/)
- word2vec (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) - implementation (https://code.google.com/archive/p/word2vec/) -
explainer blog (http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/)
- glove (https://nlp.stanford.edu/pubs/glove.pdf) - explainer blog (https://blog.acolyer.org/2016/04/22/glove-global-vectors-for-word-representation/)
- fasttext - implementation (https://github.com/facebookresearch/fastText) - paper (https://arxiv.org/abs/1607.04606) - explainer blog (https://towardsdatascience.com/fasttext-under-the-hood-11efc57b2b3)
- fasttext - implementation (https://github.com/facebookresearch/fastText) - paper (https://arxiv.org/abs/1607.04606) - explainer blog 
(https://towardsdatascience.com/fasttext-under-the-hood-11efc57b2b3)
Sentence and Language Model Based Word Embeddings
Back to Top (#contents)
- ElMo - Deep Contextualized Word Representations (https://arxiv.org/abs/1802.05365) - PyTorch implmentation (https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md) - TF Implementation 
(https://github.com/allenai/bilm-tf)
- ElMo - Deep Contextualized Word Representations (https://arxiv.org/abs/1802.05365) - PyTorch implmentation (https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md) - TF 
Implementation (https://github.com/allenai/bilm-tf)
- ULMFiT - Universal Language Model Fine-tuning for Text Classification (https://arxiv.org/abs/1801.06146) by Jeremy Howard and Sebastian Ruder
- InferSent - Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (https://arxiv.org/abs/1705.02364) by facebook
- CoVe - Learned in Translation: Contextualized Word Vectors (https://arxiv.org/abs/1708.00107)
@@ -388,12 +396,13 @@
Back to Top (#contents)
- UDPipe (https://github.com/ufal/udpipe) is a trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files. Primarily written in C++, offers a fast and 
reliable solution for multilingual NLP processing.
- NLP-Cube (https://github.com/adobe/NLP-Cube) : Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New platform, written in 
Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API).
- UralicNLP (https://github.com/mikahama/uralicNLP) is an NLP library mostly for many endangered Uralic languages such as Sami languages, Mordvin languages, Mari languages, Komi languages and so on. Also some 
non-endangered languages are supported such as Finnish together with non-Uralic languages such as Swedish and Arabic. UralicNLP can do morphological analysis, generation, lemmatization and disambiguation.
- UDPipe (https://github.com/ufal/udpipe) is a trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files. Primarily written in C++, 
offers a fast and reliable solution for multilingual NLP processing.
- NLP-Cube (https://github.com/adobe/NLP-Cube) : Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New 
platform, written in Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API).
- UralicNLP (https://github.com/mikahama/uralicNLP) is an NLP library mostly for many endangered Uralic languages such as Sami languages, Mordvin languages, Mari languages, Komi languages and
so on. Also some non-endangered languages are supported such as Finnish together with non-Uralic languages such as Swedish and Arabic. UralicNLP can do morphological analysis, generation, 
lemmatization and disambiguation.
NLP in Korean
@@ -466,8 +475,8 @@
Libraries
- spanlp (https://github.com/jfreddypuentes/spanlp) - Python library to detect, censor and clean profanity, vulgarities, hateful words, racism, xenophobia and bullying in texts written in Spanish. It contains 
data of 21 Spanish-speaking countries.
- spanlp (https://github.com/jfreddypuentes/spanlp) - Python library to detect, censor and clean profanity, vulgarities, hateful words, racism, xenophobia and bullying in texts written in 
Spanish. It contains data of 21 Spanish-speaking countries.
Data
@@ -504,8 +513,8 @@
Corpora/Datasets that need a login/access can be gained via email
- SAIL 2015 (http://amitavadas.com/SAIL/) Twitter and Facebook labelled sentiment samples in Hindi, Bengali, Tamil, Telugu.
- IIT Bombay NLP Resources (http://www.cfilt.iitb.ac.in/Sentiment_Analysis_Resources.html) Sentiwordnet, Movie and Tourism parallel labelled corpora, polarity labelled sense annotated corpus, Marathi polarity 
labelled corpus.
- IIT Bombay NLP Resources (http://www.cfilt.iitb.ac.in/Sentiment_Analysis_Resources.html) Sentiwordnet, Movie and Tourism parallel labelled corpora, polarity labelled sense annotated corpus,
Marathi polarity labelled corpus.
- TDIL-IC aggregates a lot of useful resources and provides access to otherwise gated datasets (https://tdil-dc.in/index.php?option=com_catalogue&task=viewTools&id=83&lang=en)
Language Models and Word Embeddings
@@ -521,9 +530,10 @@
- Multi-Task Deep Morphological Analyzer (https://github.com/Saurav0074/mt-dma) Deep Network based Morphological Parser for Hindi and Urdu
- Anoop Kunchukuttan (https://github.com/anoopkunchukuttan/indic_nlp_library) 18 Languages, whole host of features from tokenization to translation
- SivaReddy's Dependency Parser (http://sivareddy.in/downloads) Dependency Parser and Pos Tagger for Kannada, Hindi and Telugu. Python3 Port (https://github.com/CalmDownKarm/sivareddydependencyparser)
- iNLTK (https://github.com/goru001/inltk) - A Natural Language Toolkit for Indic Languages (Indian subcontinent languages) built on top of Pytorch/Fastai, which aims to provide out of the box support for common
NLP tasks.
- SivaReddy's Dependency Parser (http://sivareddy.in/downloads) Dependency Parser and Pos Tagger for Kannada, Hindi and Telugu. Python3 Port 
(https://github.com/CalmDownKarm/sivareddydependencyparser)
- iNLTK (https://github.com/goru001/inltk) - A Natural Language Toolkit for Indic Languages (Indian subcontinent languages) built on top of Pytorch/Fastai, which aims to provide out of the 
box support for common NLP tasks.
NLP in Thai
@@ -566,8 +576,8 @@
- VIVOS (https://ailab.hcmus.edu.vn/vivos/) - a free Vietnamese speech corpus consisting of 15 hours of recording speech by AILab
- VNTQcorpus(big).txt (http://viet.jnlp.org/download-du-lieu-tu-vung-corpus) - 1.75 million sentences in news
- ViText2SQL (https://github.com/VinAIResearch/ViText2SQL) - A dataset for Vietnamese Text-to-SQL semantic parsing (EMNLP-2020 Findings)
- EVB Corpus (https://github.com/qhungngo/EVBCorpus) - 20,000,000 words (20 million) from 15 bilingual books, 100 parallel English-Vietnamese / Vietnamese-English texts, 250 parallel law and ordinance texts, 
5,000 news articles, and 2,000 film subtitles.
- EVB Corpus (https://github.com/qhungngo/EVBCorpus) - 20,000,000 words (20 million) from 15 bilingual books, 100 parallel English-Vietnamese / Vietnamese-English texts, 250 parallel law and 
ordinance texts, 5,000 news articles, and 2,000 film subtitles.
NLP for Dutch
@@ -613,34 +623,35 @@
Libraries
- Hazm (https://github.com/roshan-research/hazm) - Persian NLP Toolkit.
- Parsivar (https://github.com/ICTRC/Parsivar): A Language Processing Toolkit for Persian
- Perke (https://github.com/AlirezaTheH/perke): Perke is a Python keyphrase extraction package for Persian language. It provides an end-to-end keyphrase extraction pipeline in which each component can be easily 
modified or extended to develop new models.
- Perke (https://github.com/AlirezaTheH/perke): Perke is a Python keyphrase extraction package for Persian language. It provides an end-to-end keyphrase extraction pipeline in which each 
component can be easily modified or extended to develop new models.
- Perstem (https://github.com/jonsafari/perstem): Persian stemmer, morphological analyzer, transliterator, and partial part-of-speech tagger
- ParsiAnalyzer (https://github.com/NarimanN2/ParsiAnalyzer): Persian Analyzer For Elasticsearch
- virastar (https://github.com/aziz/virastar): Cleaning up Persian text!
Datasets
- Bijankhan Corpus (https://dbrg.ut.ac.ir/بیژن%E2%80%8Cخان/): Bijankhan corpus is a tagged corpus that is suitable for natural language processing research on the Persian (Farsi) language. This collection is 
gathered form daily news and common texts. In this collection all documents are categorized into different subjects such as political, cultural and so on. Totally, there are 4300 different subjects. The 
Bijankhan collection contains about 2.6 millions manually tagged words with a tag set that contains 40 Persian POS tags.
- Uppsala Persian Corpus (UPC) (https://sites.google.com/site/mojganserajicom/home/upc): Uppsala Persian Corpus (UPC) is a large, freely available Persian corpus. The corpus is a modified version of the 
Bijankhan corpus with additional sentence segmentation and consistent tokenization containing 2,704,028 tokens and annotated with 31 part-of-speech tags. The part-of-speech tags are listed with explanations in 
this table (https://sites.google.com/site/mojganserajicom/home/upc/Table_tag.pdf).
- Large-Scale Colloquial Persian (http://hdl.handle.net/11234/1-3195): Large Scale Colloquial Persian Dataset (LSCP) is hierarchically organized in asemantic taxonomy that focuses on multi-task informal Persian 
language understanding as a comprehensive problem. LSCP includes 120M sentences from 27M casual Persian tweets with its dependency relations in syntactic annotation, Part-of-speech tags, sentiment polarity and 
automatic translation of original Persian sentences in English (EN), German (DE), Czech (CS), Italian (IT) and Hindi (HI) spoken languages. Learn more about this project at LSCP webpage 
(https://iasbs.ac.ir/~ansari/lscp/).
- ArmanPersoNERCorpus (https://github.com/HaniehP/PersianNER): The dataset includes 250,015 tokens and 7,682 Persian sentences in total. It is available in 3 folds to be used in turn as training and test sets. 
Each file contains one token, along with its manually annotated named-entity tag, per line. Each sentence is separated with a newline. The NER tags are in IOB format.
- FarsiYar PersianNER (https://github.com/Text-Mining/Persian-NER): The dataset includes about 25,000,000 tokens and about 1,000,000 Persian sentences in total based on Persian Wikipedia Corpus 
(https://github.com/Text-Mining/Persian-Wikipedia-Corpus). The NER tags are in IOB format. More than 1000 volunteers contributed tag improvements to this dataset via web panel or android app. They release 
updated tags every two weeks.
- PERLEX (http://farsbase.net/PERLEX.html): The first Persian dataset for relation extraction, which is an expert translated version of the “Semeval-2010-Task-8” dataset. Link to the relevant publication.
- Persian Syntactic Dependency Treebank (http://dadegan.ir/catalog/perdt): This treebank is supplied for free noncommercial use. For commercial uses feel free to contact us. The number of annotated sentences is 
29,982 sentences including samples from almost all verbs of the Persian valency lexicon.
- Bijankhan Corpus (https://dbrg.ut.ac.ir/بیژن%E2%80%8Cخان/): Bijankhan corpus is a tagged corpus that is suitable for natural language processing research on the Persian (Farsi) language. 
This collection is gathered form daily news and common texts. In this collection all documents are categorized into different subjects such as political, cultural and so on. Totally, there 
are 4300 different subjects. The Bijankhan collection contains about 2.6 millions manually tagged words with a tag set that contains 40 Persian POS tags.
- Uppsala Persian Corpus (UPC) (https://sites.google.com/site/mojganserajicom/home/upc): Uppsala Persian Corpus (UPC) is a large, freely available Persian corpus. The corpus is a modified 
version of the Bijankhan corpus with additional sentence segmentation and consistent tokenization containing 2,704,028 tokens and annotated with 31 part-of-speech tags. The part-of-speech 
tags are listed with explanations in this table (https://sites.google.com/site/mojganserajicom/home/upc/Table_tag.pdf).
- Large-Scale Colloquial Persian (http://hdl.handle.net/11234/1-3195): Large Scale Colloquial Persian Dataset (LSCP) is hierarchically organized in asemantic taxonomy that focuses on 
multi-task informal Persian language understanding as a comprehensive problem. LSCP includes 120M sentences from 27M casual Persian tweets with its dependency relations in syntactic 
annotation, Part-of-speech tags, sentiment polarity and automatic translation of original Persian sentences in English (EN), German (DE), Czech (CS), Italian (IT) and Hindi (HI) spoken 
languages. Learn more about this project at LSCP webpage (https://iasbs.ac.ir/~ansari/lscp/).
- ArmanPersoNERCorpus (https://github.com/HaniehP/PersianNER): The dataset includes 250,015 tokens and 7,682 Persian sentences in total. It is available in 3 folds to be used in turn as 
training and test sets. Each file contains one token, along with its manually annotated named-entity tag, per line. Each sentence is separated with a newline. The NER tags are in IOB format.
- FarsiYar PersianNER (https://github.com/Text-Mining/Persian-NER): The dataset includes about 25,000,000 tokens and about 1,000,000 Persian sentences in total based on Persian Wikipedia 
Corpus (https://github.com/Text-Mining/Persian-Wikipedia-Corpus). The NER tags are in IOB format. More than 1000 volunteers contributed tag improvements to this dataset via web panel or 
android app. They release updated tags every two weeks.
- PERLEX (http://farsbase.net/PERLEX.html): The first Persian dataset for relation extraction, which is an expert translated version of the “Semeval-2010-Task-8” dataset. Link to the relevant
publication.
- Persian Syntactic Dependency Treebank (http://dadegan.ir/catalog/perdt): This treebank is supplied for free noncommercial use. For commercial uses feel free to contact us. The number of 
annotated sentences is 29,982 sentences including samples from almost all verbs of the Persian valency lexicon.
- Uppsala Persian Dependency Treebank (UPDT) (http://stp.lingfil.uu.se/~mojgan/UPDT.html): Dependency-based syntactically annotated corpus.
- Hamshahri (https://dbrg.ut.ac.ir/hamshahri/): Hamshahri collection is a standard reliable Persian text collection that was used at Cross Language Evaluation Forum (CLEF) during years 2008 and 2009 for 
evaluation of Persian information retrieval systems.
- Hamshahri (https://dbrg.ut.ac.ir/hamshahri/): Hamshahri collection is a standard reliable Persian text collection that was used at Cross Language Evaluation Forum (CLEF) during years 2008 
and 2009 for evaluation of Persian information retrieval systems.
NLP in Ukrainian
@@ -666,7 +677,8 @@
Other Languages
- Russian: pymorphy2 (https://github.com/kmike/pymorphy2) - a good pos-tagger for Russian
- Asian Languages: Thai, Lao, Chinese, Japanese, and Korean ICU Tokenizer (https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-tokenizer.html) implementation in ElasticSearch
- Asian Languages: Thai, Lao, Chinese, Japanese, and Korean ICU Tokenizer (https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-tokenizer.html) implementation in 
ElasticSearch
- Ancient Languages: CLTK (https://github.com/cltk/cltk): The Classical Language Toolkit is a Python library and collection of texts for doing NLP in ancient languages
- Hebrew: NLPH_Resources (https://github.com/NLPH/NLPH_Resources) - A collection of papers, corpora and linguistic resources for NLP in Hebrew