latest research topics in natural language processing

Subscribe to the PwC Newsletter

Join the community, natural language processing, language modelling.

Long-range modeling

Protein language model, sentence pair modeling, representation learning.

Disentanglement

Graph representation learning, sentence embeddings.

Network Embedding

Classification.

Text Classification

Graph Classification

Audio Classification

Medical Image Classification

Text retrieval, deep hashing, table retrieval, question answering.

Open-Ended Question Answering

Open-Domain Question Answering

Conversational question answering.

Knowledge Base Question Answering

Nlp based person retrival, image generation.

Image-to-Image Translation

Text-to-Image Generation

Image Inpainting

Conditional Image Generation

Translation, data augmentation.

Image Augmentation

Text Augmentation

Large language model.

Knowledge Graphs

Machine translation.

Transliteration

Multimodal Machine Translation

Bilingual lexicon induction.

Unsupervised Machine Translation

Knowledge graph completion, triple classification, inductive knowledge graph completion, inductive relation prediction, text generation.

Dialogue Generation

Data-to-Text Generation

Multi-Document Summarization

Story Generation

2d semantic segmentation, image segmentation, text style transfer.

Scene Parsing

Reflection Removal

Visual question answering (vqa).

Visual Question Answering

Machine Reading Comprehension

Chart Question Answering

Chart understanding.

Topic Models

Document Classification

Sentence Classification

Emotion Classification

Sentiment analysis.

Aspect-Based Sentiment Analysis (ABSA)

Multimodal Sentiment Analysis

Aspect Sentiment Triplet Extraction

Twitter Sentiment Analysis

Named entity recognition (ner).

Nested Named Entity Recognition

Chinese named entity recognition, few-shot ner, few-shot learning.

One-Shot Learning

Few-Shot Semantic Segmentation

Cross-domain few-shot.

Unsupervised Few-Shot Learning

Optical character recognition (ocr).

Active Learning

Handwriting Recognition

Handwritten digit recognition, irregular text recognition, word embeddings.

Learning Word Embeddings

Multilingual Word Embeddings

Embeddings evaluation, contextualised word representations, continual learning.

Class Incremental Learning

Continual named entity recognition, unsupervised class-incremental learning, information retrieval.

Passage Retrieval

Cross-lingual information retrieval, table search, text summarization.

Abstractive Text Summarization

Document summarization, opinion summarization, relation extraction.

Relation Classification

Document-level relation extraction, joint entity and relation extraction, temporal relation extraction, link prediction.

Inductive Link Prediction

Dynamic link prediction, hyperedge prediction, anchor link prediction, natural language inference.

Answer Generation

Visual Entailment

Cross-lingual natural language inference, reading comprehension.

Intent Recognition

Implicit relations, active object detection, emotion recognition.

Speech Emotion Recognition

Emotion Recognition in Conversation

Multimodal Emotion Recognition

Emotion-cause pair extraction, natural language understanding, vietnamese social media text processing.

Emotional Dialogue Acts

Image captioning.

3D dense captioning

Controllable image captioning, aesthetic image captioning.

Relational Captioning

Semantic textual similarity.

Paraphrase Identification

Cross-Lingual Semantic Textual Similarity

Event extraction, event causality identification, zero-shot event extraction, in-context learning, dialogue state tracking, task-oriented dialogue systems.

Visual Dialog

Dialogue understanding, coreference resolution, coreference-resolution, cross document coreference resolution, code generation.

Code Translation

Code documentation generation, class-level code generation, library-oriented code generation, semantic parsing.

AMR Parsing

Semantic dependency parsing, drs parsing, ucca parsing, conformal prediction.

Text Simplification

Music Source Separation

Decision Making Under Uncertainty

Audio source separation, semantic similarity.

Sentence Embedding

Sentence compression, joint multilingual sentence representations, sentence embeddings for biomedical texts, specificity, instruction following, visual instruction following, dependency parsing.

Transition-Based Dependency Parsing

Prepositional phrase attachment, unsupervised dependency parsing, cross-lingual zero-shot dependency parsing, information extraction, extractive summarization, temporal information extraction, document-level event extraction, cross-lingual, cross-lingual transfer, cross-lingual document classification.

Cross-Lingual Entity Linking

Cross-language text summarization, common sense reasoning.

Physical Commonsense Reasoning

Riddle sense, memorization, response generation, prompt engineering.

Visual Prompting

Data integration.

Entity Alignment

Entity Resolution

Table annotation, mathematical reasoning.

Math Word Problem Solving

Formal logic, geometry problem solving, abstract algebra, entity linking.

Question Generation

Poll generation.

Topic coverage

Dynamic topic modeling, part-of-speech tagging.

Unsupervised Part-Of-Speech Tagging

Abuse detection, hate speech detection, open information extraction.

Hope Speech Detection

Hate speech normalization, hate speech detection crisishatemm benchmark, data mining.

Argument Mining

Opinion Mining

Subgroup discovery, cognitive diagnosis, sequential pattern mining, bias detection, selection bias, language identification, dialect identification, native language identification, word sense disambiguation.

Word Sense Induction

Fake news detection, few-shot relation classification, implicit discourse relation classification, cause-effect relation classification, intrusion detection.

Network Intrusion Detection

Relational Reasoning

Semantic Role Labeling

Predicate Detection

Semantic role labeling (predicted predicates).

Textual Analogy Parsing

Grammatical error correction.

Grammatical Error Detection

Text matching, slot filling.

Zero-shot Slot Filling

Extracting covid-19 events from twitter, symbolic regression, equation discovery, pos tagging, document text classification.

Learning with noisy labels

Multi-label classification of biomedical texts, political salient issue orientation detection, spoken language understanding, dialogue safety prediction, stance detection, zero-shot stance detection, few-shot stance detection, stance detection (us election 2020 - biden), stance detection (us election 2020 - trump), deep clustering, trajectory clustering, deep nonparametric clustering, nonparametric deep clustering, intent detection.

Open Intent Detection

Multi-modal entity alignment, word similarity, model editing, knowledge editing, document ai, document understanding, cross-modal retrieval, image-text matching, cross-modal retrieval with noisy correspondence, multilingual cross-modal retrieval.

Zero-shot Composed Person Retrieval

Cross-modal retrieval on rsitmd, fact verification, intent classification.

Text-To-Speech Synthesis

Prosody Prediction

Zero-shot multi-speaker tts, zero-shot cross-lingual transfer, cross-lingual ner, self-learning, language acquisition, grounded language learning, constituency parsing.

Constituency Grammar Induction

Entity typing.

Entity Typing on DH-KGs

Ad-hoc information retrieval, document ranking.

Word Alignment

Line items extraction, open-domain dialog, dialogue evaluation, abstract meaning representation, multimodal deep learning, multimodal text and image classification, novelty detection.

text-guided-image-editing

Text-based image editing, concept alignment.

Zero-Shot Text-to-Image Generation

Conditional text-to-image synthesis, multi-label text classification.

Shallow Syntax

Explanation generation, molecular representation, discourse parsing, discourse segmentation, connective detection, de-identification, privacy preserving deep learning, morphological analysis.

Sarcasm Detection

Text-to-Video Generation

Text-to-video editing, subject-driven video generation, conversational search, lemmatization, speech-to-text translation, simultaneous speech-to-text translation.

Aspect Extraction

Aspect category sentiment analysis, extract aspect.

Aspect-Category-Opinion-Sentiment Quadruple Extraction

Aspect-oriented Opinion Extraction

Session search.

Chinese Word Segmentation

Handwritten chinese text recognition, chinese spelling error correction, chinese zero pronoun resolution, offline handwritten chinese character recognition, entity disambiguation, authorship attribution, source code summarization, method name prediction, text clustering.

Short Text Clustering

Open Intent Discovery

Linguistic acceptability.

Column Type Annotation

Cell entity annotation, columns property annotation, row annotation, abusive language, keyphrase extraction.

Visual Storytelling

KG-to-Text Generation

Unsupervised KG-to-Text Generation

Few-shot text classification, zero-shot out-of-domain detection, multilingual nlp, protein folding, term extraction, text2text generation, keyphrase generation, figurative language visualization, sketch-to-text generation, morphological inflection, phrase grounding, grounded open vocabulary acquisition, deep attention, spam detection, context-specific spam detection, traditional spam detection, word translation, natural language transduction, image-to-text retrieval, summarization, unsupervised extractive summarization, query-focused summarization.

Cross-Lingual Word Embeddings

Knowledge base population, passage ranking, conversational response selection, text annotation, key information extraction, video generation, image to video generation.

Unconditional Video Generation

Authorship verification.

Keyword Extraction

Multimodal association, multimodal generation, biomedical information retrieval.

SpO2 estimation

Meme classification, hateful meme classification, news classification, graph-to-sequence, automated essay scoring, nlg evaluation, key point matching, component classification, argument pair extraction (ape), claim extraction with stance classification (cesc), claim-evidence pair extraction (cepe), temporal processing, timex normalization, document dating, sentence summarization, unsupervised sentence summarization, morphological tagging, long-context understanding, weakly supervised classification, weakly supervised data denoising, entity extraction using gan.

Rumour Detection

Emotional intelligence, dark humor detection, review generation, semantic retrieval, sentence ordering, comment generation.

Semantic Composition

Goal-Oriented Dialog

User simulation, lexical simplification, sentence-pair classification, conversational response generation.

Personalized and Emotional Conversation

Token classification, toxic spans detection.

Blackout Poetry Generation

Passage re-ranking, subjectivity analysis.

Taxonomy Learning

Taxonomy expansion, hypernym discovery, humor detection.

Lexical Normalization

Pronunciation dictionary creation, negation detection, negation scope resolution, question similarity, medical question pair similarity computation, intent discovery, reverse dictionary, propaganda detection, propaganda span identification, propaganda technique identification, lexical analysis, lexical complexity prediction, question rewriting, punctuation restoration, attribute value extraction.

Hallucination Evaluation

Legal reasoning, meeting summarization, table-based fact verification, pretrained multilingual language models, formality style transfer, semi-supervised formality style transfer, word attribute transfer, aspect category detection, diachronic word embeddings, extreme summarization.

Persian Sentiment Analysis

Binary classification, llm-generated text detection, cancer-no cancer per breast classification, cancer-no cancer per image classification, stable mci vs progressive mci, suspicous (birads 4,5)-no suspicous (birads 1,2,3) per image classification, clinical concept extraction.

Clinical Information Retreival

Constrained clustering.

Only Connect Walls Dataset Task 1 (Grouping)

Incremental constrained clustering, recognizing emotion cause in conversations.

Causal Emotion Entailment

trustable and focussed LLM generated content

Game design, dialog act classification, decipherment, nested mention recognition, relationship extraction (distant supervised), text compression, handwriting verification, bangla spelling error correction, clickbait detection, code repair, gender bias detection, probing language models, semantic entity labeling, ccg supertagging, linguistic steganography, toponym resolution.

Timeline Summarization

Multimodal abstractive text summarization, reader-aware summarization, stock prediction, text-based stock prediction, pair trading, event-driven trading, vietnamese visual question answering, explanatory visual question answering, arabic text diacritization, fact selection, thai word segmentation, vietnamese datasets.

Face to Face Translation

Multimodal lexical translation, semantic shift detection, similarity explanation, aggression identification, arabic sentiment analysis, commonsense causal reasoning, complex word identification, sign language production, suggestion mining, temporal relation classification, vietnamese word segmentation, speculation detection, speculation scope resolution, aspect category polarity, cross-lingual bitext mining, morphological disambiguation, multi-agent integration, scientific document summarization, lay summarization, text anonymization, text attribute transfer.

Image-guided Story Ending Generation

Abstract argumentation, chinese spell checking, dialogue rewriting, logical reasoning reading comprehension.

Unsupervised Sentence Compression

Stereotypical bias analysis, temporal tagging, anaphora resolution, bridging anaphora resolution.

Abstract Anaphora Resolution

Hope speech detection for english, hope speech detection for malayalam, hope speech detection for tamil, hidden aspect detection, latent aspect detection, personality generation, personality alignment, attribute mining, cognate prediction, japanese word segmentation, memex question answering, polyphone disambiguation, spelling correction, table-to-text generation.

KB-to-Language Generation

Vietnamese language models, zero-shot machine translation, zero-shot sentiment classification, conditional text generation, contextualized literature-based discovery, multimedia generative script learning, image-sentence alignment, open-world social event classification, action parsing, author attribution, binary condescension detection, context query reformulation, conversational web navigation, croatian text diacritization, czech text diacritization, definition modelling, document-level re with incomplete labeling, domain labelling, french text diacritization, hungarian text diacritization, irish text diacritization, latvian text diacritization, literature mining, misogynistic aggression identification, morpheme segmentaiton, multi-label condescension detection, news annotation, open relation modeling, personality recognition in conversation.

Reading Order Detection

Record linking, role-filler entity extraction, romanian text diacritization, simultaneous speech-to-speech translation, slovak text diacritization, spanish text diacritization, syntax representation, text-to-video search, turkish text diacritization, turning point identification, twitter event detection.

Vietnamese Scene Text

Vietnamese text diacritization.

Conversational Sentiment Quadruple Extraction

Attribute extraction, legal outcome extraction, automated writing evaluation, binary text classification, detection of potentially void clauses, chemical indexing, clinical assertion status detection.

Coding Problem Tagging

Collaborative plan acquisition, commonsense reasoning for rl.

Variable Disambiguation

Cross-lingual text-to-image generation, crowdsourced text aggregation.

Description-guided molecule generation

Multi-modal Dialogue Generation

Page stream segmentation.

Email Thread Summarization

Emergent communications on relations, emotion detection and trigger summarization, extractive tags summarization.

Hate Intensity Prediction

Hate span identification, job prediction, joint entity and relation extraction on scientific data, joint ner and classification, math information retrieval, meme captioning, multi-grained named entity recognition, multilingual machine comprehension in english hindi, multimodal text prediction, negation and speculation cue detection, negation and speculation scope resolution, only connect walls dataset task 2 (connections), overlapping mention recognition, paraphrase generation, multilingual paraphrase generation, phrase ranking, phrase tagging, phrase vector embedding, poem meters classification, query wellformedness.

Question-Answer categorization

Readability optimization, reliable intelligence identification, sentence completion, hurtful sentence completion, social media mental health detection, speaker attribution in german parliamentary debates (germeval 2023, subtask 1), text effects transfer, text-variation, vietnamese aspect-based sentiment analysis, sentiment dependency learning, vietnamese natural language understanding, vietnamese sentiment analysis, vietnamese multimodal sentiment analysis, web page tagging, workflow discovery, answerability prediction, incongruity detection, multi-word expression embedding, multi-word expression sememe prediction, pcl detection, semeval-2022 task 4-1 (binary pcl detection), semeval-2022 task 4-2 (multi-label pcl detection), automatic writing, complaint comment classification, counterspeech detection, extractive text summarization, face selection, job classification, multi-lingual text-to-image generation, multlingual neural machine translation, optical charater recogntion, bangla text detection, question to declarative sentence, relation mention extraction.

Tweet-Reply Sentiment Analysis

Vietnamese fact checking, vietnamese parsing.

Suggestions or feedback?

MIT News | Massachusetts Institute of Technology

• Machine learning
• Sustainability
• Black holes
• Classes and programs

Departments

• Aeronautics and Astronautics
• Brain and Cognitive Sciences
• Architecture
• Political Science
• Mechanical Engineering

Centers, Labs, & Programs

• Abdul Latif Jameel Poverty Action Lab (J-PAL)
• Picower Institute for Learning and Memory
• Lincoln Laboratory
• School of Architecture + Planning
• School of Engineering
• School of Humanities, Arts, and Social Sciences
• Sloan School of Management
• School of Science
• MIT Schwarzman College of Computing

Natural language processing

Download RSS feed: News Articles / In the Media / Audio

Using ideas from game theory to improve the reliability of language models

A new “consensus game,” developed by MIT CSAIL researchers, elevates AI’s text comprehension and generation skills.

May 14, 2024

Read full story →

Natural language boosts LLM performance in coding, planning, and robotics

Three neurosymbolic methods help language models find better abstractions within natural language, then use those representations to execute complex tasks.

May 1, 2024

Three from MIT awarded 2024 Guggenheim Fellowships

MIT professors Roger Levy, Tracy Slatyer, and Martin Wainwright appointed to the 2024 class of “trail-blazing fellows.”

April 26, 2024

3 Questions: What you need to know about audio deepfakes

MIT CSAIL postdoc Nauman Dawalatabad explores ethical considerations, challenges in spear-phishing defense, and the optimistic future of AI-created voices across various sectors.

March 15, 2024

Reasoning and reliability in AI

PhD students interning with the MIT-IBM Watson AI Lab look to improve natural language usage.

January 18, 2024

Leveraging language to understand machines

Master’s students Irene Terpstra ’23 and Rujul Gandhi ’22 use language to design new integrated circuits and make it understandable to robots.

December 22, 2023

Using language to give robots a better grasp of an open-ended world

By blending 2D images with foundation models to build 3D feature fields, a new MIT method helps robots understand and manipulate nearby objects with open-ended language prompts.

November 2, 2023

Re-imagining our theories of language

Department of Brain and Cognitive Sciences faculty members Ev Fedorenko, Ted Gibson, and Roger Levy believe they can answer a fundamental question: What is the purpose of language?

September 22, 2023

MIT researchers make language models scalable self-learners

The scientists used a natural language-based logical inference dataset to create smaller language models that outperformed much larger counterparts.

June 8, 2023

3 Questions: Jacob Andreas on large language models

The CSAIL scientist describes natural language processing research through state-of-the-art machine-learning models and investigation of how language can enhance other types of artificial intelligence.

May 11, 2023

Jacob Andreas and Mingda Li honored with Junior Bose Award for Excellence in Teaching

Award is given each year by the School of Engineering to an outstanding educator up for promotion to associate professor without tenure.

April 3, 2023

New insights into training dynamics of deep classifiers

MIT researchers uncover the structural properties and dynamics of deep classifiers, offering novel explanations for optimization, generalization, and approximation in deep networks.

March 8, 2023

Large language models are biased. Can logic help save them?

MIT researchers trained logic-aware language models to reduce harmful stereotypes like gender and racial biases.

March 3, 2023

Engineering in harmony

AeroAstro major and accomplished tuba player Frederick Ajisafe relishes the community he has found in the MIT Wind Ensemble.

January 12, 2023

Cognitive scientists develop new model explaining difficulty in language comprehension

Built on recent advances in machine learning, the model predicts how well individuals will produce and comprehend sentences.

December 22, 2022

Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA, USA

• Map (opens in new window)
• Events (opens in new window)
• People (opens in new window)
• Careers (opens in new window)
• Accessibility
• Social Media Hub
• MIT on Facebook
• MIT on YouTube
• MIT on Instagram

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts

Collection  20 April 2023

Advances in Natural Language Processing

Natural language processing (NLP) is an interdisciplinary field spanning computational science and artificial intelligence (AI), concerned with the understanding of human language, in both written and verbal forms, by machines. NLP puts an emphasis on employing machine learning and deep learning techniques to complete tasks, like language translation or question answering. In the growing NLP domain, two main methodological branches can be distinguished: natural language understanding (NLU), which aims to improve the machine's reading comprehension, and natural language generation (NLG), focused on enabling machines to produce human language text responses based on a given data input.

In the modern world, the number of NLP applications seems to be following an exponential growth curve: from highly agile chatbots, to sentiment analysis and intent classification, to personalised medicine, the NLP's capacity for improving our lives is ever-growing. At the same time, NLP progress is halted by the limited AI hardware infrastructure which struggles to accommodate more refined NLP models, the sparsity of good-quality NLP-training data, and complex linguistic problems, such as machine's understanding of homonymy or generation of polysemy.

This Collection is dedicated to the latest research on methodology in the vast field of NLP, which addresses and carries the potential to solve at least one of the many struggles the state-of-the-art NLP approaches face. We welcome theoretical-applied and applied research, proposing novel computational and/or hardware solutions.

University of Surrey, UK

Einat Liebenthal

Harvard Medical School, USA

University of Modena and Reggio Emilia, Italy

• Collection content
• How to submit
• About the Guest Editors
• Collection policies

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Natural Language Processing

Natural Language Processing (NLP) research at Google focuses on algorithms that apply at scale, across languages, and across domains. Our systems are used in numerous ways across Google, impacting user experience in search, mobile, apps, ads, translate and more.

Our work spans the range of traditional NLP tasks, with general-purpose syntax and semantic algorithms underpinning more specialized systems. We are particularly interested in algorithms that scale well and can be run efficiently in a highly distributed environment.

Our syntactic systems predict part-of-speech tags for each word in a given sentence, as well as morphological features such as gender and number. They also label relationships between words, such as subject, object, modification, and others. We focus on efficient algorithms that leverage large amounts of unlabeled data, and recently have incorporated neural net technology.

On the semantic side, we identify entities in free text, label them with types (such as person, location, or organization), cluster mentions of those entities within and across documents (coreference resolution), and resolve the entities to the Knowledge Graph.

Recent work has focused on incorporating multiple sources of knowledge and information to aid with analysis of text, as well as applying frame semantics at the noun phrase, sentence, and document level.

Recent Publications

Some of our teams.

Africa team

Impact-Driven Research, Innovation and Moonshots

We're always looking for more talented, passionate people.

Last updated 2nd August 2024: Online ordering is currently unavailable due to technical issues. As we resolve the issues resulting from this, we are also experiencing some delays to publication. We are working hard to restore services as soon as possible and apologise for the inconvenience. For further updates please visit our website https://www.cambridge.org/news-and-insights/technical-incident

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• < Back to search results
• Natural Language Processing

Natural Language Processing

• Submit your article

This journal utilises an Online Peer Review Service (OPRS) for submissions. By clicking "Continue" you will be taken to our partner site https://mc.manuscriptcentral.com/nlp . Please be aware that your Cambridge account is not valid for this OPRS and registration is required. We strongly advise you to read all "Author instructions" in the "Journal information" area prior to submitting.

• Information
• Journal home
• Journal information
• Latest issue
• FirstView articles

• You have access: full Access: Full
• Open access

• ISSN: 2977-0424 (Online)
• Editor: Ruslan Mitkov Lancaster University, UK
• Editorial board

Recently published articles

Part-of-speech tagger for bodo language using deep learning approach.

• Dhrubajyoti Pathak , Sanjib Narzary , Sukumar Nandi , Bidisha Som
• Natural Language Processing ,

Automatic generation of nominal phrases for Portuguese and Galician

• María José Domínguez Vázquez , Alberto Simões , Daniel Bardanca Outeiriño , María Caíña Hurtado , José Luis Iglesias Allones

Does learning from language family help? A case study on a low-resource question-answering task

• Hariom A. Pandya , Brijesh S. Bhatt

Calibration and context in human evaluation of machine translation

• Rebecca Knowles , Chi-kiu Lo

A case study on decompounding in Indian language IR

• Siba Sankar Sahu , Sukomal Pal

Statistical machine translation for Indic languages

• Sudhansu Bala Das , Divyajyoti Panda , Tapas Kumar Mishra , Bidyut Kr. Patra

A bidirectional LSTM-based morphological analyzer for Gujarati

• Jatayu Baxi , Brijesh Bhatt

Is Attention always needed? A case study on language identification from speech

• Atanu Mandal , Santanu Pal , Indranil Dutta , Mahidas Bhattacharya , Sudip Kumar Naskar

Evaluating optimal reference translations

• Vilém Zouhar , Věra Kloudová , Martin Popel , Ondřej Bojar

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Journal Menu

• Applied Sciences Home
• Aims & Scope
• Editorial Board
• Reviewer Board
• Topical Advisory Panel
• Instructions for Authors
• Special Issues
• Sections & Collections
• Article Processing Charge
• Indexing & Archiving
• Editor’s Choice Articles
• Most Cited & Viewed
• Journal Statistics
• Journal History
• Journal Awards
• Society Collaborations
• Conferences
• Editorial Office

Journal Browser

• arrow_forward_ios Forthcoming issue arrow_forward_ios Current issue
• Vol. 14 (2024)
• Vol. 13 (2023)
• Vol. 12 (2022)
• Vol. 11 (2021)
• Vol. 10 (2020)
• Vol. 9 (2019)
• Vol. 8 (2018)
• Vol. 7 (2017)
• Vol. 6 (2016)
• Vol. 5 (2015)
• Vol. 4 (2014)
• Vol. 3 (2013)
• Vol. 2 (2012)
• Vol. 1 (2011)

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

Current Approaches and Applications in Natural Language Processing

• Print Special Issue Flyer

Special Issue Editors

Special issue information.

• Published Papers

A special issue of Applied Sciences (ISSN 2076-3417). This special issue belongs to the section " Computing and Artificial Intelligence ".

Deadline for manuscript submissions: closed (31 January 2022) | Viewed by 103207

Share This Special Issue

Dear Colleagues,

Current approaches in Natural Language Processing (NLP) have shown impressive improvements in many major tasks: machine translation, language modelling, text generation, sentiment/emotion analysis, natural language understanding, question answering, among others. The advent of new methods and techniques like graph-based approaches, reinforcement learning or deep learning have boosted many of the tasks in NLP to reach human-level (and even further) performance. This has attracted the interest of many companies, so new products and solutions can profit from the advances of this relevant area within the artificial intelligence domain.

This Special Issue focuses on emerging techniques and trendy applications of NLP methods is an opportunity to report on all these achievements, establishing a useful reference for industry and researchers on cutting edge human language technologies. Given the focus of the journal, we expect to receive works that propose new NLP algorithms and applications of current and novel NLP tasks. Also, updated overviews on the given topics will be considered, identifying trends, potential future research areas and new commercial products.

The topics of this Special Issue include but are not limited to:

• Question answering: open-domain Q&A, knowledge-based Q&A...
• Knowledge extraction: Relation extraction, fine-grained entity recognition...
• Text generation: summarization, style transfer, dial...
• Text classification: Sentiment/emotion analysis, semi-supervised and zero-shot learning...
• Behaviour modelling: early risk detection, cyberbullying, customer modelling...
• Dialogue systems: chatbots, voice assistants...
• Reinforcement learning
• Data augmentation
• Graph based approaches
• Adversarial approaches
• Multi-modal approaches
• Multi-lingual/cross-lingual approaches

Prof. Dr. Arturo Montejo-Ráez Dr. Salud María Jiménez Zafra Guest Editors

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website . Once you are registered, click here to go to the submission form . Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Applied Sciences is an international peer-reviewed open access semimonthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2400 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Published Papers (26 papers)

Jump to: Research , Review

Jump to: Editorial , Review

Graphical abstract

Jump to: Editorial , Research

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Skip to main content
• Skip to primary sidebar
• Skip to footer

The Best of Applied Artificial Intelligence, Machine Learning, Automation, Bots, Chatbots

GPT-3 & Beyond: 10 NLP Research Papers You Should Read

November 17, 2020 by Mariya Yao

NLP research advances in 2020 are still dominated by large pre-trained language models, and specifically transformers. There were many interesting updates introduced this year that have made transformer architecture more efficient and applicable to long documents.

Another hot topic relates to the evaluation of NLP models in different applications. We still lack evaluation approaches that clearly show where a model fails and how to fix it.

Also, with the growing capabilities of language models such as GPT-3, conversational AI is enjoying a new wave of interest. Chatbots are improving, with several impressive bots like Meena and Blender introduced this year by top technology companies.

To help you stay up to date with the latest NLP research breakthroughs, we’ve curated and summarized the key research papers in natural language processing from 2020. The papers cover the leading language models, updates to the transformer architecture, novel evaluation approaches, and major advances in conversational AI.

Subscribe to our AI Research mailing list at the bottom of this article to be alerted when we release new summaries.

If you’d like to skip around, here are the papers we featured:

• WinoGrande: An Adversarial Winograd Schema Challenge at Scale
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Reformer: The Efficient Transformer
• Longformer: The Long-Document Transformer
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
• Language Models are Few-Shot Learners
• Beyond Accuracy: Behavioral Testing of NLP models with CheckList
• Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
• Towards a Human-like Open-Domain Chatbot
• Recipes for Building an Open-Domain Chatbot

Best NLP Research Papers 2020

1. winogrande: an adversarial winograd schema challenge at scale , by keisuke sakaguchi, ronan le bras, chandra bhagavatula, yejin choi, original abstract .

The Winograd Schema Challenge (WSC) (Levesque, Davis, and Morgenstern 2011), a benchmark for commonsense reasoning, is a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. However, recent advances in neural language models have already reached around 90% accuracy on variants of WSC. This raises an important question whether these models have truly acquired robust commonsense capabilities or whether they rely on spurious biases in the datasets that lead to an overestimation of the true capabilities of machine commonsense. 

To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. The best state-of-the-art methods on WinoGrande achieve 59.4-79.1%, which are 15-35% below human performance of 94.0%, depending on the amount of the training data allowed. 

Furthermore, we establish new state-of-the-art results on five related benchmarks – WSC (90.1%), DPR (93.1%), COPA (90.6%), KnowRef (85.6%), and Winogender (97.1%). These results have dual implications: on one hand, they demonstrate the effectiveness of WinoGrande when used as a resource for transfer learning. On the other hand, they raise a concern that we are likely to be overestimating the true capabilities of machine commonsense across all these benchmarks. We emphasize the importance of algorithmic bias reduction in existing and future benchmarks to mitigate such overestimation.

Our Summary 

The research group from the Allen Institute for Artificial Intelligence introduces WinoGrande , a new benchmark for commonsense reasoning. They build on the design of the famous Winograd Schema Challenge (WSC) benchmark but significantly increase the scale of the dataset to 44K problems and reduce systematic bias using a novel AfLite algorithm. The experiments demonstrate that state-of-the-art methods achieve up to 79.1% accuracy on WinoGrande, which is significantly below the human performance of 94%. Furthermore, the researchers show that WinoGrande is an effective resource for transfer learning, by using a RoBERTa model fine-tuned with WinoGrande to achieve new state-of-the-art results on WSC and four other related benchmarks.

What’s the core idea of this paper?

• The authors claim that existing benchmarks for commonsense reasoning suffer from systematic bias and annotation artifacts, leading to overestimation of the true capabilities of machine intelligence on commonsense reasoning.
• Crowdworkers were asked to write twin sentences that meet the WSC requirements and contain certain anchor words. This new requirement is aimed at improving the creativity of crowdworkers.
• Collected problems were validated through a distinct set of three crowdworkers. Out of 77K collected questions, 53K were deemed valid.
• It generalizes human-detectable biases based on word occurrences to machine-detectable biases based on embedding occurrences.
• After applying the AfLite algorithm, the debiased WinoGrande dataset contains 44K samples. 

What’s the key achievement?

• Wino Knowledge Hunting (WKH) and Ensemble LMs only achieve chance-level performance (50%);
• RoBERTa achieves 79.1% test-set accuracy;
• whereas human performance achieves 94% accuracy.
• 90.1% on WSC;
• 93.1% on DPR ; 
• 90.6% on COPA ; 
• 85.6% on KnowRef ; and 
• 97.1% on Winogender .

What does the AI community think?

• The paper received the Outstanding Paper Award at AAAI 2020, one of the key conferences in artificial intelligence.

What are future research areas?

• Exploring new algorithmic approaches for systematic bias reduction.
• Debiasing other NLP benchmarks.

Where can you get implementation code?

• The dataset can be downloaded from the WinoGrande project page .
• The implementation code is available on GitHub .
• And here is the WinoGrande leaderboard .

2. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.

The Google research team suggests a unified approach to transfer learning in NLP with the goal to set a new state of the art in the field. To this end, they propose treating each NLP problem as a “text-to-text” problem. Such a framework allows using the same model, objective, training procedure, and decoding process for different tasks, including summarization, sentiment analysis, question answering, and machine translation. The researchers call their model a Text-to-Text Transfer Transformer (T5) and train it on the large corpus of web-scraped data to get state-of-the-art results on a number of NLP tasks.

• Providing a comprehensive perspective on where the NLP field stands by exploring and comparing existing techniques.
• The mode understands which tasks should be performed thanks to the task-specific prefix added to the original input sentence (e.g., “translate English to German:”, “summarize:”).
• Presenting and releasing a new dataset consisting of hundreds of gigabytes of clean web-scraped English text, the Colossal Clean Crawled Corpus (C4) .
• Training a large (up to 11B parameters) model, called Text-to-Text Transfer Transformer (T5) on the C4 dataset.
• the GLUE score of 89.7 with substantially improved performance on CoLA, RTE, and WNLI tasks;
• the Exact Match score of 90.06 on SQuAD dataset;
• the SuperGLUE score of 88.9, which is a very significant improvement over the previous state-of-the-art result (84.6) and very close to human performance (89.8);
• the ROUGE-2-F score of 21.55 on CNN/Daily Mail abstractive summarization task.
• Researching the methods to achieve stronger performance with cheaper models.
• Exploring more efficient knowledge extraction techniques.
• Further investigating the language-agnostic models.

What are possible business applications?

• Even though the introduced model has billions of parameters and can be too heavy to be applied in the business setting, the presented ideas can be used to improve the performance on different NLP tasks, including summarization, question answering, and sentiment analysis.
• The pretrained models together with the dataset and code are released on GitHub .

3. Reformer: The Efficient Transformer , by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O( L 2 ) to O( L log L ), where L is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of N times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.

The leading Transformer models have become so big that they can be realistically trained only in large research laboratories. To address this problem, the Google Research team introduces several techniques that improve the efficiency of Transformers. In particular, they suggest (1) using reversible layers to allow storing the activations only once instead of for each layer, and (2) using locality-sensitive hashing to avoid costly softmax computation in the case of full dot-product attention. Experiments on several text tasks demonstrate that the introduced Reformer model matches the performance of the full Transformer but runs much faster and with much better memory efficiency.

• The activations of every layer need to be stored for back-propagation.
• The intermediate feed-forward layers account for a large fraction of memory use since their depth is often much larger than the depth of attention activations.
• The complexity of attention on a sequence of length L is O( L 2 ).
• using reversible layers to store only a single copy of activations;
• splitting activations inside the feed-forward layers and processing them in chunks;
• approximating attention computation based on locality-sensitive hashing .
• switching to locality-sensitive hashing attention;
• using reversible layers.
• For example, on the newstest2014 task for machine translation from English to German, the Reformer base model gets a BLEU score of 27.6 compared to Vaswani’s et al. (2017) BLEU score of 27.3. 
• The paper was selected for oral presentation at ICLR 2020, the leading conference in deep learning.
• text generation;
• visual content generation;
• music generation;
• time-series forecasting.
• The official code implementation from Google is publicly available on GitHub .
• The PyTorch implementation of Reformer is also available on GitHub .

4. Longformer: The Long-Document Transformer , by Iz Beltagy, Matthew E. Peters, Arman Cohan

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer’s attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.

Self-attention is one of the key factors behind the success of Transformer architecture. However, it also makes transformer-based models hard to apply to long documents. The existing techniques usually divide the long input into a number of chunks and then use complex architectures to combine information across these chunks. The research team from the Allen Institute for Artificial Intelligence introduces a more elegant solution to this problem. The suggested Longformer model employs an attention pattern that combines local windowed attention with task-motivated global attention. This attention mechanism scales linearly with the sequence length and enables processing of documents with thousands of tokens. The experiments demonstrate that Longformer achieves state-of-the-art results on character-level language modeling tasks, and when pre-trained, consistently outperforms RoBERTa on long-document tasks.

• The computational requirements of self-attention grow quadratically with sequence length, making it hard to process on current hardware. 
• allows memory usage to scale linearly, and not quadratically, with the sequence length;
• a windowed local-context self-attention to build contextual representations;
• an end task motivated global attention to encode inductive bias about the task and build full sequence representation.
• Since the implementation of the sliding window attention pattern requires a form of banded matrix multiplication that is not supported in the existing deep learning libraries like PyTorch and Tensorflow, the authors also introduce a custom CUDA kernel for implementing these attention operations.
• BPC of 1.10 on text8 ;
• BPC of 1.00 on enwik8 .
• accuracy of 75.0 vs. 72.4 on WikiHop ;
• F1 score of 75.2 vs. 74.2 on TriviaQA ;
• joint F1 score of 64.4 vs. 63.5 on HotpotQA ;
• average F1 score of 78.6 vs. 78.4 on the OntoNotes coreference resolution task;
• accuracy of 95.7 vs. 95.3 on the IMDB classification task;
• F1 score of 94.0 vs. 87.4 on the Hyperpartisan classification task.
• The performance gains are especially remarkable for the tasks that require a long context (i.e., WikiHop and Hyperpartisan).
• Exploring other attention patterns that are more efficient due to dynamic adaptation to the input. 
• Applying Longformer to other relevant long document tasks such as summarization.
• document classification;
• question answering;
• coreference resolution;
• summarization;
• semantic search.
• The code implementation of Longformer is open-sourced on GitHub .

5. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators , by Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30× more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.

The pre-training task for popular language models like BERT and XLNet involves masking a small subset of unlabeled input and then training the network to recover this original input. Even though it works quite well, this approach is not particularly data-efficient as it learns from only a small fraction of tokens (typically ~15%). As an alternative, the researchers from Stanford University and Google Brain propose a new pre-training task called replaced token detection . Instead of masking, they suggest replacing some tokens with plausible alternatives generated by a small language model. Then, the pre-trained discriminator is used to predict whether each token is an original or a replacement. As a result, the model learns from all input tokens instead of the small masked fraction, making it much more computationally efficient. The experiments confirm that the introduced approach leads to significantly faster training and higher accuracy on downstream NLP tasks.

• Pre-training methods that are based on masked language modeling are computationally inefficient as they use only a small fraction of tokens for learning.
• some tokens are replaced by samples from a small generator network; 
• a model is pre-trained as a discriminator to distinguish between original and replaced tokens.
• enables the model to learn from all input tokens instead of the small masked-out subset;
• is not adversarial, despite the similarity to GAN, as the generator producing tokens for replacement is trained with maximum likelihood.
• Demonstrating that the discriminative task of distinguishing between real data and challenging negative samples is more efficient than existing generative methods for language representation learning.
• ELECTRA-Small gets a GLUE score of 79.9 and outperforms a comparably small BERT model with a score of 75.1 and a much larger GPT model with a score of 78.8.
• An ELECTRA model that performs comparably to XLNet and RoBERTa uses only 25% of their pre-training compute.
• ELECTRA-Large outscores the alternative state-of-the-art models on the GLUE and SQuAD benchmarks while still requiring less pre-training compute.
• The paper was selected for presentation at ICLR 2020, the leading conference in deep learning.
• Because of its computational efficiency, the ELECTRA approach can make the application of pre-trained text encoders more accessible to business practitioners.
• The original TensorFlow implementation and pre-trained weights are released on GitHub .

6. Language Models are Few-Shot Learners , by Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10× more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

The OpenAI research team draws attention to the fact that the need for a labeled dataset for every new language task limits the applicability of language models. Considering that there is a wide range of possible tasks and it’s often difficult to collect a large labeled training dataset, the researchers suggest an alternative solution, which is scaling up language models to improve task-agnostic few-shot performance. They test their solution by training a 175B-parameter autoregressive language model, called GPT-3 , and evaluating its performance on over two dozen NLP tasks. The evaluation under few-shot learning, one-shot learning, and zero-shot learning demonstrates that GPT-3 achieves promising results and even occasionally outperforms the state of the art achieved by fine-tuned models.

• The GPT-3 model uses the same model and architecture as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization.
• However, in contrast to GPT-2, it uses alternating dense and locally banded sparse attention patterns in the layers of the transformer, as in the Sparse Transformer .
• Few-shot learning , when the model is given a few demonstrations of the task (typically, 10 to 100) at inference time but with no weight updates allowed.
• One-shot learning , when only one demonstration is allowed, together with a natural language description of the task.
• Zero-shot learning , when no demonstrations are allowed and the model has access only to a natural language description of the task.
• On the CoQA benchmark, 81.5 F1 in the zero-shot setting, 84.0 F1 in the one-shot setting, and 85.0 F1 in the few-shot setting, compared to the 90.7 F1 score achieved by fine-tuned SOTA.
• On the TriviaQA benchmark, 64.3% accuracy in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, surpassing the state of the art (68%) by 3.2%.
• On the LAMBADA dataset, 76.2 % accuracy in the zero-shot setting, 72.5% in the one-shot setting, and 86.4% in the few-shot setting, surpassing the state of the art (68%) by 18%.
• The news articles generated by the 175B-parameter GPT-3 model are hard to distinguish from real ones, according to human evaluations (with accuracy barely above the chance level at ~52%).
• “The GPT-3 hype is way too much. It’s impressive (thanks for the nice compliments!) but it still has serious weaknesses and sometimes makes very silly mistakes. AI is going to change the world, but GPT-3 is just a very early glimpse. We have a lot still to figure out.” – Sam Altman, CEO and co-founder of OpenAI .
• “I’m shocked how hard it is to generate text about Muslims from GPT-3 that has nothing to do with violence… or being killed…” – Abubakar Abid, CEO and founder of Gradio .
• “No. GPT-3 fundamentally does not understand the world that it talks about. Increasing corpus further will allow it to generate a more credible pastiche but not fix its fundamental lack of comprehension of the world. Demos of GPT-4 will still require human cherry picking.” – Gary Marcus, CEO and founder of Robust.ai .
• “Extrapolating the spectacular performance of GPT3 into the future suggests that the answer to life, the universe and everything is just 4.398 trillion parameters.” – Geoffrey Hinton, Turing Award winner .
• Improving pre-training sample efficiency.
• Exploring how few-shot learning works.
• Distillation of large models down to a manageable size for real-world applications.
• The model with 175B parameters is hard to apply to real business problems due to its impractical resource requirements, but if the researchers manage to distill this model down to a workable size, it could be applied to a wide range of language tasks, including question answering, dialog agents, and ad copy generation.
• The code itself is not available, but some dataset statistics together with unconditional, unfiltered 2048-token samples from GPT-3 are released on GitHub .

7. Beyond Accuracy: Behavioral Testing of NLP models with CheckList , by Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh

Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors. Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, as well as a software tool to generate a large and diverse number of test cases quickly. We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in an extensively tested model. In another user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.

The authors point out the shortcomings of existing approaches to evaluating performance of NLP models. A single aggregate statistic, like accuracy, makes it difficult to estimate where the model is failing and how to fix it. The alternative evaluation approaches usually focus on individual tasks or specific capabilities. To address the lack of comprehensive evaluation approaches, the researchers introduce CheckList , a new evaluation methodology for testing of NLP models. The approach is inspired by principles of behavioral testing in software engineering. Basically, CheckList is a matrix of linguistic capabilities and test types that facilitates test ideation. Multiple user studies demonstrate that CheckList is very effective at discovering actionable bugs, even in extensively tested NLP models.

• The primary approach to the evaluation of models’ generalization capabilities, which is accuracy on held-out data, may lead to performance overestimation, as the held-out data often contains the same biases as the training data. Moreover, this single aggregate statistic doesn’t help much in figuring out where the NLP model is failing and how to fix these bugs.
• The alternative approaches are usually designed for evaluation of specific behaviors on individual tasks and thus, lack comprehensiveness.
• CheckList provides users with a list of linguistic capabilities to be tested, like vocabulary, named entity recognition, and negation.
• Then, to break down potential capability failures into specific behaviors, CheckList suggests different test types , such as prediction invariance or directional expectation tests in case of certain perturbations.
• Potential tests are structured as a matrix, with capabilities as rows and test types as columns.
• The suggested implementation of CheckList also introduces a variety of abstractions to help users generate large numbers of test cases easily.
• Evaluation of state-of-the-art models with CheckList demonstrated that even though some NLP tasks are considered “solved” based on accuracy results, the behavioral testing highlights many areas for improvement.
• helps to identify and test for capabilities not previously considered;
• results in more thorough and comprehensive testing for previously considered capabilities;
• helps to discover many more actionable bugs.
• The paper received the Best Paper Award at ACL 2020, the leading conference in natural language processing.
• CheckList can be used to create more exhaustive testing for a variety of NLP tasks.
• Such comprehensive testing that helps in identifying many actionable bugs is likely to lead to more robust NLP systems.
• The code for testing NLP models with CheckList is available on GitHub .

8. Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics , by Nitika Mathur, Timothy Baldwin, Trevor Cohn

Automatic metrics are fundamental for the development and evaluation of machine translation systems. Judging whether, and to what extent, automatic metrics concur with the gold standard of human evaluation is not a straightforward problem. We show that current methods for judging metrics are highly sensitive to the translations used for assessment, particularly the presence of outliers, which often leads to falsely confident conclusions about a metric’s efficacy. Finally, we turn to pairwise system ranking, developing a method for thresholding performance improvement under an automatic metric against human judgements, which allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. Together, these findings suggest improvements to the protocols for metric evaluation and system performance evaluation in machine translation.

The most recent Conference on Machine Translation (WMT) has revealed that, based on Pearson’s correlation coefficient, automatic metrics poorly match human evaluations of translation quality when comparing only a few best systems. Even negative correlations were exhibited in some instances. The research team from the University of Melbourne investigates this issue by studying the role of outlier systems, exploring how the correlation coefficient reflects different patterns of errors (type I vs. type II errors), and what magnitude of difference in the metric score corresponds to true improvements in translation quality as judged by humans. Their findings suggest that small BLEU differences (i.e., 1–2 points) have little meaning and other metrics, such as chrF, YiSi-1, and ESIM should be preferred over BLEU. However, only human evaluations can be a reliable basis for drawing important empirical conclusions.

• Automatic metrics are used as a proxy for human translation evaluation, which is considerably more expensive and time-consuming.
• For example, the recent findings show that if the correlation between leading metrics and human evaluations is computed using a large set of translation systems, it is typically very high (i.e., 0.9). However, if only a few best systems are considered, the correlation reduces markedly and can even be negative in some cases.
• The identified problem with Pearson’s correlation is due to the small sample size and not specific to comparing strong MT systems.
• Outlier systems, whose quality is much higher or lower than the rest of the systems, have a disproportionate effect on the computed correlation and should be removed.
• The same correlation coefficient can reflect different patterns of errors. Thus, a better approach for gaining insights into metric reliability is to visualize metric scores against human scores.
• Small BLEU differences of 1-2 points correspond to true improvements in translation quality (as judged by humans) only in 50% of cases.
• Giving preference to such evaluation metrics as chrF, YiSi-1, and ESIM over BLEU and TER.
• Moving away from using small changes in evaluation metrics as the sole basis to draw important empirical conclusions, and always ensuring support from human evaluations before claiming that one MT system significantly outperforms another one.
• The paper received an Honorable Mention at ACL 2020, the leading conference in natural language processing. 
• The implementation code, data, and additional analysis will be released on GitHub .

9. Towards a Human-like Open-Domain Chatbot , by Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le

We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated. 

In contrast to most modern conversational agents, which are highly specialized, the Google research team introduces a chatbot Meena that can chat about virtually anything. It’s built on a large neural network with 2.6B parameters trained on 341 GB of text. The researchers also propose a new human evaluation metric for open-domain chatbots, called Sensibleness and Specificity Average (SSA), which can capture important attributes for human conversation. They demonstrate that this metric correlates highly with perplexity, an automatic metric that is readily available. Thus, the Meena chatbot, which is trained to minimize perplexity, can conduct conversations that are more sensible and specific compared to other chatbots. Particularly, the experiments demonstrate that Meena outperforms existing state-of-the-art chatbots by a large margin in terms of the SSA score (79% vs. 56%) and is closing the gap with human performance (86%).

• Despite recent progress, open-domain chatbots still have significant weaknesses: their responses often do not make sense or are too vague or generic.
• Meena is built on a seq2seq model with Evolved Transformer (ET) that includes 1 ET encoder block and 13 ET decoder blocks.
• The model is trained on multi-turn conversations with the input sequence including all turns of the context (up to 7) and the output sequence being the response.
• making sense,
• being specific.
• The research team discovered that the SSA metric shows high negative correlation (R2 = 0.93) with perplexity, a readily available automatic metric that Meena is trained to minimize.
• Proposing a simple human-evaluation metric for open-domain chatbots.
• The best end-to-end trained Meena model outperforms existing state-of-the-art open-domain chatbots by a large margin, achieving an SSA score of 72% (vs. 56%).
• Furthermore, the full version of Meena, with a filtering mechanism and tuned decoding, further advances the SSA score to 79%, which is not far from the 86% SSA achieved by the average human.
• “Google’s “Meena” chatbot was trained on a full TPUv3 pod (2048 TPU cores) for 30 full days – that’s more than $1,400,000 of compute time to train this chatbot model.” – Elliot Turner, CEO and founder of Hyperia .
• “So I was browsing the results for the new Google chatbot Meena, and they look pretty OK (if boring sometimes). However, every once in a while it enters ‘scary sociopath mode,’ which is, shall we say, sub-optimal” – Graham Neubig, Associate professor at Carnegie Mellon University .

• Lowering the perplexity through improvements in algorithms, architectures, data, and compute.
• Considering other aspects of conversations beyond sensibleness and specificity, such as, for example, personality and factuality.
• Tackling safety and bias in the models.
• further humanizing computer interactions; 
• improving foreign language practice; 
• making interactive movie and videogame characters relatable.
• Considering the challenges related to safety and bias in the models, the authors haven’t released the Meena model yet. However, they are still evaluating the risks and benefits and may decide otherwise in the coming months.

10. Recipes for Building an Open-Domain Chatbot , by Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston

Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to their partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent persona. We show that large scale models can learn these skills when given appropriate training data and choice of generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing failure cases of our models. 

The Facebook AI Research team shows that with appropriate training data and generation strategy, large-scale models can learn many important conversational skills, such as engagingness, knowledge, empathy, and persona consistency. Thus, to build their state-of-the-art conversational agent, called BlenderBot , they leveraged a model with 9.4B parameters, trained it on a novel task called Blended Skill Talk , and deployed beam search with carefully selected hyperparameters as a generation strategy. Human evaluations demonstrate that BlenderBot outperforms Meena in pairwise comparison 75% to 25% in terms of engagingness and 65% to 35% in terms of humanness.

• Large scale. The largest model has 9.4 billion parameters and was trained on 1.5 billion training examples of extracted conversations.
• Blended skills. The chatbot was trained on the Blended Skill Talk task to learn such skills as engaging use of personality, engaging use of knowledge, and display of empathy.
• Beam search used for decoding. The researchers show that this generation strategy, deployed with carefully selected hyperparameters, gives strong results. In particular, it was demonstrated that the lengths of the agent’s utterances is very important for chatbot performance (i.e, too short responses are often considered dull and too long responses make the chatbot appear to waffle and not listen).
• 75% of the time in terms of engagingness;
• 65% of the time in terms of humanness.
• In an A/B comparison between human-to-human and human-to-BlenderBot conversations, the latter were preferred 49% of the time as more engaging.
• a lack of in-depth knowledge if sufficiently interrogated; 
• a tendency to use simpler language; 
• a tendency to repeat oft-used phrases.
• Further exploring unlikelihood training and retrieve-and-refine mechanisms as potential avenues for fixing these issues.
• Facebook AI open-sourced BlenderBot by releasing code to fine-tune the conversational agent, the model weights, and code to evaluate it.

If you like these research summaries, you might be also interested in the following articles:

• 2020’s Top AI & Machine Learning Research Papers
• Novel Computer Vision Research Papers From 2020
• AAAI 2021: Top Research Papers With Business Applications
• ICLR 2021: Key Research Papers

Enjoy this article? Sign up for more AI research updates.

We’ll let you know when we release more summary articles like this one.

• Email Address *
• Name * First Last
• Natural Language Processing (NLP)
• Chatbots & Conversational AI
• Computer Vision
• Ethics & Safety
• Machine Learning
• Deep Learning
• Reinforcement Learning
• Generative Models
• Other (Please Describe Below)
• What is your biggest challenge with AI research? *
• Email This field is for validation purposes and should be left unchanged.

Reader Interactions

About Mariya Yao

Mariya is the co-author of Applied AI: A Handbook For Business Leaders and former CTO at Metamaven. She "translates" arcane technical concepts into actionable business advice for executives and designs lovable products people actually want to use. Follow her on Twitter at @thinkmariya to raise your AI IQ.

December 9, 2023 at 3:26 am

Thank you so much for sharing such great information with us. Your website is great. We are impressed by the details you have on your website. We have bookmarked this site. keep it up and thanks again.

March 14, 2024 at 8:49 am

quibusdam recusandae sint delectus nobis inventore dolorem nostrum omnis eum voluptas. autem blanditiis voluptatum delectus non asperiores sit. itaque accusamus voluptas repudiandae sed officiis. consectetur aut officia molestiae voluptatem sint quasi illum voluptate.

March 20, 2024 at 11:05 pm

Your article helped me a lot, is there any more related content? Thanks!

March 29, 2024 at 9:49 am

Leave a Reply

You must be logged in to post a comment.

About TOPBOTS

• Expert Contributors
• Terms of Service & Privacy Policy
• Contact TOPBOTS

Subscribe to our biweekly newsletter for the latest in applied AI, cutting-edge research updates, funding deals, and promising AI startups.

Cohere and Fujitsu Announce Strategic Partnership To Provide Japanese Enterprise AI Services

Feb 01, 2023

Medallia's On-Demand Streaming Network

• Español/Europa
• Español/América Latina
• Português/Brasil

AI-Powered Text Analytics for Everyone

Uncover high-impact insights and drive action with real-time, human-centric text analytics.

Democratized, Personalized, Actionable Text Analytics

Medallia’s omnichannel Text Analytics with Natural Language Understanding and AI – powered by Athena – enables you to quickly identify emerging trends and key insights at scale for each user role in your organization. 

Since we started building our native text analytics more than a decade ago, we’ve strived to build the most comprehensive, connected, accessible, actionable, easy-to-maintain, and scalable text analytics offering in the industry. Analyze all your unstructured data at a low cost of maintenance and unearth action-oriented insights that make your employees and customers feel seen.

"Customers looking for a fast time to value with OOTB omnichannel data models and language models tuned for multiple industries and business domains should put Medallia at the top of their shortlist."

THE FORRESTER WAVE™ Text Mining and Analytics Platforms, Q2, 2024

Generating text analytics insights for 1000+ programs worldwide

Conversations are complex; our text analytics can handle that

Pinpoint what happens – or doesn’t – in every interaction with text analytics that helps you understand complex conversations and prioritize key people, insights, and opportunities. 

Support your workflows, alerting, coaching, and other processes with Event Analytics and compound topics, which enable you to better understand how events unfold throughout an interaction.

Real-time text analytics that drive real-time actions

Automatically alert and surface emerging trends and missed opportunities to the right people based on role, prioritize support tickets, automate agent scoring, and support various workflows – all in real-time. Create alerts based on any change in categorization, sentiment, or any AI model, including effort, CX Risk, or Employee Recognition. 

Your time is precious; get more of it with real-time, action-oriented analytics.

Natural Language Understanding that is easy, robust, and cost-effective

Derive the hidden, implicit meaning behind words with AI-powered NLU that saves you time and money. Minimize the cost of ownership by combining low-maintenance AI models with the power of crowdsourcing in supervised machine learning models. 

Our NLU analyzes your data for themes, intent, empathy, dozens of complex emotions, sentiment, effort, and much more in dozens of languages and dialects so you can handle all your multilingual needs.

AI-powered and out-of-the-box topic models for all

Text analytics is no longer the domain of data wizards alone. With Medallia’s Text Analytics, you can build your own topic models in a low- to no-code environment.

You also have the option of hundreds of out-of-the-box topic models for every industry and use case at your fingertips. Gain access to accessible, easy-to-use models for the best, most accurate insights for your unique use cases, at scale.

Gain an objective view with customizable KPIs

Connect your organization to valuable insights with KPIs like sentiment and effort scoring to get an objective and accurate understanding of experiences with your organization. Leverage the power of crowd-sourced, consistent improvements to get the most accurate sentiment and effort scores.

Plus, create your own KPIs based on multiple criteria that are most important to you and your business, like empathy and competitor mentions. Generate an objective score across your text data, all automatically.

Omnichannel analytics for every industry, at scale

Medallia transforms unstructured data from billions of surveys, social media reactions, reviews, emails, conversations, voice, agent notes, SMS/ text messages, social media comments, and more into action-oriented insights. 

Our goal is simple - to empower you to focus on fostering the most impactful experiences with best-in-class omnichannel, scalable text analytics.

Related Resources

Medallia Text Analytics

Identify new trends, understand customer needs, and prioritize action with Medallia Text Analytics.

A Decade of Text Analytics Innovation

Here’s how Medallia has innovated and iterated to build the most accurate, actionable, and scalable text analytics.

The Big Story Behind Your Big Data

Six practices for making an impact with Text Analytics.

View All Resources

Oops… looks like we're having trouble. Please refresh the page and try again.

Ready to see more.

• Experience 101 Hub
• Net Promoter Score
• Voice of the Customer
• CEM Software
• Customer Retention
• Text Analytics
• Admin Suite
• Medallia CX ROI Calculator
• About Medallia
• Events & Virtual Experiences
• Developer Portal
• Documentation
• Medallia Leadership Team
• Data Protection

Drawing new insights and connections from Times journalism through Reader Q&A, Machine Translation, Topic Modeling and Document Analysis

Prioritizing Human Perspectives in Natural Language Processing

Streamlining reader q&a with zero-shot learning, generating open-ended questions from news articles.

Microsoft Research Lab – Asia

The next 10 years look golden for natural language processing research, share this page.

By Ming Zhou (opens in new tab) , Nan Duan (opens in new tab) , Furu Wei (opens in new tab) , Shujie Liu (opens in new tab) , and  Dongdong Zhang (opens in new tab) , Microsoft Research Asia

Language is the holy grail of Artificial Intelligence. The progress of Natural Language Processing (NLP) technologies will push the entire AI field forward. Here’s a view into what’s next.

Since the inception of Microsoft Research Asia, NLP has been a key research area in the field of Artificial Intelligence (AI). In the past 20 years, Microsoft Research Asia has developed NLP technologies, including those which have been shipped in Windows, Office, Bing, Microsoft Cognitive Services, Xiaoice, and Cortana.

This past work, which includes research in deep learning applied to machine translation, extractive machine reading comprehension, and grammar check, has achieved parity with human performance on related evaluation tasks

So what’s next? We believe that the next 10 years will be a golden era in NLP development, for the following reasons:

• Big data will become more easily collected, processed, and archived.
• NLP technology will extend into new applications for search engines, customer support, business intelligence, translation, education, law, finance, and more.
• Robot and IOT requirements will increasingly include text, speech, and vision capabilities.

These trends will stimulate large-scale investment in NLP and attract more talent to work in NLP research and development.

Areas of focus for the next generation of NLP research will include:

• Integrating knowledge and common sense into data-driven learning approaches.
• More attention to low-resource NLP tasks.
• Contextualized modelling and multi-turn dialogue understanding.
• Semantic analysis, leading to NLP that is knowledge-based, commonsense, and explainable.

Why NLP research is key

Natural Language Understanding (NLU) is a research area that uses the computer to analyze and extract key information from natural language sentences and texts, and then perform information retrieval, question-answering, machine translation, and text-generation activities. It is central to progress in many areas of AI, because the goal of AI overall is to make computers and smart devices listen, speak, and understand language; be able to think and solve problems; and even be able to create new things.

Recent progress in NLP includes:

Neural machine translation

Neural machine translation is a process of simulating how a human brain translates.

The task of translation is to convert a source language sentence into a target language sentence and retain the original meaning. When translating, human brains first try to understand the sentence, then form a semantic representation of the sentence in the mind, and finally transform this semantic representation into a sentence in another language. Neural machine translation simulates this human translation process, through two modular processes—encoding and decoding. The encoder is responsible for compressing source language sentences into vector representations in the semantic space, which are expected to contain the semantic information of source language sentences. The decoder generates semantically equivalent sentences of the target language based on semantic vectors provided by the encoder.

The advantage of the neural machine translation model lies in three aspects: end-to-end training, which reduces error propagation among multiple sub-models; the distributed representation of information, which can automatically learn multi-dimensional translation knowledge; and the use of global context information to complete the translation, rather than just using local context. Recurrent neural machine translation is an important foundational model, over which there have been many improvements on either advanced network structures or novel model training methods.

The translation quality of neural machine translation systems keeps improving, with the goal of reaching human-level performance. In 2018, the Chinese-English machine translation system, developed by Microsoft Research Asia in collaboration with the Microsoft Translator product team, reached a translation quality level comparable to human professional translation on the WMT 2017 news test dataset. This system combines four advanced technologies proposed by Microsoft Research Asia, including joint training and dual-learning techniques that can efficiently utilize large-scale monolingual training data to improve the model training, an agreement regularization technique to address the issue of exposure bias, as well as a deliberation network approach to improving translation quality with two-pass translations that simulate the human translation process.

Human-computer interaction

Human-computer interaction (HCI) aims to build machine intelligence that can communicate with humans by using natural language. Conversation as a Platform (CaaP) is one of the most important concepts for this.

Conversation as a Platform (CaaP) is a brand-new concept proposed by Microsoft CEO Satya Nadella in 2016. Satya thinks that conversation will become the next-generation interface, which will bring progress to both the artificial intelligence and device fields.

The reasons why this concept is important are two-fold. First, conversation-centered apps, such as WeChat and Facebook, have become part of everyone’s life, setting up our expectations for future HCI platforms. Second, a large portion of devices have small screens (such as cell phones) or even no screen (such as some IoT devices). On such devices, natural language presents the most straightforward and natural form of communication. Today, HCI using conversational systems can complete tasks such as buying coffee and booking tickets, and there are several CaaP platforms available for developers around the world to build their own conversation-based HCI systems.

In general, the technologies used for building such HCI systems can be divided into three layers: the chat layer, the search and question/answer (QA) layer, and the task-completion layer. The chat layer, such as Xiaoice, provides chat capability, which can make an HCI system more conversational and increase user engagement. The search and QA layer, such as Bing, provides search and question answering capabilities, which can return accurate search results or provide answers to a user’s questions. The task-completion layer, represented by Cortana, provides task-oriented dialogue capability that can help users complete specific tasks such as restaurant reservations, hotel booking, or weather inquiry; and once it gets to know your personal habits, it can remind you of meetings or suggest somewhere to go. A bot with these three layers can provide a natural and useful user experience.

Machine reading comprehension

Teaching a machine to read and comprehend text is a very important research challenge in natural language understanding.

The task and goal of machine reading comprehension is to design systems that can answer questions after reading a passage or document. There are a wide range of applications for this, including the ability for search engines to provide intelligent and accurate answers for natural-language queries by reading the relevant documents on the Web. In addition, machine reading comprehension can also be used in personal assistants, such as Cortana, so that Cortana can help answer customer support questions after reading documents (such as user manuals or product descriptions). It can be also used in work environments to help users read and process emails or business documents, and then summarize the relevant information. In the education domain, machine reading comprehension can be used to design tests. In legal circles, it can be used to help lawyers or judges by reading and understanding legal questions. In financial applications, machine reading comprehension can be used to extract information for making better financial decisions.

The recent advances in machine reading comprehension have been furthered by the use of large-scale, manually annotated datasets. The Stanford Question Answering Dataset (SQuAD) is the most widely used benchmark dataset for machine reading comprehension research. Stanford released SQuAD in July 2016 and it consists of 100,000 human-labeled question and answer pairs. The passages in SQuAD are from Wikipedia articles and each passage is annotated with no more than five questions, with answers that are exact sub-spans of each passage. Stanford divides the dataset into training, development, and test sets. The training set and development set are publicly available, while the test set is hidden from both researchers and participants. Participants need to submit their systems to the Stanford team to obtain the results on the test set, which will be updated on the SQuAD leaderboard. As of November 2018, there were more than 100 entries from academic and industry research labs.

The leaderboard indicates that there has been great progress in machine reading comprehension research in the last two years. In January 2018, the R-net system from Microsoft Research Asia was the first system to exceed human parity on the SQuAD dataset, in terms of the Exact Match (EM) metrics. In early 2018, systems from Alibaba and iFLYTEK also exceeded the EM test for human parity. In September 2018, the system from Microsoft Research Asia, nLnet, became the first to exceed both EM and F1 human parity on the SQuAD dataset. Google’s BERT then became the leader.

The SQuAD dataset provides a great platform and testing ground for the whole research community to develop, verify, and accumulate techniques to benefit the broader research effort in NLP. The technology stacks behind the recent progress of research on machine reading comprehension include end-to-end neural machine reading comprehension models; pretrained models, such as the ELMo from AI2 and BERT from Google AI, for machine reading comprehension and natural language processing; and system innovations on network structures, automatic data augmentation, and implementation.

AI creation

Infusing AI into creation processes and democratizing creation for ordinary people.

As early as 2005, Microsoft Research Asia successfully developed the Microsoft Couplet system, with the proposal and support of Dr. Harry Shum, who at the time was director of the lab. Given the user’s input of the first line of a couplet, the system can automatically generate the second sentence of a couplet, as well as the streamer description.

After that, we developed two intelligent AI creation systems: Metrical Poetry and Chinese Character Riddles. For example, in Chinese Character Riddles, the system is able to both solve and generate riddles based on Chinese characters.

In 2017, Microsoft Research Asia developed a system for writing modern poetry and composing music (including lyric generation and melody composition). This system of song generation has participated in the CCTV 1’s AI program (Machine vs. Human Intelligence). All of these show that deep learning technology and big data have great potential for mimicking a human’s ability to create, and that they can be used to help artists and others to create.

Taking the capability of lyrics generation as an example, the system will first generate a topic before writing the lyrics. For instance, if you would like to write a song related to “autumn,” “sundown,” and “sigh with feeling,” the user can add keywords such as “autumn wind,” “flowing year,” ” gleaming,” “changing,” and so on. The sequence-to-sequence neural networks are used to generate the sentences in the lyrics line-by-line, under the constraint of the topics.

To compose the melody for lyrics, the system should not only consider the quality of the melody, but also the correspondence between the lyrics and the melody. It requires that each note correspond to each word in the lyrics. Given the lyrics as input, we generate the lyric-conditional melody as well as the exact alignment between the generated melody and the given lyrics, simultaneously. More specifically, we develop the melody composition model based on the sequence-to-sequence framework, which is able to jointly produce musical notes and the corresponding alignment.

Hot NLP topics

We summarize the latest NLP technologies into five hot topics:

Hot topic 1: Pre-trained models (or representations)

How machines learn more general and effective pre-trained models (or representations) will continue to be one of the hottest research topics in the NLP area.

One major difficulty faced by many natural language tasks is the limited amount of training data. Today’s researchers are investigating how to learn general and effective pre-trained representations for language understanding, where words and text are represented as vectors. These are useful when task-specific training data are limited.

A Neural Probabilistic Language Model is a foundational work in neural language modeling. In this work, word embeddings are further fed into a neural sequence encoder to encode contextual information. Following this direction, many works, such Word2vec (opens in new tab) and GloVe (opens in new tab) , emerged to further improve the quality of learned word embeddings. One drawback of word embedding is its lack of context sensitivity: the representation of one word is the same regardless of the context it appears in. Work by Peters et al. with ELMo reveal that such context-sensitive representations have already been built by the neural language model. Instead of only using word embeddings, ELMo also leverages the sequence encoder from the language model; and such context-sensitive representations bring drastic improvements over traditional word embedding methods. More recently, BERT uses a transformer-based encoder and a masked word approach to train a very large bidirectional representation from large amount of text, which again brings astounding gains in a variety of tasks.

In the future, it is worth investigating new network structures, lightweight approaches, as well as incorporating world knowledge and common-sense knowledge to learn general pre-trained representations for language understanding. It is also interesting to see if further scaling up the model size and training on more text can bring further improvements.

Hot topic 2: Transfer learning and multi-task learning

Transfer learning has important and practical significance to NLP tasks that lack enough training data. Multi -task learning uses common knowledge from multiple task supervisions and improves model generalization.

In the era of deep learning, different NLP tasks often share encoders that have a homogeneous network structure, such as RNN, CNN, or transformer. This makes transfer learning more practical and straightforward. Using pre-trained word embeddings such as Word2Vec,ELMo or BERT, we employ a type of transfer learning method where the knowledge (word embeddings) learnt from a large-scale corpus via a language model is transferred to downstream tasks directly, by initializing corresponding network layers of downstream task models. Such methods are important to those tasks with little training data.

Multi-task learning is another paradigm that can use different task supervisions to improve a target task, by learning common knowledge from all involved tasks. In 2008, Collobert and Weston proposed a deep learning-based, multi-task framework, and it was the first work to combine deep learning and multi-task learning for NLP. In 2018, McCann proposed another multi-task learning framework, which treats all involved tasks as question-answering tasks and trains a unified model for ten NLP tasks. Experiments show that all tasks can benefit by using the common knowledge learnt from different task supervisions. Based on such common knowledge, each specific task can be further fine-tuned.

Hot topic 3: Knowledge and common sense

How to utilize knowledge and common sense in natural language understanding has become one of the most important topics in NLP.

With the rapid development of HCI engines (such as chat, QA, and dialogue systems), how to utilize knowledge and common sense in natural language understanding has become one of the most important topics in NLP, as they are essential for conversation engines or other types of HCI engines to understand user queries, manage conversations, and generate responses.

Wikipedia and knowledge graphs (such as Freebase and Satori) are two types of commonly used knowledge bases. Machine Reading Comprehension(MRC) is a typical NLP task based on Wikipedia, where the MRC model aims to extract an answer from the passage based on the input question. Semantic parsing is another typical NLP task based on a knowledge graph, which aims to convert an input question into a machine-readable and executable logical form. Both tasks are hot topics in NLP.

Commonsense knowledge refers to those facts that all humans are expected to know, such as lemons are sour and an elephant is bigger than a butterfly. Many HCI tasks, like QA and dialogue, need common sense to reason and generate responses. However, as most commonsense knowledge is rarely explicitly expressed in textual corpora, NLP models cannot use such knowledge directly. With the rapid development of chat, dialogue, and QA engines, how to build large-scale commonsense knowledgebases and apply them to various NLP tasks have been explored by many researchers in the last two decades.

Hot topic 4: Low-resource NLP tasks

Data augmentation methods are popularly used to enrich the data resource for low-resource NLP tasks, such as introducing domain knowledge (dictionaries and rules) and labeling more useful data with active learning.

For some NLP tasks, such as rare language translation, chatbot and customer service systems in specific domains and in multi-turn tasks, labeled data is hard to acquire and the data sparseness problem becomes serious. These are called low-resource NLP tasks. To enrich the training data, many data augmentation methods can be used. For example, we can introduce domain knowledge (dictionaries and rules) or leverage active learning to maximize the gain of labeling data. Researchers can also employ semi-supervised and unsupervised methods to use the unlabeled data. Labeled data from other tasks and other languages can also be used with multi-task learning and transfer learning.

Taking machine translation as an example, some rare language translation tasks only have a bilingual dictionary for model training, without any bilingual corpus. Based on this small dictionary of only a few thousand entries, cross-lingual word embedding methods can be used to map the source words and the target words into one semantic space, leveraging a large monolingual corpus. In this semantic space, the source word and the corresponding target word have similar word representations. Based on the cross-lingual word embedding, we can compute the semantic similarity of source and target words, which are used to build a word-based translation table. Together with the trained language model, we can build word-based statistical machine translation (SMT) systems, which are used to translate the monolingual corpus into a pseudo-bilingual corpus and turn the unsupervised translation task into a supervised one. Leveraging the pseudo-bilingual corpus, source-to-target and target-to-source neural translation models can be initialized and boosted with each other by using joint training methods and the large, monolingual corpus.

To improve the translation performance of rare languages, we also propose leveraging the large bilingual corpus between rich languages to boost four translation models for rare ones in a joint generalized EM training framework. Given two rich languages, such as X （Chinese） and Y (English), the rare language Z (such as Hebrew) is treated as a hidden state between X and Y. The translation process from X to Y can be redefined as translating X to Z first, and then translating from Z to Y, and similar for the direction from Y to X. Based on this, we can use the large bilingual data between X and Y to jointly train four translation models, which are P(Z|X), P(Y|Z), P(Z|Y), and P(X|Z), with the popularly used generalized EM training in an iterative process.

Hot topic 5: Multi-modal learning

As a typical multi-modal task, visual QA (VQA) receives great interest by researchers from both NLP and computer vision areas.

Before knowing how to speak, infants perceive the world by seeing, listening, and touching. This means language is not the only way to learn and communicate with the world. Therefore, we should substantially consider both language and other modalities for building artificial generic intelligence. This is called multi-modal learning.

As a typical multi-modal task, visual QA (VQA) receives great interest by researchers in the NLP and computer vision areas. Given an image and natural language question, VQA aims to generate the answer to the input question and depends on the deep understanding and sufficient interaction between the input question and image. Recently, researchers from Microsoft Research Asia presented two VQA approaches in this year’s CVPR and KDD, based on question generation and scene graph generation technologies respectively. We achieved state-of-the-art results on VQA benchmark datasets, including COCO and VQA 2.0. Besides VQA, video QA is another popular multi-modal learning task. Different from VQA, video QA returns a short video clip as the answer to the input query, which makes search results more vivid. With the rapid development of short music and video social platforms, live streaming apps, and mixed and artificial reality technology, how to understand, search, create, and utilize videos will become one of the keys to the next generation of HCI engines.

Future prospects

We think an ideal NLP framework could be a general-purpose architecture as described below. Note this would be one of the typical designs, there could be different design choice on using various technologies for a specific task.

As the first step, it works on the natural language sentence and obtains the word sequence, part-of-speech, dependency analysis, entity identification, intent identification, relation identification, and so on.

Then, the encoder will transform the information obtained into a semantic expression. In this procedure, the pre-trained word embedding and pre-trained entity embedding naturally bring in contextual information of a word or an entity. Furthermore, the same sentence will be encoded with other task-specific encoders and information obtained from these encoders are appended into the encoding of the main-task, with appropriate weights via transfer learning. The additional encoding from other task-specific encoding will further enrich the encoding of the input sentence.

Next, based on the semantic expression obtained from the above process, we can use a decoder to generate the expected output. Additionally, multi-task learning can be applied to introduce other NLP tasks as complementary resources to help with the main-task learning. If the task involves multi-turn modeling, we will need to record the output of the previous turn into memory and use it for the decoding and inference of the subsequent turns.

To realize this ideal NLP framework, we will need to implement the following tasks:

• Construct a large-scale commonsense knowledge base and set up effective evaluation tasks to push forward the related research.
• Study more effective expressions of words, phrases, and sentences, and build a more powerful pre-trained network for expressions at different levels.
• Push forward unsupervised learning and semi-supervised learning, by using a limited amount of knowledge to strengthen the learning ability and by building powerful, cross-lingual word-embedding models.
• Leverage the effect of multi-task learning and transfer learning in NLP tasks, and boost the effect of reinforcement learning for typical tasks such as multi-turn dialogue in customer support systems.
• Effectively model discourse and multi-turn conversation and multi-turn semantic analysis.
• Conduct user modeling and apply it to personalized recommendation and output systems.
• Build an expert system for a specific domain that uses the new generation of reasoning systems, task-completion and conversation systems, and integrates both domain knowledge and commonsense knowledge.
• Develop the explainability of NLP systems by using semantic analysis and knowledge systems.

In the next ten years, NLP research will explode. We can expect that there will be big progress in NLP fundamental research, core technologies, and important applications. As Bill Gates said, “We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten.” This is true for NLP as well.

Let’s imagine what changes NLP will bring us in the next ten years.

• In ten years, machine translation systems will be capable of modeling the context of a sentence and handling new terms. People will use a machine system as a spontaneous interpreter at meetings or presentations.
• An electronic personal assistant will understand your natural command and completes orders for food, flowers, and tickets. You will get used to being served by a robot customer support agent.
• When you climb a mountain, you can tell your phone about your thoughts and upload a photo. Then, your phone will pop up a poem with beautiful language and the photo, and that poem can be sent out to your friends.
• Many news articles will be written by a computer.
• A computer teacher corrects your English pronunciation and polishes your sentences through natural conversation.
• A robot will analyze massive documents and provide a data analysis report in a timely manner to help business leaders make decisions.
• News, books, classes, meetings, articles, and goods will be recommended to you by an intelligent recommendation system.
• Robots will help lawyers to find evidence and suggest similar cases. It can also discover the flaws of a contract or write up a legal document.
• And more, limited only by our imaginations.

While some of the above-mentioned scenarios have already emerged, they will become more mature in the next ten years. In the future, NLP and other AI technologies will dramatically change human life. To realize this bright future, we will continue to innovate boldly and solidly advance by balancing research and application. We will create a new generation of technology designed to serve all of human society.

• Follow on X
• Like on Facebook
• Follow on LinkedIn
• Subscribe on Youtube
• Follow on Instagram
• Subscribe to our RSS feed

Share this page:

• Share on Facebook
• Share on LinkedIn
• Share on Reddit

• Discovery Platform
• Innovation Scouting
• Startup Scouting
• Technology Scouting
• Tech Supplier Scouting
• Venture Clienting
• Startup Program
• Trend Intelligence
• Business Intelligence
• All Industries
• Industry 4.0
• Manufacturing
• Case Studies
• Research & Development
• Corporate Strategy
• Corporate Innovation
• Open Innovation
• New Business Development
• Product Development
• Agriculture
• Construction
• Sustainability
• All Startups
• Circularity
• All Innovation
• Business Trends
• Emerging Tech
• Innovation Intelligence
• New Companies
• Scouting Trends
• Startup Programs
• Supplier Scouting
• Tech Scouting
• Top AI Tools
• Trend Tracking
• All Reports [PDF]
• Circular Economy
• Engineering
• Oil & Gas

Share this:

• Click to share on Facebook (Opens in new window)
• Click to share on Twitter (Opens in new window)
• Click to share on LinkedIn (Opens in new window)

9 Natural Language Processing Trends in 2023

Are you curious about which natural language processing trends & startups will soon impact your business? Explore our in-depth industry research on 1 645 NLP startups & scaleups and get data-driven insights into technology-based solutions in our Natural Language Processing Innovation Map!

Natural language processing (NLP) is a subset of AI which finds growing importance due to the increasing amount of unstructured language data. The rapid growth of social media and digital data creates significant challenges in analyzing vast user data to generate insights. Further, interactive automation systems such as chatbots are unable to fully replace humans due to their lack of understanding of semantics and context. To tackle these issues, natural language models are utilizing advanced machine learning (ML) to better understand unstructured voice and text data. This article provides an overview of the top global natural language processing trends in 2023. They range from virtual agents and sentiment analysis to semantic search and reinforcement learning.

Innovation Map outlines the Top 9 Natural Language Processing Trends & 18 Promising Startups

For this in-depth research on the Top Natural Language Processing Trends & Startups, we analyzed a sample of 1 645 global startups & scaleups. The result of this research is data-driven innovation intelligence that improves strategic decision-making by giving you an overview of emerging technologies & startups advancing data processing. These insights are derived by working with our Big Data & Artificial Intelligence-powered StartUs Insights Discovery Platform , covering 2 500 000+ startups & scaleups globally. As the world’s largest resource for data on emerging companies, the SaaS platform enables you to identify relevant startups, emerging technologies & future industry trends quickly & exhaustively.

In the Innovation Map below, you get an overview of the Top 9 Natural Language Processing Trends & Innovations that impact 1 645 companies worldwide. Moreover, the Natural Language Processing Innovation Map reveals 18 hand-picked startups, all working on emerging technologies that advance their field.

Top 9 Natural Language Processing Trends

• Virtual Assistants
• Sentiment Analysis
• Multilingual Language Models
• Named Entity Recognition
• Language Transformers
• Transfer Learning
• Text Summarization
• Semantic Search
• Reinforcement Learning

Click to download

Tree Map reveals the Impact of the Top 9 Natural Language Processing Trends

Based on the Natural Language Processing Innovation Map, the Tree Map below illustrates the impact of the Top 9 NLP Trends in 2023. Virtual assistants improve customer relationships and worker productivity through smarter assistance functions. Advances in learning models, such as reinforced and transfer learning, are reducing the time to train natural language processors. Besides, sentiment analysis and semantic search enable language processors to better understand text and speech context. Named entity recognition (NER) works to identify names and persons within unstructured data while text summarization reduces text volume to provide important key points. Language transformers are also advancing language processors through self-attention. Lastly, multilingual language models use machine learning to analyze text in multiple languages.

Schedule Platform Demo

Global Startup Heat Map covers 1 645 Natural Language Processing Startups & Scaleups

The Global Startup Heat Map below highlights the global distribution of the 1 645 exemplary startups & scaleups that we analyzed for this research. Created through the StartUs Insights Discovery Platform, the Heat Map reveals that the US sees the most startup activity.

Below, you get to meet 18 out of these 1 645 promising startups & scaleups as well as the solutions they develop. These natural language processing startups are hand-picked based on criteria such as founding year, location, funding raised, & more. Depending on your specific needs, your top picks might look entirely different.

Top 8 Natural Language Processing Trends in 2023

1. virtual assistants.

There is a growing interest in virtual assistants in devices and applications as they improve accessibility and provide information on demand. However, they deliver accurate information only if the virtual assistants understand the query without misinterpretation. That is why startups are leveraging NLP to develop novel virtual assistants and chatbots. They mitigate processing errors and work continuously, unlike human virtual assistants. Additionally, NLP-powered virtual assistants find applications in providing information to factory workers, assisting academic research, and more.

Servicely advances Intelligent Service Management

Australian startup Servicely develops Sofi , an AI-powered self-service automation software solution. Its self-learning AI engine uses plain English to observe and add to its knowledge, which improves its efficiency over time. This allows Sofi to provide employees and customers with more accurate information. The flexible low-code, virtual assistant suggests the next best actions for service desk agents and greatly reduces call-handling costs.

Vox automates Conversational Experiences

Vox is a Malaysian startup that automates conversational experiences. The startup’s virtual assistant engages with customers over multiple channels and devices as well as handles various languages. Besides, its conversational AI uses predictive behavior analytics to track user intent and identifies specific personas. This enables businesses to better understand their customers and personalize product or service offerings.

2. Sentiment Analysis

Our increasingly digital world generates exponential amounts of data as audio, video, and text. While natural language processors are able to analyze large sources of data, they are unable to differentiate between positive, negative, or neutral speech. Moreover, when support agents interact with customers, they are able to adapt their conversation based on the customers’ emotional state which typical NLP models neglect. Therefore, startups are creating NLP models that understand the emotional or sentimental aspect of text data along with its context. Such NLP models improve customer loyalty and retention by delivering better services and customer experiences.

Y Meadows provides AI-based Customer Support

US-based startup Y Meadows automates customer support requests using AI. The startup’s customer service automation solution collects data from customers through multiple channels, such as emails and web forms, and understands human intent. Its deep learning-based NLP model perceives message context instead of focusing on keywords. Y Meadow’s semantics-based solution finds use across industries for customer issue handling.

Spiky delivers Video Sentiment Analytics

Spiky is a US startup that develops an AI-based analytics tool to improve sales calls, training, and coaching sessions. The startup’s automated coaching platform for revenue teams uses video recordings of meetings to generate engagement metrics. It also generates context and behavior-driven analytics and provides various unique communication and content-related metrics from vocal and non-verbal sources. This way, the platform improves sales performance and customer engagement skills of sales teams.

3. Multilingual Language Models

Communication is highly complex, with over 7000 languages spoken across the world, each with its own intricacies. Most current natural language processors focus on the English language and therefore either do not cater to the other markets or are inefficient. The availability of large training datasets in different languages enables the development of NLP models that accurately understand unstructured data in different languages. This improves data accessibility and allows businesses to speed up their translation workflows and increase their brand reach.

Lingoes offers No-Code Multilingual Text Analytics

Finnish startup Lingoes makes a single-click solution to train and deploy multilingual NLP models. It features intelligent text analytics in 109 languages and features automation of all technical steps to set up NLP models. Additionally, the solution integrates with a wide range of apps and processes as well as provides an application programming interface (API) for special integrations. This enables marketing teams to monitor customer sentiments, product teams to analyze customer feedback, and developers to create production-ready multilingual NLP classifiers.

NLP Cloud provides Pre-trained Multilingual AI Models

NLP Cloud is a French startup that creates advanced multilingual AI models for text understanding and generation. They feature custom models, customization with GPT-J, follow HIPPA, GDPR, and CCPA compliance, and support many languages. Besides, these language models are able to perform summarization, entity extraction, paraphrasing, and classification. NLP Cloud’s models thus overcome the complexities of deploying AI models into production while mitigating in-house DevOps and machine learning teams.

4. Named Entity Recognition

Data classification and annotation are important for a wide range of applications such as autonomous vehicles, recommendation systems, and more. However, classifying data from unstructured data proves difficult for nearly all traditional processing algorithms. Named entity recognition (NER) is a language processor that removes these limitations by scanning unstructured data to locate and classify various parameters. Besides identifying person names, organizations, brands, etc. NER classifies dates and times, email addresses, and numerical measurements like money and weight. NER models thus facilitate data extraction workflows across industries.

M47AI enables AI-based Data Annotation

Spanish startup M47AI offers an AI-based data annotation platform to improve data labeling. It uses NER to identify and categorize names, locations, etc. The platform also tags words based on grammar, part of speech, function, and definition. It then performs entity linking to connect entity mentions in the text with a predefined set of relational categories. Besides improving data labeling workflows, the platform reduces time and cost through intelligent automation.

HyperGlue simplifies Unstructured Text Data Analysis

HyperGlue is a US-based startup that develops an analytics solution to generate insights from unstructured text data. It utilizes natural language processing techniques such as topic clustering, NER, and sentiment reporting. Companies use the startup’s solution to discover anomalies and monitor key trends from customer data.

5. Language Transformers

Natural language solutions require massive language datasets to train processors. This training process deals with issues, like similar-sounding words, that affect the performance of NLP models. Language transformers avoid these by applying self-attention mechanisms to better understand the relationships between sequential elements. Moreover, this type of neural network architecture ensures that the weighted average calculation for each word is unique.

Build & Code aids Construction Document Processing

German startup Build & Code uses NLP to process documents in the construction industry. The startup’s solution uses language transformers and a proprietary knowledge graph to automatically compile, understand, and process data. It features automatic documentation matching, search, and filtering as well as smart recommendations. This solution consolidates data from numerous construction documents, such as 3D plans and bills of materials (BOM), and simplifies information delivery to stakeholders.

Birch.AI advances Call Center Operation Automation

Birch.AI is a US-based startup that specializes in AI-based automation of call center operations. The startup’s solution utilizes transformer-based NLPs with models specifically built to understand complex, high-compliance conversations. This includes healthcare, insurance, and banking applications. Birch.AI’s proprietary end-to-end pipeline uses speech-to-text during conversations. It also generates a summary and applies semantic analysis to gain insights from customers. The startup’s solution finds applications in challenging customer service areas such as insurance claims, debt recovery, and more.

6. Transfer Learning

Machine learning tasks are domain-specific and models are unable to generalize their learning. This causes problems as real-world data is mostly unstructured, unlike training datasets. Consequently, this affects the predictability of the trained models. However, many language models are able to share much of their training data using transfer learning to optimize the general process of deep learning. The application of transfer learning in natural language processing significantly reduces the time and cost to train new NLP models.

Got It AI creates Autonomous Conversational AI

US-based startup Got It AI offers a conversational AI platform that improves customer experience management. It uses transfer learning and NLP models with transformers such as BERT, GPT-3, and T5. Moreover, its product suite, AutoFlow , identifies the conversational paths that virtual agents follow and uses historical conversation data to improve customer engagement.

QuillBot enables AI-powered Paraphrasing

QuillBot is a US-based startup that makes an AI-powered paraphrasing tool. It uses natural language generation (NLG) and transfer learning to power its customizable text slider and AI-powered thesaurus that suggests synonyms. The tool also checks grammar, creates summaries, generates citations, and checks plagiarism. Additionally, it integrates directly into Google Chrome and Microsoft Word to enable better, faster, and smarter writing.

7. Text Summarization

Natural language processors are extremely efficient at analyzing large datasets to understand human language as it is spoken and written. However, typical NLP models lack the ability to differentiate between useful and useless information when analyzing large text documents. Therefore, startups are applying machine learning algorithms to develop NLP models that summarize lengthy texts into a cohesive and fluent summary that contains all key points. The main befits of such language processors are the time savings in deconstructing a document and the increase in productivity from quick data summarization.

SummarizeBot offers Blockchain-powered Summaries

Latvian startup SummarizeBot develops a blockchain-based platform to extract, structure, and analyze text. It leverages AI to summarize information in real time, which users share via Slack or Facebook Messenger. Besides, it provides summaries of audio content within a few seconds and supports multiple languages. SummarizeBot’s platform thus finds applications in academics, content creation, and scientific research, among others.

Zeon AI Labs provides an Intelligent Search Platform

Zeon AI Labs is an Indian startup that makes a summary generator. The startup’s summarization solution, DeepDelve , uses NLP to provide accurate and contextual answers to questions based on information from enterprise documents. Additionally, it supports search filters, multi-format documents, autocompletion, and voice search to assist employees in finding information. The startup’s other product, IntelliFAQ , finds answers quickly for frequently asked questions and features continuous learning to improve its results. These products save time for lawyers seeking information from large text databases and provide students with easy access to information from educational libraries and courseware.

8. Semantic Search

Search engines are an integral part of workflows to find and receive digital information. One of the barriers to effective searches is the lack of understanding of the context and intent of the input data. NLP enables semantic search queries that analyze search intent. This improves search accuracy and provides more relevant results. Hence, semantic search models find applications in areas such as eCommerce, academic research, enterprise knowledge management, and more.

deepset builds Natural Language Interfaces

German startup deepset develops a cloud-based software-as-a-service (SaaS) platform for NLP applications. It features all the core components necessary to build, compose, and deploy custom natural language interfaces, pipelines, and services. The startup’s NLP framework, Haystack , combines transformer-based language models and a pipeline-oriented structure to create scalable semantic search systems. Moreover, the quick iteration, evaluation, and model comparison features reduce the cost for companies to build natural language products.

Vectara develops an ML-based Search Pipeline

Vectara is a US-based startup that offers a neural search-as-a-service platform to extract and index information. It contains a cloud-native, API-driven, ML-based semantic search pipeline, Vectara Neural Rank , that uses large language models to gain a deeper understanding of questions. Moreover, Vectara’s semantic search requires no retraining, tuning, stop words, synonyms, knowledge graphs, or ontology management, unlike other platforms.

9. Reinforcement Learning

Currently, NLP-based solutions struggle when dealing with situations outside of their boundaries. Therefore, AI models need to be retrained for each specific situation that it is unable to solve, which is highly time-consuming. Reinforcement learning enables NLP models to learn behavior that maximizes the possibility of a positive outcome through feedback from the environment. This enables developers and businesses to continuously improve their NLP models’ performance through sequences of reward-based training iterations. Such learning models thus improve NLP-based applications such as healthcare and translation software, chatbots, and more.

AyGLOO creates Explainable AI

Spanish startup AyGLOO creates an explainable AI solution that transforms complex AI models into easy-to-understand natural language rule sets. The startup applies AI techniques based on proprietary algorithms and reinforcement learning to receive feedback from the front web and optimize NLP techniques. AyGLOO’s solution finds applications in customer lifetime value (CLV) optimization, digital marketing, and customer segmentation, among others.

VeracityAI specializes in Natural Language Model Training

VeracityAI is a Ghana-based startup specializing in product design, development, and prototyping using AI, ML, and deep learning. The startup’s reinforcement learning-based recommender system utilizes an experience-based approach that adapts to individual needs and future interactions with its users. This not only optimizes the efficiency of solving cold start recommender problems but also improves recommendation quality.

Discover all Natural Language Processing Trends, Technologies & Startups

Machine learning models such as reinforcement learning, transfer learning, and language transformers drive the increasing implementation of NLP systems. Text summarization, semantic search, and multilingual language models expand the use cases of NLP into academics, content creation, and so on. The cost and resource-efficient development of NLP solutions is also a necessary requirement to increase their adoption.

The Natural Language Processing Trends & Startups outlined in this report only scratch the surface of trends that we identified during our data-driven innovation & startup scouting process. Among others, transfer learning, semantic web, and behavior analysis will transform the sector as we know it today. Identifying new opportunities & emerging technologies to implement into your business goes a long way in gaining a competitive advantage. Get in touch to easily & exhaustively scout startups, technologies & trends that matter to you!

Your Name Business Email Company

Get our  free newsletter  on technology and startups.

Protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Discover our Free Technologies x Industries Report

Get free updates on Global Startups, Technologies & Trends!

Join 25k subscribers.

Business Email

Artificial Intelligence

First & Last Name Business Email Company

Advanced Analytics

Industry 4.0 22 pages report.

Leverage our unparalleled data advantage to quickly and easily find hidden gems among 4.7M+ startups, scaleups. Access the world's most comprehensive innovation intelligence and stay ahead with AI-powered precision.

Get in touch

Your Name Business Email Company How can we support you?   (optional)

Discover the Impact of Emerging Technologies on Your Industry

Uncover how over 1000 use cases across 40+ industries are transforming business landscapes. Download our comprehensive technology matrix to fast-track your innovation agenda and drive growth!

Protected by reCAPTCHA and the Google  Privacy Policy  and  Terms of Service  apply.

Publications

Daniel Jurafsky . 2014. The Language of Food . W. W. Norton.

Christopher D. Manning , Prabhakar Raghavan , and Hinrich Schütze . 2008. Introduction to Information Retrieval . Cambridge University Press.

Daniel Jurafsky and James H. Martin . 2008. Speech and Language Processing: An Introduction to Natural Language Processing, Speech Recognition, and Computational Linguistics . 2nd edition. Prentice-Hall.

Christopher D. Manning and Hinrich Schütze . 1999. Foundations of Statistical Natural Language Processing . Cambridge, MA: MIT Press.

Barbara A. Fox , Dan Jurafsky , and Laura A. Michaelis (Eds.). 1999. Cognition and Function in Language . Stanford, CA: CSLI Publications.

Avery D. Andrews and Christopher D. Manning . 1999. Complex Predicates and Information Spreading in LFG . Stanford, CA: CSLI Publications.

• Our Promise
• Our Achievements
• Our Mission
• Proposal Writing
• System Development
• Paper Writing
• Paper Publish
• Synopsis Writing
• Thesis Writing
• Assignments
• Survey Paper
• Conference Paper
• Journal Paper
• Empirical Paper
• Journal Support
• Innovative 12+ Natural Language Processing Thesis Topics

Generally, natural language processing is the sub-branch of Artificial Intelligence (AI). Natural language processing is otherwise known as NLP. It is compatible in dealing with multi-linguistic aspects and they convert the text into binary formats in which computers can understand it.  Primarily, the device understands the texts and then translates according to the questions asked. These processes are getting done with the help of several techniques. As this article is concentrated on delivering the natural language processing thesis topics , we are going to reveal each and every aspect that is needed for an effective NLP thesis .

NLP has a wide range of areas to explore in which enormous researches will be conducted. As the matter of fact, they analyses emotions, processes images, summarize texts, answer the questions & translates automatically, and so on.

Thesis writing is one of the important steps in researches. As they can deliver the exact perceptions of the researcher to the opponents hence it is advisable to frame the proper one. Let us begin this article with an overview of the NLP system . Are you ready to sail with us? Come on, guys!!!

“This is the article which is framed to the NLP enthusiasts in order to offer the natural language processing thesis topics”

What is Actually an NLP?

• NLP is the process of retrieving the meaning of the given sentence
• For this they use techniques & algorithms in order to extract the features
• They are also involved with the following,
• Audio capturing
• Text processing
• Conversion of audio into text
• Human-computer interaction

This is a crisp overview of the NLP system. NLP is one of the major technologies that are being used in the day to day life. Without these technologies, we could not even imagine a single scenario . In fact, they minimized the time of human beings by means of spelling checks, grammatical formations and most importantly they are highly capable of handling audio data . In this regard, let us have an idea of how does the NLP works in general. Shall we get into that section? Come let’s move on to that!!!

How does NLP Works?

• Unstructured Data Inputs
• Lingual Knowledge
• Domain Knowledge
• Domain Model
• Corpora Model Training
• Tools & Methods

The above listed are necessary when input is given to the model. The NLP model is in need of the above-itemized aspects to process the unstructured data in order to offer the structured data by means of parsing, stemming and lemmatization, and so on. In fact, NLP is subject to the classifications by their eminent features such as generation & understanding.  Yes my dear students we are going to cover the next sections with the NLP classifications.  

Classifications of NLP

• Natural Language-based Generation
• Natural Language-based Understanding

The above listed are the 2 major classifications of NLP technology . In these classifications let us have further brief explanations of the natural language-based understanding for your better understanding.

• Biometric Domains
• Spam Detection
• Opinion/Data Mining
• Entity Linking
• Named Entity Recognition
• Relationship Extraction

This is how the natural language-based understanding is sub-classified according to its functions. In recent days, NLP is getting boom in which various r esearches and projects are getting investigated and implemented successfully by our technical team. Generally, NLP processes are getting performed in a structural manner. That means they are overlays in several steps in crafting natural language processing thesis topics . Yes dears, we are going to envelop the next section with the steps that are concreted with the natural language processing.

NLP Natural Language Processing Steps

• Segmentation of Sentences
• Tokenization of Words
• PoS Tagging
• Parsing of Syntactic Contexts
• Removing of Stop Words
• Lemmatization & Stemming
• Classification of Texts
• Emotion/Sentiment Analysis

Here POS stands for the Parts of Speech . These are some of the steps involved in natural language processing. NLP performs according to the inputs given. Here you might need examples in these areas. For your better understanding, we are going to illustrate to you about the same with clear bulletin points. Come let us try to understand them.

• Let we take inputs as text & speech
• Text inputs are analyzed by “word tokenization”
• Speech inputs are analyzed by “phonetics”

In addition to that, they both are further processed in the same manner as they are,

• Morphological Analysis
• Syntactic Analysis
• Semantic Understanding
• Speech Processing

The above listed are the steps involved in NLP tasks in general . Word tokenization is one of the major which points out the vocabulary words presented in the word groups . Though, NLP processes are subject to numerous challenges. Our technical team is pointed out to you the challenges involved in the current days for a better understanding. Let’s move on to the current challenges sections.

Before going to the next section, we would like to highlight ourselves here. We are one of the trusted crew of technicians who are dynamically performing the NLP-based projects and researches effectively . As the matter of fact, we are offering so many successful projects all over the world by using the emerging techniques in technology. Now we can have the next section.

Current Challenges in NLP

• Context/Intention Understanding
• Voice Ambiguity/Vagueness
• Data Transformation
• Semantic Context Extracting
• Word Phrase Matching
• Vocabulary/Terminologies Creation
• PoS Tagging & Tokenization

The above listed are the current challenges that get involved in natural language processing. Besides, we can overcome these challenges by improving the NLP model by means of their performance. On the other hand, our technical experts in the concern are usually testing natural language processing approaches to abolish these constraints.

In the following passage, our technical team elaborately explained to you the various natural language processing approaches for the ease of your understanding. In fact, our researchers are always focusing on the students understanding so that they are categorizing each and every edge needed for the NLP-oriented tasks and approaches .  Are you interested to know about that? Now let’s we jump into the section.

Different NLP Approaches

Domain Model-based Approaches

• Loss Centric
• Feature Centric
• Pre-Training
• Pseudo Labeling
• Data Selection
• Model + Data-Centric

Machine Learning-based Approaches

• Association
• K-Means Clustering
• Anomalies Recognition
• Data Parsing
• Regular Emotions/Expressions
• Syntactic Interpretations
• Pattern Matching
• BFS Co-location Data
• BERT & BioBERT
• Decision Trees
• Logistic Regression
• Linear Regression
• Random Forests
• Support Vector Machine
• Gradient-based Networks
• Convolutional Neural Network
• Deep Neural Networks

Text Mining Approaches

• K-nearest Neighbor
• Naïve Bayes
• Predictive Modeling
• Association Rules
• Classification
• Document Indexing
• Term & Inverse Document Frequency
• Document Term Matrix
• Distribution
• Keyword Frequency
• Term Reduction/Compression
• Stemming/lemmatization
• Tokenization
• NLP & Log Parsing
• Text Taxonomies
• Text Classifications
• Text Categorization
• Text Clustering

The above listed are the 3 major approaches that are mainly used for natural languages processing in real-time . However, there are some demerits and merits are presented with the above-listed approaches. It is also important to know about the advantages and disadvantages of the NLP approaches which will help you to focus on the constraints and lead will lead you to the developments. Shall we discuss the pros and cons of NLP approaches? Come on, guys!

Advantages & Disadvantages of NLP Approaches

• Effortless Debugging
• Effective Precisions
• Multi-perspectives
• Short Form Reading
• Ineffective Parsing
• Poor Recalls
• Excessive Skills
• Low Scalability
• Speed Processes
• Resilient Results
• Effective Documentation
• Better Recalls
• High Scalability
• Narrow Understanding
• Poor in Reading Messages
• Huge Annotations
• Complex in Debugging

The foregoing passage conveyed to you the pros and cons of two approaches named machine learning and text mining. The best approach is also having pros and cons. If you do want further explanations or clarifications on that you can feel free to approach our researchers to get benefit from us. Generally, NLP models are trained to perform every task in order to recognize the inputs with latest natural language processing project ideas . Yes, you people guessed right! The next section is all about the training models of the NLP.

Training Models in NLP

• Scratch dataset such as language-specific BERTs & multi-linguistic BERT
• These are the datasets used in model pre-training
• Auxiliary based Pre-Training
• It is the additional data tasks used for labeled adaptive pre-training
• Multi-Phase based Pre-Training
• Domain & broad tasks are the secondary phases of pre-training
• Unlabeled data sources make differences in the multiphase pre-training
• TAPT, DAPT, AdaptaBERT & BioBERT are used datasets

As this article is named as natural language processing thesis topics , here we are going to point out to you the latest thesis topics in NLP for your reference. Commonly, a thesis is the best illustration of the projects or researches done in the determined areas. In fact, they convey the researchers’ perspectives & thoughts to the opponent by the effective structures of the thesis. If you are searching for thesis writing assistance then this is the right platform, you can surely approach our team at any time.

In the following passage, we have itemized some of the latest thesis topics in NLP .  We thought that it would help you a lot. Let’s get into the next section. As this is an important section, you are advised to pay your attention here. Are you really interested in getting into the next section? Come let us also learn them.

Latest Natural Language Processing Thesis Topics

• Cross & Multilingual based NLP Methods
• Multi-modal based NLP Methodologies
• Provocative based NLP Systems
• Graph oriented NLP Techniques
• Data Amplification in NLP
• Reinforcement Learning based NLP
• Dialogue/Voice Assistants
• Market & Customer Behavior Modeling
• Text Classification by Zero-shot/Semi-supervised Learning & Sentiment Analysis
• Text Generation & Summarization
• Relation & Knowledge Extraction for Fine-grained Entity Recognition
• Knowledge & Open-domain based Question & Answering

These are some of the latest thesis topics in NLP . As the matter of fact, we have delivered around 200 to 300 thesis with fruitful outcomes. Actually, they are very innovative and unique by means of their features. Our thesis writing approaches impress the institutes incredibly. At this time, we would like to reveal the future directions of the NLP for the ease of your understanding.

How to select the best thesis topics in NLP?

• See the latest IEEE and other benchmark papers
• Understand the NLP Project ideas recently proposed
• Highlight the problems and gaps
• Get the future scope of each existing work

Come let’s move on to the next section.

Future Research Directions of Natural Language Processing

• Logical Reasoning Chains
• Statistical Integrated Multilingual & Domain Knowledge Processing
• Combination of Interacting Modules

On the whole, NLP requires a better understanding of the texts. In fact, they understand the text’s meaning by relating to the presented word phrases. Conversion of the natural languages in reasoning logic will lead NLP to future directions. By allowing the modules to interact can enhance the NLP pipelines and modules. So far, we have come up with the areas of natural language processing thesis topics and each and every aspect that is needed to do a thesis. If you are in dilemma you could have the valuable opinions of our technical experts.

“Let’s begin to work on your experimental areas and yield the stunning outcomes”

MILESTONE 1: Research Proposal

Finalize journal (indexing).

Before sit down to research proposal writing, we need to decide exact journals. For e.g. SCI, SCI-E, ISI, SCOPUS.

Research Subject Selection

As a doctoral student, subject selection is a big problem. Phdservices.org has the team of world class experts who experience in assisting all subjects. When you decide to work in networking, we assign our experts in your specific area for assistance.

Research Topic Selection

We helping you with right and perfect topic selection, which sound interesting to the other fellows of your committee. For e.g. if your interest in networking, the research topic is VANET / MANET / any other

Literature Survey Writing

To ensure the novelty of research, we find research gaps in 50+ latest benchmark papers (IEEE, Springer, Elsevier, MDPI, Hindawi, etc.)

Case Study Writing

After literature survey, we get the main issue/problem that your research topic will aim to resolve and elegant writing support to identify relevance of the issue.

Problem Statement

Based on the research gaps finding and importance of your research, we conclude the appropriate and specific problem statement.

Writing Research Proposal

Writing a good research proposal has need of lot of time. We only span a few to cover all major aspects (reference papers collection, deficiency finding, drawing system architecture, highlights novelty)

MILESTONE 2: System Development

Fix implementation plan.

We prepare a clear project implementation plan that narrates your proposal in step-by step and it contains Software and OS specification. We recommend you very suitable tools/software that fit for your concept.

Tools/Plan Approval

We get the approval for implementation tool, software, programing language and finally implementation plan to start development process.

Pseudocode Description

Our source code is original since we write the code after pseudocodes, algorithm writing and mathematical equation derivations.

Develop Proposal Idea

We implement our novel idea in step-by-step process that given in implementation plan. We can help scholars in implementation.

Comparison/Experiments

We perform the comparison between proposed and existing schemes in both quantitative and qualitative manner since it is most crucial part of any journal paper.

Graphs, Results, Analysis Table

We evaluate and analyze the project results by plotting graphs, numerical results computation, and broader discussion of quantitative results in table.

Project Deliverables

For every project order, we deliver the following: reference papers, source codes screenshots, project video, installation and running procedures.

MILESTONE 3: Paper Writing

Choosing right format.

We intend to write a paper in customized layout. If you are interesting in any specific journal, we ready to support you. Otherwise we prepare in IEEE transaction level.

Collecting Reliable Resources

Before paper writing, we collect reliable resources such as 50+ journal papers, magazines, news, encyclopedia (books), benchmark datasets, and online resources.

Writing Rough Draft

We create an outline of a paper at first and then writing under each heading and sub-headings. It consists of novel idea and resources

Proofreading & Formatting

We must proofread and formatting a paper to fix typesetting errors, and avoiding misspelled words, misplaced punctuation marks, and so on

Native English Writing

We check the communication of a paper by rewriting with native English writers who accomplish their English literature in University of Oxford.

Scrutinizing Paper Quality

We examine the paper quality by top-experts who can easily fix the issues in journal paper writing and also confirm the level of journal paper (SCI, Scopus or Normal).

Plagiarism Checking

We at phdservices.org is 100% guarantee for original journal paper writing. We never use previously published works.

MILESTONE 4: Paper Publication

Finding apt journal.

We play crucial role in this step since this is very important for scholar’s future. Our experts will help you in choosing high Impact Factor (SJR) journals for publishing.

Lay Paper to Submit

We organize your paper for journal submission, which covers the preparation of Authors Biography, Cover Letter, Highlights of Novelty, and Suggested Reviewers.

Paper Submission

We upload paper with submit all prerequisites that are required in journal. We completely remove frustration in paper publishing.

Paper Status Tracking

We track your paper status and answering the questions raise before review process and also we giving you frequent updates for your paper received from journal.

Revising Paper Precisely

When we receive decision for revising paper, we get ready to prepare the point-point response to address all reviewers query and resubmit it to catch final acceptance.

Get Accept & e-Proofing

We receive final mail for acceptance confirmation letter and editors send e-proofing and licensing to ensure the originality.

Publishing Paper

Paper published in online and we inform you with paper title, authors information, journal name volume, issue number, page number, and DOI link

MILESTONE 5: Thesis Writing

Identifying university format.

We pay special attention for your thesis writing and our 100+ thesis writers are proficient and clear in writing thesis for all university formats.

Gathering Adequate Resources

We collect primary and adequate resources for writing well-structured thesis using published research articles, 150+ reputed reference papers, writing plan, and so on.

Writing Thesis (Preliminary)

We write thesis in chapter-by-chapter without any empirical mistakes and we completely provide plagiarism-free thesis.

Skimming & Reading

Skimming involve reading the thesis and looking abstract, conclusions, sections, & sub-sections, paragraphs, sentences & words and writing thesis chorological order of papers.

Fixing Crosscutting Issues

This step is tricky when write thesis by amateurs. Proofreading and formatting is made by our world class thesis writers who avoid verbose, and brainstorming for significant writing.

Organize Thesis Chapters

We organize thesis chapters by completing the following: elaborate chapter, structuring chapters, flow of writing, citations correction, etc.

Writing Thesis (Final Version)

We attention to details of importance of thesis contribution, well-illustrated literature review, sharp and broad results and discussion and relevant applications study.

How PhDservices.org deal with significant issues ?

1. novel ideas.

Novelty is essential for a PhD degree. Our experts are bringing quality of being novel ideas in the particular research area. It can be only determined by after thorough literature search (state-of-the-art works published in IEEE, Springer, Elsevier, ACM, ScienceDirect, Inderscience, and so on). SCI and SCOPUS journals reviewers and editors will always demand “Novelty” for each publishing work. Our experts have in-depth knowledge in all major and sub-research fields to introduce New Methods and Ideas. MAKING NOVEL IDEAS IS THE ONLY WAY OF WINNING PHD.

2. Plagiarism-Free

To improve the quality and originality of works, we are strictly avoiding plagiarism since plagiarism is not allowed and acceptable for any type journals (SCI, SCI-E, or Scopus) in editorial and reviewer point of view. We have software named as “Anti-Plagiarism Software” that examines the similarity score for documents with good accuracy. We consist of various plagiarism tools like Viper, Turnitin, Students and scholars can get your work in Zero Tolerance to Plagiarism. DONT WORRY ABOUT PHD, WE WILL TAKE CARE OF EVERYTHING.

3. Confidential Info

We intended to keep your personal and technical information in secret and it is a basic worry for all scholars.

• Technical Info: We never share your technical details to any other scholar since we know the importance of time and resources that are giving us by scholars.
• Personal Info: We restricted to access scholars personal details by our experts. Our organization leading team will have your basic and necessary info for scholars.

CONFIDENTIALITY AND PRIVACY OF INFORMATION HELD IS OF VITAL IMPORTANCE AT PHDSERVICES.ORG. WE HONEST FOR ALL CUSTOMERS.

4. Publication

Most of the PhD consultancy services will end their services in Paper Writing, but our PhDservices.org is different from others by giving guarantee for both paper writing and publication in reputed journals. With our 18+ year of experience in delivering PhD services, we meet all requirements of journals (reviewers, editors, and editor-in-chief) for rapid publications. From the beginning of paper writing, we lay our smart works. PUBLICATION IS A ROOT FOR PHD DEGREE. WE LIKE A FRUIT FOR GIVING SWEET FEELING FOR ALL SCHOLARS.

5. No Duplication

After completion of your work, it does not available in our library i.e. we erased after completion of your PhD work so we avoid of giving duplicate contents for scholars. This step makes our experts to bringing new ideas, applications, methodologies and algorithms. Our work is more standard, quality and universal. Everything we make it as a new for all scholars. INNOVATION IS THE ABILITY TO SEE THE ORIGINALITY. EXPLORATION IS OUR ENGINE THAT DRIVES INNOVATION SO LET’S ALL GO EXPLORING.

Client Reviews

I ordered a research proposal in the research area of Wireless Communications and it was as very good as I can catch it.

I had wishes to complete implementation using latest software/tools and I had no idea of where to order it. My friend suggested this place and it delivers what I expect.

It really good platform to get all PhD services and I have used it many times because of reasonable price, best customer services, and high quality.

My colleague recommended this service to me and I’m delighted their services. They guide me a lot and given worthy contents for my research paper.

I’m never disappointed at any kind of service. Till I’m work with professional writers and getting lot of opportunities.

- Christopher

Once I am entered this organization I was just felt relax because lots of my colleagues and family relations were suggested to use this service and I received best thesis writing.

I recommend phdservices.org. They have professional writers for all type of writing (proposal, paper, thesis, assignment) support at affordable price.

You guys did a great job saved more money and time. I will keep working with you and I recommend to others also.

These experts are fast, knowledgeable, and dedicated to work under a short deadline. I had get good conference paper in short span.

Guys! You are the great and real experts for paper writing since it exactly matches with my demand. I will approach again.

I am fully satisfied with thesis writing. Thank you for your faultless service and soon I come back again.

Trusted customer service that you offer for me. I don’t have any cons to say.

I was at the edge of my doctorate graduation since my thesis is totally unconnected chapters. You people did a magic and I get my complete thesis!!!

- Abdul Mohammed

Good family environment with collaboration, and lot of hardworking team who actually share their knowledge by offering PhD Services.

I enjoyed huge when working with PhD services. I was asked several questions about my system development and I had wondered of smooth, dedication and caring.

I had not provided any specific requirements for my proposal work, but you guys are very awesome because I’m received proper proposal. Thank you!

- Bhanuprasad

I was read my entire research proposal and I liked concept suits for my research issues. Thank you so much for your efforts.

- Ghulam Nabi

I am extremely happy with your project development support and source codes are easily understanding and executed.

Hi!!! You guys supported me a lot. Thank you and I am 100% satisfied with publication service.

- Abhimanyu

I had found this as a wonderful platform for scholars so I highly recommend this service to all. I ordered thesis proposal and they covered everything. Thank you so much!!!

Related Pages

A New Method for Predicting the Importance of Scientific Articles on Topics of Interest Using Natural Language Processing and Recurrent Neural Networks

• Conference paper
• First Online: 28 July 2024
• Cite this conference paper

• Adrian Lopez 13 ,
• David Dutan 13 &
• Remigio Hurtado 13  

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 1013))

Included in the following conference series:

• International Congress on Information and Communication Technology

31 Accesses

The realization of the state of the art presents significant challenges for researchers, as it involves addressing the extensive and dynamic amount of existing literature in a specific area. The explosion of information and the constant evolution of knowledge make identifying and synthesizing the most relevant contributions a complex task. Variability in source quality, diversity of methodological approaches, and lack of standardization in results presentation also hinder information systematization. Additionally, the speed at which new research emerges adds an additional layer of challenge to keeping the state of the art up to date. In this context, researchers must develop critical search skills, efficient information management, and discernment to provide a comprehensive and accurate view of existing research in a specific field. Initially, we employed the term frequency-inverse document frequency (TF-IDF) method along with total citation influence (CIT) calculations to determine the main topics and influence of articles within the scientific community. The primary innovation of this research is the development of an RNN model. This model has proven to be equally effective as the traditional TF-IDF method, complemented by CIT, in identifying the relevance and importance of articles.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Hurtado R, Picón C, Muñoz A, Hurtado J (2024) Survey of intent-based networks and a methodology based on machine learning and natural language processing. In: Proceedings of eighth international congress on information and communication technology. Springer Nature Singapore, Singapore

Google Scholar  

Park K, Kim J, Lee J (2019) Visual field prediction using recurrent neural network. Sci Rep 9(1):8385. https://doi.org/10.1038/s41598-019-44852-6

Xu M, Du J, Guan Z, Xue Z, Kou F, Shi L, Xu X, Li A (2021) A multi-RNN research topic prediction model based on spatial attention and semantic consistency-based scientific influence modeling. Comput Intell Neurosci 2021:1766743. https://doi.org/10.1155/2021/1766743

Article   Google Scholar  

Kreutz C, Schenkel R (2022) Scientific paper recommendation systems: a literature review of recent publications

Zhu Y, Lin Q, Lu H, Shi K, Qiu P, Niu Z (2021) Recommending scientific paper via heterogeneous knowledge embedding based attentive recurrent neural networks. Knowl-Based Syst 215:106744. ISSN: 0950-7051. https://doi.org/10.1016/j.knosys.2021.106744

Kwapong BA, Anarfi R, Fletcher KK

Zhou X, Li Y, Liang W (2021) CNN-RNN based intelligent recommendation for online medical pre-diagnosis support. IEEE/ACM Trans Comput Biol Bioinform 18(3):912–921. https://doi.org/10.1109/TCBB.2020.2994780

Roy D, Dutta M (2022) A systematic review and research perspective on recommender systems. J Big Data 9(1):59. ISSN: 2196-1115. https://doi.org/10.1186/s40537-022-00592-5

Kim S-W, Gil J-M (2019) Research paper classification systems based on TF-IDF and LDA schemes. Hum-Centric Comput Inf Sci. https://doi.org/10.1186/s13673-019-0192-7

Download references

Author information

Authors and affiliations.

Universidad Politécnica Salesiana, Cuenca, Ecuador

Adrian Lopez, David Dutan & Remigio Hurtado

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Adrian Lopez .

Editor information

Editors and affiliations.

Middlesex University, London, UK

Xin-She Yang

The University of Reading, Reading, UK

Simon Sherratt

Department of Computer Science and Engineering, Techno International New Town, Kolkata, West Bengal, India

Nilanjan Dey

Global Knowledge Research Foundation, Ahmedabad, Gujarat, India

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Cite this paper.

Lopez, A., Dutan, D., Hurtado, R. (2024). A New Method for Predicting the Importance of Scientific Articles on Topics of Interest Using Natural Language Processing and Recurrent Neural Networks. In: Yang, XS., Sherratt, S., Dey, N., Joshi, A. (eds) Proceedings of Ninth International Congress on Information and Communication Technology. ICICT 2024 2024. Lecture Notes in Networks and Systems, vol 1013. Springer, Singapore. https://doi.org/10.1007/978-981-97-3559-4_50

Download citation

DOI : https://doi.org/10.1007/978-981-97-3559-4_50

Published : 28 July 2024

Publisher Name : Springer, Singapore

Print ISBN : 978-981-97-3558-7

Online ISBN : 978-981-97-3559-4

eBook Packages : Engineering Engineering (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

MIT Technology Review

• Newsletters

Reimagining cloud strategy for AI-first enterprises

New research illuminates findings about cloud maturity, data readiness, and AI adoption.

• MIT Technology Review Insights archive page

In partnership with Infosys

The rise of generative artificial intelligence (AI), natural language processing, and computer vision has sparked lofty predictions: AI will revolutionize business operations, transform the nature of knowledge work, and boost companies’ bottom lines and the larger global economy by trillions of dollars.

Executives and technology leaders are eager to see these promises realized, and many are enjoying impressive results of early AI investments. Balakrishna D.R. (Bali), executive vice president, global services head, AI and industry verticals at Infosys, says that generative AI is already proving game-changing for tasks such as knowledge management, search and summarization, software development, and customer service across sectors such as financial services, retail, health care, and automotive.

Realizing AI’s full potential on a mass scale will require more than just executives’ enthusiasm; becoming a truly AI-first enterprise will require a significant, sustained investment in cloud infrastructure and strategy. In 2024, the cloud has evolved beyond its initial purpose as a storage tool and cost saver to become a crucial driver of innovation, transformation, and disruption. Now, with AI in the mix, enterprises are looking to the cloud to support large language models (LLMs) to maximize R&D performance and prevent cybersecurity attacks, among other high-impact use cases.

A 2023 report by Infosys looks at how prepared companies are to realize the combined potential of cloud and AI. To further assess this state of readiness, MIT Technology Review Insights and Infosys surveyed 500 business leaders across industries such as IT, manufacturing, financial services, and consumer goods about how their organizations are thinking about—and acting upon—an integrated cloud and AI strategy.

This research found that most companies are still experimenting and preparing their infrastructure landscape for AI from a cloud perspective—and many are planning additional investments to accelerate their progress.

Download the full report .

Artificial intelligence

How to opt out of meta’s ai training.

Your posts are a gold mine, especially as companies start to run out of AI training data.

• Melissa Heikkilä archive page

Why does AI hallucinate?

The tendency to make things up is holding chatbots back. But that’s just what they do.

• Will Douglas Heaven archive page

How generative AI could reinvent what it means to play

AI-powered NPCs that don’t need a script could make games—and other worlds—deeply immersive.

• Niall Firth archive page

Synthesia’s hyperrealistic deepfakes will soon have full bodies

With bodies that move and hands that wave, deepfakes just got a whole lot more realistic.

Stay connected

Get the latest updates from mit technology review.

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at [email protected] with a list of newsletters you’d like to receive.

A New Method for Predicting the Importance of Scientific Articles on Topics of Interest Using Natural Language Processing and Recurrent Neural Networks

• In book: Proceedings of Ninth International Congress on Information and Communication Technology (pp.625-634)
• This person is not on ResearchGate, or hasn't claimed this research yet.

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

No full-text available

To read the full-text of this research, you can request a copy directly from the authors.

• Int J Digit Libr

• Ralf Schenkel

• Comput Intell Neurosci
• Mingying Xu

• Sang-Woon Kim
• Joon-Min Gil
• Keunheung Park

• Remigio Hurtado
• Cristian Picón
• Arantxa Muñoz
• Juan Hurtado
• KNOWL-BASED SYST

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

• Frontiers in Medical Engineering
• Clinical Engineering
• Research Topics

Advanced Computing for Improving Outcomes in Healthcare

Total Downloads

Total Views and Downloads

About this Research Topic

Advances in computing, such as machine learning, natural language processing, and cognitive computing, are driving automation across multiple sectors. These computational advancements have the potential to revolutionize medicine, a field traditionally reliant on highly skilled human interaction. Among the envisioned impacts of these systems are the identification of patients at risk of adverse events, cost reduction in both patient care and hospital administration, and heightened efficiency in the medical workforce. Nevertheless, the field of medicine presents distinctive challenges to application of these automations, such as stringent privacy requirements, fragmented care models, brittle datasets, and a general lack of familiarity with these emerging technologies among medical professionals. In navigating these challenges, it is crucial for innovations to not only address technical aspects but also meet the high expectations related to privacy, interoperability, and the unique nuances of medical practice. The integration of computational methods in medicine holds significant promise but necessitates a careful approach to ensure alignment with the complex nature of healthcare practices and uphold the highest standards of patient care. Our goal is to identify new innovations and expand upon existing practices in advanced computing within medical practice. These improvements should help to optimize healthcare outcomes, navigate the intricate challenges associated with the integration of computational methods into medicine, explore the bioethics of computing in healthcare, and address the prevailing lack of familiarity within the medical community. Ultimately, this project aspires to contribute valuable insights and innovative solutions, fostering the incorporation of advanced computing technologies into the intricate landscape of medical care to improve patient care and advance the work of healthcare professionals. This research topic seeks to elucidate innovations in the introduction and integration of advanced computing within medical care. Any Frontiers article types are welcome to be submitted. Suggested topic areas include but are not limited to: 1. Development: Addresses the need for new models or analyses of datasets that improve healthcare outcomes. Examples include studies decreasing mortality rates, pinpointing patients susceptible to adverse events, and curbing healthcare costs or inefficiencies. 2. Integration: Addresses the multifaceted challenges inherent in incorporating computational methods into medical practice. Examples include ensuring user interfaces or dashboards, interoperability, or toolsets for analyzing data. 3. Bioethics of Computing in Healthcare: Addresses ethical considerations and novel dilemmas associated with the application of computational advancements in healthcare. Examples include questions of privacy, consent, and the responsible use of technology. 4. Education: Addresses the lack of familiarity or mistrust towards computing within the medical community or investigates strategies for cultivating comfort and proficiency with these tools. Examples include educational initiatives, effective use of existing curricula or methods, or other education tailored to medical professionals.

Keywords : artificial intelligence; machine learning; outcomes

Important Note : All contributions to this Research Topic must be within the scope of the section and journal to which they are submitted, as defined in their mission statements. Frontiers reserves the right to guide an out-of-scope manuscript to a more suitable section or journal at any stage of peer review.

Topic Editors

Topic coordinators, submission deadlines.

Participating Journals

Manuscripts can be submitted to this Research Topic via the following journals:

total views

• Demographics

No records found

total views article views downloads topic views

Top countries

Top referring sites, about frontiers research topics.

With their unique mixes of varied contributions from Original Research to Review Articles, Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author.

Natural Language Processing to Analyze Growing Data

What this blog covers:.

• What is NLP and how it works
• Understanding the components of NLP
• How Kyvos uses Gen AI and NLQ for supercharged analytics

Natural Language Processing (NLP) has completely changed the way we interact with technology. From getting our daily tasks completed by virtual assistants like Siri and Alexa to sophisticated chatbots for enhanced customer service, NLP is at the core of many AI innovations.

But what exactly is NLP?

NLP is a subfield of artificial intelligence (AI) that bridges the gap between human communication and machine understanding, helping computers analyze human language and leading to a treasure trove of exciting applications.

Imagine chatting with a virtual assistant that can answer your questions just like a human would or effortlessly navigating sign boards and menus in a foreign country by having real-time translation at your fingertips. These are just a few examples of how NLP makes our lives easier.

Whether it is breaking down language barriers and streamlining everyday processes, NLP is transforming user experiences while making technology more intuitive and available. According to a new study by Grand View Research, the global natural language processing market size is estimated to reach USD 439.85 billion by 2030, expanding at a CAGR of 40.4% from 2023 to 2030. The immense potential and rapid growth of this field are not ending anytime soon. Join us as we dive deeper into the world of NLP, exploring its techniques, applications and overall potential.

Understanding NLP: The Core Concepts

NLP: The modern translator between computers and humans enables machines to understand, interpret and talk to us in natural language. For NLP to achieve this remarkable feat, what really works behind the scenes is the underlying technologies, such as machine learning and deep learning to process and analyze natural language data. More on these key technologies and tasks that drive NLP applications below:

Technologies Fueling NLP

Machine Learning (ML) : The ability of NLP to understand natural language comes from ML. An ML algorithm identifies patterns and generates predictions by being trained on massive volumes of data. For example, an algorithm trained on millions of emails develops the pattern and can distinguish between work and spam emails with accuracy. NLP uses ML algorithms to perform tasks like sentiment analysis and text classification.

Deep Learning : A subset of ML, deep learning uses complex neural networks to process information similar to how the human brain would do. This capability enables NLP systems to understand the intricacies of natural language, such as different types of tones (sarcastic, positive or slang) by copying how a human brain learns these fine distinctions.

Using the combination of these powerful technologies, NLP has made human-computer interaction more intuitive than ever before, from enabling search engines to become smarter powering virtual assistants and chatbots with intelligence.

NLP Tasks and Functions

Text Classification : Text classification is a core NLP functionality. Its main purpose is to organize large amounts of unstructured text (meaning the raw text data in the system). NLP algorithms are trained on huge volumes of labeled text consisting of documents and snippets of a specific category. For example, an email classification system might be trained on millions of emails labeled as “work,” “spam” or “personal.” The system then takes this text data and structures it for further analysis for specific features, such as keywords and sentence structure. Based on these features and training data, the NLP models assign a category to a new text piece.

Sentiment Analysis : How do businesses measure or understand customer satisfaction from their reviews? NLP does the real work in the background by using sentiment analysis. It enables computers to recognize the emotional tone (positive, negative or neutral) based on the written text. It is specially used in monitoring social media interactions, analyzing customer feedback and creating chatbots that can respond based on the emotions detected in the input text.

Named Entity Recognition : NER is an NLP technique used to identify and classify specific elements within text. NER enables computers to recognize, understand and extract structured information out of the input unstructured text, allowing them to categorize these entities in a meaningful way. This technique is used for applications like text summarization, question answering, knowledge graph building, etc. Consider entities as the key characters in a story. They can be the names of people, companies, locations, quantities or dates. These pre-defined entities are specifically categorized in a way to help computers understand the context (who, what, when and where) of the input text.

Machine Translation : The world just got a whole lot smaller, thanks to machine translation. It refers to the use of AI and ML algorithms to convert one language to another in the form of text or speech. This enables seamless communication across languages in real-time. Machine translation in NLP aims to not only produce grammatically correct translations but also retain their original meaning. As an example, if a Spanish tourist reads a store sign that says “Closed” in English and uses a machine translation tool to decode its meaning in Spanish. The tool would look for words that correspond to translations in the database and give out “Cerrado” as an output to the tourist.

Simply Explained: How NLP Works?

While the specific steps involved in each NLP application can vary, here’s a brief overview of the techniques that are used in NLP.

• Text Preprocessing – As the data is fed into the system, the first step it goes through is text preprocessing. In this, the system gets rid of irrelevant information and works on organizing the given data. It removes punctuation marks, extra spaces, stop words, checks for correct spellings and makes the overall text consistent.
• Tokenization – In the second step, the sentences from given data are broken down into smaller fragments of individual words. These smaller units are called tokens (each word is assigned a token responding to it). The system then takes these tokens to understand and analyze further. For example, a sentence like “I love my dog” can be given tokens as “I=1, love=2, my=3 and dog=4”. These tokens (numbers) enable the machine to understand the data for processing.
• Lemmatization – In this step, different variations of each word are filtered and categorized for the true meaning. As an example, “running” and “ran” are two different words but their root meaning or relating action stays the same, no matter where they are used. Lemmatization converts these words into a common form to make the machine understand their meaning.
• Part-of-Speech (POS) Tagging – Like the grammatical tags we used in school, POS tagging recognizes the function (is it a noun, verb, adjective, adverb, etc.) of all the words in each statement. Defining this enables the machine to understand the structure of the sentence and how the words are related to each other.
• Text Analysis – Based on the type of NLP task, this step takes the processing further by gathering the data, prepping it and finally analyzing it for outputs. The prepping of the data is done by combining and using different techniques mentioned above. Categories are assigned to the input text, e.g., spam or important email, followed by tone identification (positive, negative or neutral) along with extracting specific entities (people, places or organizations).

Understanding the Components of NLP

NLP uses techniques from both computational linguistics and ML to analyze massive volumes of data in natural language. Broadly speaking, there are three main components of NLP:

• NLG- Natural Language Generation , as the name suggests, is the process of enabling computational machines to generate information for effective communication. It is a branch of AI that focuses on transforming data into human-readable text or speech. To execute this, the system starts selecting relevant data from the larger set and decides what information needs to be included in the generated text. Next, it creates the structure of the text including its tone, style and the overall message. As a final step, the system chooses the right words, grammatical syntax and converts planned sentences into natural language.
• NLU- Natural Language Understanding refers to the process of helping machines comprehend human language and grasp the meaning of the given words and sentences. The process requires breaking down human language into smaller components, sentences, words and phrases, as the first step. Once this is done, it converts the input into machine-understandable format. Finally, it extracts meaning by understanding the input text with context, ambiguity and synonyms.
• Search-based NLQ – In this approach, users ask their question in natural language by typing in a text box (think web searching). Once the query is run, the system analyzes the keywords and maps them to data points or earlier asked questions. The answer’s accuracy in this system depends on the detailing of the query and the capability of the system to map the user’s intent while searching for the query response. Think of it as searching for a book in a library. If someone knows the correct title and author, the chances of finding the book are higher.
• Guided NLQ – This approach provides more structure and assistance as it helps users to get in-depth information on their original query by providing prompts, suggestions and drop-down menus. Users can refine their query further and select the right data fields without having to think about the underlying data. In the context of the book-searching example, guided NLQ is like asking a librarian to guide through the library’s organization and narrow down the search to find what a person is looking for.

The Kyvos Angle: How it Uses Gen AI and NLQ for Supercharged Analytics

As enterprise data grows by the second, business users often find themselves drowning in it, struggling to find relevant insights. And that’s exactly where Kyvos Copilot enters the picture. The platform leverages the power of Gen AI and NLQ and allows users to interact with complex datasets effortlessly. Generative AI is a sub-field of AI that creates original content in the form of text, images or other formats. It learns from huge datasets and creates new, original results. Leveraging this technology, users can ask questions in plain business language, and Kyvos will translate them into powerful queries and deliver relevant visualizations.

• Conversational Analytics for Everyone – Kyvos Copilot’s chat interface lets users talk directly to their data. For any natural language question, it chooses the best-suited semantic model to deliver super-fast, accurate answers in the form of visualizations or insightful reports. It also retains the context of previous inquiries, understands its connection with new questions and tailors its response accordingly.
• From Text to Powerful Queries – It empowers power users with its text-to-query capabilities by seamlessly converting natural language questions into sophisticated MDX and SQL formulas, unlocking the true power of data.
• Natural Language Summarization – Extracting key takeaways from vast datasets is another great advantage of using Kyvos Copilot. The platform analyzes anomalies, identifies KPIs, unveils trends and summarizes business insights in a human-readable format without getting wrapped up in technical details. These summaries are then delivered directly to the users’ inboxes so that they never miss any important metric.

By harnessing the power of NLQ, Kyvos Copilot allows users to have a dynamic conversation with their data and achieve superfast, actionable insights. Contact our experts to know more and understand how we deliver true self-serve analytics to global enterprises.

Request Demo

Previous Post Assured Data-Driven Success with AI For Data Analytics

Next post an introduction to embedded analytics.

• Generative AI
• Universal Semantic Layer
• Analytical Data Warehouse
• BI & Reporting
• Data Platforms
• Google Cloud
• Visualization Tools
• Kyvos Excel Add-in
• MicroStrategy
• Documentation
• Human Rights