publications

2024

1. EACL

   Cross-lingual Editing in Multilingual Language Models

   Himanshu Beniwal, Kowsik Nandagopan D, and Mayank Singh

   2024

   PDF

2023

1. EMNLP

   Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance

   Pritam Kadasi, and Mayank Singh

   In Findings of the Association for Computational Linguistics: EMNLP 2023, Dec 2023

   Abs HTML PDF

   The NLP community has long advocated for the construction of multi-annotator datasets to better capture the nuances of language interpretation, subjectivity, and ambiguity. This paper conducts a retrospective study to show how performance scores can vary when a dataset expands from a single annotation per instance to multiple annotations. We propose a novel multi-annotator simulation process to generate datasets with varying annotation budgets. We show that similar datasets with the same annotation budget can lead to varying performance gains. Our findings challenge the popular belief that models trained on multi-annotation examples always lead to better performance than models trained on single or few-annotation examples.

2. arXiv

   Unlocking Model Insights: A Dataset for Automated Model Card Generation

   Shruti Singh, Hitesh Lodwal, Husain Malwat, and 2 more authors

   Dec 2023

   PDF
3. EACL

   MUTANT: A Multi-sentential Code-mixed Hinglish Dataset

   Rahul Gupta, Vivek Srivastava, and Mayank Singh

   In Findings of the Association for Computational Linguistics: EACL 2023, May 2023

   Abs HTML PDF

   The multi-sentential long sequence textual data unfolds several interesting research directions pertaining to natural language processing and generation. Though we observe several high-quality long-sequence datasets for English and other monolingual languages, there is no significant effort in building such resources for code-mixed languages such as Hinglish (code-mixing of Hindi-English). In this paper, we propose a novel task of identifying multi-sentential code-mixed text (MCT) from multilingual articles. As a use case, we leverage multilingual articles from two different data sources and build a first-of-its-kind multi-sentential code-mixed Hinglish dataset i.e., MUTANT. We propose a token-level language-aware pipeline and extend the existing metrics measuring the degree of code-mixing to a multi-sentential framework and automatically identify MCT in the multilingual articles. The MUTANT dataset comprises 67k articles with 85k identified Hinglish MCTs. To facilitate future research directions, we will make the dataset and the code publicly available upon publication.

2022

1. LREC

   The Bull and the Bear: Summarizing Stock Market Discussions

   Ayush Kumar, Dhyey Jani, Jay Shah, and 3 more authors

   In Proceedings of the Thirteenth Language Resources and Evaluation Conference, Jun 2022

   Abs HTML PDF

   Stock market investors debate and heavily discuss stock ideas, investing strategies, news and market movements on social media platforms. The discussions are significantly longer in length and require extensive domain expertise for understanding. In this paper, we curate such discussions and construct a first-of-its-kind of abstractive summarization dataset. Our curated dataset consists of 7888 Reddit posts and manually constructed summaries for 400 posts. We robustly evaluate the summaries and conduct experiments on SOTA summarization tools to showcase their limitations. We plan to make the dataset publicly available. The sample dataset is available here: \urlhttps://dhyeyjani.github.io/RSMC

2. ACL

   The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through

   Shruti Singh, and Mayank Singh

   In Findings of the Association for Computational Linguistics: ACL 2022, May 2022

   Abs HTML PDF

   Language models are increasingly becoming popular in AI-powered scientific IR systems. This paper evaluates popular scientific language models in handling (i) short-query texts and (ii) textual neighbors. Our experiments showcase the inability to retrieve relevant documents for a short-query text even under the most relaxed conditions. Additionally, we leverage textual neighbors, generated by small perturbations to the original text, to demonstrate that not all perturbations lead to close neighbors in the embedding space. Further, an exhaustive categorization yields several classes of orthographically and semantically related, partially related and completely unrelated neighbors. Retrieval performance turns out to be more influenced by the surface form rather than the semantics of the text.

3. CODS-COMAD

   Code-Mixed NLG: Resources, Metrics, and Challenges

   Vivek Srivastava, and Mayank Singh

   In 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD), May 2022

   Abs HTML PDF

   This tutorial will elaborate on various available resources for the natural language generation (NLG) tasks in code-mixed languages. We will also discuss the adaptability, limitations, and challenges with various evaluation metrics for the code-mixed NLG. In addition, we will put forward a set of open research questions pertaining to the tasks, resources, experiments, and metrics for the code-mixed NLG.

4. CODS-COMAD

   Program Synthesis: Does Feedback Help?

   Harsh Patel, Praveen Venkatesh, Shivam Sahni, and 3 more authors

   In 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD), May 2022

   Abs HTML PDF

   Computers are devices that execute precise instructions provided to them using various programming languages. However, the idea of delivering instructions to a computer through natural language could vastly simplify the act of programming as a specific task. Generating code from high-level descriptions for a given program is a significantly challenging task and has been an active area of research in the natural language processing domain. In this paper, we present a novel feedback-based deep learning approach for synthesizing code from human-specified descriptions. Inspired by the dual-learning mechanism, our framework uses a feedback loss to produce more consistent and robust predictions. We show how our approach fares empirically on standard code generation datasets and achieves state-of-the-art results on the NAPS (Natural Program Synthesis) dataset.

5. Phys. A: Stat.

   Identifying early-warning indicators of onset of sudden collapse in networked infrastructure systems against sequential disruptions

   Utkarsh Gangwal, Mayank Singh, Pradumn Kumar Pandey, and 3 more authors

   Physica A: Statistical Mechanics and its Applications, May 2022

   Abs HTML

   Tsunamis, power blackouts, and distribution systems failure drastically affect the networked infrastructure systems which further affect a countries economy. Moreover, if these systems reach critical thresholds, they may experience disproportionate losses in the system’s functionality. Here we propose an approach to identify the critical thresholds and observe the presence of warning regions for real-world transportation systems. While attack tolerance of networked systems has been intensively studied for the disruptions originating from a single point of failure, there have been instances where real-world systems are subject to concurrent disruptions at multiple locations. We determine the entire robustness characteristics of transportation networks of disparate architecture subject to disruptions of varying sizes. Using United States Airspace Airport network and Indian Railways Network data, and synthetic networks as prototype class of systems, we study their responses to synthetic attack strategies of varying sizes. We also observe the significant relationships between network robustness and size of simultaneous disruptions for the complex networked infrastructures for random failures and targeted attacks. Our approach can serve as a paradigm to understand the point of sudden collapse in real-world systems, and the principle can be extended to other network infrastructures to address critical issues of risk management, resilience, and system safety.

6. arXiv

   BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

   Mayank Singh

   ArXiv, Oct 2022

   PDF

2021

1. W-NUT

   PoliWAM: An Exploration of a Large Scale Corpus of Political Discussions on WhatsApp Messenger

   Vivek Srivastava, and Mayank Singh

   In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), Nov 2021

   Abs HTML PDF

   WhatsApp Messenger is one of the most popular channels for spreading information with a current reach of more than 180 countries and 2 billion people. Its widespread usage has made it one of the most popular media for information propagation among the masses during any socially engaging event. In the recent past, several countries have witnessed its effectiveness and influence in political and social campaigns. We observe a high surge in information and propaganda flow during election campaigning. In this paper, we explore a high-quality large-scale user-generated dataset curated from WhatsApp comprising of 281 groups, 31,078 unique users, and 223,404 messages shared before, during, and after the Indian General Elections 2019, encompassing all major Indian political parties and leaders. In addition to the raw noisy user-generated data, we present a fine-grained annotated dataset of 3,848 messages that will be useful to understand the various dimensions of WhatsApp political campaigning. We present several complementary insights into the investigative and sensational news stories from the same period. Exploratory data analysis and experiments showcase several exciting results and future research opportunities. To facilitate reproducible research, we make the anonymized datasets available in the public domain.

2. Eval4NLP

   MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation

   Ayush Garg, Sammed Kagi, Vivek Srivastava, and 1 more author

   In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, Nov 2021

   Abs HTML PDF

   Code-mixing is a phenomenon of mixing words and phrases from two or more languages in a single utterance of speech and text. Due to the high linguistic diversity, code-mixing presents several challenges in evaluating standard natural language generation (NLG) tasks. Various widely popular metrics perform poorly with the code-mixed NLG tasks. To address this challenge, we present a metric in- dependent evaluation pipeline MIPE that significantly improves the correlation between evaluation metrics and human judgments on the generated code-mixed text. As a use case, we demonstrate the performance of MIPE on the machine-generated Hinglish (code-mixing of Hindi and English languages) sentences from the HinGE corpus. We can extend the proposed evaluation strategy to other code-mixed language pairs, NLG tasks, and evaluation metrics with minimal to no effort.

3. ACL-IJCNLP

   TweeNLP: A Twitter Exploration Portal for Natural Language Processing

   Viraj Shah, Shruti Singh, and Mayank Singh

   In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, Aug 2021

   Abs HTML PDF

   We present TweeNLP, a one-stop portal that organizes Twitter’s natural language processing (NLP) data and builds a visualization and exploration platform. It curates 19,395 tweets (as of April 2021) from various NLP conferences and general NLP discussions. It supports multiple features such as TweetExplorer to explore tweets by topics, visualize insights from Twitter activity throughout the organization cycle of conferences, discover popular research papers and researchers. It also builds a timeline of conference and workshop submission deadlines. We envision TweeNLP to function as a collective memory unit for the NLP community by integrating the tweets pertaining to research papers with the NLPExplorer scientific literature search engine. The current system is hosted at \urlhttp://nlpexplorer.org/twitter/CFP.

4. CALCS

   Challenges and Limitations with the Metrics Measuring the Complexity of Code-Mixed Text

   Vivek Srivastava, and Mayank Singh

   In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, Jun 2021

   Abs HTML PDF

   Code-mixing is a frequent communication style among multilingual speakers where they mix words and phrases from two different languages in the same utterance of text or speech. Identifying and filtering code-mixed text is a challenging task due to its co-existence with monolingual and noisy text. Over the years, several code-mixing metrics have been extensively used to identify and validate code-mixed text quality. This paper demonstrates several inherent limitations of code-mixing metrics with examples from the already existing datasets that are popularly used across various experiments.

5. ICDAR

   TabLeX: A Benchmark Dataset for Structure and Content Information Extraction from Scientific Tables

   Harsh Desai, Pratik Kayal, and Mayank Singh

   In Document Analysis and Recognition – ICDAR 2021, Jun 2021

   Abs HTML PDF

   Information Extraction (IE) from the tables present in scientific articles is challenging due to complicated tabular representations and complex embedded text. This paper presents TabLeX, a large-scale benchmark dataset comprising table images generated from scientific articles. TabLeX consists of two subsets, one for table structure extraction and the other for table content extraction. Each table image is accompanied by its corresponding LaTeX source code. To facilitate the development of robust table IE tools, TabLeX contains images in different aspect ratios and in a variety of fonts. Our analysis sheds light on the shortcomings of current state-of-the-art table extraction models and shows that they fail on even simple table images. Towards the end, we experiment with a transformer-based existing baseline to report performance scores. In contrast to the static benchmarks, we plan to augment this dataset with more complex and diverse tables at regular intervals.

6. IEEE Access

   Augmented Convolutional LSTMs for Generation of High-Resolution Climate Change Projections

   Nidhin Harilal, Mayank Singh, and Udit Bhatia

   IEEE Access, 2021

   Abs HTML PDF

   Projection of changes in extreme indices of climate variables such as temperature and precipitation are critical to assess the potential impacts of climate change on human-made and natural systems, including critical infrastructures and ecosystems. While impact assessment and adaptation planning rely on high-resolution projections (typically in the order of a few kilometers), state-of-the-art Earth System Models (ESMs) are available at spatial resolutions of few hundreds of kilometers. Current solutions to obtain high-resolution projections of ESMs include downscaling approaches that consider the information at a coarse-scale to make predictions at local scales. Complex and non-linear interdependence among local climate variables (e.g., temperature and precipitation) and large-scale predictors (e.g., pressure fields) motivate the use of neural network-based super-resolution architectures. In this work, we present auxiliary variables informed spatio-temporal neural architecture for statistical downscaling. The current study performs daily downscaling of precipitation variable from an ESM output at 1.15 degrees (115 km) to 1/4 degrees (25 km) over one of the most climatically diversified countries, India. We showcase significant improvement gain against two popular state-of-the-art baselines with a better ability to predict statistics of extreme events. To facilitate reproducible research, we make available all the codes, processed datasets, and trained models in the public domain.

2020

1. TCSS

   Quantifying Nonrandomness in Evolving Networks

   Pradumn Kumar Pandey, and Mayank Singh

   IEEE Transactions on Computational Social Systems, Dec 2020

   Abs HTML PDF

   Complex systems have been successfully modeled as networks exhibiting the varying extent of randomness and nonrandomness. Network scientists contemplate randomness as one of the most desirable characteristics for real complex systems’ efficient performance. However, the current methodologies for randomness (or nonrandomness) quantification are nontrivial. In this article, we empirically showcase severe limitations associated with the state-of-the-art graph spectral-based quantification approaches. Addressing these limitations led to the proposal of a novel spectrum-based methodology that leverages configuration models as a reference network to quantify the nonrandomness in a given candidate network. Besides, we derive mathematical formulations for demonstrating the dependence of nonrandomness on three structural properties: modularity, clustering, and the highest degree node’s growth rate. We also introduce a novel graph signature (termed “cumulative spectral difference”) to visualize the nonrandomness in the network. Later, this article also discusses the relationship between the proposed nonrandomness measure and the diffusion affinity of networks. Toward the end, this article extensively discusses observations emerging from these signatures for both real-world and simulated networks.

2. CIKM

   CovidExplorer: A Multi-Faceted AI-Based Search and Visualization Engine for COVID-19 Information

   Heer Ambavi, Kavita Vaishnaw, Udit Vyas, and 2 more authors

   In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, Dec 2020

   Abs HTML PDF

   The entire world is engulfed in the fight against the COVID-19 pandemic, leading to a significant surge in research experiments, government policies, and social media discussions. A multi-modal information access and data visualization platform can play a critical role in supporting research aimed at understanding and developing preventive measures for the pandemic. In this paper, we present a multi-faceted AI-based search and visualization engine, CovidExplorer. Our system aims to help researchers understand current state-of-the-art COVID-19 research, identify research articles relevant to their domain, and visualize real-time trends and statistics of COVID-19 cases. In contrast to other existing systems, CovidExplorer also brings in India-specific topical discussions on social media to study different aspects of COVID-19. The system, demo video, and the datasets are available at http://covidexplorer.in.

3. W-NUT

   PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translation

   Vivek Srivastava, and Mayank Singh

   In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), Nov 2020

   Abs HTML PDF

   Code-mixing is the phenomenon of using more than one language in a sentence. In the multilingual communities, it is a very frequently observed pattern of communication on social media platforms. Flexibility to use multiple languages in one text message might help to communicate efficiently with the target audience. But, the noisy user-generated code-mixed text adds to the challenge of processing and understanding natural language to a much larger extent. Machine translation from monolingual source to the target language is a well-studied research problem. Here, we demonstrate that widely popular and sophisticated translation systems such as Google Translate fail at times to translate code-mixed text effectively. To address this challenge, we present a parallel corpus of the 13,738 code-mixed Hindi-English sentences and their corresponding human translation in English. In addition, we also propose a translation pipeline build on top of Google Translate. The evaluation of the proposed pipeline on PHINC demonstrates an increase in the performance of the underlying system. With minimal effort, we can extend the dataset and the proposed approach to other code-mixing language pairs.

4. SemEval

   IIT Gandhinagar at SemEval-2020 Task 9: Code-Mixed Sentiment Classification Using Candidate Sentence Generation and Selection

   Vivek Srivastava, and Mayank Singh

   In Proceedings of the Fourteenth Workshop on Semantic Evaluation, Dec 2020

   Abs HTML PDF

   Code-mixing is the phenomenon of using multiple languages in the same utterance. It is a frequently used pattern of communication on social media sites such as Facebook, Twitter, etc. Sentiment analysis of the monolingual text is a well-studied task. Code-mixing adds to the challenge of analyzing the sentiment of the text on various platforms such as social media, online gaming, forums, product reviews, etc. We present a candidate sentence generation and selection based approach on top of the Bi-LSTM based neural classifier to classify the Hinglish code-mixed text into one of the three sentiment classes positive, negative, or neutral. The proposed candidate sentence generation and selection based approach show an improvement in the system performance as compared to the Bi-LSTM based neural classifier. We can extend the proposed method to solve other problems with code-mixing in the textual data, such as humor-detection, intent classification, etc.

5. JCDL

   Gandhipedia: A One-Stop AI-Enabled Portal for Browsing Gandhian Literature, Life-Events and His Social Network

   Sayantan Adak, Atharva Vyas, Animesh Mukherjee, and 4 more authors

   In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, Dec 2020

   Abs HTML PDF

   We introduce an AI-enabled portal that presents an excellent visualization of Mahatma Gandhi’s life events by constructing temporal and spatial social networks from the Gandhian literature. Applying an ensemble of methods drawn from NLTK, Polyglot and Spacy we extract the key persons and places that find mentions in Gandhi’s written works. We visualize these entities and connections between them based on co-mentions within the same time frame as networks in an interactive web portal. The nodes in the network, when clicked, fire search queries about the entity and all the information about the entity presented in the corresponding book from which the network is constructed, are retrieved and presented back on the portal. Overall, this system can be used as a digital and user-friendly resource to study Gandhian literature.

6. JCDL

   PredCheck: Detecting Predatory Behaviour in Scholarly World

   Manas Satish Bedmutha, Kaushal Modi, Kevin Patel, and 2 more authors

   In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, Dec 2020

   Abs HTML PDF

   High solicitation for publishing a paper in scientific journals has led to the emergence of a large number of open-access predatory publishers. They fail to provide a rigorous peer-review process, thereby diluting the quality of research work and charge high article processing fees. Identification of such publishers has remained a challenge due to the vast diversity of the scholarly publishing ecosystem. Earlier works utilises only the objective features such as metadata. In this work, we aim to explore the possibility of identifying predatory behaviour through text-based features. We propose PredCheck, a four-step classificaton pipeline. The first classifier identifies the subject of the paper using TF-IDF vectors. Based on the subject of the paper, the Doc2Vec embeddings of the text are found. These embeddings are then fed into a Naive Bayes classifier that identifies the text to be predatory or non-predatory. Our pipeline gives a macro accuracy of 95% and an F1-score of 0.89.

7. JCDL

   Identification, Tracking and Impact: Understanding the Trade Secret of Catchphrases

   Jagriti Jalal, Mayank Singh, Arindam Pal, and 2 more authors

   In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, Dec 2020

   Abs HTML PDF

   Understanding the topical evolution in industrial innovation is a challenging problem. With the advancement in the digital repositories in the form of patent documents, it is becoming increasingly more feasible to understand the innovation secrets - ’catchphrases’ - of organizations. However, searching and understanding this enormous textual information is a natural bottleneck. In this paper, we propose an unsupervised method for the extraction of catchphrases from the abstracts of patents granted by the U.S. Patent and Trademark Office over the years. Our proposed system achieves substantial improvement, both in terms of precision and recall, against state-of-the-art techniques. As a second objective, we conduct an extensive empirical study to understand the temporal evolution of the catchphrases across various organizations. We also show how the overall innovation evolution in the form of introduction of newer catchphrases in an organization’s patents correlates with the future citations received by the patents filed by that organization. Our code and data sets will be placed in the public domain.

8. JoI

   Analysis of reference and citation copying in evolving bibliographic networks

   Pradumn Kumar Pandey, Mayank Singh, Pawan Goyal, and 2 more authors

   Journal of Informetrics, Dec 2020

   Abs HTML

   Extensive literature demonstrates how the copying of references (links) can lead to the emergence of various structural properties (e.g., power-law degree distribution and bipartite cores) in bibliographic and other similar directed networks. However, it is also well known that the copying process is incapable of mimicking the number of directed triangles in such networks; neither does it have the power to explain the obsolescence of older papers. In this paper, we propose RefOrCite, a new model that allows for copying of both the references from (i.e., out-neighbors of) as well as the citations to (i.e., in-neighbors of) an existing node. In contrast, the standard copying model (CP) only copies references. While retaining its spirit, RefOrCite differs from the Forest Fire (FF) model in ways that makes RefOrCite amenable to mean-field analysis for degree distribution, triangle count, and densification. Empirically, RefOrCite gives the best overall agreement with observed degree distribution, triangle count, diameter, h-index, and the growth of citations to newer papers.

9. ECIR

   NLPExplorer: Exploring the Universe of NLP Papers

   Monarch Parmar, Naman Jain, Pranjali Jain, and 4 more authors

   In Advances in Information Retrieval, Dec 2020

   Abs HTML PDF

   Understanding the current research trends, problems, and their innovative solutions remains a bottleneck due to the ever-increasing volume of scientific articles. In this paper, we propose NLPExplorer, a completely automatic portal for indexing, searching, and visualizing Natural Language Processing (NLP) research volume. NLPExplorer presents interesting insights from papers, authors, venues, and topics. In contrast to previous topic modelling based approaches, we manually curate five course-grained non-exclusive topical categories namely Linguistic Target (Syntax, Discourse, etc.), Tasks (Tagging, Summarization, etc.), Approaches (unsupervised, supervised, etc.), Languages (English, Chinese, etc.) and Dataset types (news, clinical notes, etc.). Some of the novel features include a list of young popular authors, popular URLs and datasets, list of topically diverse papers and recent popular papers. Also, it provides temporal statistics such as yearwise popularity of topics, datasets, and seminal papers. To facilitate future research and system development, we make all the processed dataset accessible through API calls. The current system is available at http://nlpexplorer.org.

10. CODS-COMAD

    Weakly-Supervised Deep Learning for Domain Invariant Sentiment Classification

    Pratik Kayal, Mayank Singh, and Pawan Goyal

    In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, Dec 2020

    Abs HTML PDF

    The task of learning a sentiment classification model that adapts well to any target domain, different from the source domain, is a challenging problem. Majority of the existing approaches focus on learning a common representation by leveraging both source and target data during training. In this paper, we introduce a two-stage training procedure that leverages weakly supervised datasets for developing simple lift-and-shift-based predictive models without being exposed to the target domain during the training phase. Experimental results show that transfer with weak supervision from a source domain to various target domains provides performance very close to that obtained via supervised training on the target domain itself.

11. CODS-COMAD

    Innovation and Revenue: Deep Diving into the Temporal Rank-Shifts of Fortune 500 Companies

    Mayank Singh, Arindam Pal, Lipika Dey, and 1 more author

    In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, Dec 2020

    Abs HTML PDF

    Research and innovation is an important agenda for any company to remain competitive in the market. The relationship between innovation and revenue is a key metric for companies to decide on the amount to be invested for future research. Two important parameters to evaluate innovation are the quantity and quality of scientific papers and patents. Our work studies the relationship between innovation and patenting activities for several Fortune 500 companies over a period of time. We perform a comprehensive study of the patent citation dataset available in the Reed Technology Index collected from the US Patent Office. We observe several interesting relations between parameters like the number of (i) patent applications, (ii) patent grants, (iii) patent citations and Fortune 500 ranks of companies. We also study the trends of these parameters varying over the years and derive causal explanations for these with qualitative and intuitive reasoning. To facilitate reproducible research, we make all the processed patent dataset publicly available.1

2019

1. PLOS ONE

   MSP-N: Multiple selection procedure with ‘N’ possible growth mechanisms

   Pradumn Kumar Pandey, and Mayank Singh

   PLOS ONE, Dec 2019

   Abs HTML PDF

   Network modeling is a challenging task due to non-trivial evolution dynamics. We introduce multiple-selection-procedure with ‘N’ possible growth mechanisms (MSP-N). In MSP-N, an incoming node chooses a single option among N available options to link to pre-existing nodes. Some of the potential options, in case of social networks, can be standard preferential or random attachment and node aging or fitness. In this paper, we discuss a specific case, MSP-2, and shows its efficacy in reconstructing several non-trivial characteristic properties of social networks, including networks with power-law degree distribution, power-law with an exponential decay (exponential cut-off), and exponential degree distributions. We evaluate the proposed evolution mechanism over two real-world networks and observe that the generated networks highly resembles the degree distribution of the real-world networks. Besides, several other network properties such as high clustering and triangle count, low spectral radius, and community structure, of the generated networks are significantly closer to the real-world networks.

2. ICADL

   The Evolving Ecosystem of Predatory Journals: A Case Study in Indian Perspective

   Naman Jain, and Mayank Singh

   In Digital Libraries at the Crossroads of Digital Information for the Future, Dec 2019

   Abs HTML PDF

   Digital advancement in scholarly repositories has led to the emergence of a large number of open access predatory publishers that charge high article processing fees from authors but fail to provide necessary editorial and publishing services. Identifying and blacklisting such publishers has remained a research challenge due to the highly volatile scholarly publishing ecosystem. This paper presents a data-driven approach to study how potential predatory publishers are evolving and bypassing several regularity constraints. We empirically show the close resemblance of predatory publishers against reputed publishing groups. In addition to verifying standard constraints, we also propose distinctive signals gathered from network-centric properties to understand this evolving ecosystem better. To facilitate reproducible research, we shall make all the codes and the processed dataset available in the public domain.

3. ICADL

   The Rise and Rise of Interdisciplinary Research: Understanding the Interaction Dynamics of Three Major Fields – Physics, Mathematics and Computer Science

   Rima Hazra, Mayank Singh, Pawan Goyal, and 2 more authors

   In Digital Libraries at the Crossroads of Digital Information for the Future, Dec 2019

   Abs HTML PDF

   The distinction between sciences is becoming increasingly more artificial – an approach from one area can be easily applied to the other. More exciting research nowadays is happening perhaps at the interfaces of disciplines like Physics, Mathematics and Computer Science. How do these interfaces emerge and interact? For instance, is there a specific pattern in which these fields cite each other? In this article, we investigate a collection of more than 1.2 million papers from three different scientific disciplines – Physics, Mathematics, and Computer Science. We show how over a timescale the citation patterns from the core science fields (Physics, Mathematics) to the applied and fast-growing field of Computer Science have drastically increased. Further, we observe how certain subfields in these disciplines are shrinking while others are becoming tremendously popular. For instance, an intriguing observation is that citations from Mathematics to the subfield of machine learning in Computer Science in recent times are exponentially increasing.

4. ECIR

   Automated Early Leaderboard Generation from Comparative Tables

   Mayank Singh, Rajdeep Sarkar, Atharva Vyas, and 3 more authors

   In Advances in Information Retrieval, Dec 2019

   Abs HTML PDF

   A leaderboard is a tabular presentation of performance scores of the best competing techniques that address a specific scientific problem. Manually maintained leaderboards take time to emerge, which induces a latency in performance discovery and meaningful comparison. This can delay dissemination of best practices to non-experts and practitioners. Regarding papers as proxies for techniques, we present a new system to automatically discover and maintain leaderboards in the form of partial orders between papers, based on performance reported therein. In principle, a leaderboard depends on the task, data set, other experimental settings, and the choice of performance metrics. Often there are also tradeoffs between different metrics. Thus, leaderboard discovery is not just a matter of accurately extracting performance numbers and comparing them. In fact, the levels of noise and uncertainty around performance comparisons are so large that reliable traditional extraction is infeasible. We mitigate these challenges by using relatively cleaner, structured parts of the papers, e.g., performance tables. We propose a novel performance improvement graph with papers as nodes, where edges encode noisy performance comparison information extracted from tables. Every individual performance edge is extracted from a table with citations to other papers. These extractions resemble (noisy) outcomes of ‘matches’ in an incomplete tournament. We propose several approaches to rank papers from these noisy ‘match’ outcomes. We show that our ranking scheme can reproduce various manually curated leaderboards very well. Using widely-used lists of state-of-the-art papers in 27 areas of Computer Science, we demonstrate that our system produces very reliable rankings. We also show that commercial scholarly search systems cannot be used for leaderboard discovery, because of their emphasis on citations, which favors classic papers over recent performance breakthroughs. Our code and data sets will be placed in the public domain.

5. SamEval

   IIT Gandhinagar at SemEval-2019 Task 3: Contextual Emotion Detection Using Deep Learning

   Arik Pamnani, Rajat Goel, Jayesh Choudhari, and 1 more author

   In Proceedings of the 13th International Workshop on Semantic Evaluation, Jun 2019

   Abs HTML PDF

   Recent advancements in Internet and Mobile infrastructure have resulted in the development of faster and efficient platforms of communication. These platforms include speech, facial and text-based conversational mediums. Majority of these are text-based messaging platforms. Development of Chatbots that automatically understand latent emotions in the textual message is a challenging task. In this paper, we present an automatic emotion detection system that aims to detect the emotion of a person textually conversing with a chatbot. We explore deep learning techniques such as CNN and LSTM based neural networks and outperformed the baseline score by 14%. The trained model and code are kept in public domain.

6. JCDL

   BioGen: Automated Biography Generation

   Heer Ambavi, Ayush Garg, Ayush Garg, and 5 more authors

   In 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL), Jun 2019

   Abs HTML PDF

   A biography of a person is the detailed description of several life events including his education, work, relationships and death. Wikipedia, the free web-based encyclopedia, consists of millions of manually curated biographies of eminent politicians, film and sports personalities, etc. However, manual curation efforts, even though efficient, suffers from significant delays. In this work, we propose an automatic biography generation framework BioGen. BioGen generates a short collection of biographical sentences clustered into multiple events of life. Evaluation results show that biographies generated by BioGen are significantly closer to manually written biographies in Wikipedia. A working model of this framework is available at nlpbiogen.herokuapp.com/home/

7. ICSTEM

   AgriBot: Agriculture-Specific Question Answer System

   Naman Jain, Pranjali Jain, Pratik Kayal, and 4 more authors

   Jun 2019

   PDF

2018

1. NAACL

   CL Scholar: The ACL Anthology Knowledge Graph Miner

   Mayank Singh, Pradeep Dogga, Sohan Patro, and 5 more authors

   In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, Jun 2018

   Abs HTML PDF

   We present CL Scholar, the ACL Anthology knowledge graph miner to facilitate high-quality search and exploration of current research progress in the computational linguistics community. In contrast to previous works, periodically crawling, indexing and processing of new incoming articles is completely automated in the current system. CL Scholar utilizes both textual and network information for knowledge graph construction. As an additional novel initiative, CL Scholar supports more than 1200 scholarly natural language queries along with standard keyword-based search on constructed knowledge graph. It answers binary, statistical and list based natural language queries. The current system is deployed at \urlhttp://cnerg.iitkgp.ac.in/aclakg. We also provide REST API support along with bulk download facility. Our code and data are available at \urlhttps://github.com/CLScholar.

2017

1. KDD

   Relay-Linking Models for Prominence and Obsolescence in Evolving Networks

   Mayank Singh, Rajdeep Sarkar, Pawan Goyal, and 2 more authors

   In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Jun 2017

   Abs HTML PDF

   The rate at which nodes in evolving social networks acquire links (friends, citations) shows complex temporal dynamics. Preferential attachment and link copying models, while enabling elegant analysis, only capture rich-gets-richer effects, not aging and decline. Recent aging models are complex and heavily parameterized; most involve estimating 1-3 parameters per node. These parameters are intrinsic: they explain decline in terms of events in the past of the same node, and do not explain, using the network, where the linking attention might go instead. We argue that traditional characterization of linking dynamics are insufficient to judge the faithfulness of models. We propose a new temporal sketch of an evolving graph, and introduce several new characterizations of a network’s temporal dynamics. Then we propose a new family of frugal aging models with no per-node parameters and only two global parameters. Our model is based on a surprising inversion or undoing of triangle completion, where an old node relays a citation to a younger follower in its immediate vicinity. Despite very few parameters, the new family of models shows remarkably better fit with real data. Before concluding, we analyze temporal signatures for various research communities yielding further insights into their comparative dynamics. To facilitate reproducible research, we shall soon make all the codes and the processed dataset available in the public domain.

2. JCDL

   Understanding the Impact of Early Citers on Long-Term Scientific Impact

   Mayank Singh, Ajay Jaiswal, Priya Shree, and 3 more authors

   In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), Jun 2017

   HTML PDF
3. JCDL

   Citation Sentence Reuse Behavior of Scientists: A Case Study on Massive Bibliographic Text Dataset of Computer Science

   Mayank Singh, Abhishek Niranjan, Divyansh Gupta, and 3 more authors

   In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), Jun 2017

   Abs HTML PDF

   Our current knowledge of scholarly plagiarism is largely based on the similarity between full text research articles. In this paper, we propose an innovative and novel conceptualization of scholarly plagiarism in the form of reuse of explicit citation sentences in scientific research articles. Note that while full-text plagiarism is an indicator of a gross-level behavior, copying of citation sentences is a more nuanced micro-scale phenomenon observed even for well-known researchers. The current work poses several interesting questions and attempts to answer them by empirically investigating a large bibliographic text dataset from computer science containing millions of lines of citation sentences. In particular, we report evidences of massive copying behavior. We also present several striking real examples throughout the paper to showcase widespread adoption of this undesirable practice. In contrast to the popular perception, we find that copying tendency increases as an author matures. The copying behavior is reported to exist in all fields of computer science; however, the theoretical fields indicate more copying than the applied fields.

4. WOSP

   AppTechMiner: Mining Applications and Techniques from Scientific Articles

   Mayank Singh, Soham Dan, Sanyam Agarwal, and 2 more authors

   In Proceedings of the 6th International Workshop on Mining Scientific Publications, Jun 2017

   Abs HTML PDF

   This paper presents AppTechMiner, a rule-based information extraction framework that automatically constructs a knowledge base of all application areas and problem solving techniques. Techniques include tools, methods, datasets or evaluation metrics. We also categorize individual research articles based on their application areas and the techniques proposed/improved in the article. Our system achieves high average precision ( 82%) and recall ( 84%) in knowledge base creation. It also performs well in application and technique assignment to an individual article (average accuracy  66%). In the end, we further present two use cases presenting a trivial information retrieval system and an extensive temporal analysis of the usage of techniques and application areas. At present, we demonstrate the framework for the domain of computational linguistics but this can be easily generalized to any other field of research. We plan to make the codes publicly available.

2016

1. COLING

   OCR++: A Robust Framework For Information Extraction from Scholarly Articles

   Mayank Singh, Barnopriyo Barua, Priyank Palod, and 8 more authors

   In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Dec 2016

   Abs PDF

   This paper proposes OCR++, an open-source framework designed for a variety of information extraction tasks from scholarly articles including metadata (title, author names, affiliation and e-mail), structure (section headings and body text, table and figure headings, URLs and footnotes) and bibliography (citation instances and references). We analyze a diverse set of scientific articles written in English to understand generic writing patterns and formulate rules to develop this hybrid framework. Extensive evaluations show that the proposed framework outperforms the existing state-of-the-art tools by a large margin in structural information extraction along with improved performance in metadata and bibliography extraction tasks, both in terms of accuracy (around 50% improvement) and processing time (around 52% improvement). A user experience study conducted with the help of 30 researchers reveals that the researchers found this system to be very helpful. As an additional objective, we discuss two novel use cases including automatically extracting links to public datasets from the proceedings, which would further accelerate the advancement in digital libraries. The result of the framework can be exported as a whole into structured TEI-encoded documents. Our framework is accessible online at \urlhttp://www.cnergres.iitkgp.ac.in/OCR++/home/.

2. JoI

   Is this conference a top-tier? ConfAssist: An assistive conflict resolution framework for conference categorization

   Mayank Singh, Tanmoy Chakraborty, Animesh Mukherjee, and 1 more author

   Journal of Informetrics, Dec 2016

   PDF
3. KDD

   FeRoSA: A Faceted Recommendation System for Scientific Articles

   Tanmoy Chakraborty, Amrith Krishna, Mayank Singh, and 3 more authors

   In Advances in Knowledge Discovery and Data Mining, Dec 2016

   Abs PDF

   The overwhelming number of scientific articles over the years calls for smart automatic tools to facilitate the process of literature review. Here, we propose for the first time a framework of faceted recommendation for scientific articles (abbreviated as FeRoSA) which apart from ensuring quality retrieval of scientific articles for a query paper, also efficiently arranges the recommended papers into different facets (categories). Providing users with an interface which enables the filtering of recommendations across multiple facets can increase users’ control over how the recommendation system behaves. FeRoSA is precisely built on a random walk based framework on an induced subnetwork consisting of nodes related to the query paper in terms of either citations or content similarity. Rigorous analysis based an experts’ judgment shows that FeRoSA outperforms two baseline systems in terms of faceted recommendations (overall precision of 0.65). Further, we show that the faceted results of FeRoSA can be appropriately combined to design a better flat recommendation system as well. An experimental version of FeRoSA is publicly available at www.ferosa.org(receiving as many as 170 hits within the first 15 days of launch).

2015

1. CIKM

   The Role Of Citation Context In Predicting Long-Term Citation Profiles: An Experimental Study Based On A Massive Bibliographic Text Dataset

   Mayank Singh, Vikas Patidar, Suhansanu Kumar, and 3 more authors

   In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, Dec 2015

   Abs HTML PDF

   The impact and significance of a scientific publication is measured mostly by the number of citations it accumulates over the years. Early prediction of the citation profile of research articles is a significant as well as challenging problem. In this paper, we argue that features gathered from the citation contexts of the research papers can be very relevant for citation prediction. Analyzing a massive dataset of nearly 1.5 million computer science articles and more than 26 million citation contexts, we show that average countX (number of times a paper is cited within the same article) and average citeWords (number of words within the citation context) discriminate between various citation ranges as well as citation categories. We use these features in a stratified learning framework for future citation prediction. Experimental results show that the proposed model significantly outperforms the existing citation prediction models by a margin of 8-10% on an average under various experimental settings. Specifically, the features derived from the citation context help in predicting long-term citation behavior.

2. JCDL

   ConfAssist: A Conflict Resolution Framework for Assisting the Categorization of Computer Science Conferences

   Mayank Singh, Tanmoy Chakraborty, Animesh Mukherjee, and 1 more author

   In Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital Libraries, Dec 2015

   Abs HTML PDF

   Classifying publication venues into top-tier or non top-tier is quite subjective and can be debatable at times. sIn this paper, we propose ConfAssist, a novel assisting framework for conference categorization that aims to address the limitations in the existing systems and portals for venue classification. We identify various features related to the stability of conferences that might help us separate a top-tier conference from the rest of the lot. While there are many clear cases where expert agreement can be almost immediately achieved as to whether a conference is a top-tier or not, there are equally many cases that can result in a conflict even among the experts. ConfAssist tries to serve as an aid in such cases by increasing the confidence of the experts in their decision. A human judgment survey was conducted with 28 domain experts. The results were quite impressive with 91.6% classification accuracy.

3. D-Lib

   PubIndia: A Framework for Analyzing Indian Research Publications in Computer Science

   Mayank Singh, Soumajit Pramanick, and Tanmoy Chakraborty

   D-Lib Magazine, Nov 2015

   HTML
4. WACV

   The Role Of Citation Context In Predicting Long-Term Citation Profiles: An Experimental Study Based On A Massive Bibliographic Text Dataset

   Mayank Singh, Vikas Patidar, Suhansanu Kumar, and 3 more authors

   In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, Nov 2015

   Abs HTML PDF

   The impact and significance of a scientific publication is measured mostly by the number of citations it accumulates over the years. Early prediction of the citation profile of research articles is a significant as well as challenging problem. In this paper, we argue that features gathered from the citation contexts of the research papers can be very relevant for citation prediction. Analyzing a massive dataset of nearly 1.5 million computer science articles and more than 26 million citation contexts, we show that average countX (number of times a paper is cited within the same article) and average citeWords (number of words within the citation context) discriminate between various citation ranges as well as citation categories. We use these features in a stratified learning framework for future citation prediction. Experimental results show that the proposed model significantly outperforms the existing citation prediction models by a margin of 8-10% on an average under various experimental settings. Specifically, the features derived from the citation context help in predicting long-term citation behavior.