publications

2023

1. 

   Efficient Large Language Models: A Survey

   Wan, Zhongwei, Wang, Xin, Liu, Che,  Alam, Samiul, Zheng, Yu, Qu, Zhongnan, Yan, Shen, Zhu, Yi, Zhang, Quanlu, Chowdhury, Mosharaf, and others,

   2023

   Abs arXiv PDF

   Large Language Models (LLMs) have demonstrated remarkable capabilities in important tasks such as natural language understanding, language generation, and complex reasoning and have the potential to make a substantial impact on our society. Such capabilities, however, come with the considerable resources they demand, highlighting the strong need to develop effective techniques for addressing their efficiency challenges. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We have also created a GitHub repository where we compile the papers featured in this survey at https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey, this https URL, and will actively maintain this repository and incorporate new research as it emerges. We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of the research developments in efficient LLMs and inspire them to contribute to this important and exciting field.

2. 

   Sparse Multichannel Decomposition of Electrodermal Activity With Physiological Priors

   Alam, Samiul, Amin, Md. Rafiul, and Faghih, Rose T.

   IEEE Open Journal of Engineering in Medicine and Biology 2023

   Abs PDF

   Goal: Inferring autonomous nervous system (ANS) activity is a challenging issue and has critical applications in stress regulation. Sweat secretions caused by ANS activity influence the electrical conductance of the skin. Therefore, the variations in skin conductance (SC) measurements reflect the sudomotor nerve activity (SMNA) and can be used to infer the underlying ANS activity. These variations are strongly correlated with emotional arousal as well as thermoregulation. However, accurately recovering ANS activity and the corresponding state-space system from a single channel signal is difficult due to artifacts introduced by measurement noise. To minimize the impact of noise on inferring ANS activity, we utilize multiple channels of SC data. Methods: We model skin conductance using a second-order differential equation incorporating a time-shifted sparse impulse train input in combination with independent cubic basis spline functions. Finally, we develop a block coordinate descent method for SC signal decomposition by employing a generalized cross-validation sparse recovery approach while including physiological priors. Results: We analyze the experimental data to validate the performance of the proposed algorithm. We demonstrate its capacity to recover the ANS activations, the underlying physiological system parameters, and both tonic and phasic components. Finally, we present an overview of the algorithm’s comparative performance under varying conditions and configurations to substantiate its ability to accurately model ANS activity. Our results show that our algorithm performs better in terms of multiple metrics like noise performance, AUC score, the goodness of fit of reconstructed signal, and lower missing impulses compared with the single channel decomposition approach. Conclusion: In this study, we highlight the challenges and benefits of concurrent decomposition and deconvolution of multichannel SC signals.

3. 

   FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things

   Alam, Samiul, Zhang, Tuo, Feng, Tiantian, Shen, Hui, Cao, Zhichao, Zhao, Dong, Ko, JeongGil, Somasundaram, Kiran, Narayanan, Shrikanth S, Avestimehr, Salman, and others,

   2023

   Abs arXiv PDF

   There is a significant relevance of federated learning (FL) in the realm of Artificial Intelligence of Things (AIoT). However, most existing FL works are not conducted on datasets collected from authentic IoT devices that capture unique modalities and inherent challenges of IoT data. In this work, we introduce FedAIoT, an FL benchmark for AIoT to fill this critical gap. FedAIoT includes eight datatsets collected from a wide range of IoT devices. These datasets cover unique IoT modalities and target representative applications of AIoT. FedAIoT also includes a unified end-to-end FL framework for AIoT that simplifies benchmarking the performance of the datasets. Our benchmark results shed light on the opportunities and challenges of FL for AIoT. We hope FedAIoT could serve as an invaluable resource to foster advancements in the important field of FL for AIoT. The repository of FedAIoT is maintained at https://github.com/AIoT-MLSys-Lab/FedAIoT

4. 

   OOD-Speech: A Large Bengali Speech Recognition Dataset for Out-of-Distribution Benchmarking

   Rakib, Fazle Rabbi, Dip, Souhardya Saha,  Alam, Samiul, Tasnim, Nazia, Shihab, Md. Istiak Hossain, Ansary, Md. Nazmuddoha, Hossen, Syed Mobassir, Meghla, Marsia Haque, Mamun, Mamunur, Sadeque, Farig, Chowdhury, Sayma Sultana, Reasat, Tahsin, Sushmit, Asif, and Humayun, Ahmed Imtiaz

   In Proc. INTERSPEECH 2023 2023

   arXiv
5. Gpt-fl: Generative pre-trained model-assisted federated learning

   Zhang, Tuo, Feng, Tiantian,  Alam, Samiul, Zhang, Mi, Narayanan, Shrikanth S, and Avestimehr, Salman

   arXiv preprint arXiv:2306.02210 2023
6. Federated Learning Benchmarks and Frameworks for Artificial Intelligence of Things

   Alam, Samiul

   2023
7. FedAudio: A Federated Learning Benchmark for Audio Tasks

   Zhang, Tuo, Feng, Tiantian,  Alam, Samiul, Lee, Sunwoo, Zhang, Mi, Narayanan, Shrikanth S, and Avestimehr, Salman

   In IEEE International Conference on Acoustics, Speech, and Signal Processing 2023

   Abs arXiv HTML

   Federated learning (FL) has gained substantial attention in recent years due to the data privacy concerns related to the pervasiveness of consumer devices that continuously collect data from users. While a number of FL benchmarks have been developed to facilitate FL research, none of them include audio data and audio-related tasks. In this paper, we fill this critical gap by introducing a new FL benchmark for audio tasks which we refer to as FedAudio. FedAudio includes four representative and commonly used audio datasets from three important audio tasks that are well aligned with FL use cases. In particular, a unique contribution of FedAudio is the introduction of data noises and label errors to the datasets to emulate challenges when deploying FL systems in real-world settings. FedAudio also includes the benchmark results of the datasets and a PyTorch library with the objective of facilitating researchers to fairly compare their algorithms. We hope FedAudio could act as a catalyst to inspire new FL research for audio tasks and thus benefit the acoustic and speech research community. The datasets and benchmark results can be accessed at this https URL.

2022

1. 

   FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction

   Alam, Samiul, Liu, Luyang, Yan, Ming, and Zhang, Mi

   In Advances in Neural Information Processing Systems 2022

   Abs arXiv HTML PDF

   Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical. However, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains clients from training large models due to on-device resource bottlenecks. In this work, we propose FedRolex, a partial training (PT)-based approach that enables model-heterogeneous FL and can train a global server model larger than the largest client model. At its core, FedRolex employs a rolling sub-model extraction scheme that allows different parts of the global server model to be evenly trained, which mitigates the client drift induced by the inconsistency between individual client models and server model architectures. Empirically, we show that FedRolex outperforms state-of-the-art PT-based model-heterogeneous FL methods (e.g. Federated Dropout) and reduces the gap between model-heterogeneous and model-homogeneous FL, especially under the large-model large-dataset regime. In addition, we provide theoretical statistical analysis on its advantage over Federated Dropout. Lastly, we evaluate FedRolex on an emulated real-world device distribution to show that FedRolex can enhance the inclusiveness of FL and boost the performance of low-end devices that would otherwise not benefit from FL. Our code is available at: https://github.com/AIoT-MLSys-Lab/FedRolex.

2. FedSEA: A Semi-Asynchronous Federated Learning Framework for Extremely Heterogeneous Devices

   Sun, Jingwei, Li, Ang, Duan, Lin,  Alam, Samiul, Deng, Xuliang, GUO, Xin, Wang, Haiming, Gorlatova, Maria, Zhang, Mi, and Chen, Yiran

   In ACM Conference on Embedded Networked Sensor Systems Dec 2022

   Abs HTML PDF

   Federated learning (FL) has attracted increasing attention as a promising technique to drive a vast number of edge devices with artificial intelligence. However, it is very challenging to guarantee the efficiency of a FL system in practice due to the heterogeneous computation resources on different devices. To improve the efficiency of FL systems in the real world, asynchronous FL (AFL) and semi-asynchronous FL (SAFL) methods are proposed such that the server does not need to wait for stragglers. However, existing AFL and SAFL systems suffer from poor accuracy and low efficiency in realistic settings where the data is non-IID distributed across devices and the on-device resources are extremely heterogeneous. In this work, we propose FedSEA - a semi-asynchronous FL framework for extremely heterogeneous devices. We theoretically disclose that the unbalanced aggregation frequency is a root cause of accuracy drop in SAFL. Based on this analysis, we design a training configuration scheduler to balance the aggregation frequency of devices such that the accuracy can be improved. To improve the efficiency of the system in realistic settings where the devices have dynamic on-device resource availability, we design a scheduler that can efficiently predict the arriving time of local updates from devices and adjust the synchronization time point according to the devices’ predicted arriving time. We also consider the extremely heterogeneous settings where there exist extremely lagging devices that take hundreds of times as long as the training time of the other devices. In the real world, there might be even some extreme stragglers which are not capable of training the global model. To enable these devices to join in training without impairing the systematic efficiency, Fed-SEA enables these extreme stragglers to conduct local training on much smaller models. Our experiments show that compared with status quo approaches, FedSEA improves the inference accuracy by 44.34% and reduces the systematic time cost and local training time cost by 87.02× and 792.9×. FedSEA also reduces the energy consumption of the devices with extremely limited resources by 752.9×

3. 

   Bengali Common Voice Speech Dataset for Automatic Speech Recognition

   Alam, Samiul, Sushmit, Asif, Abdullah, Zaowad, Nakkhatra, Shahrin, Ansary, MD, Hossen, Syed Mobassir, Mehnaz, Sazia Morshed, Reasat, Tahsin, and Humayun, Ahmed Imtiaz

   arXiv preprint arXiv:2206.14053 Dec 2022

   Abs arXiv HTML

   Bengali is one of the most spoken languages in the world with over 300 million speakers globally. Despite its popularity, research into the development of Bengali speech recognition systems is hindered due to the lack of diverse open-source datasets. As a way forward, we have crowdsourced the Bengali Common Voice Speech Dataset, which is a sentence-level automatic speech recognition corpus. Collected on the Mozilla Common Voice platform, the dataset is part of an ongoing campaign that has led to the collection of over 400 hours of data in 2 months and is growing rapidly. Our analysis shows that this dataset has more speaker, phoneme, and environmental diversity compared to the OpenSLR Bengali ASR dataset, the largest existing open-source Bengali speech dataset. We present insights obtained from the dataset and discuss key linguistic challenges that need to be addressed in future versions. Additionally, we report the current performance of a few Automatic Speech Recognition (ASR) algorithms and set a benchmark for future research.

2021

1. 

   A large multi-target dataset of common bengali handwritten graphemes

   Alam, Samiul, Reasat, Tahsin, Sushmit, Asif Shahriyar, Siddique, Sadi Mohammad, Rahman, Fuad, Hasan, Mahady, and Humayun, Ahmed Imtiaz

   Dec 2021

   Abs arXiv HTML PDF

   Latin has historically led the state-of-the-art in handwritten optical character recognition (OCR) research. Adapting existing systems from Latin to alpha-syllabary languages is particularly challenging due to a sharp contrast between their orthographies. Due to a cursive writing system and frequent use of diacritics, the segmentation and/or alignment of graphical constituents with corresponding characters becomes significantly convoluted. We propose a labeling scheme based on graphemes (linguistic segments of word formation) that makes segmentation inside alpha-syllabary words linear and present the first dataset of Bengali handwritten graphemes that are commonly used in everyday context. The dataset contains 411k curated samples of 1295 unique commonly used Bengali graphemes. Additionally, the test set contains 900 uncommon Bengali graphemes for out of dictionary performance evaluation. The dataset is open-sourced as a part of a public Handwritten Grapheme Classification Challenge on Kaggle to benchmark vision algorithms for multi-target grapheme classification. The unique graphemes present in this dataset are selected based on commonality in the Google Bengali ASR corpus. From competition proceedings, we see that deep learning methods can generalize to a large span of out of dictionary graphemes which are absent during training (Kaggle Competition kaggle.com/c/bengaliai-cv19, Supplementary materials and Appendix https://github.com/AhmedImtiazPrio/ICDAR2021supplementary).

2018

1. 

   Numtadb-assembled bengali handwritten digits

   Alam, Samiul, Reasat, Tahsin, Doha, Rashed Mohammad, and Humayun, Ahmed Imtiaz

   arXiv preprint arXiv:1806.02452 Dec 2018

   Abs arXiv HTML PDF

   To benchmark Bengali digit recognition algorithms, a large publicly available dataset is required which is free from biases originating from geographical location, gender, and age. With this aim in mind, NumtaDB, a dataset consisting of more than 85,000 images of hand-written Bengali digits, has been assembled. This paper documents the collection and curation process of numerals along with the salient statistics of the dataset.

2. AI Learns to Recognize Bengali Handwritten Digits: Bengali. AI Computer Vision Challenge 2018

   Kamran, Sharif Amit, Humayun, Ahmed Imtiaz,  Alam, Samiul, Doha, Rashed Mohammad, Mandal, Manash Kumar, Reasat, Tahsin, and Rahman, Fuad

   arXiv preprint arXiv:1810.04452 Dec 2018
3. 

   Predictive real-time beat tracking from music for embedded application

   Al-Hussaini, Irfan, Humayun, Ahmed Imtiaz,  Alam, Samiul, Foysal, Shariful Islam, Al Masud, Abdullah, Mahmud, Arafat, Chowdhury, Rakibul Islam, Ibtehaz, Nabil, Zaman, Sums Uz, Hyder, Rakib, and others,

   Dec 2018

   Abs HTML PDF

   Beat tracking from music signals has significant importance in multimedia information retrieval systems, especially in cover song detection. A predictive real-time beat tracking system can also be used to assist musicians performing live. In this paper we present a real-time beat tracking algorithm, fast enough to be implemented on an embedded system. The onset of a note is detected using a maximum filter approach that suppresses the effect of vibrato. Beats are predicted a second in advance using a causal variant of Dynamic Programming. We have employed an onset memoization algorithm, to reduce the computational resources required. Raspberry Pi was chosen as our preferred development board. We have demonstrated through experimental results that the proposed approach can satisfactorily estimate beat positions from a music signal in real-time with an average continuity score (AMLt) of 0.67.