Large language models have dominated the scientific and media landscape due to their impressive performance in processing large amounts of data and producing humanlike text. Nevertheless, their high energy demand and slow processing remain bottlenecks to further improving model quality while making the models broadly accessible. To address these bottlenecks, we investigate how reservoir computing performs in natural language processing, which could enable fast and energy-efficient hardware implementations. Studies investigating the use of reservoir computing as a language model remain sparse. In this paper, we compare three distinct approaches to character-level language modeling: two reservoir-computing approaches, in which only an output layer is trainable, and a well-known transformer-based architecture, which learns a fully trainable attention-based sequence representation. We explore the performance, computational cost, and prediction accuracy of both paradigms while varying the number of trainable parameters consistently across all models. Using a consistent pipeline for all three approaches, we demonstrate that transformers excel in prediction quality, whereas reservoir computers remain highly efficient, substantially reducing training and inference times. Furthermore, we investigate two types of reservoir computing: a traditional reservoir with a static linear readout and an attention-enhanced reservoir that dynamically adapts its output weights through an attention mechanism. Our findings illustrate how these paradigms scale and provide guidelines for balancing resource constraints with performance.

Close

PrevNext

View figure in article

• Dynamics of networks
• Neuromorphic computing
• Physics of computation
• Artificial neural networks
• Language models
• Artificial intelligence
• Natural language processing