Try our interactive maze-solving demo: https://pub.sakana.ai/ctm/

Continuous Thought Machines

• arXiv: https://arxiv.org/abs/2505.05522
• Interactive Website: https://pub.sakana.ai/ctm/
• Blog Post: https://sakana.ai/ctm/
• GitHub Repo: https://github.com/SakanaAI/continuous-thought-machines

Hey r/MachineLearning!

We're excited to share our new research on Continuous Thought Machines (CTMs), a novel approach aiming to bridge the gap between computational efficiency and biological plausibility in artificial intelligence. We're sharing this work openly with the community and would love to hear your thoughts and feedback!

What are Continuous Thought Machines?

Most deep learning architectures simplify neural activity by abstracting away temporal dynamics. In our paper, we challenge that paradigm by reintroducing neural timing as a foundational element. The Continuous Thought Machine (CTM) is a model designed to leverage neural dynamics as its core representation.

Core Innovations:

The CTM has two main innovations:

1. Neuron-Level Temporal Processing: Each neuron uses unique weight parameters to process a history of incoming signals. This moves beyond static activation functions to cultivate richer neuron dynamics.
2. Neural Synchronization as a Latent Representation: The CTM employs neural synchronization as a direct latent representation for observing data (e.g., through attention) and making predictions. This is a fundamentally new type of representation distinct from traditional activation vectors.

Why is this exciting?

Our research demonstrates that this approach allows the CTM to:

• Perform a diverse range of challenging tasks: Including image classification, solving 2D mazes, sorting, parity computation, question-answering, and RL tasks.
• Exhibit rich internal representations: Offering a natural avenue for interpretation due to its internal process.
• Perform tasks requirin sequential reasoning.
• Leverage adaptive compute: The CTM can stop earlier for simpler tasks or continue computing for more challenging instances, without needing additional complex loss functions.
• Build internal maps: For example, when solving 2D mazes, the CTM can attend to specific input data without positional embeddings by forming rich internal maps.
• Store and retrieve memories: It learns to synchronize neural dynamics to store and retrieve memories beyond its immediate activation history.
• Achieve strong calibration: For instance, in classification tasks, the CTM showed surprisingly strong calibration, a feature that wasn't explicitly designed for.

Our Goal:

It is crucial to note that our approach advocates for borrowing concepts from biology rather than insisting on strict, literal plausibility. We took inspiration from a critical aspect of biological intelligence: that thought takes time.

The aim of this work is to share the CTM and its associated innovations, rather than solely pushing for new state-of-the-art results. We believe the CTM represents a significant step toward developing more biologically plausible and powerful artificial intelligence systems. We are committed to continuing work on the CTM, given the potential avenues of future work we think it enables.

We encourage you to check out the paper, interactive demos on our project page, and the open-source code repository. We're keen to see what the community builds with it and to discuss the potential of neural dynamics in AI!