Skip to content
Fossfits

attention-mechanisms.

Listed 17Updated Aug 28, 2026

Sorted by Most popular

  1. PaLM-rlhf-pytorchImplementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLMmit · linux7.9k
  2. audiolm-pytorchImplementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorchmit · linux2.6k
  3. alphafold3-pytorchImplementation of Alphafold 3 from Google Deepmind in Pytorchmit · linux1.7k
  4. BS-RoFormerImplementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labsmit · linux935
  5. magvit2-pytorchImplementation of MagViT2 Tokenizer in Pytorchmit · linux668
  6. mmditImplementation of a single layer of the MMDiT, proposed in Stable Diffusion 3, in Pytorchmit · linux559
  7. iTransformerUnofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant groupmit · linux541
  8. local-attentionAn implementation of local windowed attention for language modelingmit · linux505
  9. recurrent-memory-transformer-pytorchImplementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorchmit · linux425
  10. q-transformerImplementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmindmit · linux406
  11. clinical-calculator-tooluseExplorations into training LLMs to use clinical calculators from patient history, using open sourced models. Will start with Wells' Criteriamit · linux316
  12. CoLT5-attentionImplementation of the conditionally routed attention in the CoLT5 architecture, in Pytorchmit · linux231
  13. MambaTransformerIntegrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modelingmit · linux228
  14. simple-hierarchical-transformerExperiments around a simple idea for inducing multiple hierarchical predictive model within a GPTmit · linux228
  15. flash-cosine-sim-attentionImplementation of fused cosine similarity attention in the same style as Flash Attentionmit · linux221
  16. recurrent-interface-network-pytorchImplementation of Recurrent Interface Network (RIN), for highly efficient generation of images and video without cascading networks, in Pytorchmit · linux210
  17. coconut-pytorchImplementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorchmit · linux184