I've recently been wondering about that as well! Atm, simple-parsing seems to be the closest, but datargs or the configs used in olmo-core seem to be cleaner.. and then there's jsonargparse which aims to do everything but is also reeally big. Did you find anything else?
Why you should probe more than just the final layer of your Vision Transformer to maximize performance. 🧵👇
Happy to present our NeurIPS Spotlight paper with @flxsa.bsky.social, Nicola Muça Cirone, and Antonio Orvieto:
Fixed-Point RNNs: Interpolating from Diagonal to Dense
Here's a summary of the paper.
At the opening ceremony in Singapore they said that the city is still unclear.
Agreed that it's important to consider depth in addition to work. But for work to be negligible, the hardware would need to be infinitely scalable with input size because the asymptotic runtime is O(work/workers + depth). I doubt that this will change for future hardware due to physical limitations.
You could have a look at litmaps. They allow to explore new papers from a set of starting nodes. Also nice ways of sorting in time, relevancy, clustering etc.
It's not fake:
bsky.app/profile/yann...
Congrats Anastasia!!
Just a heads up to everyone: @deep-mind.bsky.social is unfortunately a fake account and has been reported. Please do not follow it nor repost anything from it.
You can use @bookmarks.bluecanary.dev to send messages to or report feature which are stored in a feed of your bookmarks
🍏 New preprint alert! 🍏
PoM: Efficient Image and Video Generation with the Polynomial Mixer
arxiv.org/abs/2411.12663
This is my latest "summer project" and it was so big I had to call in reinforcements (Thanks @nicolasdufour.bsky.social)
TL;DR Transformers are for boomers, welcome to the future
🧵👇
I also wonder how one would go about normalizing the sum in a variable sequence length mode.. How are you handling that at the moment for video? One potential solution could be a geometric decay on top of the mask, but that would lead to somewhat of an SSMs formulation? Interesting food for thought!
Interesting idea! This reminds me of the Symmetric Power Transformer (manifestai.com/articles/symmetric-power-transformers), but in your case the values are subject to the polynomial kernel instead of queries/keys while the selection resembles a Gated Linear Unit which is used in many backbones:)
