and in particular this image which clarifies the key essential difference between liner attention and delta network by examining the case of two tokens with same key but different value
I think for comparison it would also have been good to have how original quadratic attention handles it: since both keys are identical, the attention would be "evenly divided" between both values so the final output would be the average of both values, as opposed to the latest value
https://x.com/waterloo_intern/article/2081762065392541951
and in particular this image which clarifies the key essential difference between liner attention and delta network by examining the case of two tokens with same key but different value
https://pbs.twimg.com/media/HOPCc7BaEAAQDtO.jpg?format=jpg&n...
I think for comparison it would also have been good to have how original quadratic attention handles it: since both keys are identical, the attention would be "evenly divided" between both values so the final output would be the average of both values, as opposed to the latest value