Urgent.News

What's breaking now, across thousands of outlets.

More in AI

RoPE vs sinusoidal positional encoding, measured: 55 logits of drift against 0.0005

Take two token embeddings. Keep them exactly five positions apart and slide the pair down a 2048-token sequence. Under sinusoidal absolute position encoding the attention score for that one unchanging…

  • RoPE showed consistent -0.6102 logits, while sinusoidal encoding fluctuated
  • Sinusoidal encoding had 55.5-logit spread changing sign 157 times
  • Gap-0 rows confirmed RoPE matched no-positional-encoding score

More from Thursday 1 October →