Skip to content
Snippets Groups Projects
Commit aa8f0f4b authored by Jan Ebert's avatar Jan Ebert
Browse files

Fix xPos numerical stability optimization

parent c8f3fcd8
Branches
No related tags found
No related merge requests found
...@@ -230,7 +230,7 @@ class XPosEncoding(RotaryPositionalEncoding): ...@@ -230,7 +230,7 @@ class XPosEncoding(RotaryPositionalEncoding):
dampening = ( dampening = (
(torch.arange(0, d_model, 2) + gamma * d_model) (torch.arange(0, d_model, 2) + gamma * d_model)
/ (1.0 + gamma) * d_model / ((1.0 + gamma) * d_model)
).repeat_interleave(2) ).repeat_interleave(2)
self.register_buffer('dampening', dampening) self.register_buffer('dampening', dampening)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment