Difference in accuracy question

[adam-hartshorne]

It appears somebody else has written a pull request for an implementation of the Adan optimizer for Optax. I pointed out that your implementation already existed and it is claimed that in testing that your version is "3 order of magnitude larger error".

google-deepmind/optax#401

Is there potentially a bug in your code?

[hr0nix]

Thanks for pointing that out! I'll take a look

[hr0nix]

Took a look at it, the difference comes from where epsilon is placed: google-deepmind/optax#401 (comment)

Not sure if that's important, but I'll push this change so that this implementation is more similar to reference.