My goal in this project is building mathematical transformer architecture from scratch and training it to learn basic integer based mathematics like addition, subtraction and multiplication. I implemented the original "Attention Is All You Need" paper architecture in PyTorch and expand from there.
Can tokens be used for math calculations and inference? How well embeddings can represent numerical values and their relations? I want to verify how to expand these in order to create more symbolic-like math representations with a flavor of Wolfram Mathematica or Theano.