softmax activation in GRU

[fgvbrt]

Hi, I noticed that you put softmax activation inside GRU cell, as I understand in this case you wont get sum of activations for each timestep equals to 1. Here is link for GRU cell and the same situation for terminal GRU https://github.com/HIPS/molecule-autoencoder/blob/master/autoencoder/train_autoencoder.py#L225

I also checked with you version of keras that it does not sum to 1, here is link to ghist https://gist.github.com/fgvbrt/1f2e1828c6d8c0eb88614f14c60874ad

Was it done on purpose or was it mistake?
Thanks in advance.

[fgvbrt]

I also want to add, that this softmax in GRU would be valid if initialization of initial state would represent probability distribution (i.e. initial states sums to one), but in code there is initialization with zeros.

[duvenaud]

Thanks for catching this! I'll bring it up with Rafa G-B next time I talk to him.