improve reproducability

[loeken]

Description

in the requirements.txt not all dependencies have a fixed version. with those other projects changing/bugs introduced its a bit hit/miss to not get a broken version when installing. I suggest to also add specific versions to all ( including a hash for the transformers which is pulled from git ). this will ensure users have the same dependcies.

Additional Context

Dependency pinning is useful because it ensures that a software application will use the exact version of its dependencies that have been tested and confirmed to work with that application. Without pinning, the application may use newer versions of its dependencies that have introduced breaking changes or other incompatibilities that cause the application to fail.

Dependency pinning also helps to ensure that the application will continue to work as expected even as the dependencies are updated over time. If a new version of a dependency is released that introduces breaking changes or other incompatibilities, the application will not automatically start using that version and potentially break as a result.

Overall, dependency pinning helps to ensure the stability and reliability of software applications, particularly in complex systems with many interdependent components.

[oobabooga]

There is a trade off. Pinning packages requires additional maintenance, so I have only pinned packages that are more likely to break things, and left stable packages like numpy and requests unpinned.

[loeken]

while testing the docker stuff i did notice that both:

git+https://github.com/huggingface/transformers
sentencepiece

made some builds fail

What is the additional maintenance work - keeping track of new versions and testing/updating when those are released?