Skip to content

dirty-cat/dirty_cat

 
 

Repository files navigation

dirty_cat

Warning

dirty_cat has migrated to skrub. This repository will no longer be maintained.

dirty_cat logo

py_ver pypi_var pypi_dl codecov circleci black

dirty_cat is a Python library that facilitates machine-learning on dirty categorical variables.

For a detailed description of the problem of encoding dirty categorical data, see Similarity encoding for learning with dirty categorical variables [1] and Encoding high-cardinality string categorical variables [2].

If you like the package, please spread the word, and ⭐ the repository!

What can dirty_cat do?

dirty_cat provides tools (TableVectorizer, fuzzy_join...) and encoders (GapEncoder, MinHashEncoder...) for morphological similarities, for which we usually identify three common cases: similarities, typos and variations

The first example notebook goes in-depth on how to identify and deal with dirty data using the dirty_cat library.

What dirty_cat does not

Semantic similarities are currently not supported. For example, the similarity between car and automobile is outside the reach of the methods implemented here.

This kind of problem is tackled by Natural Language Processing methods.

dirty_cat can still help with handling typos and variations in this kind of setting.

Installation

dirty_cat can be easily installed via pip:

pip install dirty_cat

Dependencies

Dependencies and minimal versions are listed in the setup file.

Related projects

Are listed on the dirty_cat's website

Contributing

If you want to encourage development of dirty_cat, the best thing to do is to spread the word!

If you encounter an issue while using dirty_cat, please open an issue and/or submit a pull request. Don't hesitate, you're helping to make this project better for everyone!

Additional resources

References

[1]Patricio Cerda, Gaël Varoquaux, Balázs Kégl. Similarity encoding for learning with dirty categorical variables. 2018. Machine Learning journal, Springer.
[2]Patricio Cerda, Gaël Varoquaux. Encoding high-cardinality string categorical variables. 2020. IEEE Transactions on Knowledge & Data Engineering.

About

Machine learning on dirty tabular data (legacy clone of skrub)

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 97.6%
  • Shell 2.4%