Token Viz

Token Viz is a project that utilizes FastAPI, Hugging Face, and basic HTML, JS, and CSS to visually demonstrate tokenization.

Installation

1. Clone the repository:

git clone https://github.com/your-username/token-viz.git

2. Navigate to the project directory:

cd token-viz

3. Install the required dependencies:

pip install -r requirements.txt

Usage

1. Start the FastAPI server:

uvicorn main:app --port 8000  --reload

2. Open your web browser and go to http://localhost:8000 to access the Token Viz interface.