-
Notifications
You must be signed in to change notification settings - Fork 277
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Finetuning for multiple classes #114
Comments
Hi, If you're using this implementation, you can just individually train one model for each concept and then use the merge embedding script to combine them into a single model. There's instructions for that in the readme. You can also changr your placeholder token from * to 'ship' or 'plane'. Look at either the config file, or main.py's run arguments for how to do this. |
Thanks for your response @rinongal. I tried to invert for 2 classes "airplane" and "truck" Inversion command (for airplane):
And the inference command is:
The images in the And the images generated in which I believe is a fair generation of airplane capturing the style/features present in the above 5 images. But I've few questions and need suggestions from your end
and it did generate the airplane_truck.pt file. Now my question is,
Thanks a lot for your time :) |
You can find an explanation of the logged images here: #19 and here: #34
If you are using Stable Diffusion, their text encoder is significantly weaker and more prone to overfitting, which may make some modifications harder. In that case, you may have to use more complex prompts or some prompt-weighting methods (for which I'd recommend the AUTOMATIC1111 WebUI 3(+4). The merged embedding file is just putting both of your new words into one single file. It won't create better trucks than just the truck file, and it won't create better airplanes than just the airplane file. It just stores them in a way that lets you access both at the same time. |
Thank you so much. Have additional questions which are not in this thread, so closing this thread :) |
@rinongal @NamburiSrinath Hi, I have a relate question: I use Stable Diffusion and I find it's very hard to modification. For example, below are images generated by using prompts "a photo of * " and "a photo of * in river". The embedding is at step 499 and should not be overfitting. |
@NamburiSrinath hi, I'd like to ask the merge of plane and truck, the output images are mixture of airplanes and trucks, or a separate airplane and truck? Thank you. |
Hi,
I tried playing with Stable diffusion (https://github.com/huggingface/diffusers) and tried to generate images, but didn't achieve good quality ones.
Example prompt: "Aeroplane"
![plane](https://user-images.githubusercontent.com/40389487/197682468-bd7f02e6-3c13-491f-bfac-ae8a82e6c81d.png)
Generated image
Note: Some generated images are of nice quality (eg:
![plane2](https://user-images.githubusercontent.com/40389487/197682994-f5abcb8a-5254-457d-9704-166b50162f54.png)
)
I came across your repo and found that I can use "Textual Inversion" and fine tune to get the concept transferred. But I have multiple classes (ship, aeroplane etc;) and would like to know how can I fine tune on multiple classes.
In short, while inferencing, you mentioned that we need to prompt it as "A photo of *", but I would like to do "A photo of an aeroplane", "A photo of a ship" etc; after fine tuning the model. (I checked this and am curious to know if this repo can work for my case - #8)
Thanks in advance
Srinath
The text was updated successfully, but these errors were encountered: