Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Pixtral #33449

Merged
merged 62 commits into from
Sep 14, 2024
Merged

Add support for Pixtral #33449

merged 62 commits into from
Sep 14, 2024
+2,707 −2

Conversation

ArthurZucker
Copy link
Collaborator

@ArthurZucker ArthurZucker commented Sep 12, 2024 •
edited by LysandreJik
Loading

What does this PR do?

Pixtral is basically LLAVA with a new vision encoder that uses ROPE.

Closes #33441

@LysandreJik LysandreJik mentioned this pull request Sep 12, 2024
Closed
@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ArthurZucker and others added 5 commits September 12, 2024 16:57
@ArthurZucker
nits
eb76b0c
@ArthurZucker
nits
334d7a9
@ArthurZucker
updates to support the tokenizer :)
30439a1
@ArthurZucker
updates
6544127
@amyeroberts
Pixtral processor (#33454)
a45122b 
* rough outline

* Add in image break and end tokens

* Fix

* Udo some formatting changes

* Set patch_size default

* Fix
amyeroberts and others added 26 commits September 13, 2024 17:41
@amyeroberts
Apply suggestions from code review
ff04e9f
@amyeroberts
Fix up
e7fae23
@ArthurZucker
updates
324ba36
@ArthurZucker
revert modeling changes
c4ad4e5
@ArthurZucker
update
9f2d98b
@ArthurZucker
Merge branch 'add-pixtral' of github.com:huggingface/transformers int…
4562fa4 
…o add-pixtral
@ArthurZucker
update
97b4d93
@ArthurZucker
fix load safe
ce23dc3
@ArthurZucker
addd liscence
bbf516c
@ArthurZucker
update
b783e7a
@ArthurZucker
use pixel_values as required by the model
443917f
@ArthurZucker
skip some tests and refactor
f9291ea
@amyeroberts
Add pixtral image processing tests (#33476)
db84a7d 
* Image processing tests

* Add processing tests

* woops

* defaults reflect pixtral image processor
@ArthurZucker
fixup post merge
908233f
@ArthurZucker
images -> pixel values
8c0f8f6
@ArthurZucker
oups sorry Mr docbuilder
b7d7760
@ArthurZucker
isort
be154e1
@ArthurZucker
fix
bd721e2
@ArthurZucker
fix processor tests
2eda353
@ArthurZucker
small fixes
36e5525
@ArthurZucker
nit
7d4bb19
@ArthurZucker
update
df84fe7
@ArthurZucker
last nits
0da8b22
@ArthurZucker
oups this was really breaking!
8cfff1a
@ArthurZucker
nits
1effab2
@ArthurZucker
is composition needs to be true
f33fe19
@ArthurZucker ArthurZucker merged commit 8bd2b1e into main Sep 14, 2024
22 of 24 checks passed
@ArthurZucker ArthurZucker deleted the add-pixtral branch September 14, 2024 10:28
@NielsRogge NielsRogge mentioned this pull request Sep 14, 2024
Open
@YorkieDev YorkieDev mentioned this pull request Sep 14, 2024
Open
4 tasks
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyeroberts amyeroberts amyeroberts approved these changes

Assignees
No one assigned
Labels
New model
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support Pixtral
3 participants
@ArthurZucker @HuggingFaceDocBuilderDev @amyeroberts