-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Not sure how to interpret the output #9
Comments
Hi Bart
Have a look at the
https://github.com/jveitchmichaelis/edgetpu-yolo/blob/784d9be1bb13ce4b8b3c1bad729a02a69cca97bb/edgetpumodel.py#L226
process predictions method.
And also
https://github.com/jveitchmichaelis/edgetpu-yolo/blob/784d9be1bb13ce4b8b3c1bad729a02a69cca97bb/nms.py#L52
which might make it clearer how the array is handled.
You should have the four bounding coordinates, object confidence and a
score for each class. That's a lot of detections though, so normally you'd
want to run thresholding and non max suppression on the output. You also
need to rescale the coords back to absolute pixels. So there's a bit of
post processing required, but just dropping low confidence predictions
should clean things up a bit.
Cheers
…On Sun, 14 May 2023 at 23:18, bartbutenaers ***@***.***> wrote:
Hi @jveitchmichaelis <https://github.com/jveitchmichaelis>,
Thank you for sharing these models!
When I run a coco ssd model (from the coral.ai
<https://coral.ai/models/object-detection/> site) via tfjs
<https://github.com/tensorflow/tfjs>, then I get as expected 4 output
sensors as prediction result: scores, classes, bboxes, and detected object
count.
I can load your models (i.e. edgetpu.tflite) without problems in tfjs, but
afterwards the object detection output contains only one tensor. That
tensor contains an array of 3087 sub-arrays (each containing 85 integer
numbers):
[image: image]
<https://user-images.githubusercontent.com/14224149/238209740-e5e2bdf0-b564-400a-8a72-87eabe112507.png>
I see that you use python instead of TensorflowJs, so I assume you don't
use tfjs... But do you perhaps have an idea how I can interpret this
output, or what I might be doing wrong? The tfjs-tflite package
***@***.***/tfjs-tflite> has status *"work
in progress"* so perhaps it contains some bugs...
Thanks!!!
Bart
—
Reply to this email directly, view it on GitHub
<#9>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYDMJZZUCF2LZ3JXKGPOFLXGFDZFANCNFSM6AAAAAAYBMO5LU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hi Josh, Thanks for the pointers to the code!! Although I am not very familiar with Python, the code was very illuminating. I am getting closer to a solution, but my bounding boxes do not fit my objects yet. So I would appreciate if you could help me a bit more with this... Can you please confirm whether my assumptions are correct:
Or perhaps you have any other info that could help me determine why my bounding boxes do not match my objects inside the image... |
I don't get the bounding boxes fixed when running your model in tfjs ;-( When debugging your code, I see that the But when I debug my own code, I see that your model works with int32 data: And all values in my output are indeed integers... I would appreciate a lot if you could give me some advice based on your knowledge!! Because I think you do some extra postprocessing - next to the things I listed above - but not all Python code is entirely clear to me Thanks! |
Hi Bart,
Could you send over a test image and the tensor that comes out of the model?
Best
Josh
…On Fri, 19 May 2023 at 07:22, bartbutenaers ***@***.***> wrote:
@jveitchmichaelis <https://github.com/jveitchmichaelis>,
I don't get the bounding boxes fixed when running your model in tfjs ;-(
When debugging your code, I see that the x contains all floats:
[image: image]
<https://user-images.githubusercontent.com/14224149/239443814-7ab72b49-84d1-4d21-b887-b06d77ef14e1.png>
But when I debug my own code, I see that your model works with int32 data:
[image: image]
<https://user-images.githubusercontent.com/14224149/239443994-d087633d-bff3-4bb8-ae65-be7c1326aa16.png>
And all values in my output are indeed integers...
I would appreciate a lot if you could give me some advice based on your
knowledge!! Because I think you do some extra postprocessing - next to the
things I listed above - but not all Python code is entirely clear to me
Thanks!
Bart
—
Reply to this email directly, view it on GitHub
<#9 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYDMJ5IEPYUW2TMZGYMO5TXG37QZANCNFSM6AAAAAAYBMO5LU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hi Josh, For example if I use this image, the I get this output tensor: In the following file you can find the tensor's data array: |
I tried lots of things, but can't get this running unfortunately... It would be very appreciated if you could find some free time to have a look at my tensordata.txt file above. |
Hi Bart, One thing that seems obviously wrong is your tensor data is all integers. It should be float data - they represent probabilities (mostly). Are you casting somewhere? The model does use integer math but there is some scaling that happens to the output tensor which converts back to float. Line 163 in 784d9be
And see a few lines above, you should also scale the input to the model. If you're just using the weights on their own (without scaling the input image or the predictions) I guess this won't work. The shape looks good though! |
Hello Josh, The only input processing I do is a bilinear resizing of my input image tensor, to resize the image to the resolution requirements of your model. I also have a normalization preprocessing part, but that is not being executed for your model since both the input image tensor and the requirement of your model is int32: If I force this normalization to be executed for your model, then my prediction gives an exception So I assume your input scaling does something different? Would be nice if you could explain it in pseudo code, so I can convert it to javascript. Because unfortunately my Python knowledge is a bit lacking... |
Sure, the process is described here: https://www.tensorflow.org/lite/performance/post_training_quantization https://www.tensorflow.org/lite/performance/quantization_spec See the bit at the bottom about representation for quantized tensors. You need to apply the scaling to both the image going in (float > int) and on the tensor coming out (int > float). So going in, rearrange for the int8 bit and going out, you want the real bit. If you're getting an int output then tfjs isn't doing it for you I think. The scaling parameters are stored with the model as they're calibrated by running a bunch of images through. The Python code here reads them from the checkpoint. This is not the usual 1/255 scaling you might do for a normal CNN. You need to do that first, and then apply the conversion to a scaled integer. I guess you could also roll it into one scale factor but sensible to separate it for clarity. We do the 1/255 here: Line 65 in 784d9be
Another example here https://www.tensorflow.org/lite/performance/post_training_integer_quant In this example they check for a uint8 input (ie the model spec, not if the image is 8 bit!) and if so they apply scaling. I've not had much luck looking for a tfjs version but there may be one somewhere. Note they read the "quantization" parameter to get the scaling values. See at the top where test_images is defined, they also scale by 255. I guess tfjs doesn't have an 8 bit datatype so you have int32? Something like: const norm_image = image.cast('float32').div(255);
const scale_image = norm_image.div(scale).add(zero_point);
const pred = model.infer(norm_image.cast('int32'));
const float_pred = pred.cast('float32').add(zero_point).mul(scale);
// now do NMS, filter low probs etc. |
Thanks for the clarification! Never heard of quantization before... I only had time last night to implement scaling on the output tensor (i.e. detectionResult):
When applying this, the bounding boxes already start to making sense (when I filter out the ones with a low score): Although there are still some things I don't get:
Perhaps this is caused because I didn't scale the input yet. Will try to find some time tonight for that... |
About scaling the input image tensor: how can you determine the scalefactor that you use in your code snippet BEFORE the prediction is executed? Because to determine the scaleFactor you need qmin and qmax, which are based on the prediction result that you don't have yet. Hmm I assume my factor calculation is not correct :-( |
The quantisation values are essentially weights. They are computed once
when you calibrate the model and are the same for every subsequent input.
They're stored in the model checkpoint - I assume you can access it with
tfjs but I'm not familiar. We load the in/out scale factors here:
https://github.com/jveitchmichaelis/edgetpu-yolo/blob/784d9be1bb13ce4b8b3c1bad729a02a69cca97bb/edgetpumodel.py#L90
This uses functions from pycoral, but there must be an equivalent
…On Thu, 24 Aug 2023 at 07:55, bartbutenaers ***@***.***> wrote:
About scaling the input image tensor: how can you determine the
scalefactor that you use in your code snippet BEFORE the prediction is
executed? Because to determine the scaleFactor you need qmin and qmax,
which are based on the prediction result that you don't have yet. Hmm I
assume my factor calculation is not correct :-(
—
Reply to this email directly, view it on GitHub
<#9 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYDMJ66KMDALDNQCDYXLL3XW33GVANCNFSM6AAAAAAYBMO5LU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Also note there are different input and output scale values (I forgot about
that) stored in the model.
The model will predict a lot of very similar boxes due to the way yolo
works (you'll get a fixed number of predictions each time). You normally
drop most of them based on the confidence score and run non-max suppression
to get rid of duplicates.
On Thu, 24 Aug 2023 at 08:46, Josh Veitch-Michaelis <
***@***.***> wrote:
… The quantisation values are essentially weights. They are computed once
when you calibrate the model and are the same for every subsequent input.
They're stored in the model checkpoint - I assume you can access it with
tfjs but I'm not familiar. We load the in/out scale factors here:
https://github.com/jveitchmichaelis/edgetpu-yolo/blob/784d9be1bb13ce4b8b3c1bad729a02a69cca97bb/edgetpumodel.py#L90
This uses functions from pycoral, but there must be an equivalent
On Thu, 24 Aug 2023 at 07:55, bartbutenaers ***@***.***>
wrote:
> About scaling the input image tensor: how can you determine the
> scalefactor that you use in your code snippet BEFORE the prediction is
> executed? Because to determine the scaleFactor you need qmin and qmax,
> which are based on the prediction result that you don't have yet. Hmm I
> assume my factor calculation is not correct :-(
>
> —
> Reply to this email directly, view it on GitHub
> <#9 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAYDMJ66KMDALDNQCDYXLL3XW33GVANCNFSM6AAAAAAYBMO5LU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
BTW I could not find a way to get the quantization parameters from the model in Tfjs. So I have asked in the Tensorflow forum for help. Hopefully somebody of their community can provide me the golden tip... |
Hi @jveitchmichaelis,
Thank you for sharing these models!
When I run a coco ssd model (from the coral.ai site) via tfjs, then I get as expected 4 output sensors as prediction result: scores, classes, bboxes, and detected object count.
I can load your models (i.e. edgetpu.tflite) without problems in tfjs, but afterwards the object detection output contains only one tensor. That tensor contains an array of 3087 sub-arrays (each containing 85 integer numbers):
I see that you use python instead of TensorflowJs, so I assume you don't use tfjs... But do you perhaps have an idea how I can interpret this output, or what I might be doing wrong? The tfjs-tflite package has status "work in progress" so perhaps it contains some bugs...
Thanks!!!
Bart
The text was updated successfully, but these errors were encountered: