-
Notifications
You must be signed in to change notification settings - Fork 525
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TensorflowPredictVGGish #1333
Comments
Hi @seunggookim, For now, you could reconstruct the patches by: np.reshape(activation, (-1, 6, 4, 512)) Regarding your experiment, I would expect slightly different activations since we approach the patch generation in a slightly different way which could result in different amounts of padding applied. |
Thanks @palonso! The dimension of the input image is 96 frames x 64 mel-bands, not 64x96. With (96, 64), the pool4 should be (-1, 6, 4, 512), not (-1, 4, 6, 512). While this is not critical for my purpose, I just wonder if the input image was transposed in what Essentia uses. |
Right @seunggookim, I confused the axes. Essentia generates data as (batch, channel, timestamps, features), so the only transposition done internally is putting the channel as the last dimension. |
I updated my previous answer with your comment |
Hi @seunggookim , If you want to retrieve outputs with an arbitrary shape, you can use the generic algorithm TensorflowPredict, which requires additional configuration work in exchange for the extra flexibility. For your case: from essentia.standard import TensorflowInputVGGish, TensorflowPredict, FrameGenerator
from essentia import Pool
import numpy as np
vggish_path = (
"/media/data/models/essentia-models/feature-extractors/vggish/audioset-vggish-3.pb"
)
output_node = "model/vggish/pool4/MaxPool"
input_node = "model/Placeholder"
frame_size = 400
hop_size = 160
patch_size = 96
n_melbands = 64
vggish = TensorflowPredict(
graphFilename=vggish_path, outputs=[output_node], inputs=[input_node]
)
melband_extractor = TensorflowInputVGGish()
pool = Pool()
audio = np.ones(2 * 16000).astype("float32")
# Compute mel bands
melbands = np.array([
melband_extractor(frame)
for frame in FrameGenerator(audio, frameSize=frame_size, hopSize=hop_size)
])
# Reshape to patches
trim = len(melbands) % patch_size
melbands = np.reshape(melbands[:-trim], (-1, patch_size, n_melbands))
# Add channel dimension
melbands = np.expand_dims(melbands, axis=1)
# Put the data into a pool
pool.set(input_node, melbands)
output = vggish(pool)[output_node]
print(output.shape)
|
In other TensorflowPredict algorithm we have prefered to return 2D outputs since they typically fit in the schema (timestamps, embeddings), or (timestamps, activations). Hoewer in some cases more dimensions are required. For example, we need 3D to return the attention layers (batch, tokens, dimensions). A similar problem has happened before when trying to retrieve the internal representations of VGGish: MTG#1333
I would like to find corresponding patches for shallow layer activation (MaxPool). But, because they are flattened, it is difficult for me. For example, if I feed a short audio segment for a single patch (15600 samples at 16 kHz),
model/vggish/pool4/MaxPool
is (24, 512). But for a longer audio file, it is (24*#patches, 512). For me, it's unclear whether the first dimension, which is flattened, is along patches first ([p1f1, p2f1, p3f1, ..., p1f2, p2f2, p3f2, ...]) or features ([p1f1, p1f2, p1f3, ..., p2f1, p2f2, p2f3, ...]).To figure this out myself, I manually segmented the audio file into 15600 samples before feeding them to the models as follows:
Because the final layer "embeddings" is clear to me, I compared the results for the final layer. What I tried got me the same number of patches, but slightly different activation values (r = 0.6349 to 1; mean r = 0.9743), which is still disturbing. Should I add anything to make these identical to the original function?
The text was updated successfully, but these errors were encountered: