Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Got error when executing Hive on Snappy supported Hadoop #252

Open
changzong opened this issue Nov 25, 2016 · 1 comment
Open

Got error when executing Hive on Snappy supported Hadoop #252

changzong opened this issue Nov 25, 2016 · 1 comment

Comments

@changzong
Copy link

Hi, I just tried to run hive script with azkaban 3.0 with the hive plugin, and this error came up:

"Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec."

My Hadoop supports SnappyCodec and every configuration in hadoop and azkaban plugin seem correct. I think the problem may come from missing native library configuration but I can't find any place to set. Any suggestion?

Thanks

@changzong
Copy link
Author

I just knew how this came up. The problem is from my tez configuration. I set tez.runtime.compress to true to compress my intermediate data and use SnappyCodec as the compression method. After I set the config to false, every thing succeeded. I still don't understand why this parameter cannot be used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant