[SUPPORT] Missing dependency on hive-exec (core) #8147
Comments
danny0405
added
flink
|
On cluster, you should use the bundle jar instead, and yeah, the default bundler jar does not package the hive-exec, which should be fixed: https://issues.apache.org/jira/browse/HUDI-5916
hudi/hudi-flink-datasource/hudi-flink/pom.xml Line 287 in 2675118 |
|
The error stack trace confused me a log, because |
|
I was also confused by that. I think when the |
|
The |
|
You are right, the bundle jar should keep more slim. |
I see, in thie PR #7055, I have moved the utilities method into another class which is located in |
|
@gamblewin - if you can, I think the easiest solution is to ad the |
@kkrugler thx for replying
|
|
The error indicates some parquet class conflict. |
|
I think we can close the issue, this is a known behavior of flink bundle jar packaging. |
I met the same issues, did you fix the problem finally? @gamblewin |
|
I have the same issue when I'm trying to use hudi-flink1.17-bundle-0.14.1.jar with Flink and sync to Hive. Is there any way to fix it? Adding hive-exec like above results in the same error as above. |
|
I think the issue you see is due to having multiple versions of Parquet support classes in different jars. Unfortunately the only way to fix it is to carefully review versions of all transitive dependencies of all jars, to figure out which jar(s) are in conflict. I've re-purposed the "JarHell" class from OpenSearch to track down issues like this (e.g. call it from the open method of some UDF in your case, to find run-time class conflicts). |
Describe the problem you faced
When using Flink to do an incremental query read from a table, using the 0.12.2 and Flink 1.15, I occasionally get a ClassNotFoundException for
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat. This usually happens when running the test from inside Eclipse, occasionally from the command line.To Reproduce
Steps to reproduce the behavior:
git clone https://github.com/kkrugler/flink-hudi-query-testcd flink-hudi-query-testpom.xmlto remove explicit dependency onhive-exec.mvn clean packageExpected behavior
The tests should all pass.
Environment Description
Hudi version : 0.12.2
Flink version : 1.15.1
Additional context
I believe the problem is that the
hudi-hadoop-mrdependency onhive-exec(with classifiercore) is marked as provided, but when running a Flink workflow in a typical Flink cluster you don't have Hive jars installed. I think maybe it's OK forhudi-hadoop-mrto say this is provided, buthudi-flinkshould then have an explicit dependency on this artifact, something like:Note the exclusion of all transitive dependencies. All that Hudi needs from
hive-execis the one missing class, as Hudi uses HoodieParquetInputFormatBase as the base class, as per:And if you don't do this exclusion, you wind up pulling in lots of additional code that's not needed (AFAICT).
Stacktrace
The text was updated successfully, but these errors were encountered: