You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The test case is verifying the correct reading of a Parquet file with nested structures and arrays: nested-array-struct.zip
When parsing then provided parquet here with the content:
scala> val df=spark.read.parquet("/home/fejiang/Downloads/nested-array-struct.parquet")
df: org.apache.spark.sql.DataFrame = [primitive: int, myComplex: array<struct<id:int,repeatedMessage:array<struct<someId:int>>>>]
scala> df.show()
24/09/18 11:17:21 WARN GpuOverrides:
!Exec <CollectLimitExec> cannot run on GPU because the Exec CollectLimitExec has been disabled, and is disabled by default because Collect Limit replacement can be slower on the GPU, if huge number of rows in a batch it could help by limiting the number of rows transferred from GPU to CPU. Set spark.rapids.sql.exec.CollectLimitExec to true if you wish to enable it
@Partitioning <SinglePartition$> could run on GPU
24/09/18 11:17:23 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)/ 1]
java.lang.ClassCastException: optional int32 someId is not a group
at org.apache.parquet.schema.Type.asGroupType(Type.java:248)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.checkSchemaCompat(GpuParquetScan.scala:805)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.$anonfun$checkSchemaCompat$3(GpuParquetScan.scala:830)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.$anonfun$checkSchemaCompat$3$adapted(GpuParquetScan.scala:821)
at scala.Option.foreach(Option.scala:407)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.$anonfun$checkSchemaCompat$2(GpuParquetScan.scala:821)
at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.$anonfun$checkSchemaCompat$2$adapted(GpuParquetScan.scala:820)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
The text was updated successfully, but these errors were encountered:
The test case is verifying the correct reading of a Parquet file with nested structures and arrays:
nested-array-struct.zip
When parsing then provided parquet here with the content:
CPU:
GPU:
The text was updated successfully, but these errors were encountered: