Hi I am trying to read a timestamp field (datetime) from A parquet file using a case class (in a typedpipe).
The parquet file is from an impala table.
the parquet-tools lib gives the type as
optional int96
Does anyone know what the field type should be ?
If I use Option[Long]
case class Columns(request_id:Option[String],datetime: Option[Long])
val input = TypedParquet[Columns](input1)
then it gives the below error
Caused by: org.apache.parquet.io.ParquetDecodingException: The requested schema is not compatible with the file schema. incompatible types: optional int64 datetime != optional int96 datetime
which kind of makes sense.
If I try to use BigInt
case class Columns(request_id:Option[String],datetime: Option[Bigint])
val input = TypedParquet[Columns](input1)
It does not even compile it gives the below error
Parquetreadsupport - could not find implicit value for parameter