spark sql failed to parse parquet -

-

spark sql parse parquet file, failed. cmd 

create temporary table testparquet     using org.apache.spark.sql.parquet options (                           path "file:////tmp/table_generic.parquet"              );

this error log:

java.lang.nullpointerexception     @ parquet.format.converter.parquetmetadataconverter.fromparquetstatistics(parquetmetadataconverter.java:249)     @ parquet.format.converter.parquetmetadataconverter.fromparquetmetadata(parquetmetadataconverter.java:543)     @ parquet.format.converter.parquetmetadataconverter.readparquetmetadata(parquetmetadataconverter.java:520)     @ parquet.hadoop.parquetfilereader.readfooter(parquetfilereader.java:426)     @ org.apache.spark.sql.parquet.parquetrelation2$metadatacache$$anonfun$refresh$6.apply(newparquet.scala:298)     @ org.apache.spark.sql.parquet.parquetrelation2$metadatacache$$anonfun$refresh$6.apply(newparquet.scala:297)     @ scala.collection.parallel.mutable.pararray$map.leaf(pararray.scala:658)     @ scala.collection.parallel.task$$anonfun$tryleaf$1.apply$mcv$sp(tasks.scala:54)     @ scala.collection.parallel.task$$anonfun$tryleaf$1.apply(tasks.scala:53)     @ scala.collection.parallel.task$$anonfun$tryleaf$1.apply(tasks.scala:53)     @ scala.collection.parallel.task$class.tryleaf(tasks.scala:56)     @ scala.collection.parallel.mutable.pararray$map.tryleaf(pararray.scala:650)     @ scala.collection.parallel.adaptiveworkstealingtasks$wrappedtask$class.compute(tasks.scala:165)     @ scala.collection.parallel.adaptiveworkstealingforkjointasks$wrappedtask.compute(tasks.scala:514)     @ scala.concurrent.forkjoin.recursiveaction.exec(recursiveaction.java:160)     @ scala.concurrent.forkjoin.forkjointask.doexec(forkjointask.java:260)     @ scala.concurrent.forkjoin.forkjoinpool$workqueue.runtask(forkjoinpool.java:1339)     @ scala.concurrent.forkjoin.forkjoinpool.runworker(forkjoinpool.java:1979)     @ scala.concurrent.forkjoin.forkjoinworkerthread.run(forkjoinworkerthread.java:107)

the parquet's format has no errors drill has parsed correctly. why that?


Comments

Post a Comment

Popular posts from this blog

javascript - oscilloscope of speaker input stops rendering after a few seconds -

-
the following script reads audio user's microphone , renders oscilloscope on html canvas. the source taken example of mozilla developer network: visualizations web audio api and here fiddle: http://jsfiddle.net/b7j8pktp/ mozgetusermedia (note: code has no fork mechanism different browsers: works firefox) it works fine few seconds , stops rendering. whereas works totally stable: http://mdn.github.io/voice-change-o-matic/ the problem can reduced following code. microphone activation icon (next the address bar in firefox) disappears after 5 seconds: navigator.mozgetusermedia({audio: true}, function() {}, function() {} ); ( http://jsfiddle.net/b7j8pktp/2/ ) this known bug in firefox. take stream getusermedia call , hook window so: navigator.mozgetusermedia({audio: true}, function(stream) { window.stream = stream; // rest of code }, function err() { // handle error }); hopefully can fixed soon. problem we're f
Read more

javascript - gulp-nodemon - nodejs restart after file change - Error: listen EADDRINUSE events.js:85 -

-
i'm running nodejs 0.12.3 - gulp 3.10.8 , gulp-nodemon 2.0.3. gulpfile.js : gulp.task('serve', ['build'], function() { gulp.start('fb-flo'); nodemon({ script: paths.server, env: { 'node_env': 'development' }, // env: { 'node_env': 'production' }, execmap: { 'js': 'node_modules/babel/bin/babel-node --stage 1' //es7 cote server }, delay: '0ms', watch: ['src/shared/', 'src/server/'], ignore: ['src/shared/components'], tasks: function (changedfiles) { var tasks = []; changedfiles.foreach(function (file) { if (path.extname(file) === '.jsx' && !~tasks.indexof('bundlejs')) tasks.push('bundlejs'); if (path.extname(file) === '.less' && !~tasks.indexof('compileless')) tasks.push('compileless'); }); return tasks; }
Read more

Fatal Python error: Py_Initialize: unable to load the file system codec. ImportError: No module named 'encodings' -

-
i trying make simple python program opening list of webpages user manually download reports site. don't have previous experience preparing exe files.. , i'm in learning process python coding. of done on windows 7 x64 this python code: #!c:/python34/python.exe -u splinter import * import time import os import csv #---------------------------------- raporty = [] open('../raporty.csv', newline='') csvfile: contents = csv.reader(csvfile, delimiter=' ', quotechar='|') row in contents: r = ', '.join(row) r = r.replace(',','') raporty.append(r) #--not implemented yet zmienne = [] open('../zmienne.csv', newline='') csvfile: contents = csv.reader(csvfile, delimiter=' ', quotechar='|') row in contents: r = ', '.join(row) r = r.replace(',','') zmienne.append(r) print("start") browser = browser(
Read more