apache spark - Serialization error when using a non-serializable object in driver code -

-

i'm using spark streaming process stream processing each partition (saving events hbase), ack last event in each rdd driver receiver, receiver can ack source in turn.

public class streamprocessor {    final ackclient ackclient;    public streamprocessor(ackclient ackclient) {     this.ackclient = ackclient;   }    public void process(final javareceiverinputdstream<event> inputdstream)     inputdstream.foreachrdd(rdd -> {       javardd<event> lastevents = rdd.mappartition(events -> {         // ------ code executes on worker -------         // process events 1 one; don't use ackclient here         // return event max delivery tag here       });       // ------ code executes on driver -------       event lastevent = .. // find event max delivery tag across partitions       ackclient.ack(lastevent); // use ackclient ack last event     });   } }

the problem here following error (even though seems work fine):

org.apache.spark.sparkexception: task not serializable     @ org.apache.spark.util.closurecleaner$.ensureserializable(closurecleaner.scala:166)     @ org.apache.spark.util.closurecleaner$.clean(closurecleaner.scala:158)     @ org.apache.spark.sparkcontext.clean(sparkcontext.scala:1435)     @ org.apache.spark.rdd.rdd.mappartitions(rdd.scala:602)     @ org.apache.spark.api.java.javarddlike$class.mappartitions(javarddlike.scala:141)     @ org.apache.spark.api.java.javardd.mappartitions(javardd.scala:32) ... caused by: java.io.notserializableexception: <some non-serializable object used ackclient> ...

it seems spark trying serialize ackclient send workers, thought code inside mappartitions serialized/shipped workers, , code @ rdd level (i.e. inside foreachrdd not inside mappartitions) not serialized/shipped workers.

can confirm if thinking correct or not? , if correct, should reported bug?

you correct, fixed in 1.1. however, if @ stack trace, cleaner throwing being invoked in mappartitions

at org.apache.spark.sparkcontext.clean(sparkcontext.scala:1435)

at org.apache.spark.rdd.rdd.mappartitions(rdd.scala:602)

so, problem has mappartitions. make sure aren't accidentally wrapping this, common issue


Comments

Post a Comment

Popular posts from this blog

javascript - oscilloscope of speaker input stops rendering after a few seconds -

-
the following script reads audio user's microphone , renders oscilloscope on html canvas. the source taken example of mozilla developer network: visualizations web audio api and here fiddle: http://jsfiddle.net/b7j8pktp/ mozgetusermedia (note: code has no fork mechanism different browsers: works firefox) it works fine few seconds , stops rendering. whereas works totally stable: http://mdn.github.io/voice-change-o-matic/ the problem can reduced following code. microphone activation icon (next the address bar in firefox) disappears after 5 seconds: navigator.mozgetusermedia({audio: true}, function() {}, function() {} ); ( http://jsfiddle.net/b7j8pktp/2/ ) this known bug in firefox. take stream getusermedia call , hook window so: navigator.mozgetusermedia({audio: true}, function(stream) { window.stream = stream; // rest of code }, function err() { // handle error }); hopefully can fixed soon. problem we're f
Read more

javascript - gulp-nodemon - nodejs restart after file change - Error: listen EADDRINUSE events.js:85 -

-
i'm running nodejs 0.12.3 - gulp 3.10.8 , gulp-nodemon 2.0.3. gulpfile.js : gulp.task('serve', ['build'], function() { gulp.start('fb-flo'); nodemon({ script: paths.server, env: { 'node_env': 'development' }, // env: { 'node_env': 'production' }, execmap: { 'js': 'node_modules/babel/bin/babel-node --stage 1' //es7 cote server }, delay: '0ms', watch: ['src/shared/', 'src/server/'], ignore: ['src/shared/components'], tasks: function (changedfiles) { var tasks = []; changedfiles.foreach(function (file) { if (path.extname(file) === '.jsx' && !~tasks.indexof('bundlejs')) tasks.push('bundlejs'); if (path.extname(file) === '.less' && !~tasks.indexof('compileless')) tasks.push('compileless'); }); return tasks; }
Read more

Fatal Python error: Py_Initialize: unable to load the file system codec. ImportError: No module named 'encodings' -

-
i trying make simple python program opening list of webpages user manually download reports site. don't have previous experience preparing exe files.. , i'm in learning process python coding. of done on windows 7 x64 this python code: #!c:/python34/python.exe -u splinter import * import time import os import csv #---------------------------------- raporty = [] open('../raporty.csv', newline='') csvfile: contents = csv.reader(csvfile, delimiter=' ', quotechar='|') row in contents: r = ', '.join(row) r = r.replace(',','') raporty.append(r) #--not implemented yet zmienne = [] open('../zmienne.csv', newline='') csvfile: contents = csv.reader(csvfile, delimiter=' ', quotechar='|') row in contents: r = ', '.join(row) r = r.replace(',','') zmienne.append(r) print("start") browser = browser(
Read more