How to find spark master URL on Amazon EMR -

-

i new spark , trying install spark on amazon cluster version 1.3.1. when 

sparkconf sparkconfig = new sparkconf().setappname("sparksqltest").setmaster("local[2]");

it work me , came know testing purpose can set local[2]

when tried use cluster mode changed 

sparkconf sparkconfig = new sparkconf().setappname("sparksqltest").setmaster("spark://localhost:7077");

with getting below error

tried associate unreachable remote address [akka.tcp://sparkmaster@localhost:7077]. address gated 5000 ms, messages address delivered dead letters. reason: connection refused 15/06/10 15:22:21 info client.appclient$clientactor: connecting master akka.tcp://sparkmaster@localhost:7077/user/master..

could please let me how set master url.

if using bootstrap action https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark configuration setup spark on yarn. set master yarn-client or yarn-cluster. sure define number of executors memory , cores. more details spark on yarn @ https://spark.apache.org/docs/latest/running-on-yarn.html

addition regarding executor settings memory , core sizing:

take @ default yarn node manager configs each type @ http://docs.aws.amazon.com/elasticmapreduce/latest/developerguide/taskconfiguration_h2.html, yarn.scheduler.maximum-allocation-mb. can determine number of cores basic ec2 info url (http://aws.amazon.com/ec2/instance-types/). max size of executor memory has fit within max allocation less spark's overhead , in increments of 256mb. description of calculation @ http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/. don't forget little on half executor memory can used rdd cache.


Comments

Post a Comment

Popular posts from this blog

javascript - gulp-nodemon - nodejs restart after file change - Error: listen EADDRINUSE events.js:85 -

-
i'm running nodejs 0.12.3 - gulp 3.10.8 , gulp-nodemon 2.0.3. gulpfile.js : gulp.task('serve', ['build'], function() { gulp.start('fb-flo'); nodemon({ script: paths.server, env: { 'node_env': 'development' }, // env: { 'node_env': 'production' }, execmap: { 'js': 'node_modules/babel/bin/babel-node --stage 1' //es7 cote server }, delay: '0ms', watch: ['src/shared/', 'src/server/'], ignore: ['src/shared/components'], tasks: function (changedfiles) { var tasks = []; changedfiles.foreach(function (file) { if (path.extname(file) === '.jsx' && !~tasks.indexof('bundlejs')) tasks.push('bundlejs'); if (path.extname(file) === '.less' && !~tasks.indexof('compileless')) tasks.push('compileless'); }); return tasks; } ...
Read more

Fatal Python error: Py_Initialize: unable to load the file system codec. ImportError: No module named 'encodings' -

-
i trying make simple python program opening list of webpages user manually download reports site. don't have previous experience preparing exe files.. , i'm in learning process python coding. of done on windows 7 x64 this python code: #!c:/python34/python.exe -u splinter import * import time import os import csv #---------------------------------- raporty = [] open('../raporty.csv', newline='') csvfile: contents = csv.reader(csvfile, delimiter=' ', quotechar='|') row in contents: r = ', '.join(row) r = r.replace(',','') raporty.append(r) #--not implemented yet zmienne = [] open('../zmienne.csv', newline='') csvfile: contents = csv.reader(csvfile, delimiter=' ', quotechar='|') row in contents: r = ', '.join(row) r = r.replace(',','') zmienne.append(r) print("start") browser = browser(...
Read more

javascript - oscilloscope of speaker input stops rendering after a few seconds -

-
the following script reads audio user's microphone , renders oscilloscope on html canvas. the source taken example of mozilla developer network: visualizations web audio api and here fiddle: http://jsfiddle.net/b7j8pktp/ mozgetusermedia (note: code has no fork mechanism different browsers: works firefox) it works fine few seconds , stops rendering. whereas works totally stable: http://mdn.github.io/voice-change-o-matic/ the problem can reduced following code. microphone activation icon (next the address bar in firefox) disappears after 5 seconds: navigator.mozgetusermedia({audio: true}, function() {}, function() {} ); ( http://jsfiddle.net/b7j8pktp/2/ ) this known bug in firefox. take stream getusermedia call , hook window so: navigator.mozgetusermedia({audio: true}, function(stream) { window.stream = stream; // rest of code }, function err() { // handle error }); hopefully can fixed soon. problem we're f...
Read more