database - What are the approaches to the Big-Data problems? -

-

let consider following problem. have system containing huge amount of data (big-data). so, in fact have data base. first requirement want able write , read data base quickly. want have web-interface data-bases (so different clients can write , read data base remotely).

but system want have should more data base. first, want able run different data-analysis algorithm on data find regularities, correlations, abnormalities , on (as before care lot performance). second, want bind machine learning machinery data-base. means want run machine learning algorithms on data able learn "relations" present on data , based on predict values of entries not yet in data base.

finally, want have nice clicks based interface visualize data. users can see data in form of nice graphics, graphs , other interactive visualisation objects.

what standard , recognised approaches above described problem. programming languages have used deal described problems?

i approach question this: assume firmly interested in big data database use , have real need one, instead of repeating textbooks upon textbooks of information them, highlight meet 5 requirements - cassandra , hadoop.

1) first requirement want able write , read database quickly.

you'll want explore nosql databases used storing “unstructured” big data. open-source databases include hadoop , cassandra. regarding cassandra,

facebook needed fast , cheap handle billions of status updates, started project , moved apache it's found plenty of support in many communities (ref).

references:

2) want have web interface database

see list of 150 nosql databases see various interfaces available, including web interfaces.

cassandra has cluster admin, web-based environment, web-admin based on angularjs, , gui clients.

references:

3) want able run different data-analysis algorithm on data

cassandra, hive, , hadoop well-suited data analytics. example, ebay uses cassandra managing time-series data.

references:

4) want run machine learning algorithms on data able learn "relations"

again, cassandra , hadoop well-suited. regarding apache spark + cassandra,

spark developed in 2009 @ uc berkeley amplab, open sourced in 2010, , became top-level apache project in february, 2014. has since become 1 of largest open source communities in big data, on 200 contributors in 50+ organizations (ref).

regarding hadoop,

with rapid adoption of apache hadoop, enterprises use machine learning key technology extract tangible business value massive data assets.

references:

5) finally, want have nice clicks-based interface visualize data.

visualization tools (paid) work above databases include pentaho, jasperreports, , datameer analytics solutions. alternatively, there several open-source interactive visualization tools such d3 , dygraphs (for big data sets).

references:


Comments

Post a Comment

Popular posts from this blog

javascript - gulp-nodemon - nodejs restart after file change - Error: listen EADDRINUSE events.js:85 -

-
i'm running nodejs 0.12.3 - gulp 3.10.8 , gulp-nodemon 2.0.3. gulpfile.js : gulp.task('serve', ['build'], function() { gulp.start('fb-flo'); nodemon({ script: paths.server, env: { 'node_env': 'development' }, // env: { 'node_env': 'production' }, execmap: { 'js': 'node_modules/babel/bin/babel-node --stage 1' //es7 cote server }, delay: '0ms', watch: ['src/shared/', 'src/server/'], ignore: ['src/shared/components'], tasks: function (changedfiles) { var tasks = []; changedfiles.foreach(function (file) { if (path.extname(file) === '.jsx' && !~tasks.indexof('bundlejs')) tasks.push('bundlejs'); if (path.extname(file) === '.less' && !~tasks.indexof('compileless')) tasks.push('compileless'); }); return tasks; } ...
Read more

Fatal Python error: Py_Initialize: unable to load the file system codec. ImportError: No module named 'encodings' -

-
i trying make simple python program opening list of webpages user manually download reports site. don't have previous experience preparing exe files.. , i'm in learning process python coding. of done on windows 7 x64 this python code: #!c:/python34/python.exe -u splinter import * import time import os import csv #---------------------------------- raporty = [] open('../raporty.csv', newline='') csvfile: contents = csv.reader(csvfile, delimiter=' ', quotechar='|') row in contents: r = ', '.join(row) r = r.replace(',','') raporty.append(r) #--not implemented yet zmienne = [] open('../zmienne.csv', newline='') csvfile: contents = csv.reader(csvfile, delimiter=' ', quotechar='|') row in contents: r = ', '.join(row) r = r.replace(',','') zmienne.append(r) print("start") browser = browser(...
Read more

javascript - oscilloscope of speaker input stops rendering after a few seconds -

-
the following script reads audio user's microphone , renders oscilloscope on html canvas. the source taken example of mozilla developer network: visualizations web audio api and here fiddle: http://jsfiddle.net/b7j8pktp/ mozgetusermedia (note: code has no fork mechanism different browsers: works firefox) it works fine few seconds , stops rendering. whereas works totally stable: http://mdn.github.io/voice-change-o-matic/ the problem can reduced following code. microphone activation icon (next the address bar in firefox) disappears after 5 seconds: navigator.mozgetusermedia({audio: true}, function() {}, function() {} ); ( http://jsfiddle.net/b7j8pktp/2/ ) this known bug in firefox. take stream getusermedia call , hook window so: navigator.mozgetusermedia({audio: true}, function(stream) { window.stream = stream; // rest of code }, function err() { // handle error }); hopefully can fixed soon. problem we're f...
Read more