amazon web services - Error with Data Pipeline backup when I transfer my data from DynamoDb to S3 -
i have backup dynamodb table s3 when launch service receive error after 3 attempts:
private.com.amazonaws.amazonserviceexception: user: arn:aws:sts::769870455028:assumed-role/datapipelinedefaultresourcerole/i-3678d99c not authorized perform: elasticmapreduce:modifyinstancegroups (service: amazonelasticmapreduce; status code: 400; error code: accessdeniedexception; request id: 9065ea77-0f95-11e5-8f35-39a70915a1ef) @ private.com.amazonaws.http.amazonhttpclient.handleerrorresponse(amazonhttpclient.java:1077) @ private.com.amazonaws.http.amazonhttpclient.executeonerequest(amazonhttpclient.java:725) @ private.com.amazonaws.http.amazonhttpclient.executehelper(amazonhttpclient.java:460) @ private.com.amazonaws.http.amazonhttpclient.execute(amazonhttpclient.java:295) @ private.com.amazonaws.services.elasticmapreduce.amazonelasticmapreduceclient.invoke(amazonelasticmapreduceclient.java:1391) @ private.com.amazonaws.services.elasticmapreduce.amazonelasticmapreduceclient.modifyinstancegroups(amazonelasticmapreduceclient.java:785) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) @ java.lang.reflect.method.invoke(method.java:606) @ private.com.amazonaws.services.datapipeline.retrier.retryproxy.invokeinternal(retryproxy.java:36) @ private.com.amazonaws.services.datapipeline.retrier.retryproxy.invoke(retryproxy.java:48) @ com.sun.proxy.$proxy33.modifyinstancegroups(unknown source) @ amazonaws.datapipeline.cluster.emrutil.acquirecorenodes(emrutil.java:325) @ amazonaws.datapipeline.activity.abstractclusteractivity.resizeifrequired(abstractclusteractivity.java:47) @ amazonaws.datapipeline.activity.abstracthiveactivity.runactivity(abstracthiveactivity.java:113) @ amazonaws.datapipeline.objects.abstractactivity.run(abstractactivity.java:16) @ amazonaws.datapipeline.taskrunner.taskpoller.executeremoterunner(taskpoller.java:132) @ amazonaws.datapipeline.taskrunner.taskpoller.executetask(taskpoller.java:101) @ amazonaws.datapipeline.taskrunner.taskpoller$1.run(taskpoller.java:77) @ private.com.amazonaws.services.datapipeline.poller.pollworker.executework(pollworker.java:76) @ private.com.amazonaws.services.datapipeline.poller.pollworker.run(pollworker.java:53) @ java.lang.thread.run(thread.java:745)
how can backup? have error?
edit: new policy
{ "version": "2012-10-17", "statement": [ { "effect": "allow", "action": [ "s3:", "dynamodb:", "ec2:describe*", "elasticmapreduce:describe*", "elasticmapreduce:listinstance*", "elasticmapreduce:addjobflowsteps", "elasticmapreduce:", "rds:describe", "datapipeline:", "cloudwatch:", "redshift:describeclusters", "redshift:describeclustersecuritygroups", "sdb:", "sns:", "sqs:" ], "resource": [ "" ] } ]
this new exception :
error during job, obtaining debugging information... examining task id: task_1434014832347_0001_m_000008 (and more) job job_1434014832347_0001 examining task id: task_1434014832347_0001_m_000013 (and more) job job_1434014832347_0001 examining task id: task_1434014832347_0001_m_000005 (and more) job job_1434014832347_0001 examining task id: task_1434014832347_0001_m_000034 (and more) job job_1434014832347_0001 examining task id: task_1434014832347_0001_m_000044 (and more) job job_1434014832347_0001 examining task id: task_1434014832347_0001_m_000004 (and more) job job_1434014832347_0001 task failures(4): ----- task id: task_1434014832347_0001_m_000002 url: http://ip-10-37-138-149.eu-west-1.compute.internal:9026/taskdetails.jsp?jobid=job_1434014832347_0001&tipid=task_1434014832347_0001_m_000002 ----- diagnostic messages task: error: java heap space failed: execution error, return code 2 org.apache.hadoop.hive.ql.exec.mr.mapredtask mapreduce jobs
datapipeline agent (taskrunner) running on emr cluster trying resize emr cluster , failing. resource role passed emr cluster not have permissions invoke following api amazonelasticmapreduce::modifyinstancegroups.
i looked @ defaultresourcerolepolicy, created using wizard in console, (http://docs.aws.amazon.com/datapipeline/latest/developerguide/dp-iam-roles.html ) these allowed policies emr: "elasticmapreduce:describe*", "elasticmapreduce:listinstance*", "elasticmapreduce:addjobflowsteps"
and found not allow modifyinstancegroups.
please update resource role policy allow that. e.g.,"elasticmapreduce:*"
thx reporting bug. in meanwhile, work on fixing console wizard generated default resource role policy.
aravind r.
Comments
Post a Comment