scala - How to use s3 with Apache spark 2.2 in the Spark shell -


i'm trying load data amazon aws s3 bucket, while in spark shell.

i have consulted following resources:

parsing files amazon s3 apache spark

how access s3a:// files apache spark?

hortonworks spark 1.6 , s3

cloudera

custom s3 endpoints

i have downloaded , unzipped apache spark 2.2.0. in conf/spark-defaults have following (note replaced access-key , secret-key):

spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.s3afilesystem spark.hadoop.fs.s3a.access.key=access-key  spark.hadoop.fs.s3a.secret.key=secret-key

i have downloaded hadoop-aws-2.8.1.jar , aws-java-sdk-1.11.179.jar mvnrepository, , placed them in jars/ directory. start spark shell:

bin/spark-shell --jars jars/hadoop-aws-2.8.1.jar,jars/aws-java-sdk-1.11.179.jar

in shell, here how try load data s3 bucket:

val p = spark.read.textfile("s3a://sparkcookbook/person")

and here error results:

java.lang.noclassdeffounderror: org/apache/hadoop/fs/globalstoragestatistics$storagestatisticsprovider   @ java.lang.class.forname0(native method)   @ java.lang.class.forname(class.java:348)   @ org.apache.hadoop.conf.configuration.getclassbynameornull(configuration.java:2134)   @ org.apache.hadoop.conf.configuration.getclassbyname(configuration.java:2099)   @ org.apache.hadoop.conf.configuration.getclass(configuration.java:2193)   @ org.apache.hadoop.fs.filesystem.getfilesystemclass(filesystem.java:2654)   @ org.apache.hadoop.fs.filesystem.createfilesystem(filesystem.java:2667)   @ org.apache.hadoop.fs.filesystem.access$200(filesystem.java:94)   @ org.apache.hadoop.fs.filesystem$cache.getinternal(filesystem.java:2703)   @ org.apache.hadoop.fs.filesystem$cache.get(filesystem.java:2685)   @ org.apache.hadoop.fs.filesystem.get(filesystem.java:373)   @ org.apache.hadoop.fs.path.getfilesystem(path.java:295)

when instead try start spark shell follows:

bin/spark-shell --packages org.apache.hadoop:hadoop-aws:2.8.1

then 2 errors: 1 when interperter starts, , when try load data. here first:

:: problems summary :: :::: errors     unknown resolver null      unknown resolver null      unknown resolver null      unknown resolver null      unknown resolver null      unknown resolver null   :: use verbose or debug message level more details

and here second:

val p = spark.read.textfile("s3a://sparkcookbook/person") java.lang.illegalaccesserror: tried access method org.apache.hadoop.metrics2.lib.mutablecounterlong.<init>(lorg/apache/hadoop/metrics2/metricsinfo;j)v class org.apache.hadoop.fs.s3a.s3ainstrumentation   @ org.apache.hadoop.fs.s3a.s3ainstrumentation.streamcounter(s3ainstrumentation.java:195)   @ org.apache.hadoop.fs.s3a.s3ainstrumentation.streamcounter(s3ainstrumentation.java:216)   @ org.apache.hadoop.fs.s3a.s3ainstrumentation.<init>(s3ainstrumentation.java:139)   @ org.apache.hadoop.fs.s3a.s3afilesystem.initialize(s3afilesystem.java:174)   @ org.apache.hadoop.fs.filesystem.createfilesystem(filesystem.java:2669)   @ org.apache.hadoop.fs.filesystem.access$200(filesystem.java:94)   @ org.apache.hadoop.fs.filesystem$cache.getinternal(filesystem.java:2703)   @ org.apache.hadoop.fs.filesystem$cache.get(filesystem.java:2685)   @ org.apache.hadoop.fs.filesystem.get(filesystem.java:373)   @ org.apache.hadoop.fs.path.getfilesystem(path.java:295)   @ org.apache.spark.sql.execution.datasources.datasource.hasmetadata(datasource.scala:301)   @ org.apache.spark.sql.execution.datasources.datasource.resolverelation(datasource.scala:344)   @ org.apache.spark.sql.dataframereader.load(dataframereader.scala:152)   @ org.apache.spark.sql.dataframereader.text(dataframereader.scala:506)   @ org.apache.spark.sql.dataframereader.textfile(dataframereader.scala:542)   @ org.apache.spark.sql.dataframereader.textfile(dataframereader.scala:515)

could suggest how working? thanks.

if using apache spark 2.2.0, should use hadoop-aws-2.7.3.jar , aws-java-sdk-1.7.4.jar.

$ spark-shell --jars jars/hadoop-aws-2.7.3.jar,jars/aws-java-sdk-1.7.4.jar

after that, when try load data s3 bucket in shell, able so.


Comments

Post a Comment

Popular posts from this blog

Is there a better way to structure post methods in Class Based Views -

i'm using class based views in django , never specify url in ajax calls , omit action parameter in form because know these requests go through post method of class based view. notice view - profile page make many requests depending on user updating on page - post method become convoluted , if/else mess example: def post(self, request, *args, **kwargs): # handle user changing available date if request.post.get('availabledate') != none: self.updateavailabledate(request) return jsonresponse({'result':'success'}) if request.post.get('newprojectname') != none: creative_user = creativeuserprofile.objects.get(id = request.user.id) project = project.create_project(creative_user, request.post['newprojectname']) project.save() return jsonresponse({'projectid': project.id }) if request.post.get('projectimage') != none , request.files['file']: proj...
Read more

performance - Why is XCHG reg, reg a 3 micro-op instruction on modern Intel architectures? -

i'm doing micro-optimization on performance critical part of code , came across sequence of instructions (in at&t syntax): add %rax, %rbx mov %rdx, %rax mov %rbx, %rdx i thought had use case xchg allow me shave instruction , write: add %rbx, %rax xchg %rax, %rdx however, dimay found agner fog's instruction tables , xchg 3 micro-op instruction 2 cycle latency on sandy bridge, ivy bridge, broadwell, haswell , skylake. 3 whole micro-ops , 2 cycles of latency! 3 micro-ops throws off 4-1-1-1 cadence , 2 cycle latency makes worse original in best case since last 2 instructions in original might execute in parallel. now... cpu might breaking instruction micro-ops equivalent to: mov %rax, %tmp mov %rdx, %rax mov %tmp, %rdx where tmp anonymous internal register , suppose last 2 micro-ops run in parallel latency 2 cycles. given register renaming occurs on these micro-architectures, though, doesn't make sense me done way. why wouldn't register renamer ...
Read more

c# - Asp.net web api : redirect unauthorized requst to forbidden page -

im trying redirect unauthorized request forbidden page instead i'm getting forbidden page in response body , how can fix ? here's startup class : app.createperowincontext(storecontext.create); app.createperowincontext<applicationusermanager>(applicationusermanager.create); app.createperowincontext<applicationsigninmanager>(applicationsigninmanager.create); app.usecookieauthentication(new cookieauthenticationoptions { authenticationtype = defaultauthenticationtypes.applicationcookie, expiretimespan = timespan.fromdays(30), }); app.useexternalsignincookie(defaultauthenticationtypes.externalcookie); app.usetwofactorsignincookie(defaultauthenticationtypes.twofactorcookie, timespan.fromminutes(5)); app.usetwofactorrememberbrowsercookie(defaultauthenticationtypes.twofactorrememberbrowsercookie); the method im trying reach : [httpget] [authorize(roles = "admin")] public string getcurrentusername() { return userman...
Read more