scala - How to use s3 with Apache spark 2.2 in the Spark shell -


i'm trying load data amazon aws s3 bucket, while in spark shell.

i have consulted following resources:

parsing files amazon s3 apache spark

how access s3a:// files apache spark?

hortonworks spark 1.6 , s3

cloudera

custom s3 endpoints

i have downloaded , unzipped apache spark 2.2.0. in conf/spark-defaults have following (note replaced access-key , secret-key):

spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.s3afilesystem spark.hadoop.fs.s3a.access.key=access-key  spark.hadoop.fs.s3a.secret.key=secret-key

i have downloaded hadoop-aws-2.8.1.jar , aws-java-sdk-1.11.179.jar mvnrepository, , placed them in jars/ directory. start spark shell:

bin/spark-shell --jars jars/hadoop-aws-2.8.1.jar,jars/aws-java-sdk-1.11.179.jar

in shell, here how try load data s3 bucket:

val p = spark.read.textfile("s3a://sparkcookbook/person")

and here error results:

java.lang.noclassdeffounderror: org/apache/hadoop/fs/globalstoragestatistics$storagestatisticsprovider   @ java.lang.class.forname0(native method)   @ java.lang.class.forname(class.java:348)   @ org.apache.hadoop.conf.configuration.getclassbynameornull(configuration.java:2134)   @ org.apache.hadoop.conf.configuration.getclassbyname(configuration.java:2099)   @ org.apache.hadoop.conf.configuration.getclass(configuration.java:2193)   @ org.apache.hadoop.fs.filesystem.getfilesystemclass(filesystem.java:2654)   @ org.apache.hadoop.fs.filesystem.createfilesystem(filesystem.java:2667)   @ org.apache.hadoop.fs.filesystem.access$200(filesystem.java:94)   @ org.apache.hadoop.fs.filesystem$cache.getinternal(filesystem.java:2703)   @ org.apache.hadoop.fs.filesystem$cache.get(filesystem.java:2685)   @ org.apache.hadoop.fs.filesystem.get(filesystem.java:373)   @ org.apache.hadoop.fs.path.getfilesystem(path.java:295)

when instead try start spark shell follows:

bin/spark-shell --packages org.apache.hadoop:hadoop-aws:2.8.1

then 2 errors: 1 when interperter starts, , when try load data. here first:

:: problems summary :: :::: errors     unknown resolver null      unknown resolver null      unknown resolver null      unknown resolver null      unknown resolver null      unknown resolver null   :: use verbose or debug message level more details

and here second:

val p = spark.read.textfile("s3a://sparkcookbook/person") java.lang.illegalaccesserror: tried access method org.apache.hadoop.metrics2.lib.mutablecounterlong.<init>(lorg/apache/hadoop/metrics2/metricsinfo;j)v class org.apache.hadoop.fs.s3a.s3ainstrumentation   @ org.apache.hadoop.fs.s3a.s3ainstrumentation.streamcounter(s3ainstrumentation.java:195)   @ org.apache.hadoop.fs.s3a.s3ainstrumentation.streamcounter(s3ainstrumentation.java:216)   @ org.apache.hadoop.fs.s3a.s3ainstrumentation.<init>(s3ainstrumentation.java:139)   @ org.apache.hadoop.fs.s3a.s3afilesystem.initialize(s3afilesystem.java:174)   @ org.apache.hadoop.fs.filesystem.createfilesystem(filesystem.java:2669)   @ org.apache.hadoop.fs.filesystem.access$200(filesystem.java:94)   @ org.apache.hadoop.fs.filesystem$cache.getinternal(filesystem.java:2703)   @ org.apache.hadoop.fs.filesystem$cache.get(filesystem.java:2685)   @ org.apache.hadoop.fs.filesystem.get(filesystem.java:373)   @ org.apache.hadoop.fs.path.getfilesystem(path.java:295)   @ org.apache.spark.sql.execution.datasources.datasource.hasmetadata(datasource.scala:301)   @ org.apache.spark.sql.execution.datasources.datasource.resolverelation(datasource.scala:344)   @ org.apache.spark.sql.dataframereader.load(dataframereader.scala:152)   @ org.apache.spark.sql.dataframereader.text(dataframereader.scala:506)   @ org.apache.spark.sql.dataframereader.textfile(dataframereader.scala:542)   @ org.apache.spark.sql.dataframereader.textfile(dataframereader.scala:515)

could suggest how working? thanks.

if using apache spark 2.2.0, should use hadoop-aws-2.7.3.jar , aws-java-sdk-1.7.4.jar.

$ spark-shell --jars jars/hadoop-aws-2.7.3.jar,jars/aws-java-sdk-1.7.4.jar

after that, when try load data s3 bucket in shell, able so.


Comments

Post a Comment

Popular posts from this blog

Is there a better way to structure post methods in Class Based Views -

i'm using class based views in django , never specify url in ajax calls , omit action parameter in form because know these requests go through post method of class based view. notice view - profile page make many requests depending on user updating on page - post method become convoluted , if/else mess example: def post(self, request, *args, **kwargs): # handle user changing available date if request.post.get('availabledate') != none: self.updateavailabledate(request) return jsonresponse({'result':'success'}) if request.post.get('newprojectname') != none: creative_user = creativeuserprofile.objects.get(id = request.user.id) project = project.create_project(creative_user, request.post['newprojectname']) project.save() return jsonresponse({'projectid': project.id }) if request.post.get('projectimage') != none , request.files['file']: proj...
Read more

What is happening when Matlab is starting a "parallel pool"? -

running parallel cpu processes in matlab starts command parpool() according documentation , function: [creates] special job on pool of workers, , [connects] matlab client parallel pool. this function takes bit of time execute, on order of 30 seconds. in other multi-cpu paradigms openmp, parallel execution seems totally transparent -- i've never noticed behavior analogous matlab (granted i'm not experienced parallel programming). so, happening between time parpool() called , when finishes executing? takes long? parallel computing toolbox enables run matlab code in parallel using several different paradigms (e.g. jobs , tasks, parfor , spmd , parfeval , batch processing), , run either locally (parallelised across cores in local machine) or remotely (parallelised across machines in cluster - either 1 own, or 1 in cloud). in of these cases, code run on matlab workers , copies of matlab without interactive desktop. if you're intending run on remo...
Read more

php - Cannot override Laravel Spark authentication with own implementation -

during login process of laravel spark need request token of remote api. api stores exact same users , passwords laravel spark application. therefore need username , non hashed password of user during authentication process. i thought overriding authenticated method solution problem. in routes/web.php override post /login endpoint , pointing endpoint own logincontroller: <?php namespace app\http\controllers\auth; use illuminate\support\facades\log; use illuminate\http\request; use laravel\spark\http\controllers\auth\logincontroller sparklogincontroller; class logincontroller extends sparklogincontroller { /** * create new login controller instance. * * @return void */ public function __construct() { parent::__construct(); } /** * handle successful authentication attempt. * * @param request $request * @param \illuminate\contracts\auth\authenticatable $user * @return response */ public fu...
Read more