hadoop - Scala Spark-Shell ArrayIndexOutOfBoundsException: 7 when inserting into table using sqlContext -


here's full error:

error tasksetmanager: task 55 in stage 7.0 failed 4 times; aborting job org.apache.spark.sparkexception: job aborted due stage failure: task 55 in  stage 7.0 failed 4 times, recent failure: lost task 55.3 in stage 7.0  (tid 17, <url>): java.lang.arrayindexoutofboundsexception: 7

i'm getting error when insert table dataframe created extracting table.

here's code (some code divided multiple lines readability):

sqlcontext.sql("drop table if exists db.final_scala")  val createtable = "create table if not exists db.final_scala (                        key1 string,                        key2 string,                        col3 string,                        col4 string,                        col5 string,                        col6 string,                        col7 string,                        col8 string,                        col9 string                    )"  sqlcontext.sql(createtable)  val insertintotable= "insert table db.final_scala      select         key1,         key2,          coalesce(colx, coly) col3,          concat(date,' 04:01:00.000') col4,          'text' col5,          'text' col6,          col7,          col8,          col9     (          select              *,              row_number() on (                 partition key2                 order from_unixtime(unix_timestamp(date, 'yyyy-mm-dd'))             desc) rnk          db2.temp                               cola= 'a'                  , (colb null or colb = 'c')                  , date >= '2017-08-09'                  , date < '2017-08-10'                  , key2 not null                  , (col7 not null , col7 > 0)          ) rnk = 1"  sqlcontext.sql(insertintotable)

when run query in hive directly, same exact query executes, runs, , populates table fine. example output:

101     1011 null    2017-08-05 04:01:00.000 text   text    70000   0       2017-08-09 08:52:15 102     1022 null    2017-08-06 04:01:00.000 text   text    52000   0       2017-08-09 08:52:15 103     1033 null    2017-08-05 04:01:00.000 text   text    1200000 0       2017-08-09 08:52:15 104     1044 null    2017-08-06 04:01:00.000 text   text    57000   0       2017-08-09 08:52:15 105     1055 null    2017-06-17 04:01:00.000 text   text    28080   0       2017-08-09 08:52:15

what issue, , how can resolve it?


Comments

Post a Comment

Popular posts from this blog

What is happening when Matlab is starting a "parallel pool"? -

running parallel cpu processes in matlab starts command parpool() according documentation , function: [creates] special job on pool of workers, , [connects] matlab client parallel pool. this function takes bit of time execute, on order of 30 seconds. in other multi-cpu paradigms openmp, parallel execution seems totally transparent -- i've never noticed behavior analogous matlab (granted i'm not experienced parallel programming). so, happening between time parpool() called , when finishes executing? takes long? parallel computing toolbox enables run matlab code in parallel using several different paradigms (e.g. jobs , tasks, parfor , spmd , parfeval , batch processing), , run either locally (parallelised across cores in local machine) or remotely (parallelised across machines in cluster - either 1 own, or 1 in cloud). in of these cases, code run on matlab workers , copies of matlab without interactive desktop. if you're intending run on remo...
Read more

angular - DownloadURL return null in below code -

when change file updated in firebase storage when try downloadurl show null. filechange(event: any) { const imagefolder: string = this.employee.id; const filelist: filelist = event.target.files; const file: file = filelist[0]; const storageref = firebase.storage().ref().child(`${imagefolder}/profile.jpg`).put(file); console.log(storageref.snapshot.downloadurl); } the put() asynchronous. if want url after file uploaded have this: firebase.storage().ref().child(`${imagefolder}/profile.jpg`).put(file).then((snapshot) => { storageref = snapshot.downloadurl: console.log(snapshot.downloadurl); });
Read more

php - Cannot override Laravel Spark authentication with own implementation -

during login process of laravel spark need request token of remote api. api stores exact same users , passwords laravel spark application. therefore need username , non hashed password of user during authentication process. i thought overriding authenticated method solution problem. in routes/web.php override post /login endpoint , pointing endpoint own logincontroller: <?php namespace app\http\controllers\auth; use illuminate\support\facades\log; use illuminate\http\request; use laravel\spark\http\controllers\auth\logincontroller sparklogincontroller; class logincontroller extends sparklogincontroller { /** * create new login controller instance. * * @return void */ public function __construct() { parent::__construct(); } /** * handle successful authentication attempt. * * @param request $request * @param \illuminate\contracts\auth\authenticatable $user * @return response */ public fu...
Read more