scala - How can I convert an arbitrary number of columns in a Spark dataframe from Timestamps to Longs? -


i'm writing in scala , using spark 1.6, , don't have option switch newer version. i'm attempting merge 2 dataframes, 1 pulled in avro files on hadoop cluster , 1 pulled in teradata db. can read them both in fine, , both guaranteed have same column names in same order, when try merge them using 

 data1.unionall(data2)

i hit error because avro converts timestamps longs, datatypes of 2 don't match fields. process repeated several times , know there @ least 1 timestamp field in tables, there can possibly more , not know names, i'm attempting make general method convert arbitrary number of columns timestamp longs. have far:

def transformtimestamps(df: dataframe): dataframe = {     val convert_timestamp_udf = udf((time:timestamp) => time.gettime())     df.dtypes.foreach { f =>          val fname = f._1         val ftype = f._2         if (ftype == "timestamptype:) {             println("found timestamp col: " + fname)             df.withcolumn(fname, convert_timestamp_udf(df.col(fname)))             df.printschema()         }     }    return df }

with printouts can tell method correctly recognizing timestamp columns, .withcolumn transformation not working. printing schema in next line not show updated column. additionally, i've tried creating entirely new column transformed values, , not added df either. can spot why isn't working?

the following line transformation 

df.withcolumn(fname, convert_timestamp_udf(df.col(fname)))

which not reflected on original dataframe until action performed. assignment work action can create temporary dataframe , assign in loop 

   def transformtimestamps(df: dataframe): dataframe = {       val convert_timestamp_udf = udf((time:timestamp) => time.gettime())       var tempdf = df       df.schema.map(f => {         val fname = f.name         val ftype = f.datatype         if (ftype.tostring == "timestamptype") {           println("found timestamp col: " + fname)           tempdf = tempdf.withcolumn(fname, convert_timestamp_udf(df.col(fname)))           tempdf.printschema()         }       })       return tempdf     }

i hope answer helpful


Comments

Post a Comment

Popular posts from this blog

Is there a better way to structure post methods in Class Based Views -

i'm using class based views in django , never specify url in ajax calls , omit action parameter in form because know these requests go through post method of class based view. notice view - profile page make many requests depending on user updating on page - post method become convoluted , if/else mess example: def post(self, request, *args, **kwargs): # handle user changing available date if request.post.get('availabledate') != none: self.updateavailabledate(request) return jsonresponse({'result':'success'}) if request.post.get('newprojectname') != none: creative_user = creativeuserprofile.objects.get(id = request.user.id) project = project.create_project(creative_user, request.post['newprojectname']) project.save() return jsonresponse({'projectid': project.id }) if request.post.get('projectimage') != none , request.files['file']: proj...
Read more

performance - Why is XCHG reg, reg a 3 micro-op instruction on modern Intel architectures? -

i'm doing micro-optimization on performance critical part of code , came across sequence of instructions (in at&t syntax): add %rax, %rbx mov %rdx, %rax mov %rbx, %rdx i thought had use case xchg allow me shave instruction , write: add %rbx, %rax xchg %rax, %rdx however, dimay found agner fog's instruction tables , xchg 3 micro-op instruction 2 cycle latency on sandy bridge, ivy bridge, broadwell, haswell , skylake. 3 whole micro-ops , 2 cycles of latency! 3 micro-ops throws off 4-1-1-1 cadence , 2 cycle latency makes worse original in best case since last 2 instructions in original might execute in parallel. now... cpu might breaking instruction micro-ops equivalent to: mov %rax, %tmp mov %rdx, %rax mov %tmp, %rdx where tmp anonymous internal register , suppose last 2 micro-ops run in parallel latency 2 cycles. given register renaming occurs on these micro-architectures, though, doesn't make sense me done way. why wouldn't register renamer ...
Read more

c# - Asp.net web api : redirect unauthorized requst to forbidden page -

im trying redirect unauthorized request forbidden page instead i'm getting forbidden page in response body , how can fix ? here's startup class : app.createperowincontext(storecontext.create); app.createperowincontext<applicationusermanager>(applicationusermanager.create); app.createperowincontext<applicationsigninmanager>(applicationsigninmanager.create); app.usecookieauthentication(new cookieauthenticationoptions { authenticationtype = defaultauthenticationtypes.applicationcookie, expiretimespan = timespan.fromdays(30), }); app.useexternalsignincookie(defaultauthenticationtypes.externalcookie); app.usetwofactorsignincookie(defaultauthenticationtypes.twofactorcookie, timespan.fromminutes(5)); app.usetwofactorrememberbrowsercookie(defaultauthenticationtypes.twofactorrememberbrowsercookie); the method im trying reach : [httpget] [authorize(roles = "admin")] public string getcurrentusername() { return userman...
Read more