apache spark - Unbagging a dataset in pyspark -


i have dataset looks this. 

(34521658, 0001-01-01, 2500-01-01, 2 , a, y, 15, p, a, 4776, 4776, 4776, {(11, p, a, 4776,4766, 4776), (12, p, a, 4776,4766, 4776), (13, p, a, 4776,4766, 4776)})

and want un-bag make 

(34521658, 0001-01-01, 2500-01-01, 2 , a, y, 15, p, a, 4776, 4776, 4776, 11, p, a, 4776,4766, 4776)     (34521658, 0001-01-01, 2500-01-01, 2 , a, y, 15, p, a, 4776, 4776, 4776, 12, p, a, 4776,4766, 4776) (34521658, 0001-01-01, 2500-01-01, 2 , a, y, 15, p, a, 4776, 4776, 4776, 13, p, a, 4776,4766, 4776)

.

how in pyspark?

as suggested in comment flatmap or explode can used. here how can using explode sql function (explode, name says, expand array or map column more rows) keep meaningful columns sake of simplifying approach. assuming first column id , columns want explode named bag, here how initial dataset

+--------+--------------------+ |      id|                 bag| +--------+--------------------+ |34521658|[[11,p,a,4776,476...| +--------+--------------------+

the schema dataset :

scala> df.printschema root  |-- id: integer (nullable = true)  |-- bag: array (nullable = true)  |    |-- element: struct (containsnull = true)  |    |    |-- _1: integer (nullable = true)  |    |    |-- _2: string (nullable = true)  |    |    |-- _3: string (nullable = true)  |    |    |-- _4: integer (nullable = true)  |    |    |-- _5: integer (nullable = true)  |    |    |-- _6: integer (nullable = true)

note bag column array of elements. on colum can apply explode function this:

df.withcolumn("bag", explode($"bag"))

the resulting dataset/dataframe is:

+--------+--------------------+ |      id|                 bag| +--------+--------------------+ |34521658|[11,p,a,4776,4766...| |34521658|[12,p,a,4776,4766...| |34521658|[13,p,a,4776,4766...| +--------+--------------------+

hope helps


Comments

Post a Comment

Popular posts from this blog

Is there a better way to structure post methods in Class Based Views -

i'm using class based views in django , never specify url in ajax calls , omit action parameter in form because know these requests go through post method of class based view. notice view - profile page make many requests depending on user updating on page - post method become convoluted , if/else mess example: def post(self, request, *args, **kwargs): # handle user changing available date if request.post.get('availabledate') != none: self.updateavailabledate(request) return jsonresponse({'result':'success'}) if request.post.get('newprojectname') != none: creative_user = creativeuserprofile.objects.get(id = request.user.id) project = project.create_project(creative_user, request.post['newprojectname']) project.save() return jsonresponse({'projectid': project.id }) if request.post.get('projectimage') != none , request.files['file']: proj...
Read more

reflection - How to access the object-members of an object declaration in kotlin -

say have following, nested object declaration: object father { val fathersfield = "value" object child { val childsfield = 3.141592654 } } when use reflection starting father , i'm able find field fathersfield no member referencing child instance. is possible find inner object declarations via reflection? , if so, how? use nestedclasses kotlin-reflect : father::class.nestedclasses.find { it.simplename == "child" } or, @s1m0nw1 suggested, use java reflection , convert class kclass .kotlin if needed: father::class.java.classes.first { it.simplename == "child" }.kotlin
Read more

php - Doctrine Query Builder Error on Join: [Syntax Error] line 0, col 87: Error: Expected Literal, got 'JOIN' -

i building shipping system ecommerce site using doctrine. have appropriate shipping methods , prices based on product , region data in checkout. i using following code querybuilder: $shippingpricereccords = $this->em->createquerybuilder() ->select('price') ->from('ordershippingprice', 'price') ->innerjoin('ordershippingmethod', 'method', 'price.fkordershippingmethod = method.id') ->innerjoin('ordershippingmethodregionmapping', 'map', 'map.fkordershippingmethod = method.id') ->where('price.fkproducttype = :fkproducttype') ->andwhere('price.fkbrand = :fkbrand') ->andwhere('map.fkregion = :fkregion') ->setparameters([ 'fkproducttype' => $fkproducttype, 'fkbrand' => $fkbrand, ...
Read more