java - Processing very large xml file with Hadoop MapReduce -


i new hadoop , want read data large xml file (8gb).as read many blogs in-order-to process xml file, have write custom xmlinputformat class. used on of custom xmlinputformat class in blog. after run hadoop job takes large amount of time complete job using 150mb xml file input. need pass large xml chunk @ once mapper. bellow custom inputformat class

public class xmlinputformat extends textinputformat {  public static final string start_tag_key = "xmlinput.start"; public static final string end_tag_key = "xmlinput.end";  public recordreader<longwritable, text> createrecordreader(inputsplit split, taskattemptcontext context) {     return new xmlrecordreader(); }  public static class xmlrecordreader extends         recordreader<longwritable, text> {     private byte[] starttag;     private byte[] endtag;     private long start;     private long end;     private fsdatainputstream fsin;     private dataoutputbuffer buffer = new dataoutputbuffer();      private longwritable key = new longwritable();     private text value = new text();      @override     public void initialize(inputsplit split, taskattemptcontext context)             throws ioexception, interruptedexception {         configuration conf = context.getconfiguration();         starttag = conf.get(start_tag_key).getbytes("utf-8");         endtag = conf.get(end_tag_key).getbytes("utf-8");         filesplit filesplit = (filesplit) split;          // open file , seek start of split         start = filesplit.getstart();         end = start + filesplit.getlength();         path file = filesplit.getpath();         filesystem fs = file.getfilesystem(conf);         fsin = fs.open(filesplit.getpath());         fsin.seek(start);         }      @override     public boolean nextkeyvalue() throws ioexception, interruptedexception {         if (fsin.getpos() < end) {             if (readuntilmatch(starttag, false)) {                 try {                     buffer.write(starttag);                     if (readuntilmatch(endtag, true)) {                          key.set(fsin.getpos());                         value.set(buffer.getdata(), 0, buffer.getlength());                         return true;                     }                  } {                     buffer.reset();                 }             }         }         return false;     }      @override     public longwritable getcurrentkey() throws ioexception,             interruptedexception {         return key;     }      @override     public text getcurrentvalue() throws ioexception, interruptedexception {         return value;     }      @override     public void close() throws ioexception {         fsin.close();     }      @override     public float getprogress() throws ioexception {         return math.min(1.0f, (fsin.getpos() - start) / (float)(end - start));     }          private boolean readuntilmatch(byte[] match, boolean withinblock) throws ioexception {      int = 0;     while (true) {         int noofbyteindicator = fsin.read();          // end of file:         if (noofbyteindicator == -1)             return false;          // save buffer:         if (withinblock)             buffer.write(noofbyteindicator);          string buffertoprint = new string(buffer.getdata());          // check if we're matching:         if (noofbyteindicator == match[i]) {             i++;             if (i >= match.length)                 return true;         } else             = 0;          // see if we've passed stop point:         if (!withinblock && == 0 && fsin.getpos() >= end)             return false;     }    }   }}

does have working example? searched found same custom inputformat class.as read file 200mb should processed less minute.

i adding mapper class in case of need.

public class xmlmapper extends mapper<longwritable, text, text, text> {  @override protected void map(longwritable key, text value, context context)         throws ioexception, interruptedexception {     map<string,string> mapvalue = null;     string document = value.tostring();     // system.out.println("‘" + document + "‘");     try {         xmlstreamreader reader = xmlinputfactory.newinstance()                 .createxmlstreamreader(                         new bytearrayinputstream(document.getbytes()));          string propertyid = "";         string currentelement = "";         string newvalue= "";          stringbuffer sb = new stringbuffer();              while (reader.hasnext()) {             int code = reader.next();             switch (code) {             case xmlstreamconstants.start_element:                 currentelement = reader.getlocalname();                 break;             case xmlstreamconstants.end_element:                 currentelement = "";                 break;             case xmlstreamconstants.characters:                 if (currentelement.equalsignorecase("id")) {                     propertyid = reader.gettext();                     sb.append(propertyid).append(",");                 } else if (currentelement.equalsignorecase("supplier")) {                     sb.append(reader.gettext()).append(",");                 } else if (currentelement.equalsignorecase("value")) {                     sb.append(reader.gettext()).append("\n");                 }                 break;             }         }         reader.close();          context.write(new text(propertyid), new text(sb.tostring()));      } catch (exception e) {         throw new ioexception(e);      }  }}

bellow sample xml need process.

<root> <row>   <id>0000000001</id>   <supplier>1111111111</supplier>   <value>tpk100</value> </row> <row>   <id>0000000002</id>   <supplier>1111111111</supplier>   <value>tpk101</value> </row> </root>

i need process data within each <row></row> tag


Comments

Post a Comment

Popular posts from this blog

Is there a better way to structure post methods in Class Based Views -

i'm using class based views in django , never specify url in ajax calls , omit action parameter in form because know these requests go through post method of class based view. notice view - profile page make many requests depending on user updating on page - post method become convoluted , if/else mess example: def post(self, request, *args, **kwargs): # handle user changing available date if request.post.get('availabledate') != none: self.updateavailabledate(request) return jsonresponse({'result':'success'}) if request.post.get('newprojectname') != none: creative_user = creativeuserprofile.objects.get(id = request.user.id) project = project.create_project(creative_user, request.post['newprojectname']) project.save() return jsonresponse({'projectid': project.id }) if request.post.get('projectimage') != none , request.files['file']: proj...
Read more

performance - Why is XCHG reg, reg a 3 micro-op instruction on modern Intel architectures? -

i'm doing micro-optimization on performance critical part of code , came across sequence of instructions (in at&t syntax): add %rax, %rbx mov %rdx, %rax mov %rbx, %rdx i thought had use case xchg allow me shave instruction , write: add %rbx, %rax xchg %rax, %rdx however, dimay found agner fog's instruction tables , xchg 3 micro-op instruction 2 cycle latency on sandy bridge, ivy bridge, broadwell, haswell , skylake. 3 whole micro-ops , 2 cycles of latency! 3 micro-ops throws off 4-1-1-1 cadence , 2 cycle latency makes worse original in best case since last 2 instructions in original might execute in parallel. now... cpu might breaking instruction micro-ops equivalent to: mov %rax, %tmp mov %rdx, %rax mov %tmp, %rdx where tmp anonymous internal register , suppose last 2 micro-ops run in parallel latency 2 cycles. given register renaming occurs on these micro-architectures, though, doesn't make sense me done way. why wouldn't register renamer ...
Read more

c# - Asp.net web api : redirect unauthorized requst to forbidden page -

im trying redirect unauthorized request forbidden page instead i'm getting forbidden page in response body , how can fix ? here's startup class : app.createperowincontext(storecontext.create); app.createperowincontext<applicationusermanager>(applicationusermanager.create); app.createperowincontext<applicationsigninmanager>(applicationsigninmanager.create); app.usecookieauthentication(new cookieauthenticationoptions { authenticationtype = defaultauthenticationtypes.applicationcookie, expiretimespan = timespan.fromdays(30), }); app.useexternalsignincookie(defaultauthenticationtypes.externalcookie); app.usetwofactorsignincookie(defaultauthenticationtypes.twofactorcookie, timespan.fromminutes(5)); app.usetwofactorrememberbrowsercookie(defaultauthenticationtypes.twofactorrememberbrowsercookie); the method im trying reach : [httpget] [authorize(roles = "admin")] public string getcurrentusername() { return userman...
Read more