java - Spring Batch asynchronous processor configuration for best performance -


i have problem creating asynchronous processor in spring batch. processor getting id reader , creating object based on response soap call. 1 input (id) there must e.g. 60-100 soap calls , 1. tried make multithreaded step processing e.g 50 inputs @ time useless because 49 threads done job in 1 second , blocked, waiting 1 doing 60-100 soap calls. use asyncitemprocessor+asyncitemwriter solution works me. input (ids) large, around 25k items read db start ~50-100 inputs @ time.

here configuration:

@configuration public class batchconfig {      @autowired     public jobbuilderfactory jobbuilderfactory;     @autowired     public stepbuilderfactory stepbuilderfactory;     @autowired     private databaseconfig databaseconfig;     @value(value = "classpath:categories.txt")     private resource categories;      @bean     public job processjob() throws exception {         return jobbuilderfactory.get("processjob").incrementer(new runidincrementer()).listener(listener()).flow(orderstep1()).end().build();     }      @bean     public step orderstep1() throws exception {         return stepbuilderfactory.get("orderstep1").<category, categorydailyresult>chunk(1).reader(reader()).processor(asyncitemprocessor()).writer(asyncitemwriter()).taskexecutor(taskexecutor()).build();     }      @bean     public jobexecutionlistener listener() {         return new jobcompletionlistener();     }       @bean     public itemwriter asyncitemwriter() {         asyncitemwriter<categorydailyresult> asyncitemwriter = new asyncitemwriter<>();         asyncitemwriter.setdelegate(itemwriter());         return asyncitemwriter;     }      @bean     public itemwriter<categorydailyresult> itemwriter(){         return new writer();     }      @bean     public itemprocessor asyncitemprocessor() {         asyncitemprocessor<category, categorydailyresult> asyncitemprocessor = new asyncitemprocessor<>();         asyncitemprocessor.setdelegate(itemprocessor());         asyncitemprocessor.settaskexecutor(taskexecutor());         return asyncitemprocessor;     }      @bean     public itemprocessor<category, categorydailyresult> itemprocessor(){         return new processor();     }      @bean     public taskexecutor taskexecutor(){         simpleasynctaskexecutor taskexecutor = new simpleasynctaskexecutor();         taskexecutor.setconcurrencylimit(50);         return taskexecutor;     }      @bean(destroymethod = "")     public itemreader<category> reader() throws exception {         string query = "select c category c not exists elements(c.children)";          jpapagingitemreader<category> reader = new jpapagingitemreader<>();         reader.setsavestate(false);         reader.setquerystring(query);         reader.setentitymanagerfactory(databaseconfig.entitymanagerfactory().getobject());         reader.setpagesize(1);          return reader;     } }

how can boost application? maybe doing wrong? feedback welcome ;)

@edit: input of ids: 1 100 want e.g 50 threads executing processor. want them not block each other: thread1 process input "1" 2 minutes , @ time want thread2 process input "2", "8", "64" small , execute in few seconds.

@edit2: my goal: have 25k ids in database, read them jpapagingitemreader , every id processed processor. each item independent of each other. each id make soap call 0-100 times in loop , create object pass writer , save in database. how can obtain best performance such task?


Comments

Post a Comment

Popular posts from this blog

Is there a better way to structure post methods in Class Based Views -

i'm using class based views in django , never specify url in ajax calls , omit action parameter in form because know these requests go through post method of class based view. notice view - profile page make many requests depending on user updating on page - post method become convoluted , if/else mess example: def post(self, request, *args, **kwargs): # handle user changing available date if request.post.get('availabledate') != none: self.updateavailabledate(request) return jsonresponse({'result':'success'}) if request.post.get('newprojectname') != none: creative_user = creativeuserprofile.objects.get(id = request.user.id) project = project.create_project(creative_user, request.post['newprojectname']) project.save() return jsonresponse({'projectid': project.id }) if request.post.get('projectimage') != none , request.files['file']: proj...
Read more

performance - Why is XCHG reg, reg a 3 micro-op instruction on modern Intel architectures? -

i'm doing micro-optimization on performance critical part of code , came across sequence of instructions (in at&t syntax): add %rax, %rbx mov %rdx, %rax mov %rbx, %rdx i thought had use case xchg allow me shave instruction , write: add %rbx, %rax xchg %rax, %rdx however, dimay found agner fog's instruction tables , xchg 3 micro-op instruction 2 cycle latency on sandy bridge, ivy bridge, broadwell, haswell , skylake. 3 whole micro-ops , 2 cycles of latency! 3 micro-ops throws off 4-1-1-1 cadence , 2 cycle latency makes worse original in best case since last 2 instructions in original might execute in parallel. now... cpu might breaking instruction micro-ops equivalent to: mov %rax, %tmp mov %rdx, %rax mov %tmp, %rdx where tmp anonymous internal register , suppose last 2 micro-ops run in parallel latency 2 cycles. given register renaming occurs on these micro-architectures, though, doesn't make sense me done way. why wouldn't register renamer ...
Read more

c# - Asp.net web api : redirect unauthorized requst to forbidden page -

im trying redirect unauthorized request forbidden page instead i'm getting forbidden page in response body , how can fix ? here's startup class : app.createperowincontext(storecontext.create); app.createperowincontext<applicationusermanager>(applicationusermanager.create); app.createperowincontext<applicationsigninmanager>(applicationsigninmanager.create); app.usecookieauthentication(new cookieauthenticationoptions { authenticationtype = defaultauthenticationtypes.applicationcookie, expiretimespan = timespan.fromdays(30), }); app.useexternalsignincookie(defaultauthenticationtypes.externalcookie); app.usetwofactorsignincookie(defaultauthenticationtypes.twofactorcookie, timespan.fromminutes(5)); app.usetwofactorrememberbrowsercookie(defaultauthenticationtypes.twofactorrememberbrowsercookie); the method im trying reach : [httpget] [authorize(roles = "admin")] public string getcurrentusername() { return userman...
Read more