Spring Batch - Need help in design approach -


we need merge set of table data 1 data source based on last run config date. had implemented spring batch , working fine performance slow. taken around 18 hours process around 5 million records. haven't used multi threading or partionin yet. need in finding right design approach increase performance. task done through sql loader , completed in 3 hours. have around 8 table merged datasource. please let me know if info needed. in advance.

spring batch designed allow incremental enhancement of batch jobs basic single threaded processing full blown multi-jvm scaled solutions minimal configuration changes each step. without knowing use case, approach take depend on requirements:

  1. do need restartability? if so, eliminates basic multi-threaded steps since readers not support multi-threaded processing restartability.
  2. is process io bound? i'm assuming eliminate remote chunking option.

if above assumptions correct, leaves partitioning. can read more partitioning vs chunking here: difference between spring batch remote chunking , remote partitioning.

once you've chosen partitioning model, other questions you'll need answer are:

  1. what partitioning strategy? partitioning sends descriptions of data processed master each slave. you'll need determine description consists of (in db, id ranges common option).
  2. local or remote? can throughput need single jvm using threads execute slaves or need more horse power? if so, you'll want @ remote partitioning.

Comments

Post a Comment

Popular posts from this blog

java - How to specify maven bin in eclipse maven plugin? -

i have eclipse maven plugin coming on centos. how can execute maven command on linx like: mvn clean package [abigail@localhost ~]$ mvn bash: mvn: command not found how should specify path executable of maven plugin in eclipse? or have install maven seperately？ thanks. eclipse (m2e) have bundled central parts of maven inside eclipse plugin, why not detected, , not usable. investigation find correct way bootstrap it, have create launcher file manually. if want run maven outside of eclipse, far better off standalone version of maven. mvn command bootstrap maven runtime system correctly, , secondly, able use advice concerning use of maven. maven binaries quite stable, wouldn't need upgrade upgrade eclipse. you can subsequently switch eclipse use external installation if prefer.
Read more

single sign on - Logging into Plone site with credentials passed through HTTP -

i using plone 4.3.3 , new plone development. here trying achieve: i have site going redirect plone site. in http url line redirects plone site pass username whereas plone site should automatically log user username in without asking credentials password. not care safety or encryption @ point. how achieve that? (i guess need somehow modify existing login view, how do that?) thanks. i had same problem , @ end i've developed own pas plugin. i've followed next steps: create pas plugin implement extractcredentials method following this example implement authenticatecredentials method following this example add plugin develop section in build.conf. start plone , open zope console. go acl_users folder , add plugin pas system there examples here can use templates.
Read more

php - Why does AJAX not process login form? -

i writing simple login script in php form. can't work. there wrong ajax. when click submit button doesn't anything. this login html form: <form id="login_form"> <input type="text" class="input_field" name="username" id="username" value="gebruikersnaam" onclick="if(value=='gebruikersnaam'){ this.value='' }"/> <input type="password" class="input_field" name="password" id="password"/> <input type="submit" class="login_button" id="login_button" value="ga door"/> </form> this ajax: $(document).ready(function(){ $("#login_form").submit(function() { $.ajax({ type: "post", url: 'php/login_check.php', data: $("#login_form").serialize(), // serializes form...
Read more