r - Turn on all CPUs for all nodes on a cluster: snow/snowfall package -


i working on cluster , using snowfall package establish socket cluster on 5 nodes 40 cpus each following command:

 > sfinit(parallel=true, cpus = 200, type="sock", sockethosts=c("host1", "host2", "host3", "host4", "host5"));  r version:  r version 3.1.0 (2014-04-10)    snowfall 1.84-6 initialized (using snow 0.3-13): parallel execution on 5 cpus.

i seeing lower load on slaves expected when check cluster report , disconcerted fact says "parallel execution on 5 cpus" instead of "parallel execution on 200 cpus". merely ambiguous reference cpus or hosts running 1 cpu each?

edit: here example of why concerns me, if use local machine , specify max number of cores, have:

 > sfinit(parallel=true, type="sock", cpus = 40);  snowfall 1.84-6 initialized (using snow 0.3-13): parallel execution on 40 cpus.

i ran identical job on single node, 40 cpu cluster , took 1.4 minutes while 5 node, apparently 5 cpu cluster took 5.22 minutes. me confirms suspicions running parallelism on 5 nodes turning on 1 of cpus on each node.

my question then: how turn on cpus use across available nodes?

edit: @simong used underlying snow package's intialization , can see 5 nodes being turned on:

 > cl <- makesockcluster(names = c("host1", "host2", "host3", "host4", "host5"), count = 200)  > clustercall(cl, runif, 3)  [[1]]  [1] 0.9854311 0.5737885 0.8495582   [[2]]  [1] 0.7272693 0.3157248 0.6341732   [[3]]  [1] 0.26411931 0.36189866 0.05373248   [[4]]  [1] 0.3400387 0.7014877 0.6894910   [[5]]  [1] 0.2922941 0.6772769 0.7429913   > stopcluster(cl)  > cl <- makesockcluster(names = rep("localhost", 40), count = 40)  > clustercall(cl, runif, 3)  [[1]]  [1] 0.6914666 0.7273244 0.8925275   [[2]]  [1] 0.3844729 0.7743824 0.5392220   [[3]]  [1] 0.2989990 0.7256851 0.6390770        [[4]]  [1] 0.07114831 0.74290601 0.57995908   [[5]]  [1] 0.4813375 0.2626619 0.5164171   .  .  .   [[39]]  [1] 0.7912749 0.8831164 0.1374560   [[40]]  [1] 0.2738782 0.4100779 0.0310864

i think shows pretty clearly. tried in desperation:

 > cl <- makesockcluster(names = rep(c("host1", "host2", "host3", "host4", "host5"), each = 40), count = 200)

and predictably got:

 error in socketconnection(port = port, server = true, blocking = true,  :     connections in use

after thoroughly reading snow documentation, have come (partial) solution.

i read 128 connections may opened @ once distributed r version, , have found true. can open 25 cpus on each node, cluster not start if try start 26 on each. here proper structure of host list needs passed makecluster:

> library(snow);  > unixhost13 <- list(host = "host1"); > unixhost14 <- list(host = "host2"); > unixhost19 <- list(host = "host3"); > unixhost29 <- list(host = "host4"); > unixhost30 <- list(host = "host5");  > kcpus <- 25; > hostlist <- c(rep(list(unixhost13), kcpus), rep(list(unixhost14), kcpus),               rep(list(unixhost19), kcpus), rep(list(unixhost29), kcpus), rep(list(unixhost30), kcpus)); > cl <- makecluster(hostlist, type = "sock") > clustercall(cl, runif, 3) [[1]] [1] 0.08430941 0.64479036 0.90402362  [[2]] [1] 0.1821656 0.7689981 0.2001639  [[3]] [1] 0.5917363 0.4461787 0.8000013 . . . [[123]] [1] 0.6495153 0.6533647 0.2636664  [[124]] [1] 0.75175580 0.09854553 0.66568129  [[125]] [1] 0.79336203 0.61924813 0.09473841

i found reference saying in order connections, r needed rebuilt nconnections set higher (see here).


Comments

Post a Comment

Popular posts from this blog

javascript - Jquery show_hide, what to add in order to make the page scroll to the bottom of the hidden field once button is clicked -

i have button on website when clicked, reveals map. button can seen when page first loads up, when clicked, section holding map slides open , goes out of screen since page loads @ top (normal). want page scroll down bottom of div select when button pushed, if map out of screen. if page scrolled down, , clicks show map button, don't want page jump bottom of map. i tried figure out myself, no luck. seasoned jquery coders able give me guidance? many thanks! map = mapdiv.gmap3("get"); infobox = new infobox({ pixeloffset: new google.maps.size(-220, -310), closeboxurl: '', enableeventpropagation: true }); mapdiv.delegate('.infobox .close','click',function () { infobox.close(); }); jquery(".slidingdiv").hide(); jquery(".show_hide").show(); jquery('.show_hide').click(function(){ lastcenter=map.getcenter(); jquery(".slidingdiv&quo
Read more

javascript - Highcharts multi-color line -

Image
i looking create highchart multi-colored line. if goes down under value, become red, , when comes on value, become green . here picture example: or maybe highcharts have other charts models differentiate high or low level color? you can achieve coloring using lineargradient coloring: color: { lineargradient: { x1: 0, x2: 0, y1: 0, y2: 1 }, stops: [ [0, 'red'], [0.25, 'yellow'], [0.50, 'green'], [0.75, 'yellow'], [1, 'red'] ] } here 0 in stops relates minimum value of series , 1 maximum value, , in between percentages between values. if find percentages according values prior creating chart use correct stops values make relate specific values in data (at least approximately). see this jsfiddle example .
Read more

javascript - Enter key does not work in search box -

tried search guys sorry. input in search box doesnt work (or search) when hit enter when user clicks search button. doing wrong? here code. thanks. <script type="text/javascript"> $(document).ready(function() { $("#submit").click(function() { var url = "http://asla.org/awardssearch.html"; url += "?s=" + $('#googlecse').val(); window.location = url; }); $('#googlecse').keydown(function(event) { if (event.keycode == '13') { $("#submit").click(); } else { //do nothing :) } }); }); </script> <form class="navbar-form navbar-right" role="search"> <div class="search"> <input id="googlecse" type="text" onblur="if(this.value=='')this.value=this.defaultvalue;" onfocus="if(this.value==this.defaultvalue)t
Read more