c# - Performance Counters NextValue() Very Slow (1,000+ Counters) -

in our application, using windows performance counters store of our application metrics, later retrieved in web services.

i'm having issue amount of time takes read value counters. i've looked through rest of app , fine, performance wise, reading counters within loop (from list or array) takes painful amount of time.

example code:

// triggers read of counter's initial value (1000ms delay following calculated counters) counters.tolist().foreach(counter => counter.nextvalue()); 

in testing of loop above, list of 1,359 counters takes 20 seconds, , stopwatch in place, seems average time read counter value either 0-10ms, or 80-90ms. lot of them take 0ms, highest being 170ms, average non-zero being 80-90ms.

maybe being optimistic, have figured reading 1,000 numeric values should take few milliseconds. there lot more processing going on here i'm aware of?

i have loop later in logic gets second value calculated counters. makes doubly worse. :)

thanks!

---

update 1

i wrapped counter retrieval in stopwatch , i'm surprised results. reading simple property of .rawvalue still takes excessive amount of time. it's understanding counters work same, , retrieval should incredibly fast; strangely i'm seeing pattern counters networking categories take longer.

according http://joe.blog.freemansoft.com/2014/03/windows-performance-counters.html, performance performance counter service shouldn't consideration.

i've posted stopwatch results following pastebin: http://pastebin.com/raw.php?i=adjk2tru

my code follows:

stopwatch t; foreach (var c in counters) {     t = stopwatch.startnew();     var r = c.rawvalue;     debug.writeline(t.elapsedmilliseconds.tostring("000") + " - " + c.categoryname + ":" + c.countername + "(" + c.countertype + ") = " + r); } 

as can see in paste, lot of reads 0, there lot in 50-100ms range. don't understand how can be. surely 1 counter value should fast other, right?

here’s i've been able find out counters. please forgive grammar; extracted email sent out regarding problem.

• there 4-5 second processing time, on machine @ least (may better or worse on server, not sure), read instance names counter category. varies negligibly number of counters in category. if not using instance counters, can avoid this.
• we store of counters in single category, it’s inevitable category end thousands of counters, given our situation. in testing, more counters in category, worse performance. seems should make sense, performance of individual counter affected number of counters in memory, odd correlation, maybe:
  • with 8 total counters, read time 1-2ms per counter
  • with 256 total counters, read time 15-18ms per counter
  • with 512 total counters, read time 30ms per counter
  • with 3,584 total counters (reading counters), read time 200ms per counter
  • with 3,584 total counters in system (filtered down in memory, reading 512 counters), read time anywhere 50-90ms per counter. not sure why these slower previous batch of 512 counters.
  • i ran each of these tests few times using system.diagnostics.stopwatch time them.
• of importance note fact counters have read twice because many counters calculated on span of time , present average between start , end read times, these bad numbers made worse in real-world scenario.

given numbers above, on machine, 512 counters @ 50ms each on slower end, plus instance query, , second counter read, we’re looking @ 60 seconds per request. given we’re working 512 counters @ time. i’ve run full query against service on machine several times , request consistently completes in 60-65 seconds.

i not have assumed type of performance degradation of single counters based on number of other counters being assessed. in reading, windows performance monitor system supposed fast, , small collections is. it’s possible our use case not fit , may abusing system.

update

given have control on how create counters, have decided change our approach bit. instead of few categories many counters, instead create many categories, each having fewer counters (4-8 counters per category). approach has allowed avoid performance issue, , counter read times in 0-1ms range. in our experience far, having 100 new categories few counters each not affect performance in system @ all.

it's important note when dealing large number of additional counters, need address memory limitation set default performance counters. can done either via machine.config or registry entry. more information can found here: http://msdn.microsoft.com/en-us/library/ms229387(v=vs.110).aspx