reactivemongo - MongoDB is giving inconsistent write times -

i using scala, reactive mongo 0.10.5 , mongo 2.6.4 running on ubuntu. have tested on few machine configurations right working 15gb of memory, 2 cores , 60gb of ssd storage (aws)

i have set test mongo instance , have been using benchmark few things, seeing inconsistency can't explain.

i writing consistent amount of data using 10 separate threads single collection. each write consists of document containing array contains 1000 elements. each element complex document consisting of several fields , nested fields. have tested arrays of 1000, 10000 , 100 , have seen same behavior all. each write unique (i.e. never write same document twice)

the write speed tends around 100-200ms per write current hardware using. better isn't main issue.

my main issue write times spike. when do, can take single write several seconds complete. complete takes while. have timeouts built app doing writing (10 seconds) , when spikes happen hit timeout. have increased timeout , verified write complete can take long time (30+ seconds).

i have worked mongo before using mongo java driver in scala , have not noticed problem. unclear whether issue result of driver, or mongo setup.

i have looked @ logs , while report when query taking longer, don't provide information why taking longer. have done same profiling , again report long query don't why long.

i have run mongostat while running , seems when writes start taking long time notice similar slow down in mongostat. i.e. mongostat pause several seconds before continuing.

the mongo machine bored while happening. load averages minimal cpu , memory usage. not appear going swap.

i suspect have configured incorrectly in mongo haven't been able find indicates what.

has seen behavior before? in configuration or perhaps reactive mongo driver?

update:

using iostat able determine normal writes/second hitting around 1mb/second. during slow periods spikes 6-7mb/second.

i found following in mongo logs.

[datafilesync] flushing mmaps took 15621ms 35 files

[datafilesync] flushing mmaps took 14816ms 22 files

in @ least 1 case log statement corresponds 1 of slow downs.

this seems disk flush problem based on these observations.

does imply pushing more data current mongo configuration can handle? or there other configuration can done reduce impact of flushes?

it appears in case problem may have been related thread locking within application itself. once resolved issues thread locking these other issues seemed go away.

to honest don't know why thread locking result in observed behavior in mongo, if problem gone not going complain.