c# - Working with "very very" large arrays -

i need work large arrays of small types (int or float arrays), i'm targeting x64 on machines lot of ram, physical memory never issue in scenarios. while looking @ doc gcallowverylargeobjects noticed point :

•the maximum index in single dimension 2,147,483,591 (0x7fffffc7) byte arrays , arrays of single-byte structures, , 2,146,435,071 (0x7fefffff) other types.

now issue "need" work larger arrays that, appropriate workaround here? creating arrays of arrays or other abstractions?

knowing need access arrays sequencially (never random reads, diferent segments getting read sequencially diferent threads, potentially 100+ threads @ once) best bet be?

i may need hold arrays of 65 536 000 000 elements or more.

if must break array length limit you'll have split array chunks of suitable size. can wrap chunks in container has appropriate semantics, bigarrayoflong object james mccaffrey blogged while back. there numerous others it.

the basic idea use jagged array allocate space you're going use. note multi-dimensional array won't give advantage since still single object, while jagged array smaller array of arrays, each of own object in (probably not contiguous) memory.

here's very simple (and not particular optimal) implementation:

public class hugearray<t> : ienumerable<t>     t : struct {     public static int arysize = (int32.maxvalue >> 4) / marshal.sizeof<t>();      public readonly long capacity;     private readonly t[][] content;      public t this[long index]     {                 {             if (index < 0 || index >= capacity)                 throw new indexoutofrangeexception();             int chunk = (int)(index / arysize);             int offset = (int)(index % arysize);             return content[chunk][offset];         }         set         {             if (index < 0 || index >= capacity)                 throw new indexoutofrangeexception();             int chunk = (int)(index / arysize);             int offset = (int)(index % arysize);             content[chunk][offset] = value;         }     }      public hugearray(long capacity)     {         capacity = capacity;         int nchunks = (int)(capacity / arysize);         int nremainder = (int)(capacity % arysize);          if (nremainder == 0)             content = new t[nchunks][];         else             content = new t[nchunks + 1][];          (int = 0; < nchunks; i++)             content[i] = new t[arysize];         if (nremainder > 0)             content[content.length - 1] = new t[nremainder];     }      public ienumerator<t> getenumerator()     {         return content.selectmany(c => c).getenumerator();     }      ienumerator system.collections.ienumerable.getenumerator() { return getenumerator(); } } 

this 1 statically allocated, it's not hard make 1 grows fit demand. make sure block size specify isn't out of range. i've gone calculation based on item size in case.