lazy loading - Exchange Time for Memory in Python -

challenge here in evaluating multiple large files.

what coding instruct python "load" limited number of files memory, process them, garbage collect , load next set?

def main(directory):     """     create audioanalysis objects directory , call object_analysis(). """        ff = os.listdir(directory)     f in ff:         # can limit number load @ 1 time?         audiofile = audio.localaudiofile(os.path.join(directory,f)) # hungry! 

tried adding audiofile = 0 loop, memory allocation same.

as understand it, lazy evaluation "is evaluation strategy delays evaluation of expression until value needed", in case need delay evaluation until there's memory available.

am expecting decorator, descriptor and/or use of pythons property() function may involved, or possibly buffering or queueing input.

here's 1 solution: have python spawn process, run function on 1 file, exit. parent proc collect results each of files.

this in no way graceful, if localaudiofile refuses dislodged memory, allows flexibility in getting results.

this code runs runs function on each python file in current directory, returning message parent process, prints out.

source

import glob, multiprocessing, os  def proc(path):     """     create audioanalysis objects directory , call object_analysis(). """        # audiofile = audio.localaudiofile(path) # hungry!     return 'woot: {}'.format(path)  if __name__=='__main__':  # required windows     pool = multiprocessing.pool()   # 1 process per cpu     output in pool.map(proc, [             os.path.abspath(name) name in glob.glob('q*.py')             ]):         print 'output:',output 

output

output: woot: /home/johnm/src/johntellsall/karma/qpopen.py output: woot: /home/johnm/src/johntellsall/karma/quotes.py