python - Linux Time Command causes Exception for Multi-context With Statement -


background

i using python 2.7.6 parse chunks of large files (20+ gb) in parallel multiprocessing module. have worker processes extract information input file , put results in shelved dictionary later processing. prevent simultaneous writes pseudo-database, using managed lock. have implemented context manager database access ensure closed, because shelve module doesn't natively support context manager functionality until python 3.4.

the problem

i measure overall run time linux time command. however, when run script time command, syntaxerror exception don't if run normally. example code:

import multiprocessing import shelve contextlib import contextmanager  db_name = 'temp_db'  # manually implemented context manager - not natively implemented until python 3.4 # use contextlib.closing, method makes "with" statements cleaner @contextmanager def open_db(db_name, flag='c'):     db = shelve.open(db_name, flag=flag)     try:         yield db     finally:         db.close()  db_lock = multiprocessing.manager().lock()  db_lock, open_db(db_name) db:     db['1'] = 'test_value1'     db['2'] = 1.5  db_lock, open_db(db_name) db:     key, val in db.iteritems():         print("{0} : {1}\n".format(key, val)) 

running python test_script.py produces expected output:

2 : 1.5

1 : test_value1

on other hand, running time python test_script.py causes exception:

  file "test_script.py", line 21
    with db_lock, open_db(db_name) db:
                        ^
syntaxerror: invalid syntax
0.005u 0.002s 0:00.01 0.0%      0+0k 0+0io 0pf+0w

the question

why time command affect interpreter considers valid syntax?

other notes

  1. i assume time command being invoked correctly because produce timing information, , presence of exception shows interpreter finding correct script.
  2. if eliminate either acquisition of lock or database opening, exception disappears, problem appears caused comma in with statement.

something causing python executable (and version) change. try these commands:

which python python -v  time python time python -v 

for overall project, consider having each worker return data parent, stores info in file or database. simplifies code because don't need locking -- parent has access.


Comments

Popular posts from this blog

javascript - Jquery show_hide, what to add in order to make the page scroll to the bottom of the hidden field once button is clicked -

javascript - Highcharts multi-color line -

javascript - Enter key does not work in search box -