python - Linux Time Command causes Exception for Multi-context With Statement -
background
i using python 2.7.6 parse chunks of large files (20+ gb) in parallel multiprocessing
module. have worker processes extract information input file , put results in shelved dictionary later processing. prevent simultaneous writes pseudo-database, using managed lock. have implemented context manager database access ensure closed, because shelve
module doesn't natively support context manager functionality until python 3.4.
the problem
i measure overall run time linux time command. however, when run script time command, syntaxerror exception don't if run normally. example code:
import multiprocessing import shelve contextlib import contextmanager db_name = 'temp_db' # manually implemented context manager - not natively implemented until python 3.4 # use contextlib.closing, method makes "with" statements cleaner @contextmanager def open_db(db_name, flag='c'): db = shelve.open(db_name, flag=flag) try: yield db finally: db.close() db_lock = multiprocessing.manager().lock() db_lock, open_db(db_name) db: db['1'] = 'test_value1' db['2'] = 1.5 db_lock, open_db(db_name) db: key, val in db.iteritems(): print("{0} : {1}\n".format(key, val))
running python test_script.py
produces expected output:
2 : 1.5
1 : test_value1
on other hand, running time python test_script.py
causes exception:
file "test_script.py", line 21
with db_lock, open_db(db_name) db:
^
syntaxerror: invalid syntax
0.005u 0.002s 0:00.01 0.0% 0+0k 0+0io 0pf+0w
the question
why time command affect interpreter considers valid syntax?
other notes
- i assume time command being invoked correctly because produce timing information, , presence of exception shows interpreter finding correct script.
- if eliminate either acquisition of lock or database opening, exception disappears, problem appears caused comma in
with
statement.
something causing python
executable (and version) change. try these commands:
which python python -v time python time python -v
for overall project, consider having each worker return data parent, stores info in file or database. simplifies code because don't need locking -- parent has access.
Comments
Post a Comment