io - Reading a very large file word by word in Python -


i have pretty large text files (>2g) process word word. files space-delimited text files no line breaks (all words in single line). want take each word, test if dictionary word (using enchant), , if so, write new file.

this code right now:

with open('big_file_of_words', 'r') in_file:         open('output_file', 'w') out_file:             words = in_file.read().split(' ')             word in word:                 if d.check(word) == true:                     out_file.write("%s " % word)

i looked @ lazy method reading big file in python, suggests using yield read in chunks, concerned using chunks of predetermined size split words in middle. basically, want chunks close specified size while splitting on spaces. suggestions?

combine last word of 1 chunk first of next:

def read_words(filename):     last = ""     open(filename) inp:         while true:             buf = inp.read(10240)             if not buf:                 break             words = (last+buf).split()             last = words.pop()             word in words:                 yield word         yield last  open('output.txt') output:     word in read_words('input.txt'):         if check(word):             output.write("%s " % word)

Comments

Post a Comment

Popular posts from this blog

java - How to specify maven bin in eclipse maven plugin? -

i have eclipse maven plugin coming on centos. how can execute maven command on linx like: mvn clean package [abigail@localhost ~]$ mvn bash: mvn: command not found how should specify path executable of maven plugin in eclipse? or have install maven seperately？ thanks. eclipse (m2e) have bundled central parts of maven inside eclipse plugin, why not detected, , not usable. investigation find correct way bootstrap it, have create launcher file manually. if want run maven outside of eclipse, far better off standalone version of maven. mvn command bootstrap maven runtime system correctly, , secondly, able use advice concerning use of maven. maven binaries quite stable, wouldn't need upgrade upgrade eclipse. you can subsequently switch eclipse use external installation if prefer.
Read more

single sign on - Logging into Plone site with credentials passed through HTTP -

i using plone 4.3.3 , new plone development. here trying achieve: i have site going redirect plone site. in http url line redirects plone site pass username whereas plone site should automatically log user username in without asking credentials password. not care safety or encryption @ point. how achieve that? (i guess need somehow modify existing login view, how do that?) thanks. i had same problem , @ end i've developed own pas plugin. i've followed next steps: create pas plugin implement extractcredentials method following this example implement authenticatecredentials method following this example add plugin develop section in build.conf. start plone , open zope console. go acl_users folder , add plugin pas system there examples here can use templates.
Read more

php - Why does AJAX not process login form? -

i writing simple login script in php form. can't work. there wrong ajax. when click submit button doesn't anything. this login html form: <form id="login_form"> <input type="text" class="input_field" name="username" id="username" value="gebruikersnaam" onclick="if(value=='gebruikersnaam'){ this.value='' }"/> <input type="password" class="input_field" name="password" id="password"/> <input type="submit" class="login_button" id="login_button" value="ga door"/> </form> this ajax: $(document).ready(function(){ $("#login_form").submit(function() { $.ajax({ type: "post", url: 'php/login_check.php', data: $("#login_form").serialize(), // serializes form...
Read more