python - Panda's read_csv always crashes on small file -


i trying import rather small (217 rows, 87 colums, 15k) csv file analysis in python using panda. file rather poorly structured, still import it, since raw data not want manipulate manually outside python (e.g. excel). unfortunately leads crash "the kernel appears have died. restart automatically".

https://www.wakari.io/sharing/bundle/uniquely/readcsv

did research indicated possible crashes read_csv, large files, not understand problem. crash happens both using local installation (anaconda 64-bit, ipython (py 2.7) notebook) , wakari.

can me? appreciated. lot!

code:

# have somehow ugly, illustrative csv file, not big, 217 rows, 87 colums. # file can downloaded @ http://www.win2day.at/download/lo_1986.csv  # in[1]:  file_csv = 'lo_1986.csv' f = open(file_csv, mode="r") x = 0 line in f:     print x, ": ", line     x = x + 1 f.close()   # i'd import csv python using pandas - lead crash: # "the kernel appears have died. restart automatically."  # in[ ]:  import pandas pd pd.read_csv(file_csv, delimiter=';')  # doing wrong?

it because of invalid character (e.g. 0xe0) in file

if add encoding parameter read_csv() call, see stacktrace instead of segfault 

>>> df = pandas.read_csv("/tmp/lo_1986.csv", delimiter=";", encoding="utf-8") traceback (most recent call last):   file "<stdin>", line 1, in <module>   file "/users/antkong/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py", line 400, in parser_f     return _read(filepath_or_buffer, kwds)   file "/users/antkong/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py", line 205, in _read     return parser.read()   file "/users/antkong/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py", line 608, in read     ret = self._engine.read(nrows)   file "/users/antkong/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py", line 1028, in read     data = self._reader.read(nrows)   file "parser.pyx", line 706, in pandas.parser.textreader.read (pandas/parser.c:6745)   file "parser.pyx", line 728, in pandas.parser.textreader._read_low_memory (pandas/parser.c:6964)   file "parser.pyx", line 804, in pandas.parser.textreader._read_rows (pandas/parser.c:7780)   file "parser.pyx", line 890, in pandas.parser.textreader._convert_column_data (pandas/parser.c:8793)   file "parser.pyx", line 950, in pandas.parser.textreader._convert_tokens (pandas/parser.c:9484)   file "parser.pyx", line 1026, in pandas.parser.textreader._convert_with_dtype (pandas/parser.c:10642)   file "parser.pyx", line 1051, in pandas.parser.textreader._string_convert (pandas/parser.c:10905)   file "parser.pyx", line 1278, in pandas.parser._string_box_utf8 (pandas/parser.c:15657) unicodedecodeerror: 'utf8' codec can't decode byte 0xe0 in position 0: unexpected end of data

you can preprocessing remove these characters before asking pandas read in file

attached picture highlight invalid characters in file

enter image description here


Comments

Post a Comment

Popular posts from this blog

java - How to specify maven bin in eclipse maven plugin? -

i have eclipse maven plugin coming on centos. how can execute maven command on linx like: mvn clean package [abigail@localhost ~]$ mvn bash: mvn: command not found how should specify path executable of maven plugin in eclipse? or have install maven seperately？ thanks. eclipse (m2e) have bundled central parts of maven inside eclipse plugin, why not detected, , not usable. investigation find correct way bootstrap it, have create launcher file manually. if want run maven outside of eclipse, far better off standalone version of maven. mvn command bootstrap maven runtime system correctly, , secondly, able use advice concerning use of maven. maven binaries quite stable, wouldn't need upgrade upgrade eclipse. you can subsequently switch eclipse use external installation if prefer.
Read more

single sign on - Logging into Plone site with credentials passed through HTTP -

i using plone 4.3.3 , new plone development. here trying achieve: i have site going redirect plone site. in http url line redirects plone site pass username whereas plone site should automatically log user username in without asking credentials password. not care safety or encryption @ point. how achieve that? (i guess need somehow modify existing login view, how do that?) thanks. i had same problem , @ end i've developed own pas plugin. i've followed next steps: create pas plugin implement extractcredentials method following this example implement authenticatecredentials method following this example add plugin develop section in build.conf. start plone , open zope console. go acl_users folder , add plugin pas system there examples here can use templates.
Read more

php - Why does AJAX not process login form? -

i writing simple login script in php form. can't work. there wrong ajax. when click submit button doesn't anything. this login html form: <form id="login_form"> <input type="text" class="input_field" name="username" id="username" value="gebruikersnaam" onclick="if(value=='gebruikersnaam'){ this.value='' }"/> <input type="password" class="input_field" name="password" id="password"/> <input type="submit" class="login_button" id="login_button" value="ga door"/> </form> this ajax: $(document).ready(function(){ $("#login_form").submit(function() { $.ajax({ type: "post", url: 'php/login_check.php', data: $("#login_form").serialize(), // serializes form...
Read more