using nltk regex example in scikit-learn CountVectorizer -


i trying use example nltk book regex pattern inside countvectorizer scikit-learn. see examples simple regex not this:

pattern = r''' (?x)         # set flag allow verbose regexps      ([a-z]\.)+          # abbreviations (e.g. u.s.a.)     | \w+(-\w+)*        # words optional internal hyphens     | \$?\d+(\.\d+)?%?  # currency & percentages     | \.\.\.            # ellipses '''  text = 'i love n.y.c. 100% of traffic-ridden streets...' vectorizer = countvectorizer(stop_words='english',token_pattern=pattern) analyze = vectorizer.build_analyzer() analyze(text)

this produces:

[(u'', u'', u''),  (u'', u'', u''),  (u'', u'', u''),  (u'', u'', u''),  (u'', u'', u''),  (u'', u'', u''),  (u'', u'', u''),  (u'', u'', u''),  (u'', u'', u''),  (u'', u'', u''),  (u'', u'', u''),  (u'', u'-ridden', u''),  (u'', u'', u''),  (u'', u'', u'')]

with nltk, entirely different:

nltk.regexp_tokenize(text,pattern)

['i', 'love', 'n.y.c.', '100', 'even', 'with', 'all', 'of', 'its', 'traffic-ridden', 'streets', '...']

is there way skl countvectorizer output same thing? hoping use of other handy features incorporated in same function call.

tl;dr

from functools import partial countvectorizer(analyzer=partial(regexp_tokenize, pattern=pattern))

is vectorizer uses nltk tokenizer.

now actual problem: apparently nltk.regexp_tokenize quite special pattern, whereas scikit-learn re.findall pattern give it, , findall doesn't pattern:

in [33]: re.findall(pattern, text) out[33]:  [('', '', ''),  ('', '', ''),  ('c.', '', ''),  ('', '', ''),  ('', '', ''),  ('', '', ''),  ('', '', ''),  ('', '', ''),  ('', '', ''),  ('', '-ridden', ''),  ('', '', ''),  ('', '', '')]

you'll either have rewrite pattern make work in scikit-learn style, or plug nltk tokenizer scikit-learn:

in [41]: functools import partial  in [42]: v = countvectorizer(analyzer=partial(regexp_tokenize, pattern=pattern))  in [43]: v.build_analyzer()(text) out[43]:  ['i',  'love',  'n.y.c.',  '100',  'even',  'with',  'all',  'of',  'its',  'traffic-ridden',  'streets',  '...']

Comments

Post a Comment

Popular posts from this blog

javascript - Jquery show_hide, what to add in order to make the page scroll to the bottom of the hidden field once button is clicked -

i have button on website when clicked, reveals map. button can seen when page first loads up, when clicked, section holding map slides open , goes out of screen since page loads @ top (normal). want page scroll down bottom of div select when button pushed, if map out of screen. if page scrolled down, , clicks show map button, don't want page jump bottom of map. i tried figure out myself, no luck. seasoned jquery coders able give me guidance? many thanks! map = mapdiv.gmap3("get"); infobox = new infobox({ pixeloffset: new google.maps.size(-220, -310), closeboxurl: '', enableeventpropagation: true }); mapdiv.delegate('.infobox .close','click',function () { infobox.close(); }); jquery(".slidingdiv").hide(); jquery(".show_hide").show(); jquery('.show_hide').click(function(){ lastcenter=map.getcenter(); jquery(".slidingdiv&quo
Read more

javascript - Highcharts multi-color line -

Image
i looking create highchart multi-colored line. if goes down under value, become red, , when comes on value, become green . here picture example: or maybe highcharts have other charts models differentiate high or low level color? you can achieve coloring using lineargradient coloring: color: { lineargradient: { x1: 0, x2: 0, y1: 0, y2: 1 }, stops: [ [0, 'red'], [0.25, 'yellow'], [0.50, 'green'], [0.75, 'yellow'], [1, 'red'] ] } here 0 in stops relates minimum value of series , 1 maximum value, , in between percentages between values. if find percentages according values prior creating chart use correct stops values make relate specific values in data (at least approximately). see this jsfiddle example .
Read more

javascript - Enter key does not work in search box -

tried search guys sorry. input in search box doesnt work (or search) when hit enter when user clicks search button. doing wrong? here code. thanks. <script type="text/javascript"> $(document).ready(function() { $("#submit").click(function() { var url = "http://asla.org/awardssearch.html"; url += "?s=" + $('#googlecse').val(); window.location = url; }); $('#googlecse').keydown(function(event) { if (event.keycode == '13') { $("#submit").click(); } else { //do nothing :) } }); }); </script> <form class="navbar-form navbar-right" role="search"> <div class="search"> <input id="googlecse" type="text" onblur="if(this.value=='')this.value=this.defaultvalue;" onfocus="if(this.value==this.defaultvalue)t
Read more