FIxing tokenizer URL pattern to handle URL shortener URLs, among other things. #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
URL pattern copied from here: http://daringfireball.net/2010/07/improved_regex_for_matching_urls Should now handle non-http(s) urls, .co, .ly, .gl TLDs, unicode characters, etc.
|
Thanks, this look great! Have you run it on a bunch of tweets and looked at the differences? And ideally post them here. I usually spit out space-separated tokenizations, one per line (python twokenize.py does that i think). Run both versions on the same data, then "diff -u" works fine to see the changes. |
|
Diff run over 1000 tweets sampled from the stream pasted below. Some improvements, some problems. Needs more work, I believe. guacamole:twitter jelsas$ diff -u stream_sample_oldtok.txt stream_sample_newtok.txt --- stream_sample_oldtok.txt 2011-04-14 11:45:18.000000000 -0400 +++ stream_sample_newtok.txt 2011-04-14 11:42:30.000000000 -0400 @@ -166,7 +166,7 @@ Tomorrow is going to be a very busy day ! Present hunt for Sheila , @mixedupradio interview with @Robbie_Taylor and at night birthday bash ;) Angga anax siapa sich ? Bz gantenk gituu ych ? Dress hangin off yo shoulder barely sober -Africkan Boy No Pretender , Killing Mc 's Out There , Like Lucas EASTENDER LOOOLhttp :/ / www.youtube.com/watch?v=sJmT3Ooo-V8&feature=related +Africkan Boy No Pretender , Killing Mc 's Out There , Like Lucas EASTENDER LOOOLhttp://www.youtube.com/watch?v=sJmT3Ooo-V8&feature=related Photo : http://tumblr.com/xra247qu6l @chloefantastic same easy get fat ... RT @Xstrology #Sagittarius could profit from their charm and wit . @@ -176,7 +176,7 @@ RT @faktacowok : Cowok tdk suka diburu-buru dlm mengatakan cinta . so , tunggu sampai dia yg pertama mengatakannya #fc @sophielockley @missmez20 *reaches to unfollow mezzy and soph * :O :P it 's a good job I loves ya lol :D xxx @sloanfans @sloanmusic @sonicboommusic oops I had Record Store Day on my mind but this in- store is set for May ! -Mal so nebenbei : Bis 2014 fallen an meiner Uni 20 Professuren weg . In der Eingangshalle meines Fachbereichs gibts jetzt ein LCD TV Notizbrett . +Mal so nebenbei:Bis 2014 fallen an meiner Uni 20 Professuren weg . In der Eingangshalle meines Fachbereichs gibts jetzt ein LCD TV Notizbrett . @simonjgray I can say , hand on heart , I have no idea what you're talking about lol ! #Iisadiv Realmente todo el mundo se tiene que enterar ? @ppcastrog Tenang ada aku dsni , hahahahahRT @yuliasharen : Kenapa malem ini serem dan mistis yah , mamah : | @@ -323,7 +323,7 @@ Martin St . Louis you are crazy . RT @labelleavery : As if I don't stay up late enough already , talking to @anxiouslee usually results in me staying up even later . #worthit RT @fondafebrina : Baca ini : @aslisuroboyo : nang fh unair lante 2 & 3 ng kelas pojok lek bengi ono mahasiswa n dosen tp lek melbu jeding/ ... -The @IndianolaTrib is looking for 2011 Prom photos . Do you have any photos to share with us ? E-mail your photos to recordherald@ live.com . +The @IndianolaTrib is looking for 2011 Prom photos . Do you have any photos to share with us ? E-mail your photos to recordherald@live . com . @Bartvanloon1996 nee En dan nu de filemeldingen . Op de Ceintuurbaan in de richting van de N35 een metertje of 500 ! #hotellumen http://twitpic.com/4krxvv Auto 10K tournament in Poker Texas Holdem is now open , buyin 10K +R , guaranteed prize 200,000 GC http://gamedesire.com/t3540066 @@ -339,7 +339,7 @@ RT @DjPrettybOyTank : " @SpinzHoodrich : SHOUTS OUT TO @DJPRETTYBOYTANK & @1FUTURE . . . WE BOUT TO DROP THE BOMBS ON THESE CLOWNS . #WATCHTHIS " @RehStolochi KKKKKKKKK . pera ae No woman should ever be riding the bench . Either you're the star player or you're on the wrong Team ! -New Event : " Ibiza Opening Special " on 06/13/2011 at roxiz.com - share good times : http://bit.ly/fjfBql +New Event : " Ibiza Opening Special " on 06/13/2011 at roxiz . com - share good times : http://bit.ly/fjfBql my hair is rocked =/ lol . but i'll be good . philly in 2days ! llol @ZanneSusuMoomoo JAY FANS !! Help vote Jay Chou as most influential in Time 2011 ... Voting ends today !! http://t.co/NwDjp7u I def do love sneakerhead ass females doe , but only ones that are up on their shxt and can dress , not ones just rockin kicks just to rock em @@ -388,7 +388,7 @@ Photo : http://tumblr.com/xda247qx2l http://www.yagarden.net/american-air-filter-220-782-051-disposable-panel-20-x-24-x-1-case-of-12.html #filtration #applications #air #fil comida esta deitando hahaha !!!!!!!!!!!! #garotoprendado -Free #domains : #bayernlb . me # brotfilme.com http://tumblr.com/x8f247qx38 +Free #domains : #bayernlb . me #brotfilme . com http://tumblr.com/x8f247qx38 Pending menghancurkan segalanya Surprise lunch guest- off to enjoy the sun for a quick bit with Benny O ! @Sannekuilenburg iloveyouuuu @@ -473,7 +473,7 @@ RT @DefJamRecords : HUSTLE HUSTLE HUSTLE HUSTLE HUSTLE #HUSTLEHARD ! @TheRealAceHood @PAMLAAAAAAAAAAA Hahahaha magkakasama kame ni sindol at wes sa eco . Hahahaha @PRessureAkaPRe ur to much fam tooo much and them bars are heated -5 Simple Steps To Start Your Day Productively | ow . ly/4zqob +5 Simple Steps To Start Your Day Productively | ow.ly/4zqob Katherine Jackson Fires Her Lawyer http://tiny.ly/T8ao @AbiiGrapes hahahahaha yo lil homie I get what you get in ten years in two days http://www.youtube.com/user/Yinkaaa?feature=mhum @@ -579,7 +579,7 @@ Photo : learntocarrylove : http://tumblr.com/xj3247r02s North Myrtle Beach resorts http://bit.ly/gkfYYi #MyrtleBeach MOLTEN FINAL 4 : Baruch to Play Springfield on Friday , 5pm ; Live Stats & Video Available http://bit.ly/dN81Rb -rakyatmerdekaonline.com - PDIP : Sayang , Militer Tangguh Dipimpin Presiden Lelet http://bit.ly/dVcLdM +rakyatmerdekaonline . com - PDIP : Sayang , Militer Tangguh Dipimpin Presiden Lelet http://bit.ly/dVcLdM boa tarde vcs @arie1969 ik mag alleen deze maar meenemen voor je . Ook goed ? http://twitpic.com/4kry2o @ladyfae10408 sounds good ! @@ -704,7 +704,7 @@ " I rarely end up where I was intending to go , but often I end up somewhere that I needed to be ." -- Dirk Gently ( Douglas Adams ) RT @epistolitas : #teodiomasque la comida de hospital del IMSS RT @mateusaguiar_ : boa tarde amores [ educados ( as ) respondem ] -Six Tips For SMEs To Get Their Digital House In Order http%3a%2f% 2fc.moreover.com%2fclick%2fhere.pl%3fr4511429497%26w%3d2390512 +Six Tips For SMEs To Get Their Digital House In Order http%3a%2f%2fc . moreover . com%2fclick%2fhere . pl%3fr4511429497%26w%3d2390512 @VemmaNEXT can give you peace of mind knowing your child is getting all the vitamins , antioxidants and over 65 plant-sourced minerals . @x_Blackheart_x porque la carita ? @OfficialAndu hay wena !!!!!!! Hahahahaha I'm bored @@ -741,7 +741,7 @@ Man is definitely a political animal . Right now , Nigeria is full of animals RT @kmh4eva : " just bc she dresses like a pirate doesnt mean its okay for people to pop her balloons !" hahaha @oliviaaanderson @lolagetslife It 's a damn shame , is what it is -BRANDING ! Bring us skills on : PMs , Designers , Copywriters & Production Wanted ! Email ap@ updatehq.com #jobs #branding +BRANDING ! Bring us skills on : PMs , Designers , Copywriters & Production Wanted ! Email ap@updatehq . com #jobs #branding @Lisettexappel jij krijgt hemm . :D :D @HockeyBroad is GAA kind of like ERA and based on time ? I didn't think so but just checking At 1:30 , we will discuss on Fox Business the recent arrest of the Harvard surgeon turned insider info trader |
|
the interaction between the URL and the surrounding text is tricky -- I think that's probably the issue with many of the regressions. The gruber regex probably makes different assumptions. and then they interact with the rest of the tokenizer too, kind of. bleah. maybe it's simpler tho |
|
It looks like a few things are still needed:
might work on this a bit more in a few days, but not now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
URL pattern copied from here: http://daringfireball.net/2010/07/improved_regex_for_matching_urls
Should now handle non-http(s) urls, .co, .ly, .gl TLDs, unicode characters, etc.