问题
I have few text(SMS) messages and I want to segment them using period('.') as a delimiter. I am unable to handle following types of messages. How can I segment these messages using Regex in Python.
Before segmentation:
'hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u' 'no of beds 8.please inform person in-charge.tq'
After segmentation:
'hyper count 16.8mmol/l' 'plz review b4 5pm' 'just to inform u' 'thank u' 'no of beds 8' 'please inform person in-charge' 'tq'
Each line is a separate message
Updated:
I am doing natural language processing and I feel its okay to treat '16.8mmmol/l' and 'no of beds 8.2 cups of tea.' as same. 80% accuracy is enough for me but I want to reduce False Positive as much as possible.
回答1:
What about
re.split('(?<!\d)\.|\.(?!\d)', 'hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u')
The lookarounds ensure that either on one or the other side is not a digit. So this covers also the 16.8 case. This expression will not split if there are on both sides digits.
回答2:
Some weeks ago, I searched for a regex that would catch every string representing a number in a string, whatever the form in which the number is written, even the ones in scientific notation, even the indian numbers having commas: see this thread
I use this regex in the following code to give a solution to your problem.
Contrary to the other answers, in my solution a dot in '8.' isn't considered as a dot on which a split must be done, because it can be read as a float having no digit after the dot.
import re
regx = re.compile('(?<![\d.])(?!\.\.)'
'(?<![\d.][eE][+-])(?<![\d.][eE])(?<!\d[.,])'
'' #---------------------------------
'([+-]?)'
'(?![\d,]*?\.[\d,]*?\.[\d,]*?)'
'(?:0|,(?=0)|(?<!\d),)*'
'(?:'
'((?:\d(?!\.[1-9])|,(?=\d))+)[.,]?'
'|\.(0)'
'|((?<!\.)\.\d+?)'
'|([\d,]+\.\d+?))'
'0*'
'' #---------------------------------
'(?:'
'([eE][+-]?)(?:0|,(?=0))*'
'(?:'
'(?!0+(?=\D|\Z))((?:\d(?!\.[1-9])|,(?=\d))+)[.,]?'
'|((?<!\.)\.(?!0+(?=\D|\Z))\d+?)'
'|([\d,]+\.(?!0+(?=\D|\Z))\d+?))'
'0*'
')?'
'' #---------------------------------
'(?![.,]?\d)')
simpler_regex = re.compile('(?<![\d.])0*(?:'
'(\d+)\.?|\.(0)'
'|(\.\d+?)|(\d+\.\d+?)'
')0*(?![\d.])')
def split_outnumb(string, regx=regx, a=0):
excluded_pos = [x for mat in regx.finditer(string) for x in range(*mat.span()) if string[x]=='.']
li = []
for xdot in (x for x,c in enumerate(string) if c=='.' and x not in excluded_pos):
li.append(string[a:xdot])
a = xdot + 1
li.append(string[a:])
return li
for sentence in ('hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u',
'no of beds 8.please inform person in-charge.tq',
'no of beds 8.2 cups of tea.tarabada',
'this number .977 is a float',
'numbers 214.21E+45 , 478945.E-201 and .12478E+02 are in scientific.notation',
'an indian number 12,45,782.258 in this.sentence and 45,78,325. is another',
'no dot in this sentence',
''):
print 'sentence =',sentence
print 'splitted eyquem =',split_outnumb(sentence)
print 'splitted eyqu 2 =',split_outnumb(sentence,regx=simpler_regex)
print 'splitted gurney =',re.split(r"\.(?!\d)", sentence)
print 'splitted stema =',re.split('(?<!\d)\.|\.(?!\d)',sentence)
print
result
sentence = hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u
splitted eyquem = ['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']
splitted eyqu 2 = ['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']
splitted gurney = ['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']
splitted stema = ['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']
sentence = no of beds 8.please inform person in-charge.tq
splitted eyquem = ['no of beds 8.please inform person in-charge', 'tq']
splitted eyqu 2 = ['no of beds 8.please inform person in-charge', 'tq']
splitted gurney = ['no of beds 8', 'please inform person in-charge', 'tq']
splitted stema = ['no of beds 8', 'please inform person in-charge', 'tq']
sentence = no of beds 8.2 cups of tea.tarabada
splitted eyquem = ['no of beds 8.2 cups of tea', 'tarabada']
splitted eyqu 2 = ['no of beds 8.2 cups of tea', 'tarabada']
splitted gurney = ['no of beds 8.2 cups of tea', 'tarabada']
splitted stema = ['no of beds 8.2 cups of tea', 'tarabada']
sentence = this number .977 is a float
splitted eyquem = ['this number .977 is a float']
splitted eyqu 2 = ['this number .977 is a float']
splitted gurney = ['this number .977 is a float']
splitted stema = ['this number ', '977 is a float']
sentence = numbers 214.21E+45 , 478945.E-201 and .12478E+02 are in scientific.notation
splitted eyquem = ['numbers 214.21E+45 , 478945.E-201 and .12478E+02 are in scientific', 'notation']
splitted eyqu 2 = ['numbers 214.21E+45 , 478945.E-201 and .12478E+02 are in scientific', 'notation']
splitted gurney = ['numbers 214.21E+45 , 478945', 'E-201 and .12478E+02 are in scientific', 'notation']
splitted stema = ['numbers 214.21E+45 , 478945', 'E-201 and ', '12478E+02 are in scientific', 'notation']
sentence = an indian number 12,45,782.258 in this.sentence and 45,78,325. is another
splitted eyquem = ['an indian number 12,45,782.258 in this', 'sentence and 45,78,325. is another']
splitted eyqu 2 = ['an indian number 12,45,782.258 in this', 'sentence and 45,78,325. is another']
splitted gurney = ['an indian number 12,45,782.258 in this', 'sentence and 45,78,325', ' is another']
splitted stema = ['an indian number 12,45,782.258 in this', 'sentence and 45,78,325', ' is another']
sentence = no dot in this sentence
splitted eyquem = ['no dot in this sentence']
splitted eyqu 2 = ['no dot in this sentence']
splitted gurney = ['no dot in this sentence']
splitted stema = ['no dot in this sentence']
sentence =
splitted eyquem = ['']
splitted eyqu 2 = ['']
splitted gurney = ['']
splitted stema = ['']
EDIT 1
I added a simpler_regex detecting numbers, from a post of mine in this thread
I doesn't detect indian numbers and numbers in scientific notation but it gives in fact the same results
回答3:
you can use a negative lookahead assertion to match a "." not followed by a digit, and use re.split on this:
>>> import re
>>> splitter = r"\.(?!\d)"
>>> s = 'hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u'
>>> re.split(splitter, s)
['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']
>>> s = 'no of beds 8.please inform person in-charge.tq'
>>> re.split(splitter, s)
['no of beds 8', 'please inform person in-charge', 'tq']
回答4:
It depends on your exact sentence, but you could try:
.*?[a-zA-Z0-9]\.(?!\d)
See if that works. This will keep in the quotes, but you can then remove them if needed.
回答5:
"...".split(".")
split is a Python builtin that separates a string at a specific character.
来源：https://stackoverflow.com/questions/6745592/sentence-segmentation-using-regex