Convert non-breaking spaces to spaces in Ruby
I have cases where user-entered data from an html textarea or input is sometimes sent with \\u00a0 (non-breaking spaces) instead of spaces when encoded as utf-8
-
While not related to Ruby (and not directly to this question), the core of the problem might be that Alt+Space on Macs produces a non-breaking space.
This can cause all kinds of weird behaviour (especially in the terminal).
For those who are interested in more details, I wrote "Why chaining commands with pipes in Mac OS X does not always work" about this topic some time ago.
讨论(0) -
Use
/\u00a0/to match non-breaking spaces. For instances.gsub(/\u00a0/, ' ')converts all non-breaking spaces to regular spaces.Use
/[[:space:]]/to match all whitespace, including Unicode whitespace like non-breaking spaces. This is unlike/\s/, which matches only ASCII whitespace.See also: Ruby Regexp documentation
讨论(0) -
Actual functioning IRB code examples that answer the question, with latest Rubies (May 2012)
Ruby 1.9
require 'rubygems' require 'nokogiri' RUBY_DESCRIPTION # => "ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]" doc = '<html><body> </body></html>' page = Nokogiri::HTML(doc) s = page.inner_text s.each_codepoint {|c| print c, ' ' } #=> 32 160 32 s.strip.each_codepoint {|c| print c, ' ' } #=> 160 s.gsub(/\s+/,'').each_codepoint {|c| print c, ' ' } #=> 160 s.gsub(/\u00A0/,'').strip.empty? #trueRuby 1.8
require 'rubygems' require 'nokogiri' RUBY_DESCRIPTION # => "ruby 1.8.7 (2012-02-08 patchlevel 358) [x86_64-linux]" doc = '<html><body> </body></html>' page = Nokogiri::HTML(doc) s = page.inner_text # " \302\240 " s.gsub(/\s+/,'') # "\302\240" s.gsub(/\302\240/,'').strip.empty? #true讨论(0) -
For the old versions of ruby (1.8.x), the fixes are the ones described in the question.
This is fixed in the newer versions of ruby 1.9+.
讨论(0) -
If you cannot use
\sfor Unicode whitespace, that’s a bug in the Ruby regex implementation, because according to UTS#18 “Unicode Regular Expressions” Annex C on Compatibility Properties a\s, is absolutely required to match any Unicode whitespace code point.There is no wiggle-room allowed since the two columns detailing the Standard Recommendation and the POSIX Compatibility are the same for the
\scase. You cannot document your way around this: you are out of compliance with The Unicode Standard, in particular, with UTS#18’s RL1.2a, if you do not do this.If you do not meet RL1.2a, you do not meet the Level 1 requirements, which are the most basic and elementary functionality needed to use regular expressions on Unicode. Without that, you are pretty much lost. This is why standards exist. My recollection is that Ruby also fails to meet several other Level 1 requirements. You may therefore wish to use a programming language that meets at least Level 1 if you actually need to handle Unicode with regular expressions.
Note that you cannot use a Unicode General Category property like
\p{Zs}to stand for\p{Whitespace}. That’s because the Whitespace property is a derived property, not a general category. There are also control characters included in it, not just separators.讨论(0) -
For whatever reason \s doesn't match \u00a0.
I think the "whatever reason" is that is not supposed to. Only the POSIX and \p construct character classes are Unicode aware. The character-class abbreviations are not:
Sequence As[...] Meaning \d [0-9] ASCII decimal digit character \D [^0-9] Any character except a digit \h [0-9a-fA-F] Hexadecimal digit character \H [^0-9a-fA-F] Any character except a hex digit \s [ \t\r\n\f] ASCII whitespace character \S [^ \t\r\n\f] Any character except whitespace \w [A-Za-z0-9\_] ASCII word character \W [^A-Za-z0-9\_] Any character except a word character讨论(0)
加载中...
- 热议问题