Why is it that ~2 is equal to -3? How does ~ operator work?
tl;dr ~ flips the bits. As a result the sign changes. ~2 is a negative number (0b..101). To output a negative number ruby prints -, then two's complement of ~2: -(~~2 + 1) == -(2 + 1) == 3. Positive numbers are output as is.
There's an internal value, and its string representation. For positive integers, they basically coincide:
irb(main):001:0> '%i' % 2
=> "2"
irb(main):002:0> 2
=> 2
The latter being equivalent to:
irb(main):003:0> 2.to_s
"2"
~ flips the bits of the internal value. 2 is 0b010. ~2 is 0b..101. Two dots (..) represent an infinite number of 1's. Since the most significant bit (MSB) of the result is 1, the result is a negative number ((~2).negative? == true). To output a negative number ruby prints -, then two's complement of the internal value. Two's complement is calculated by flipping the bits, then adding 1. Two's complement of 0b..101 is 3. As such:
irb(main):005:0> '%b' % 2
=> "10"
irb(main):006:0> '%b' % ~2
=> "..101"
irb(main):007:0> ~2
=> -3
To sum it up, it flips the bits, which changes the sign. To output a negative number it prints -, then ~~2 + 1 (~~2 == 2).
The reason why ruby outputs negative numbers like so, is because it treats the stored value as a two's complement of the absolute value. In other words, what's stored is 0b..101. It's a negative number, and as such it's a two's complement of some value x. To find x, it does two's complement of 0b..101. Which is two's complement of two's complement of x. Which is x (e.g ~(~2 + 1) + 1 == 2).
In case you apply ~ to a negative number, it just flips the bits (which nevertheless changes the sign):
irb(main):008:0> '%b' % -3
=> "..101"
irb(main):009:0> '%b' % ~-3
=> "10"
irb(main):010:0> ~-3
=> 2
What is more confusing is that ~0xffffff00 != 0xff (or any other value with MSB equal to 1). Let's simplify it a bit: ~0xf0 != 0x0f. That's because it treats 0xf0 as a positive number. Which actually makes sense. So, ~0xf0 == 0x..f0f. The result is a negative number. Two's complement of 0x..f0f is 0xf1. So:
irb(main):011:0> '%x' % ~0xf0
=> "..f0f"
irb(main):012:0> (~0xf0).to_s(16)
=> "-f1"
In case you're not going to apply bitwise operators to the result, you can consider ~ as a -x - 1 operator:
irb(main):018:0> -2 - 1
=> -3
irb(main):019:0> --3 - 1
=> 2
But that is arguably of not much use.
An example Let's say you're given a 8-bit (for simplicity) netmask, and you want to calculate the number of 0's. You can calculate them by flipping the bits and calling bit_length (0x0f.bit_length == 4). But ~0xf0 == 0x..f0f, so we've got to cut off the unneeded bits:
irb(main):014:0> '%x' % (~0xf0 & 0xff)
=> "f"
irb(main):015:0> (~0xf0 & 0xff).bit_length
=> 4
Or you can use the XOR operator (^):
irb(main):016:0> i = 0xf0
irb(main):017:0> '%x' % i ^ ((1 << i.bit_length) - 1)
=> "f"