shouldn't `where.not(field: “something”)` include `where(field: nil)`?

Question

Perhaps I'm going crazy, or just need a break, but in my rails console Order.where(state: nil).count returns 1010, but Order.where.not(state: "pending").count returns 0... If an order's state is nil, then it is not "pending", so I expect the set returned by not(state: "pending") to include the set where(state: nil).

Does arel not work this way? If not, does arel work a different way?

EDIT: more info! When I go to another database, where some records have a state other than nil, and I run Order.where.not(state: "pending").count I get back a bunch of orders, none of which are "pending" but also none of which are nil. It seems that where.not is implicitly adding a and not nil to the query?

EDIT: in desperation, I have turned to darker spirits.

# look into another shop, that has records
o = Order.where(shop_id: 2)

# summon dread spirits
t = Order.arel_table[:state]

o.where(t.eq(nil).or(t.eq("pending").not)).count

=> 1569

o.where(t.eq(nil)).count

=> 1471

So in this case, I get the 98 records whose status is neither nil nor "pending", and I get all the records whose status is nil. I would really like to know why I can't just say where.not("pending") and have the same effect. If there maybe an option I can invoke? Like, where.not("pending", include_nil: true)?

EDIT: as requested in a comment by @Filip Bartuzi

Order.where.not(state: "pending").to_sql

=> "SELECT \"orders\".* FROM \"orders\"  WHERE \"orders\".\"shop_id\" = 2 AND (\"orders\".\"state\" != 'pending')"

Orders.where(state: nil).to_sql

=> "SELECT \"orders\".* FROM \"orders\"  WHERE \"orders\".\"shop_id\" = 2 AND \"orders\".\"state\" IS NULL"

Answer 1

Order.where.not(state: "pending").to_sql generates:

=> "SELECT \"orders\".* FROM \"orders\" WHERE \"orders\".\"shop_id\" = 2 AND (\"orders\".\"state\" != 'pending')"

It will return all records with VALUES which are not 'pending'. When u set pending to nil (like Order.first.update! state: nil it assigns NULL in database.

NULL isn't interpreted as value in SQL so it will not be included in SELECT response

So the answer is : where.not(field: “something”) does NOT include where(field: nil)!

You can check how it works here:

http://www.w3schools.com/sql/trysql.asp?filename=trysql_select_all

Go to the categories table and firstly execute

UPDATE Categories
SET CategoryName = NULL
WHERE CategoryName = 'Beverages'

So now we have Categories of Count 8 which one of them has NULL on column CategoryName

Now execute:

SELECT CategoryName FROM Categories
WHERE CategoryName != 'Condiments'

As you see 6 records where returned (so it skipped one with NULL)

Answer 2

SQL implementations follow 3 valued logic where NULL is not a value but marks the absence of a value. 3 valued logic defines the following truth table for logical operations: (null is Unknown here)

p       |q      |p OR q |p AND q|p = q
True    |True   |True   |True   |True
True    |False  |True   |False  |False
True    |Unknow |True   |Unknown|Unknown
False   |True   |True   |False  |False
False   |False  |False  |False  |True
False   |Unknown|Unknown|False  |Unknown
Unknown |True   |True   |Unknown|Unknown
Unknown |False  |Unknown|False  |Unknown
Unknown |Unknown|Unknown|Unknown|Unknown

Since where tests for equality (row.state == 'pending') on each entry in the data set, by the truth table above if row.state is NULL the result is 'Unknown'. 'Unknown' isn't true so the row isn't included in the result set.

More info at Wikipedia.