[bits 32]
global _start
section .data
str_hello db \"HelloWorld\", 0xa
str_hello_length db $-str_hello
section .text
_start
The issue is probably that the offset of str_hello_length is greater than 32 bits. IA-32 doesn't support displacements of greater than 32 bits. The way around that is to use RIP-relative addressing, under the (often correct) assumption that the distance between the RIP and the address you're trying to reach fits in 32 bits. In this case, the base is RIP and the index is the instruction length, so if the instruction already has a base or an index, RIP-Relative can't be used.
Let's examine your various attempts:
str_hello_length equ $-str_hello
...
...
mov edx, str_hello_length
There's no memory access here, only a simply move with an immediate, so there's no addressing at all.
Next:
mov eax, str_hello_length
mov edx, [eax] ; count
Now the first instruction is a move with an immediate, which is still not a memory access. The second instruction has a memory access, but it uses eax as a base, and there's no displacement. RIP-relative is only relevant when there's a displacement, so there's no RIP-relative here.
Finally:
str_hello_length db $-str_hello
...
...
mov edx, [str_hello_length] ; of course, without the brackets it'll load the address, which I don't want. I want the value stored at that address
Here you're using str_hello_length as your displacement. As I explained above, this will result in RIP-Relative addressing.