for self educational purposes I tried to find a way to create a height map all by myself. I googled a little bit and found a function that creates pseudo-random numbers.
Why do I get an OverflowException in VB.net but not in C#?
Because VB projects check for overflow by default, while C# does not.  You'd see the same exception in C# if you either set the project to check for overflow or put the code in a checked{ } block:
public static float Noise(int x)
{
    x = (x << 13) ^ x;
    checked{
        return (1f - ((x * (x * x * 15731) + 1376312589) & 0x7FFFFFFF) / 1073741824.0f);
    }
}
What can I do to make it work in VB.net as well?
I doubt it "works" for C#. Since it overflows silently you are probably getting incorrect results.
However, you can prevent the overflow by casting x to a double:
Private Function Noise(ByVal x As Integer) As Double
    x = (x << 13) Xor x
    Dim y As Double = x
    Return (1.0R - ((y * (y * y * 15731.0R) + 1376312589.0R) And &H7FFFFFFF) / 1073741824.0R)
End Function
You can also turn on the "Remove integer overflow checks" property of your VB project, but that's a project wide setting. VB currently doesn't have a way of turning off overflow checks for specific lines of code like C# does.
Whatever you do, I would test it thoroughly since you are computing a floating-point value, bit-masking it (And &H7FFFFFFF) and then dividing by another floating-point value.  Mixing integer and floating point math can be very tricky.
Also note that your two code segments are NOT equivalent.  Your C# function returns a float while VB returns a Double.  And you are using floating-point literals in VB but integer literals in C#.  It may not seem like a big difference but it can change the math unexpectedly.
Like D Stanley pointed out, the reason for the OverFlowException in VB.net is that VB.net is automatically doing an overflow check on integer operations while C# is not.
In case someone else is interested in getting it to work in VB.net you can do the following:
Add this structure to your project
<StructLayout(LayoutKind.Explicit)>
Public Structure LongToIntegerNoOverflow
    <FieldOffset(0)>
    Public LongValue As Long
    <FieldOffset(0)>
    Public IntValue As Integer
End Structure
and change the Noise function to
Private Function Noise(ByVal x As Integer) As Single
    Dim x1 As Long
    Dim x2 As LongToIntegerNoOverflow
    Dim x3 As LongToIntegerNoOverflow
    Dim x4 As Long
    x1 = (x << 13) Xor x
    x2.LongValue = x1 * x1 * 15731 + 789221
    x3.LongValue = x1 * x2.IntValue + 1376312589
    x4 = x3.IntValue And &H7FFFFFFF
    Return (1.0R - x4 / 1073741824.0R)
End Function
After these changes I got the same results in VB.net and in C#.