I would suggest starting from the same algorithm used in Maple, the Quadratic Sieve.
- Choose your odd number n to factorize,
- Choose a natural number k,
- Search all p <= k so that k^2 is not congruent to (n mod p) to obtain a factor base B = p1, p2, ..., pt,
- Starting from r > floor(n) search at least t+1 values so that y^2 = r^2 - n all have just factors in B,
- For every y1, y2, ..., y(t+1) just calculated you generate a vector v(yi) = (e1, e2, ..., et) where ei is calculated by reducing over modulo 2 the exponent pi in yi,
- Use Gaussian Elimination to find some of the vectors that added together give a null vector
- Set x as the product of ri related to yi found in the previous step and set y as p1^a * p2^b * p3^c * .. * pt^z where exponents are the half of the exponents found in the factorization of yi
- Calculate d = mcd(x-y, n), if 1 < d < n then d is a non-trivial factor of n, otherwise start from step 2 choosing a bigger k.
The problem about these algorithms is that they really imply a lot of theory in numerical calculus..