Server info: Ubuntu Server, Nginx, PHP FPM
in /etc/php5/fpm/php.ini I have: memory_limit = 512M
in /etc/php5/fpm/php-fpm.conf
You can take Nginx out of the picture, it just passes the request on, it's in php-fpm running your php code.
you might check your suhosin settings, suhosin.memory_limit.
when you see your phpinfo output there should be 2 columns, master and local for memory_limit, what do these each show?
128M is the default for memory_limit in php 5.3