Server info: Ubuntu Server, Nginx, PHP FPM
in /etc/php5/fpm/php.ini
I have: memory_limit = 512M
in /etc/php5/fpm/php-fpm.conf
Just FYI,
you should also check in /etc/php5/fpm/pool.d for any files.
It just happened to me that no matter what I wrote on /etc/php5/fpm/php-fpm.conf it would get overrided by /etc/php5/fpm/pool.d/www.conf !
So if none of the above answers are not working for you, you should check this (I just lost half hour to this).