I am using Jenkins for CI, I have a master and two slaves, master is running Jenkins and I want only slaves doing the actual building task, is there anywhere I can configure
You can set where a job will be run using the "Restrict where this project can be run" option in your job. This setting can be used together with tags you have added to your slaves.
For example two slaves having the tag "Linux-buildserver" and using that tag will split the job up on those two slaves. Setting the IP-address as a tag in the job will make sure only that buildserver / slave is used.