I have .doc and .zip files in download directory on my server.
whoever visit my site page (download-file.php) only those user should
You should be able to restrict people from directly accessing your content using a method similar to the following code:
RewriteCond %{HTTP_REFERER} !^http://your_domain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://your_domain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.your_domain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.your_domain.com$ [NC]
RewriteRule .*.(jpg|jpeg|gif|png|bmp|pdf|doc)$ http://your_domain.com/no_access.php [R,NC]
This would prevent people from directly linking images, pdfs or docs from your site. Then, files with these extensions would only be able to accessed from your site. If someone attempts to directly link or access your files, they will experience the whatever you choose for them to see (that you place in the no_access.php).