I want to have docker CLI to connect to remote daemon but do I need to install the whole engine including daemon on the local machine?
You can (like the other answer suggests) download it direct from Docker:
docker_url=https://download.docker.com/linux/static/stable/x86_64
docker_version=18.03.1-ce
curl -fsSL $docker_url/docker-$docker_version.tgz | \
tar zxvf - --strip 1 -C /usr/bin docker/docker
The difference from the other answer is that there is no intermediate tar file. I use this in a Dockerfile RUN layer.