As for current Chrome version (56) you can't remove it yet. Solution provided in other posts leads to overflowing some part of the video.
I've found another solution - you can make the preceding button to overlap the download button and simply cover it, by using this technique:
video::-webkit-media-controls-fullscreen-button {
margin-right: -48px;
z-index: 10;
position: relative;
background: #fafafa;
background-image: url(https://image.flaticon.com/icons/svg/151/151926.svg);
background-size: 35%;
background-position: 50% 50%;
background-repeat: no-repeat;
}
Example: https://jsfiddle.net/dk4q6hh2/
PS You might want to customise the icon, since it's for example only.