$ psql -E --host=xxx --port=yyy --username=chi --dbname=C_DB -c \'DELETE FROM \"Stock_Profile\" WHERE \"Symbol\" = \'MSFT\'; \'
ERROR: column \"msft\"
It's because the single quote before MSFT terminates the string as far as psql is concerned.
As @imsop points out case sensitivity is not preserved when removing double quotes from table names and column names so you can escape the double quotes with backward slash (\) when this is required.
psql -E --host=xxx --port=yyy --username=chi --dbname=C_DB -c "DELETE FROM \"Stock_Profile\" WHERE \"Symbol\" = 'MSFT';"