I have GitLab CE setup with my domain (Ubuntu) and the git clone with http not working but ssh works well.

git clone git@gitlab.mydomain.com:anshad/test.git works.

git clone http://gitlab.mydomain.com/anshad/test.git not working.

Cloning into 'test'... Username for 'http://gitlab.mydomain.com': anshad Password for 'http://anshad@gitlab.mydomain.com':  fatal: unable to access 'http://gitlab.mydomain.com/anshad/test.git/': The requested URL returned error: 500 

sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production says

System information System:     Ubuntu 16.04 Current User:   git Using RVM:  no Ruby Version:   2.3.1p112 Gem Version:    2.5.1 Bundler Version:1.13.6 Rake Version:   10.5.0 Sidekiq Version:4.2.1  GitLab information Version:    8.14.0-pre Revision:   358e814 Directory:  /home/git/gitlab DB Adapter: postgresql URL:        http://gitlab.mydomain.com HTTP Clone URL: http://gitlab.mydomain.com/some-group/some-project.git SSH Clone URL:  git@gitlab.mydomain.com:some-group/some-project.git Using LDAP: no Using Omniauth: no  GitLab Shell Version:    4.0.0 Repository storage paths: - default:  /home/git/repositories/ Hooks:      /home/git/gitlab-shell/hooks/ Git:        /usr/bin/git 

sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production says

Checking GitLab Shell ...  GitLab Shell version >= 4.0.0 ? ... OK (4.0.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ...  Anshad Vattapoyil / test ... repository is empty Running /home/git/gitlab-shell/bin/check Check GitLab API access: OK Access to /home/git/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful  Checking GitLab Shell ... Finished  Checking Sidekiq ...  Running? ... yes Number of Sidekiq processes ... 1  Checking Sidekiq ... Finished  Checking Reply by email ...  Reply by email is disabled in config/gitlab.yml  Checking Reply by email ... Finished  Checking LDAP ...  LDAP is disabled in config/gitlab.yml  Checking LDAP ... Finished  Checking GitLab ...  Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... yes Init script exists? ... yes Init script up-to-date? ... yes projects have namespace: ...  Anshad Vattapoyil / test ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.1) Your git bin path is "/usr/bin/git" Git version >= 2.7.3 ? ... yes (2.7.4) Active users: 1  Checking GitLab ... Finished 

Apache virtual host configuration for the domain,

<VirtualHost *:80>     ServerAdmin anshad@mydomain.com     ServerName gitlab.mydomain.com     ServerAlias www.gitlab.mydomain.com      ServerSignature Off      ProxyPreserveHost On      AllowEncodedSlashes NoDecode      <Location />        Require all granted        ProxyPassReverse http://localhost:8080        ProxyPassReverse http://gitlab.mydomain.com/     </Location>      RewriteEngine on      RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]     RewriteCond %{REQUEST_URI} ^/uploads/.*     RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA,NE]      DocumentRoot /home/git/gitlab/public      LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded     ErrorLog /var/log/apache2/gitlab_error.log     CustomLog /var/log/apache2/gitlab_forwarded.log common_forwarded     CustomLog /var/log/apache2/gitlab_access.log combined env=!dontlog     CustomLog /var/log/apache2/gitlab.log combined  </VirtualHost> 

Config /home/git/gitlab/config/gitlab.yml

production: &base    gitlab:     host: gitlab.mydomain.com     port: 80      https: false      trusted_proxies:      email_from: noreply@mydomain.com     email_display_name: GitLab     email_reply_to: noreply@mydomain.com     email_subject_suffix: ''      default_projects_features:       issues: true       merge_requests: true       wiki: true       snippets: true       builds: true       container_registry: true    incoming_email:     enabled: false     address: "gitlab-incoming+%{key}@gmail.com"     user: "gitlab-incoming@gmail.com"     password: "[REDACTED]"     host: "imap.gmail.com"     port: 993     ssl: true     start_tls: false     mailbox: "inbox"    artifacts:     enabled: true   lfs:     enabled: true    gravatar:    cron_jobs:     stuck_ci_builds_worker:       cron: "0 0 * * *"     expire_build_artifacts_worker:       cron: "50 * * * *"     repository_check_worker:       cron: "20 * * * *"     admin_email_worker:       cron: "0 0 * * 0"     repository_archive_cache_worker:       cron: "0 * * * *"    registry:     gitlab_ci:    ldap:     enabled: false     servers:       main:          label: 'LDAP'         host: '_your_ldap_server'         port: 389         uid: 'sAMAccountName'         method: 'plain' # "tls" or "ssl" or "plain"         bind_dn: '_the_full_dn_of_the_user_you_will_bind_with'         password: '_the_password_of_the_bind_user'         timeout: 10         active_directory: true         allow_username_or_email_login: false         block_auto_created_users: false         base: ''         user_filter: ''         attributes:           username: ['uid', 'userid', 'sAMAccountName']           email:    ['mail', 'email', 'userPrincipalName']           name:       'cn'           first_name: 'givenName'           last_name:  'sn'    omniauth:     enabled: false     allow_single_sign_on: ["saml"]     block_auto_created_users: true     auto_link_ldap_user: false     auto_link_saml_user: false     external_providers: []     providers:    shared:    satellites:     path: /home/git/gitlab-satellites/    repositories:     storages: # You must have at least a `default` storage path.       default: /home/git/repositories/   backup:     path: "tmp/backups"      gitlab_shell:     path: /home/git/gitlab-shell/     hooks_path: /home/git/gitlab-shell/hooks/     upload_pack: true     receive_pack: true   git:     bin_path: /usr/bin/git     max_size: 20971520 # 20.megabytes     timeout: 10    extra:    rack_attack:     git_basic_auth: 

In /home/git/gitlab/unicorn.rb, I have

worker_processes 4 working_directory "/home/git/gitlab" # available in 0.94.0+ listen "/home/git/gitlab/tmp/sockets/gitlab.socket", :backlog => 1024 listen "127.0.0.1:8080", :tcp_nopush => true 

The production.log says only this

Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:45 +0000 Started GET "/anshad/test.git/info/refs?service=git-upload-pack" for 137.97.204.60 at 2016-11-19 08:16:47 +0000 Processing by Projects::GitHttpController#info_refs as */*   Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"anshad", "project_id"=>"test.git"} Filter chain halted as :authenticate_user rendered or redirected Completed 401 Unauthorized in 23ms (Views: 0.3ms | ActiveRecord: 1.6ms) Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:48 +0000 Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:51 +0000 Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:54 +0000 Started GET "/anshad/test.git/info/refs?service=git-upload-pack" for 137.97.204.60 at 2016-11-19 08:16:56 +0000 Processing by Projects::GitHttpController#info_refs as */*   Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"anshad", "project_id"=>"test.git"} Filter chain halted as :authenticate_user rendered or redirected Completed 401 Unauthorized in 25ms (Views: 0.3ms | ActiveRecord: 2.2ms) Started GET "/anshad/test.git/info/refs?service=git-upload-pack" for 137.97.204.60 at 2016-11-19 08:16:57 +0000 Processing by Projects::GitHttpController#info_refs as */*   Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"anshad", "project_id"=>"test.git"} Completed 500 Internal Server Error in 98ms (ActiveRecord: 3.1ms)  JWT::DecodeError (Nil JSON web token):   lib/gitlab/workhorse.rb:120:in `verify_api_request!'   app/controllers/projects/git_http_client_controller.rb:154:in `verify_workhorse_api!'   lib/gitlab/request_profiler/middleware.rb:15:in `call'   lib/gitlab/middleware/go.rb:16:in `call'   Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:57 +0000 

Issue on GitLab

It's important to note that this system referenced in question was built from source code and supported nginx was replaced with Apache (not officially supported by gitlab).
Here is the deal - in the standard nginx config on my system I can see this

upstream gitlab-workhorse {   server unix:/var/opt/gitlab/gitlab-workhorse/socket; }  proxy_pass http://gitlab-workhorse; 

Which means - it's using socket. Not a network port. If I try to see if the workhorse even listening on network - I will see that it's not.

ps -ef|grep -i workhorse lsof -p pid 

Would not show any network ports open by workhorse pid. So perhaps apache config is incorrect? It should be using socket instead of port?

First, double-check your gitlab workhorse version and if it is compatible with your current GitLab installation.

Of all the GitLab issues you reference, the comments on 22484 seem the most promising:

In my case, workhorse's logs showed an error accessing ./.gitlab_workhorse_secret

After some digging, the fix was to add the following to the workhorse startup command line in /etc/systemd/system/gitlab-workhorse.service:

-secretPath /home/git/gitlab/.gitlab_workhorse_secret  

For reference, the full ExecStart is now:

ExecStart=/home/git/gitlab/bin/daemon_with_pidfile /home/git/gitlab/tmp/pids/gitlab-workhorse.pid \ /home/git/gitlab-workhorse/gitlab-workhorse -listenUmask 0 -listenNetwork unix \ -listenAddr /home/git/gitlab/tmp/sockets/gitlab-workhorse.socket \ -authBackend http://127.0.0.1:8080 -authSocket /home/git/gitlab/tmp/sockets/gitlab.socket \ -documentRoot /home/git/gitlab/public -secretPath /home/git/gitlab/.gitlab_workhorse_secret \ >> /home/git/gitlab/log/gitlab-workhorse.log 2>&1 

The other possibility is:

In my case 500 error was caused by bad nginx configuration in /etc/gitlab/gitlab.rb.

In case where I had something "before" the nginx, like in my case haproxy. I overlooked this fact. It is described in NGiNX settings.

In my case in haproxy sends backend to 8081 where is listening nginx now (originally I setted 8080 - default unicorn service) -
I was not able configure gitlab only with haproxy, without nginx layer.

So in my configuration was important

nginx['listen_port'] = 8081 nginx['listen_https'] = false 

Note that both issues are for NGiNX (there is one when Apache2 is used)

There is also a mention about 403 (permission denied) errors:

We were able to resolve the 403 issue by enabling both HTTPS and SSH cloning; we only had SSH cloning enabled which seemed to be causing the problem. This can be changed by going to https:///admin/application_settings and double checking Enabled Git access protocols

Those conclusions are summarized in merge request 6843

But there is more:

Looking at the default files, it looks like there is some sort of confusion with upgrades and what the defaults should be.
With the default configuration file examples (init.d and nginx), gitlab-workhorse will listen on a Unix socket and not an IP:port.
The Nginx example config file does have some lines for Unix sockets, but the proxy pass goes to an address.

I thought setting gitlab up for the first time I needed point my nginx config to the Unicorn port bind as it was the only port I was seeing in netstat get setup when I started the gitlab services.
If you send the git clone request to Unicorn you will get the 500 error.
What I had to do is change gitlab-workhorse to listen to my lookback address and point Nginx there. That cleared up my HTTP 500 error with cloning.
See more with A Brief History of GitLab Workhorse