Methods for using Git with Google Colab

Question

Are there any recommended methods to integrate git with colab?

For example, is it possible to work off code from google source repositories or the likes?

Nei

Answer 1

This works if you want to share your repo and colab. Also works if you have multiple repos. Just throw it in a cell.

import ipywidgets as widgets
from IPython.display import display
import subprocess

class credentials_input():
    def __init__(self, repo_name):
        self.repo_name = repo_name
        self.username = widgets.Text(description='Username', value='')
        self.pwd = widgets.Password(description = 'Password', placeholder='password here')

        self.username.on_submit(self.handle_submit_username)
        self.pwd.on_submit(self.handle_submit_pwd)        
        display(self.username)

    def handle_submit_username(self, text):
        display(self.pwd)
        return

    def handle_submit_pwd(self, text):
        cmd = f'git clone https://{self.username.value}:{self.pwd.value}@{self.repo_name}'
        process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
        output, error = process.communicate()
        print(output, error)
        self.username.value, self.pwd.value = '', ''

get_creds = credentials_input('github.com/username/reponame.git')
get_creds

Answer 2

Cloning a private repo to google colab :

Generate a token:

Settings -> Developer settings -> Personal access tokens -> Generate new token

Copy the token and clone the repo (replace username and token accordingly)

!git clone https://username:token@github.com/username/repo_name.git

Answer 3

git is installed on the machine, and you can use ! to invoke shell commands.

For example, to clone a git repository:

!git clone https://github.com/fastai/courses.git

Here's a complete example that clones a repository and loads an Excel file stored therein. https://colab.research.google.com/notebook#fileId=1v-yZk-W4YXOxLTLi7bekDw2ZWZXWW216

Answer 4

You can use ssh protocol to connect your private repository with colab

1. Generate ssh key pairs on your local machine, don't forget to keep
   the paraphrase empty, check this tutorial.

2. Upload it to colab, check the following screenshot

   from google.colab import files
uploaded = files.upload()

3. Move the ssh kay pairs to /root and connect to git

   • remove previously ssh files
     ! rm -rf /root/.ssh/*
! mkdir /root/.ssh
   • uncompress your ssh files
     ! tar -xvzf ssh.tar.gz
   • copy it to root
     ! cp ssh/* /root/.ssh && rm -rf ssh && rm -rf ssh.tar.gz ! chmod 700 /root/.ssh
   • add your git server e.g gitlab as a ssh known host
     ! ssh-keyscan gitlab.com >> /root/.ssh/known_hosts
! chmod 644 /root/.ssh/known_hosts
   • set your git account
     ! git config --global user.email "email"
! git config --global user.name "username"
   • finally connect to your git server
     ! ssh git@gitlab.com

4. Authenticate your private repository, please check this Per-repository deploy keys.

5. Use ! git@gitlab.com:{account}/{projectName}.git
   note: to use push, you have to give write access for
   the public ssh key that you authenticate git server with.

Answer 5

Three steps to use git to sync colab with github or gitlab.

1. Generate a private-public key pair. Copy the private key to the system clibboard for use in step 2. Paste the public key to github or gitlab as appropriate.

   In Linux, ssh-keygen can be used to generate the key-pair in ~/.ssh. The resultant private key is in the file id_rsa, the public key is in the file id_rsa.pub.

2. In Colab, execute

   key = \
   '''
   paste the private key here
   '''
   ! mkdir -p /root/.ssh
   with open(r'/root/.ssh/id_rsa', 'w', encoding='utf8') as fh:
       fh.write(key)
   ! chmod 600 /root/.ssh/id_rsa
   ! ssh-keyscan github.com >> /root/.ssh/known_hosts 
   

3. Use git to pull/push as usual.

The same idea can be used for rsync bewteen colab and HostA with minor changes:

1. Generate a private-public key pair. Copy the private key to the system clibboard for use in step 2. Paste the public key to authorized_keys in .ssh in HostA.

2. In Colab, execute

   key = \
   '''
   paste the private key here
   '''
   ! mkdir -p /root/.ssh
   with open(r'/root/.ssh/id_rsa', 'w', encoding='utf8') as fh:
       fh.write(key)
   ! chmod 600 /root/.ssh/id_rsa
   ! ssh -oStrictHostKeyChecking=no root@HostA hostnam  # ssh-keyscan 
   

HostA >> /root/.ssh/known_hosts does not seem to work with IP.

3. Use rsync to sync files bewtenn colab and HostA as usual.

Answer 6

The very simple and easy way to clone your private github repo in Google colab is as below.

1. Your password won't be exposed
2. Though your password contains special character also it works
3. Just run the below snippet in Colab cell and it will execute in an interactive way

import os
from getpass import getpass
import urllib

user = input('User name: ')
password = getpass('Password: ')
password = urllib.parse.quote(password) # your password is converted into url format
repo_name = input('Repo name: ')

cmd_string = 'git clone https://{0}:{1}@github.com/{0}/{2}.git'.format(user, password, repo_name)

os.system(cmd_string)
cmd_string, password = "", "" # removing the password from the variable