Methods for using Git with Google Colab

问题

Are there any recommended methods to integrate git with colab?

For example, is it possible to work off code from google source repositories or the likes?

Neither google drive nor cloud storage can be used for git functionality.

So I was wondering if there is a way to still do it?

回答1:

git is installed on the machine, and you can use ! to invoke shell commands.

For example, to clone a git repository:

!git clone https://github.com/fastai/courses.git

Here's a complete example that clones a repository and loads an Excel file stored therein. https://colab.research.google.com/notebook#fileId=1v-yZk-W4YXOxLTLi7bekDw2ZWZXWW216

回答2:

If you want to clone a private repository, the quickest way would be:

!git clone https://username:password@github.com/username/repository.git

回答3:

In order to protect your account username and password, you can use getPass and concatenate them in the shell command:

from getpass import getpass
import os

user = getpass('BitBucket user')
password = getpass('BitBucket password')
os.environ['BITBUCKET_AUTH'] = user + ':' + password

!git clone https://$BITBUCKET_AUTH@bitbucket.org/{user}/repository.git

回答4:

You can use ssh protocol to connect your private repository with colab

Generate ssh key pairs on your local machine, don't forget to keep
the paraphrase empty, check this tutorial.
Upload it to colab, check the following screenshot

from google.colab import files
uploaded = files.upload()
Move the ssh kay pairs to /root and connect to git
- remove previously ssh files
  ! rm -rf /root/.ssh/*
  ! mkdir /root/.ssh
- uncompress your ssh files
  ! tar -xvzf ssh.tar.gz
- copy it to root
  ! cp ssh/* /root/.ssh && rm -rf ssh && rm -rf ssh.tar.gz ! chmod 700 /root/.ssh
- add your git server e.g gitlab as a ssh known host
  ! ssh-keyscan gitlab.com >> /root/.ssh/known_hosts
  ! chmod 644 /root/.ssh/known_hosts
- set your git account
  ! git config --global user.email "email"
  ! git config --global user.name "username"
- finally connect to your git server
  ! ssh git@gitlab.com
Authenticate your private repository, please check this Per-repository deploy keys.
Use ! git@gitlab.com:{account}/{projectName}.git
note: to use push, you have to give write access for
the public ssh key that you authenticate git server with.

回答5:

The very simple and easy way to clone your private github repo in Google colab is as below.

Your password won't be exposed
Though your password contains special character also it works
Just run the below snippet in Colab cell and it will execute in an interactive way

import os
from getpass import getpass
import urllib

user = input('User name: ')
password = getpass('Password: ')
password = urllib.parse.quote(password) # your password is converted into url format
repo_name = input('Repo name: ')

cmd_string = 'git clone https://{0}:{1}@github.com/{0}/{2}.git'.format(user, password, repo_name)

os.system(cmd_string)
cmd_string, password = "", "" # removing the password from the variable

回答6:

You can almost use this link: https://qiita.com/Rowing0914/items/51a770925653c7c528f9

as a summary of the above link you should do this steps:

1- connect your google colab runtime to your Google Drive using this commands:

from google.colab import drive
drive.mount('/content/drive')

It would need a authentication process. Do whatever it needs.

2- Set current directory the path you want to clone the Git project there:

in my example:

path_clone = "drive/My Drive/projects"
!cd path_clone

don't forget to use ! in the beginning of cd command.

3- Clone the Git project:

!git clone <Git project URL address>

now you would have the cloned Git project in projects folder in you Google Drive (which is also connected to your Google Colab runtime machine)

4- Go to your Google Drive (using browser or etc) and then go to the "projects" folder and open the .ipynb file that you want to use in Google Colab.

5- Now you have Google Colab runtime with the .ipynb that you wanted to use which is also connected to your Google Drive and all cloned git files are in the Colab runtime's storage.

Note:

1- Check that your Colab runtime is connected to Google Drive. If it's not connected, just repeat the step #1 above.

2- Double check by using "pwd" and "cd" commands that the current directory is related to the cloned git project in google Drive (step #2 above).

回答7:

Three steps to use git to sync colab with github or gitlab.

Generate a private-public key pair. Copy the private key to the system clibboard for use in step 2. Paste the public key to github or gitlab as appropriate.

In Linux, ssh-keygen can be used to generate the key-pair in ~/.ssh. The resultant private key is in the file id_rsa, the public key is in the file id_rsa.pub.

In Colab, execute

key = \
'''
paste the private key here
'''
! mkdir -p /root/.ssh
with open(r'/root/.ssh/id_rsa', 'w', encoding='utf8') as fh:
    fh.write(key)
! chmod 600 /root/.ssh/id_rsa
! ssh-keyscan github.com >> /root/.ssh/known_hosts

Use git to pull/push as usual.

The same idea can be used for rsync bewteen colab and HostA with minor changes:

Generate a private-public key pair. Copy the private key to the system clibboard for use in step 2. Paste the public key to authorized_keys in .ssh in HostA.

In Colab, execute

key = \
'''
paste the private key here
'''
! mkdir -p /root/.ssh
with open(r'/root/.ssh/id_rsa', 'w', encoding='utf8') as fh:
    fh.write(key)
! chmod 600 /root/.ssh/id_rsa
! ssh -oStrictHostKeyChecking=no root@HostA hostnam  # ssh-keyscan

HostA >> /root/.ssh/known_hosts does not seem to work with IP.

Use rsync to sync files bewtenn colab and HostA as usual.

回答8:

Mount the drive using:

from google.colab import drive
drive.mount('/content/drive/')

Then:

%cd /content/drive/

To clone the repo in your drive

!git clone <github repo url>

Access other files from the repo(example: helper.py is another file in repo):

import imp 
helper = imp.new_module('helper')
exec(open("drive/path/to/helper.py").read(), helper.__dict__)

回答9:

This works if you want to share your repo and colab. Also works if you have multiple repos. Just throw it in a cell.

import ipywidgets as widgets
from IPython.display import display
import subprocess

class credentials_input():
    def __init__(self, repo_name):
        self.repo_name = repo_name
        self.username = widgets.Text(description='Username', value='')
        self.pwd = widgets.Password(description = 'Password', placeholder='password here')

        self.username.on_submit(self.handle_submit_username)
        self.pwd.on_submit(self.handle_submit_pwd)        
        display(self.username)

    def handle_submit_username(self, text):
        display(self.pwd)
        return

    def handle_submit_pwd(self, text):
        cmd = f'git clone https://{self.username.value}:{self.pwd.value}@{self.repo_name}'
        process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
        output, error = process.communicate()
        print(output, error)
        self.username.value, self.pwd.value = '', ''

get_creds = credentials_input('github.com/username/reponame.git')
get_creds

回答10:

The solution https://stackoverflow.com/a/53094151/3924118 did not work for me because the expression {user} was not being converted to the actual username (I was getting a 400 bad request), so I slightly changed that solution to the following one.

from getpass import getpass
import os

os.environ['USER'] = input('Enter the username of your Github account: ')
os.environ['PASSWORD'] = getpass('Enter the password of your Github account: ')
os.environ['REPOSITORY'] = input('Enter the name of the Github repository: ')
os.environ['GITHUB_AUTH'] = os.environ['USER'] + ':' + os.environ['PASSWORD']

!rm -rf $REPOSITORY # To remove the previous clone of the Github repository
!git clone https://$GITHUB_AUTH@github.com/$USER/$REPOSITORY.git

os.environ['USER'] = os.environ['PASSWORD'] = os.environ['REPOSITORY'] = os.environ['GITHUB_AUTH'] = ""

If you are able to clone your-repo, you should not see any password in the output of this command. If you get an error, the password could be displayed to the output, so make sure you do not share your notebook whenever this command fails.

来源：https://stackoverflow.com/questions/48350226/methods-for-using-git-with-google-colab

标签

google-colaboratory