1. 技术文章
  2. gitlab上加速运行Django项目测试代码

gitlab上加速运行Django项目测试代码

由 戏子无情 提交于 2020-03-10 10:46:20

项目环境：

  • python 3.6
  • Django 1.11
  • Coverage 5.0

减少测试代码运行时间的方法：

  • 把单一的管道改为多管道并行，最后合并测试结果
  • 减少数据库迁移的次数

具体做法：
1、把测试代码拆分成四部分，分别放在不同的文件夹
配置文件.gitlab.yml参考：

stages:
  - test
  - result



Part_one:
  stage: test
  script:
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi
   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate
   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ -r $CI_PROJECT_DIR/requirements.txt
   - cd $CI_PROJECT_DIR
   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi
   - touch $CI_PROJECT_DIR/logs/request.log
   - touch $CI_PROJECT_DIR/logs/biotools.log
   - echo $CI_JOB_ID >job_id.txt
   - python manage.py runscript check_test_database_migrations --traceback  --settings=base_settings
   - coverage run -p manage.py test app.tests.part_one --noinput --settings=base_settings --debug-mode  --keepdb
   - python manage.py runscript rename_database --traceback  --settings=base_settings
   - python manage.py runscript remove_testdb --traceback  --settings=base_settings
  retry: 2
  artifacts:
    untracked: true
    name: "$CI_JOB_NAME"
    expire_in: 1 week


Part_two:
  stage: test
  script:
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi
   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate
   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ -r $CI_PROJECT_DIR/requirements.txt
   - cd $CI_PROJECT_DIR
   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi
   - touch $CI_PROJECT_DIR/logs/request.log
   - touch $CI_PROJECT_DIR/logs/biotools.log
   - echo $CI_JOB_ID >job_id.txt
   - python manage.py runscript check_test_database_migrations --traceback  --settings=base_settings
   - coverage run -p manage.py test app.tests.part_two --noinput --settings=base_settings --debug-mode  --keepdb
   - python manage.py runscript remove_testdb --traceback  --settings=base_settings
  retry: 2
  artifacts:
    untracked: true
    name: "$CI_JOB_NAME"
    expire_in: 1 week


Part_three:
  stage: test
  script:
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi
   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate
   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ -r $CI_PROJECT_DIR/requirements.txt
   - cd $CI_PROJECT_DIR
   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi
   - touch $CI_PROJECT_DIR/logs/request.log
   - touch $CI_PROJECT_DIR/logs/biotools.log
   - echo $CI_JOB_ID >job_id.txt
   - python manage.py runscript check_test_database_migrations --traceback  --settings=base_settings
   - coverage run -p manage.py test app.tests.part_three --noinput --settings=base_settings --debug-mode  --keepdb
   - python manage.py runscript remove_testdb --traceback  --settings=base_settings
  retry: 2
  artifacts:
    untracked: true
    name: "$CI_JOB_NAME"
    expire_in: 1 week


Part_four:
  stage: test
  script:
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi
   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate
   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ -r $CI_PROJECT_DIR/requirements.txt
   - cd $CI_PROJECT_DIR
   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi
   - touch $CI_PROJECT_DIR/logs/request.log
   - touch $CI_PROJECT_DIR/logs/biotools.log
   - echo $CI_JOB_ID >job_id.txt
   - python manage.py runscript check_test_database_migrations --traceback  --settings=base_settings
   - coverage run -p manage.py test app.tests.part_four --noinput --settings=base_settings --debug-mode  --keepdb
   - python manage.py runscript remove_testdb --traceback  --settings=base_settings
  retry: 2
  artifacts:
    untracked: true
    name: "$CI_JOB_NAME"
    expire_in: 1 week


result:
  stage: result
  script:
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi
   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi
   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate
   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ coverage
   - cd $CI_PROJECT_DIR
   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi
   - touch $CI_PROJECT_DIR/logs/request.log
   - touch $CI_PROJECT_DIR/logs/biotools.log
   - coverage combine --append
   - coverage report
  dependencies:
   - Part_one
   - Part_two
   - Part_three
   - Part_four

减少数据库迁移的次数的具体做法是把测试用到的sqlite数据库文件放在/dev/shm/目录下，每条管道运行测试代码之前，先在/dev/shm/目录下用job id创建一个文件夹，job id是每个job运行时唯一的id，可以查阅gitlab的官方文档了解更多，把sqlite数据库复制一份到新建的文件夹里面，再运行测试代码。运行测试代码的语句是：

 coverage run -p manage.py test app.tests.part_one --noinput --settings=base_settings --debug-mode  --keepdb

注意一定要加上–keepdb

还有其他细节，例如检查数据库有没有更新，测试完成后删除测试用到的数据库等，我全都粘贴出来。
check_test_database_migrations.py

# -*- coding: utf-8 -*-
import logging
import os
import sqlite3
import sys
import base_settings
import shutil
import os
import stat
from django.conf import settings

__author__ = 'JayChen'


logger = logging.getLogger('scripts')


def read_text():
    """
    读取文件中的job_id
    :return:
    """
    with open(settings.PROJECT_PATH + '/job_id.txt', 'r') as f:
        for line in f.readlines():
            job_id = line.strip()
    return job_id


def remove_db_files():
    logger.info("model有更新，删除本地数据库")
    for root, dirs, files in os.walk('/dev/shm', topdown=False):
        for name in files:
            os.remove(os.path.join(root, name))
        for name in dirs:
            os.rmdir(os.path.join(root, name))
    job_id = read_text()
    os.mkdir("/dev/shm/test_{}".format(job_id))
    if os.path.exists("/dev/shm/test_{}".format(job_id)):
        logger.info('文件夹-------' + "/dev/shm/test_{}".format(job_id) + '------创建成功')


def coppy_and_rename():
    job_id = read_text()
    new_file_name = '/dev/shm/test_{}/vb_en.test_{}.db.sqlite3'.format(job_id, job_id)  # 文件新名字
    origin_path = '/dev/shm/vb_en.test.db.sqlite3'  # 原始文件完整目录

    # 数据库文件存在
    if os.path.exists(origin_path):
        if os.path.exists("/dev/shm/test_{}".format(job_id)):
            logger.info('文件夹-------' + "/dev/shm/test_{}".format(job_id) + '------创建成功')
            shutil.copy(origin_path, new_file_name)
            os.chmod(new_file_name, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
            logger.info('复制数据库并重新命名为:--------' + new_file_name.split('/')[4] + '------------')
        else:
            os.mkdir("/dev/shm/test_{}".format(job_id))
            logger.info('文件夹-------' + "/dev/shm/test_{}".format(job_id) + '------创建成功')
            shutil.copy(origin_path, new_file_name)
            os.chmod(new_file_name, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
            logger.info('复制数据库并重新命名为:--------' + new_file_name.split('/')[4] + '------------')
    else:
        logger.info('找不到数据库源文件')
        if os.path.exists("/dev/shm/test_{}".format(job_id)):
            for root, dirs, files in os.walk("/dev/shm/test_{}".format(job_id), topdown=False):
                for name in files:
                    os.remove(os.path.join(root, name))
                for name in dirs:
                    os.rmdir(os.path.join(root, name))
        else:
            os.mkdir("/dev/shm/test_{}".format(job_id))
            logger.info('文件夹-------' + "/dev/shm/test_{}".format(job_id) + '------创建成功')


    for i, j, k in os.walk('/dev/shm'):
        print(i, j, k)


def check_migrations():
    base_dir = os.path.dirname(__file__)
    migrations_data = []
    if os.path.exists('/dev/shm/vb_en.test.db.sqlite3'):
        try:
            conn = sqlite3.connect('/dev/shm/vb_en.test.db.sqlite3')
            migrations_data = conn.cursor().execute("select name from django_migrations where app='app'").fetchall()
            conn.close()
            data_length = len(migrations_data)
            migrations_files = []
            for f in os.listdir(base_settings.PROJECT_PATH + '/app/migrations'):
                if f.endswith('.py') and f != '__init__.py':
                    migrations_files.append(f)

            files_length = len(migrations_files)

            if data_length != files_length:
                remove_db_files()
            else:
                for m in migrations_data:
                    if m[0] + '.py' not in migrations_files:
                        remove_db_files()
        except sqlite3.OperationalError:
            logger.info('读取本地数据库失败')
        coppy_and_rename()
    else:
        coppy_and_rename()


def run(*args):
    logger.info('Start checking the test database migrations file.')
    logger.info(args)
    check_migrations()
    logger.info('End checking.')


'''
./manage.py runscript check_test_database_migrations --traceback  --settings=base_settings  --script-args=one
'''

remove_testdb.py

# -*- coding: utf-8 -*-
import os
import logging
from django.conf import settings

__author__ = 'JayChen'

logger = logging.getLogger('scripts')


def read_text():
    """
    读取文件中的job_id
    :return:
    """
    with open(settings.PROJECT_PATH + '/job_id.txt', 'r') as f:
        for line in f.readlines():
            job_id = line.strip()
    return job_id


def remove_db():
    # 清空文件夹里面的文件，再删除文件夹
    job_id = read_text()
    for root, dirs, files in os.walk('/dev/shm/test_{}'.format(job_id), topdown=False):
        for name in files:
            os.remove(os.path.join(root, name))
        for name in dirs:
            os.rmdir(os.path.join(root, name))
    os.rmdir("/dev/shm/test_{}".format(job_id))


def remove_db_history():
    """
    job_id 是从小排到大的，只保留最近的20个文件夹
    清理gitlab上运行失败后留下的数据库文件夹和文件
    """
    job_id = read_text()
    job_id = int(round(float(job_id) - 20, 0))
    filePath = '/dev/shm'
    name_list = os.listdir(filePath)
    for name in name_list:
        if os.path.isfile('/dev/shm/' + name):
            """
            如果是文件而不是文件夹，则跳过，因为这个文件就是我用于复制到其他文件夹的数据库
            """
            continue
        else:
            data = int(round(float(name.split('_')[1]), 0))
            if data < job_id:
                for root, dirs, files in os.walk('/dev/shm/{}'.format(name), topdown=False):
                    for file in files:
                        os.remove(os.path.join(root, file))
                    for dir in dirs:
                        os.rmdir(os.path.join(root, dir))
                os.rmdir("/dev/shm/{}".format(name))
            else:
                continue


def run():
    logger.info('删除测试数据库')
    remove_db()
    remove_db_history()


'''
./manage.py runscript remove_testdb --traceback  --settings=base_settings
'''

rename_database.py

# -*- coding: utf-8 -*-
import os
import logging
import shutil
from django.conf import settings

__author__ = 'JayChen'

logger = logging.getLogger('scripts')


def read_text():
    """
    读取文件中的job_id
    :return:
    """
    with open(settings.PROJECT_PATH + '/job_id.txt', 'r') as f:
        for line in f.readlines():
            job_id = line.strip()
    return job_id


def move_and_rename():
    job_id = read_text()
    origin_path = '/dev/shm/test_{}/vb_en.test_{}.db.sqlite3'.format(job_id, job_id)
    new_file_name = '/dev/shm/vb_en.test.db.sqlite3'
    if os.path.exists(origin_path):
        shutil.copy(origin_path, new_file_name)
        logger.info('更新数据库文件')
    else:
        pass


def run():
    move_and_rename()


'''
./manage.py runscript rename_database --traceback  --settings=base_settings
'''

settings.py文件最后加上：

if os.path.exists(PROJECT_PATH + '/job_id.txt'):
    # 读取文件中的job_id, gitlab上每条管道的job_id, 这个id是唯一的
    with open(PROJECT_PATH + '/job_id.txt', 'r') as f:
        for line in f.readlines():
            job_id = line.strip()
    TEST_SQLITE_PREFIX = '/dev/shm/test_{}/vb_en.test_{}.db'.format(job_id, job_id)
else:
    TEST_SQLITE_PREFIX = 'vb_en_testdb'
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': TEST_SQLITE_PREFIX + '.sqlite3',
        'TEST': {
            'NAME': TEST_SQLITE_PREFIX + '.sqlite3'
        },
        'OPTIONS': {
            'timeout': 20,
        }
    }
}
标签
test
数据库文件
django
gitlab
dev
sqlite3
python
ci
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!