gil

multi-threading in python: is it really performance effiicient most of the time?

由 a 夏天 提交于 2020-01-13 20:20:20
问题 In my little understanding, it is the performance factor that drives programming for multi-threading in most cases but not all. (irrespective of Java or Python). I was reading this enlightening article on GIL in SO. The article summarizes that python adopts GIL mechanism; i.e only a single Thread can execute python byte code at any given time. This makes single thread application really faster. My question is as follows: Since if only one Thread is served at a given point, does

GIL锁 验证Cpython的并发效率 GIL与互斥锁的关系 进程池,线程池

由 夙愿已清 提交于 2020-01-11 16:09:39
一丶GIL锁 什么是GIL锁: 存在Cpython解释器,全名:全局解释器锁.(解释器级别的锁) GIL是一把互斥锁,将并发运行变成串行. 在同一个进程下开启的多个线程,同时只能有一个线程执行,无法利用多核优势 GIL锁的作用: 保证同一时间内,共享数据只能被一个任务修改.保证数据的完整性和安全性 自动上锁和解锁,不需要人为的添加.减轻开发人员的负担 所谓诟病: 单进程的多线程不能利用多核 通常有人会认为GIL锁不能利用多核处理任务,是Python语言的诟病.个人认为任何一门语言都不是完美的,python问世于20世纪末,当时的CPU也只是单核. So,不能利用多核的原因是:同一个进程下,只允许一个线程对共享内容进行修改.不允许多线程同时处理共享数据. (打个比方:现在你家有10升(共享数据:10)牛掰的牛奶,你免费送给大家每人1升. 现在有100个人(100个线程)知道了这件事,都上你家来取奶.肯定谁先来,先得. 现在来了10个人(10个线程),每个人都说我去了1升,还剩9升.你真的还剩9升吗? 毛线~~ ,你球都不剩了. 后面来的90个人还能拿到奶吗? 肯定拿拿不到了. So你是不是得上锁,比如你家大门就是一把锁.每次只允许进来(上锁)1个人去奶.取完之后把总的奶量 减一. 第一个取完了出了大门(解锁).剩余的99个人开始抢着进你家门,公平竞争,谁先到你家门

A question on python GIL

由 这一生的挚爱 提交于 2020-01-10 04:22:04
问题 Does the presence of python GIL imply that in python multi threading the same operation is not so different from repeating it in a single thread?. For example, If I need to upload two files, what is the advantage of doing them in two threads instead of uploading them one after another?. I tried a big math operation in both ways. But they seem to take almost equal time to complete. This seems to be unclear to me. Can someone help me on this?. Thanks. 回答1: Python's threads get a slightly worse

400 threads in 20 processes outperform 400 threads in 4 processes while performing an I/O-bound task

由 谁说胖子不能爱 提交于 2020-01-09 10:29:11
问题 Experimental Code Here is the experimental code that can launch a specified number of worker processes and then launch a specified number of worker threads within each process and perform the task of fetching URLs: import multiprocessing import sys import time import threading import urllib.request def main(): processes = int(sys.argv[1]) threads = int(sys.argv[2]) urls = int(sys.argv[3]) # Start process workers. in_q = multiprocessing.Queue() process_workers = [] for _ in range(processes): w

Python GIL and globals

由 跟風遠走 提交于 2020-01-03 15:58:55
问题 In python, I have a global variable defined that gets read/incremented by different threads. Because of the GIL, will this ever cause problems without using any kind of locking mechanism? 回答1: The GIL only requires that the interpreter completely executes a single bytecode instruction before another thread can take over. However, there is no reason to assume that an increment operation is a single instruction. For example: >>> import dis >>> dis.dis(compile("x=753","","exec")) 1 0 LOAD_CONST

Does using the subprocess module release the python GIL?

由 …衆ロ難τιáo~ 提交于 2020-01-02 01:09:30
问题 When calling a linux binary which takes a relatively long time through Python's subprocess module, does this release the GIL? I want to parallelise some code which calls a binary program from the command line. Is it better to use threads (through threading and a multiprocessing.pool.ThreadPool ) or multiprocessing ? My assumption is that if subprocess releases the GIL then choosing the threading option is better. 回答1: When calling a linux binary which takes a relatively long time through

python Global Interpreter Lock GIL problem

由 泄露秘密 提交于 2019-12-31 02:29:20
问题 I want to provide a service on the web that people can test out the performance of an algo, which is written in python and running on the linux machine basically what I want to do is that, there is a very trivial PHP handler, let's say start_algo.php, which accepts the request coming from browser, and in the php code through system() or popen() (something like exec( "python algo.py" ) ) to issue a new process running the python script, I think it is doable in this part problem is that since

并发编程-线程-死锁现象-GIL全局锁-线程池

由 孤者浪人 提交于 2019-12-27 05:39:18
一堆锁 死锁现象 (重点) 死锁指的是某个资源被占用后,一直得不到释放,导致其他需要这个资源的线程进入阻塞状态. 产生死锁的情况 对同一把互斥锁加了多次 一个共享资源,要访问必须同时具备多把锁,但是这些锁被不同线程或者进程所持有,就会导致相互等待对方释放从而程序就卡死了 第二种情况的解决方法: 抢锁一定按照相同的顺序去抢 给抢锁加上超时,如果超时则放弃执行 递归锁 (了解) 与普通的区别 相同: 多线程之间都有互斥的效果 不同: 同一个线程可以对这个锁执行多次acquire 解决方法 同一个线程必须保证,加锁的次数和解锁的次数相同,其他线程才能够抢到这把锁 信号量 (了解) 可以限制同时并执行公共代码的线程数量 如果限制数量为1,则与普通互斥锁没有区别(默认为1) from threading import Semaphore,current_thread,Thread import time s = Semaphore(2) def task(): s.acquire() time.sleep(1) print(current_thread().name) s.release() for i in range(10): Thread(target=task).start() # 结果是每次都会执行两个子线程 GIL全局锁 (重点) 什么是GIL锁? 在cpython中

cython memoryviews slices without GIL

由 こ雲淡風輕ζ 提交于 2019-12-25 12:40:09
问题 I want to release the GIL in order to parallelise loop in cython, where different slices of memoryviews are passed to a some function inside the loop. The code looks like this: cpdef void do_sth_in_parallel(bint[:,:] input, bint[:] output, int D): for d in prange(D, schedule=dynamic, nogil=True): ouput[d] = some_function_not_requiring_gil(x[d,:]) This is not possible, since selecting the slice x[d,:], seems to require GIL. Running cython -a , and using a normal for loop, I get the code posted

Usage of threadpoolexecutor in conjunction with cython's nogil

由 耗尽温柔 提交于 2019-12-25 01:55:11
问题 I have read this question and answer -Cython nogil with ThreadPoolExecutor not giving speedups and I have a similar problem with my Cython code not getting the speedup that is expected in spite of my system having multiple cores. I have 4 physical cores on a Ubuntu 18.04 instance and if I make the number of jobs to be 1 in the code below it runs faster than when I make it 4. Looking at the CPU usage using top I see the CPU usage go upto 300 %. I am doing the lookup of a data structure in a C+
订阅 gil