I actually kind of like the GIL. It allows me to write threaded code in a "lazy" way sometimes, when I don't want to take the effort to do it properly. (e.g. implementing a message queue using a simple list, using a global bool or int as a state flag.) Even though it's not using the correct structures for these things, it sometimes allows to prototype a simple threaded approach quickly, and at least be sure it won't cause weird clashes in the interpreter. When I want to later port it to a proper solution, I'll switch to the multiprocess model which forces you to use queues and other IPC mechanisms instead of shared data structures-- this is more effort, but usually worth it, if not for efficiency then for robustness. But for one-off scripts and little proofs of concept it can be nice to just carefully share some variables and know with some assurance that you're not going to break anything in the run-time. That's not to say that transactional memory wouldn't be welcome, but sometimes the GIL allows a certain pragmatism in a prototyping scenario.
In Python, threads are "enough" to implement event-based concurrency, as well as asynchronous I/O and calling out to long-running C functions for number crunching.
Beyond that, the main reason to remove the GIL and allow true multicore parallelism is purely for efficiency reasons. The catch is, if I'm serious about efficiency, I probably won't be using Python. Or at least porting only my CPU-intensive functions to C.
Of course, if PyPy eventually provides Java-like speeds, maybe it makes sense to go this route, as Python could start to be depended on for efficiency, so the gains of true parallelism would be visible. But removing the GIL from CPython seems kind of pointless, since the interpreter has so much overhead regardless.
> I actually kind of like the GIL. It allows me to write threaded code in a "lazy" way sometimes
As far as I understand, this is exactly the reason why the PyPy team went for the STM route: they want to guarantee the same semantics of CPython with the GIL, but with more parallelism.
The idea here is that most of the time threads do not conflict, which means that they do not mutate objects while other threads are accessing them. Based on this assumption, you can optimistically run a snippet of code assuming that no conflict will happen, but if there is such a conflict all the effects of the code (the "transaction") are rolled back and the code is run again.
Since the interpreter can decide where to put the transaction boundaries, it can do it where CPython would acquire/release the GIL. This would give basically GIL-equivalent semantics.
you're getting lucky. as I understand it, the GIL just protects certain pieces of interpreter state. you still need to lock to get correct behavior from your code. the presence of the GIL doesn't remove the need to lock your data structures and each thread created with the threading stdlib is actually a native thread on your platform.
Yep. The GIL means that individual Python bytecode operations are atomic, and so by extension single Python statements that translate to individual bytecode instructions can be treated as atomic (if you don't care about portability to alternate Python implementations that lack a GIL). But complicated Python statements that translate into multiple bytecode instructions (like dict.setdefault) can't. And, of course, neither can sequences of multiple statements. So you still need locks for most cases where you mutate objects that are shared across threads.
Ah yes, that's a good point. I wouldn't go as far as to say I'm just getting lucky though, as I do tend to reserve my lazy bad practices to very simple cases. But I would never use such code in production anyways.
In Python, threads are "enough" to implement event-based concurrency, as well as asynchronous I/O and calling out to long-running C functions for number crunching.
Beyond that, the main reason to remove the GIL and allow true multicore parallelism is purely for efficiency reasons. The catch is, if I'm serious about efficiency, I probably won't be using Python. Or at least porting only my CPU-intensive functions to C.
Of course, if PyPy eventually provides Java-like speeds, maybe it makes sense to go this route, as Python could start to be depended on for efficiency, so the gains of true parallelism would be visible. But removing the GIL from CPython seems kind of pointless, since the interpreter has so much overhead regardless.