There should be one – and preferably only one – obvious way to do it.
— Tim Peters in The Zen of Python
There are multiple ways to manage resources with Python, but only one of them is save, reliable and Pythonic.
Before we dive in, let's examine what resources can mean in this context. The
most obvious examples are open files, but the concept is broader: it includes
locked mutexes, started client processes, or a temporary directory change
os.chdir(). The common theme is that all of these require some sort
of cleanup that must reliably be executed in the future.
The file must be closed, the mutex unlocked, the process terminated, and the
current directory must be changed back.
So the core question is: how to ensure that this cleanup really happens?
Manually calling the cleanup function at the end of a code block is the most obvious solution:
f = open('file.txt', 'w') do_something(f) f.close()
The problem with this is that
f.close() will never be executed if
do_something(f) throws an exception. So we'll need a better solution.
C++ programmers see this and try to apply the C++ solution: RAII, where resources are acquired in an object's constructor and released in the destructor:
class MyFile(object): def __init__(self, fname): self.f = open(fname, 'w') def __del__(self): self.f.close() my_f = MyFile('file.txt') do_something(my_f.f) # my_f.__del__() automatically called once my_f goes out of scope
Apart from being verbose and a bit un-Pythonic, it's also not necessarily
__del__() is only called once the object's refcount reaches zero,
which can be prevented by reference cycles or leaked references.
Additionally, until Python 3.4 some
__del__() methods were not called during
A workable solution
The way to ensure that cleanup code is called in the face of exceptions is the
try ... finally construct:
f = open('file.txt', 'w') try: do_something(f) finally: f.close()
In contrast to the previous two solutions, this ensures that the file is closed
no matter what (short of an interpreter crash). It's a bit unwieldy, especially
when you think about
try ... finally statements sprinkled all over a large
code base. Fortunately, Python provides a better way.
The correct solution™
The Pythonic solution is to use the
with open('file.txt', 'w') as f: do_something(f)
It is concise and correct even if
do_something(f) raises an exception. Nearly
all built-in classes that manage resources can be used in this way.
Under the covers, this functionality is implemented using objects known as
context managers, which provide
methods that are called at the beginning and end of the
with block. While
it's possible to write such classes manually, an easier way is to use the
from contextlib import contextmanager @contextmanager def managed_resource(name): r = acquire_resource(name) try: yield r finally: release_resource(r) with managed_resource('file.txt') as r: do_something(r)
contextmanager decorator turns a generator function (a function with a
yield statement) into a context manager. This way it is possible to make
arbitrary code compatible with the
with statement in just a few lines of
try ... finally is used as a building block here. In contrast
to the previous solution, it is hidden away in a utility resource manager
function, and doesn't clutter the main program flow, which is nice.
If the client code doesn't need to obtain an explicit reference to the resource, things are even simpler:
@contextmanager def managed_resource(name): r = acquire_resource(name) try: yield finally: release_resource(r) with managed_resource('file.txt'): do_something()
Sometimes the argument comes up that this makes it harder to use those
resources in interactive Python sessions – you can't wrap your whole session in
with block, after all. The solution is simple: just call
__enter__() on the context manager manually to obtain the resource:
cm_r = managed_resource('file.txt') r = cm_r.__enter__() # Work with r... cm_r.__exit__(None, None, None)
__exit__() method takes three arguments, passing
None here is fine
(these are used to pass exception information, where applicable). Another option
in interactive sessions is to not call
__exit__() at all, if you can live
with the consequences.
Concise, correct, Pythonic. There is no reason to ever manage resources in any other way in Python. If you aren't using it yet - start now!