Python Relative Imports

Intro

I thought I understood python modules. To my embarrassment, I tried doing a relative import just now and, having researched the ensuing error, realized that I had still failed to understand some fundamentals of relative imports.

Background

I created the following file structure:

.
└── src
    ├── __init__.py
    ├── main.py
    └── temp
        └── __init__.py

… with the following demo code:

### src/main.py
from .temp import foo
print('>>>'+foo)

### src/temp/__init__.py
foo = 'bar'

… and tried running python3 src/main.py expecting to get the simple message “>>>bar” printed out. Instead, I got the error:

Traceback (most recent call last):
  File ".../src/main.py", line 1, in <module>
    from .temp import foo
ImportError: attempted relative import with no known parent package

So something about the relative import was not working. What really threw me off understanding the problem was the fact that VSCode (with Pylance) was resolving the variable foo in the import without any apparent problem or warning, so I figured that the command-line interpreter must just be struggling to know where to look.

But no matter how many extra __init__.py files I scattered around the repo, or how many more directories I told the interpreter to look in for modules (viz. PYTHONPATH=".:src:src/temp" python src/main.py), I could not get this error about “no known parent package” to go away.

TL;DR

When you kick off a python process, you can choose to run your entry-point script either as a “top-level script” (TLS) or as a “module”. If you run main.py as a TLS, then you cannot import stuff from elsewhere using relative imports; if you run it as a module, then you can if the script is not in the same directory as the interpreter’s launch location. Remind please me why everyone loves python?

# TLS: relative imports not ok
python3 src/main.py

# Module: relative imports ok
python3 -m src.main

Explanation

This SO answer was key to my eventual understanding.^​*​

The key thing is that, in order to move up/down a “package hierarchy”, the interpreter needs to establish a position in a hierarchy, and it simply does not establish such a position if you start a process off as a TLS. You can see that that is the case by running print(">>>"+str(__package__)) within main.py and running it as a TLS (which prints “>>>None“).

Now, yes, this is counter-intuitive (as everyone in the python blogosphere agrees), and, yes, the authors could have built the language with a common-sense decision tree so as to establish a TLS’s placement within a package,^​†​ but they did not, and you just have to treat that as a brute fact to work around.

And since your initial position within a package is not determined, you can’t perform relative imports up/down the package hierarchy. By contrast, if you start off the process by executing a module then the “src.main” dot syntax that is passed to the interpreter is used to establish a position within a package hierarchy that can be examined within the __package__ variable.^​‡​

Further Notes

This topic involves a bunch of pesky details/subtleties that it’s worth trying to remain aware of.

• “Relative imports” begin with a dot, indicating placement within a package hierarchy. “Absolute imports” begin with a package/module name with dots to indicate descent into subdirectories. Absolute imports rely on the contents of the sys.path list, which you can set/augment with the PYTHONPATH env var.
• If you perform an absolute import on a module, then the __name__ and __package__ variables of that script will reflect the position of the module in the package as it was called. For example, if you run from package1.subpackage.somefile import something, then the __name__ variable within somefile will be “package1.subpackage.somefile“, and the __package__ variable will be package1.subpackage; the interpreter knows how to use this information to then interpret imports relative to somefile. For example, you can run from .. import something_else within somefile to access other modules within the subpackage module.
• If you run a module from the same directory that you are in when starting the interpret, even if you are specifying a module (e.g. python3 -m main), then no package structure has been communicated and, like with a TLS, you cannot perform relative imports right off the bat.
• If you have a package with nested subdirectories then you do not need to have an __init__.py file in each of them in order to access the modules in a nested file/dir. The only real practical consequence of having an __init__.py file (that I can discern) is to turn the “directory itself” into a module, so you can import the dir by name without having to name any subdirectory or file.
• I had thought that to be a package, a dir had to have an __init__.py file in it; perhaps that’s true in some sort of definitional sense but, practically speaking, a dir does NOT need to have an __init__.py file in order to function “like a package”, i.e. to just represent a collection of modules. In particular, as mentioned above, you do not need an __init__.py file in order to do imports from within a dir.
• In writing this article I worried that my explanations would sound circular since I sort of assumed that the reader knew what I meant by the term “hierarchical package” and, ideally, I would define this first before building off of the concept. However, part of the challenge here is that a “package hierarchy” is clearly a notion that must be defined with respect to the design of the interpreter, and trying to understand how the interpreter works with respect to package hierarchies is the really the point of this article. So, apologies if you feel dissatisfied; however, it feels sufficiently clear in my head (right now at least) what is going on here to close this chapter.
• Having researched this matter recently, I can report that a lot of people feel python’s package/module system is confusing and sub-optimally designed.

Summary

As a rule of thumb, you can only perform relative imports after performing an absolute import, or kicking things off as a module with hierarchical details (i.e. dots in the path to the entry script), since this info is used to establish placement within a package hierarchy.

Relative imports are tricky and actually generally discouraged in the python community since absolute imports (e.g. from package1.some_module import something) tend to read clearer than relative imports (e.g. from ..some_module import something).

So make sure that when you name packages in your code you are mindful not to introduce confusion/conflicts with packages installed from 3rd parties. (Sounds “imprecise”, but python just is far from perfect.)