ValueError: Attempted relative import in non-package for running standalone scripts In Flask Web App

Question

I have flask web app and its structure is as follows:

/app  
    /__init__.py  
    /wsgi.py
    /app  
        /__init__.py
        /views.py  
        /models.py 
        /method.py
        /common.py
        /db_client.py
        /amqp_client.py
        /cron
            /__init.py__
            /daemon1.py
            /daemon2.py
        /static/  
            /main.css
        /templates/  
            /base.html
    /scripts  
    /nginx
    /supervisor 
    /Dockerfile 
    /docker-compose.yml

In app/app/cron i have written standalone daemons which i want to call outside the docker. e.g. python daemon1.py

daemon1.py code

from ..common import stats

from ..method import msapi, dataformater

from ..db_client import db_connection

def run_daemon():

......

......

......

if name =="main":

run_daemon()

So when i am trying to run this daemon1.py its throwing ValueError: Attempted relative import in non-package

Please suggest the right approach for import as well as to structure these daemons.

Thanks in advance.

Answer 1

@greenbergé Thank you for your solution. i tried but didn't worked for me.

So to make things work I have changed my code little bit. Apart from calling run_daemon() in main of daemon1.py, i have called function run_daemon() directly.

python -m 'from app.cron.daemon1 import run_daemon(); run_daemon()'

As it is not exact solution of the problem but things worked for me.

Answer 2

I ran into the exact same problem with an app that was running Flask and Celery. I spent far too many hours Googling for what should be an easy answer. Alas, there was not.

I did not like the "python -m" syntax, as that was not terribly practical for calling functions within running code. And on account of of my seemingly small brain, I was not able to come to grips with any of the other answers out there.

So...there is the wrong way and the long way. Both of them work (for me) and I'm sure I'll get a tongue lashing from the community.

The Wrong Way

You can call a module directly using the imp package like so:

import imp
common = imp.load_source('common', os.path.dirname(os.path.abspath('__file__')) + '/common.py')
result = common.stats()  #not sure how you call stats, but you hopefully get the idea

I had a quick search for the references that said that is a no-no, but I can't find them...sorry.

The Long Way

This method involves temporarily appending each of your modules to you PATH. This has worked for me on my Docker deploys and works nicely regardless of the container's directory structure. Here are the steps:

1) You must import the relevant modules from the parent directories in your __init__ files. This is really the whole point of the __init__ - allowing the modules in its package to be callable. So, in your case, cron/__init__ should contain:

from . import common

It doesn't look like your directories go any higher than that, but you would do the same for any other packages levels up as well.

2) Now you need to append the path of the module to the PATH variable. You can see what is in there right now by running:

sys.path

As expected, you probably won't see any of your modules in there. That means, that Python can't figure out what you want when you call the common module. In order to add the path, you need to figure out the directory structure. You will want to make this dynamic to account for changing directories.

It's worth noting that this will need to run each time your module runs. I'm not sure what your cron module is, but in my case it is Celery. So, this runs only when I fire up workers and the initial crontabs.

Here is the hack I threw together (I'm sure there is a cleaner way to do it):

curr_path = os.getcwd()   #current path where cron is running
parrent_path = os.path.abspath(os.path.join(os.getcwd(), '..'))   #the parent directory path
parrent_dir = os.path.basename(os.path.abspath(parrent_path))   #the parent directory name
while parrent_dir <> 'project_name':    #loop until you get to the top directory - should be the project name
    parrent_path = os.path.abspath(os.path.join(par_path, '..'))  
    parrent_dir = os.path.basename(os.path.abspath(parrent_path))

In your case, this might be a challenge since you have two directories named 'app'. Your top level 'app' is my 'project_name'. For the next step, let's assume you have changed it to 'project_name'.

3) Now you can append the path for each of your modules to the PATH variable:

sys.path.append(parrent_dir + '/app')

Now if you run sys.path again, you should see the path to /app in there.

In summary: make sure all of your __init__'s have imports, determine the paths to the modules you want to import, append the paths to the PATH variable.

I hope that helps.