help needed with Lighttpd and Apache cofiguration with Django

Question

I am using Django as web framework and then Apache and Lighttpd as web server and static media server respectively. Lighty serves all my static content well and good, but I need to configure it to serve the new files uploaded by the user. Lighttpd is running on a different machine from the Apache(Django) one. My django code of creating a directory and then creating an image file gets executed on my Apache machine, making it currently getting saved in the same machine itself. I want this directory and file creation happen on my static media server, which should then be served by the media server itself. I am using os.mkdir and urllib.urlretrieve functions respectively to create directory and save files on the Django(Apache) machine. Is there anyway that I can do some configuration tweaking to get these things work or I need to write scripts on the media server and call em from Django machine??

Answer 1

The simplest answer is that the user uploads to a shared directory that both web servers can access. Then it is available instantly. If you are using unix (sounds like it) then NFS is a possible solution. If you think your site will scale to multiple servers a la flickr then using rsync to push to multiple edge servers and possibly even implementing a sharding scheme is another solution.

Just be careful. There are a lot of security concerns that depending on your app you have to consider.

If all files go to a publicly accessible directory it could be possible for users to guess the names of other peoples files and download them. In that case you'll want to serve them from Django with a thin layer of security on top.

Never trust your users! Verify that what they upload is in a certain allowable set. Under no circumstances should you allow them to upload whatever they want to. Unless of course your users are a trusted few. Even then you should do some checks. They probably shouldn't be uploading .php files for one. The last thing you want to give them is the ability to run arbitrary scripts on your server. At least configure the directory to just serve up files and not execute anything.

Good luck

Answer 2

This is the kind of stuff I use rsync for. Do whatever you like on the main server and then periodically (or on demand) rsync push to the static server. Rsync is faster (and more featureful) than anything you can produce with a quick hack.

Just because I'm paranoid I do hourly rsync's of all my customer sites to 2 backup servers, one of which is in my garage. I just timed an "rsync -a" against a 1.7GB customer site (which didn't have any changes) and it took 9.92 seconds wall-clock-time, including the 3 network handshakes for rsyncing 3 different directories. Had anything changed, presto-bango, it's done, complete with timestamps, owner/group, etc.

It's amazing how blasé you can become about server failure when you get true, multi-machine, no-humans-involved backup working. I sleep real good.