Brad Fitzpatrick (brad) wrote in fotobilder,
Brad Fitzpatrick

Need advice

Here's the problem:

S2 layouts/themes depend on images. The S2 layer files list their image dependencies.

When you run --populate, it recognizes the dependencies and needs to somehow get the images in the database, such that they're available for palette modification later.

This means allocating a gpic, and allocating a upic for user "system".

Okay, but when the image data itself is stored on disk (under var/picroot), as is the only case now, here's the problem:

95% of the time the user running isn't the same user the webserver runs as.

Thus, the permissions under picroot will get fucked, or the admin user won't be able to write to it anyway (owned by www-data, nobody, or whatnot)

Various solutions I've considered:

Upload to webserver from
Can't. Webserver shouldn't be running during upgrades. (Can't hurt, but the recommended upgrade procedure is to shut it down first....)

umask fun
Change umask temporarily in make_dirs such that directories created under picroot allow group writing, and the admin user is in the www-data group. Lame. Requires a lot of work for admins. This needs to be easy to use & install.

Convert on apache startup
update-db could add to db info on what apache must do on start-up. Then the code is running as www-data. But it's not! The first time perl runs in apache is right before it forks, and then it's running as root. Even worse to be doing work there.

So then ... In a child init handler? Nope... huge latency on first request. And if you do it right after the fork, you need to allocate a db handle right away to grab a lock (so all children don't do the work at once), so then you have a database connection storm, as 100+ child processes try to connect right away.

gpics with filenames outside var/picroot
Mart wants this option, for his own reasons, but I've looked at the code paths required to support it, and it'd be too slow in too many places for me to consider it an option. Plus, I don't want to go down that road.

Magic gpic record
We got this 16 byte md5sum field we could do some magic with... maybe if the first 15 bytes are zero, the last byte is an app-defined magic value action. We'll say md5sum magic value of 0x01 is "delayed gpic creation from picture on filesystem". Then, some other table (new one, probably) maps gpicids to files on disk to copy to var/picroot ... the gpic & upic records will already have everything else populated. Then, what happens:

-- recalculate the md5sum and replace the magic value.
-- update the named entities table with the gpicid, which is either entirely new, or replacing an old one
-- in the case of replacing an old named entity, we then delete the old gpic.

So then all needs to do is calculate the md5 and see if it differs from the gpic already in the database for the named entity it wants to populate.


The magic gpic record is the option I'm currently leaning towards, as the others aren't really options. Plus, the implementation is pretty easy, even if it sounds long-winded.

Still, I'll wait until tomorrow morning (er, afternoon) until I implement this. Maybe one of you has a better idea.

UPDATE: I thought of the best solution before I ended up falling asleep, so I'm back on the computer.....

Don't use the database at all! Put all images in a directory off the top like /palimage/ and then you avoid not only the normal db overhead, but also the named entity lookup .... so I'll just map all requests to that location to the Pic handler, which'll know to get the file off disk, and it'll still be able to do the palette swapping.

The ETag computation will be from file size & modtime instead of md5sum, since we couldn't get the md5sum until we spit the filehandle all the way out, anyway.

Score! Screw sleep, time to code!

  • 302: lj_dev

    In the interests of consolidating all FotoBilder development-related discussion, we're going to be closing down this community. The same…

  • Development stalled?

    Is the development of Fotobilder held? Stalled? I am asking because there is no activity on the community and there is no link to the Fotobilder…

  • (no subject)

    Does FotoBilder works with Apache2? I installed all the required modules on my debian sarge, and when I restart my apache server, it dies horribly…

  • Post a new comment


    Comments allowed for members only

    Anonymous comments are disabled in this journal

    default userpic