July 28th, 2002

  • brad

Need advice

Here's the problem:

S2 layouts/themes depend on images. The S2 layer files list their image dependencies.

When you run update-db.pl --populate, it recognizes the dependencies and needs to somehow get the images in the database, such that they're available for palette modification later.

This means allocating a gpic, and allocating a upic for user "system".

Okay, but when the image data itself is stored on disk (under var/picroot), as is the only case now, here's the problem:

95% of the time the user running update-db.pl isn't the same user the webserver runs as.

Thus, the permissions under picroot will get fucked, or the admin user won't be able to write to it anyway (owned by www-data, nobody, or whatnot)

Various solutions I've considered:

Upload to webserver from update-db.pl
Can't. Webserver shouldn't be running during upgrades. (Can't hurt, but the recommended upgrade procedure is to shut it down first....)

umask fun
Change umask temporarily in make_dirs such that directories created under picroot allow group writing, and the admin user is in the www-data group. Lame. Requires a lot of work for admins. This needs to be easy to use & install.

Convert on apache startup
update-db could add to db info on what apache must do on start-up. Then the code is running as www-data. But it's not! The first time perl runs in apache is right before it forks, and then it's running as root. Even worse to be doing work there.

So then ... In a child init handler? Nope... huge latency on first request. And if you do it right after the fork, you need to allocate a db handle right away to grab a lock (so all children don't do the work at once), so then you have a database connection storm, as 100+ child processes try to connect right away.

gpics with filenames outside var/picroot
Mart wants this option, for his own reasons, but I've looked at the code paths required to support it, and it'd be too slow in too many places for me to consider it an option. Plus, I don't want to go down that road.

Magic gpic record
We got this 16 byte md5sum field we could do some magic with... maybe if the first 15 bytes are zero, the last byte is an app-defined magic value action. We'll say md5sum magic value of 0x01 is "delayed gpic creation from picture on filesystem". Then, some other table (new one, probably) maps gpicids to files on disk to copy to var/picroot ... the gpic & upic records will already have everything else populated. Then, what happens:

-- recalculate the md5sum and replace the magic value.
-- update the named entities table with the gpicid, which is either entirely new, or replacing an old one
-- in the case of replacing an old named entity, we then delete the old gpic.

So then all update-db.pl needs to do is calculate the md5 and see if it differs from the gpic already in the database for the named entity it wants to populate.


The magic gpic record is the option I'm currently leaning towards, as the others aren't really options. Plus, the implementation is pretty easy, even if it sounds long-winded.

Still, I'll wait until tomorrow morning (er, afternoon) until I implement this. Maybe one of you has a better idea.

UPDATE: I thought of the best solution before I ended up falling asleep, so I'm back on the computer.....

Don't use the database at all! Put all images in a directory off the top like /palimage/ and then you avoid not only the normal db overhead, but also the named entity lookup .... so I'll just map all requests to that location to the Pic handler, which'll know to get the file off disk, and it'll still be able to do the palette swapping.

The ETag computation will be from file size & modtime instead of md5sum, since we couldn't get the md5sum until we spit the filehandle all the way out, anyway.

Score! Screw sleep, time to code!
  • mart

S2View Really Early Release Thingy

I figured people probably want to play with S2, but FotoBilder doesn't make it easy to just mess around right now since the UI isn't there and the layer settings are hardcoded. Since S2View is now developed to the extent that it's actually useful, even if it's not particularly elegant yet, I thought I'd make an early release so that people can more easily tinker.

I hacked together a really simple site for S2View which has the executable and the FotoBilder support files available for download. It was written in VB6 (sorry!) so you're going to need the Visual Basic 6 runtime libraries. I also have no idea if all of the libraries I used are standard Windows stuff. In particular, you might find that you don't have MSSCRIPT.OCX as I've no idea if that comes with the base Windows Script distribution or whether it's something that got added to my system later.

To use the FotoBilder support stuff, just make a directory under your S2View directory (assuming you actually make one for it) and extract the files to there. I have mine called 'init' under the installed directory, but S2View shouldn't care as it will just load from whatever you type into the combo boxes. (Which, incidentally, are just textboxes right now because there's no code to populate the lists with anything.)

You'll need to specify the InitScript and Core layer at minimum to get anything useful out of it. You can then overlay other layers on top of the core layer, but not all of the builtin functions are defined in the distribution I'm offering, so not everything's going to work right at the moment. Also, Brad's really ugly test layout won't work because the core layer in the distribution is my modified version which Brad is currently reviewing. (And, thus, you should consider it volatile and not get stroppy when your layers need changing as my core layer is reorganised by Brad.)

As noted on the page, I know it has bugs already. They're mostly trivial stuff which most people shouldn't even notice unless they go looking for them. Feel free to report bugs if you want, but I'm more releasing this for the convenience of those tinkering than as an actual testing release. (It doesn't even have a version number with any meaning)

Update @ 14:09 GMT: I accidentally uploaded a version I built with my debugging S2 library rather than the 'real' one. I've updated the executable with the same name on the site, so if you're behind an aggressive caching proxy you might not be able to get a new copy. Sorry. The debug version reports some harmless errors during the loading of the core layer to do with functions that call functions which haven't appeared in the file yet, but after that the page will often render just fine.

  • brad

style makin' time!

My previously posted problem is solved....

Images that S2 layers depend on are now served right from the filesystem, with no database metadata.

(featuring mart's "supersimple" navigation icons)

The actual file, on disk:

But we can still do palette modifications:

If you want to trace the code path, look at lib/Apache/Fotobilder.pm, and search for Apache::FotoBilder::Pic and see where one path (the normal one) goes to 'handler' (upic with metadata in database), and the other goes to 'img_dir', which is a lot simpler. The look at Pic.pm and see how they then share a ton of code to do the palette stuff and send the file out.

Fun fun.

So, um.... I think it's style makin' time!
computer crap


I talked to brad about my friends group editor (/manage/groups). He showed me a lot of things that suck about it, so I will be changing all that tomorrow probably, before they fester.

I just spent the last few hours learning CSS so that I could implement niko's style in a BML template... the sticky part was that brad wanted to use the strict html DTD... which meant that the browsers ignored HEIGHT="100%" on tables, although mozilla and IE both supported it with no DTD specified. Solution: learn CSS, implement the whole thing using nothing but divs. Heh.

I'm not sure if CSS is really okay for this though. I mainly just had fun learning/trying it out. Looks fine for me in Mozilla and IE, but maybe there are other objections. ?
  • brad

New BML scheme

new look:


niko ... HTML
whitaker .. BML template from that
me ... change some stuff. :P

Now, I know many people will be offended by the colors.

But here's the good news: I'm adding config vars so you can change the color theme. And we'll use the palette modification code to dynamically make the two images you need for it.
concerned, frustrated
  • mart

FotoBilder on Windows crash update

You may recall that my server was crashing whenever it tried to handle a picture resize request. I've tracked down the part of the code which is making it crash, and as I suspected it's part of ImageMagick, although not the part I was expecting.

The offending line is the call to $buff->Write() in FB::gpic_append (which can be found in fotobilder.pl). I'm not familiar with what this function actually does other than the obvious "write stuff to this filehandle" so I can't diagnose any further, but hopefully this information could be useful to someone, sometime.

I'm off to update my ImageMagick install as a last-ditch attempt to get it to work. I'll update if I find a solution.

Update @ 20:31 GMT: I fixed it, but it ain't pretty. I had to change the call to the Write method described above to just include a filename rather than a filehandle and a filename. This means that, for one thing, my temp file needed to have the extension added, which was not really a big deal. However, it also means that after ImageMagick has gone to the trouble of opening a file, writing to it and closing it, my code then has to open it again, read from it, and close it once more. This is obviously not very efficient, but it seems to be the only way to get it to work since ImageMagick will, by default, write out images in the wrong format if there's no file extension or a file extension it doesn't know.

Of course, I don't expect Brad to incorporate my ugly code into FotoBilder. Instead, I'm writing a small bit of code which overwrites the gpic_append function with my version and putting that in the Apache configuration so that my test installation will use the ugly version of the code and everyone with their pretty UNIX servers can have relative efficiency. (I say relative, because that code isn't exactly pretty anyway: writing a file to disk just so it can be read in again.)

  • brad

more core S2 work

Weekend mostly over, I'm getting back to working on the core S2 file. I'm slowly merging the bits and pieces from Mart's version that I can make sense of.

Some things I just have no clue, though:

class PictureView { # A way of viewing a picture
var int width;
var int height;
var string url;
function toString() : string;
function toString(string alt) : string;
function print();
function print(string alt);

Is that an HTML page? It doesn't contain an Image or Picture, either, and it doesn't descend from either of those. Hence, not in my version.

Also, some things are just unnecessary:

class GalleryPicturePage extends PicturePage { # A page for showing a picture form a gallery
var ItemRange pictures; # Some Range information for other pictures in the gallery
var GalleryBasic gallery;

There is no GalleryPicturePage because when a gallery does override the default style, it has its own style... all the layers. So there's just PicturePage.

I changed the Picture class quite a bit, too.

Some things I extended. Mart had:

class ItemRange {
var bool alldisplayed;
var int current;
var int total;
var string url_next; # --.
var string url_prev; # |__ Any of these can be blank if there is nothing for them
var string url_first; # | to point to.
var string url_last; # --'

I changed that to:

class ItemRange {
var bool alldisplayed;
var int pages;
var int page;
var int items;
var int pageitemfrom;
var int pageitemto;
var string url_next; # --.
var string url_prev; # |__ Any of these can be blank if there is nothing for them
var string url_first; # | to point to.
var string url_last; # --
function builtin url_of_page(int n);

The builtin function's implementation will check for a subref pointer hanging off the hashref at '_url_of_page' and run it, else return an empty string.

Mart, I cut down your string class a ton. We can add some of that stuff as we need it (IF we need it ... some of that stuff I couldn't foresee ever needing.)

Also, I don't merge things too quickly when they're accompanied by, "Well, this breaks things so it doesn't run anymore...".

Let's just add bits and pieces as we go, discussing their rationale, and keeping the implementation working throughout.
concerned, frustrated
  • mart

Concern about Global Filehandles

The code I was working with to solve my FotoBilder-on-Windows problem opens a file with the global filehandle TEMP. Now, I may be wrong here, but don't globals exist for all requests that mod_perl is currently handling, so that when there's high traffic one request might clobber another's TEMP filehandle and cause both resized images to get corrupted?

I switched everything to using the local variable $temp and all was happy, and even if it's not necessary I suggest doing that because I think that the symbolless global filehandles are ugly and nasty and should die along with ampersands at the start of calls to subroutines.