arbingersysWrit

Plake: Morph a File Based on Targets

noreply@blogger.com (JA) — Tue, 10 Jun 2008 16:59:00 +0000

This blog was always intended as a means to talk about projects I'm working on, as well as a way to voice my opinions to the world. So far, it's been largely skewed to the latter.

So, I'd like to talk about a little build tool I've written called Plake. In a nutshell:

Plake is a tool that allows you to maintain sections within a single file (usually, variations of the same code/markup/content) and then assemble variations of that file according to which target you call. It was inspired by Make, can be used in conjunction with Make, and is written in Perl, hence the name "Plake".

Make is a nearly ubiquitous build tool. It's used in countless software projects and is even the basis of the CPAN installer that's part of any Perl distribution --

perl -MCPAN -e shell

Make does a really simple, powerful thing. It sets up rules (aka targets) that execute commands or invokes other targets, which is known as dependency chaining. From these rather simple concepts, you are able to orient a project for different variations, nicely denoted by a single target name.

For example, you might type

make linux_build

to build a Linux platform binary, which may consist of X number of steps that must execute in a certain order. Or, you might say

make apache_modperl

to include files from your web application specifically for an Apache/mod_perl web environment, along with the more general non-platform specific files.

What Make can't do, however, is snag bits of code (or markup) from individual files for a given build. If you've ever looked at cross-platfrom C/C++ code, you've probably noticed the #ifdef directives in the header files. These are used because sometimes there are small portions of code that need to be excluded when compiling for a certain platform or target, and keeping totally separate files to accommodate this is excessive.

Plake allows you to define sections within a single file, and then "assemble" only the sections you want at build time. Here's an example.

Let's say you have a C++ source file that gets built for the Windows platform and also for Linux. Keep the differences as sections in a single Plake file. Then when you assemble the .cpp file for the given platform, it only contains that platform's code.

The following commands both produce "myfile.cpp" (but possibly at different folder locations) with only the code that each platform needs:

plake file=myfile.plk target=windows_build

plake file=myfile.plk target=linux_build

Because Make is generally made up of shell commands, you would put the above commands under the appropriate Make target, and when you type make target, Plake assembles the file with only the parts you need prior to compiling it. The advantage you get, in the scenario above, is that reviewing code is easier, since after a specific target is assembled, only the code you need to see is there.

There are some other uses for Plake, which I've discussed over at Perlmonks, here and here. This is the short list:

Setting variations for builds. A convenience for me since I have yet to implement a more complex (i.e. overrides) configuration system, but still have to make subtle changes (usually, by hand-editing) for various implementations at various stages of development.
Assemble C/C++ files for specific platforms, in the stead of #ifdef, etc. The resulting .c/.cpp/.h file would be assembled dynamically when the project was make'd for a given platform, just prior to compilation. The code generated for that platform would be a bit simpler to review, since it only includes code that a person cares about in that build.
Remove experimental features, stubs, or extra debugging from code prior to generating distros, i.e. "Cleanup".
Branching, like what source control does. You could keep some client or "branch" specific features out of a specific build, but still maintain it in a single file.
Template variations, like letter writing. Instead of a single boiler plate template, you have targets like "standard_greeting", "enthusiastic_greeting", "familiar_greeting", etc.
Target-based programming for Perl. Sort of a side-effect, and one I don't see all the ramifications of, but you could use Plake to assemble code targets wholly or partially independent of each other by storing Perl code in a Plake file and doing an eval against the assembled content for a given target. (Just think -- you could keep your entire project of hundreds of modules and code files all in one single, massive text file! I can see everyone lining up now...)

The last item above, target-based programming, is particularly interesting, I think, so I'll cover it briefly before finishing up. Plake was written in Perl, and uses the eval() function to execute code on the fly. With a minimal change in the code, you could take the content you return from the plk file and eval() it, effectively creating a target-based interpreter. (I include a sample that does this in the download. See plakeval.pl.)

So, if you have a Plake file like

!plake:

target('helowrld', "helowrld", '');
target('oneplus', "oneplus", '');
target('both', "helowrld oneplus", '');

!plake helowrld
print "helowrld\n";

!plake oneplus:
# Add value to one
print 1+3.14, "\n";

and you called it with plakeval.pl, you would get the following:

perl t\plakeval.pl file="t/plakeval.plk" target="helowrld"
helowrld


perl t\plakeval.pl file="t/plakeval.plk" target="oneplus"
4.14


perl t\plakeval.pl file="t/plakeval.plk" target="both"
helowrld
4.14

When the target both is called, you can see that we are printing helowrld and also adding 3.14+1.

What this means is that you can stick things together in a file that perhaps make sense in a certain context, but wouldn't otherwise. Like I said, target-based programming is sort of a side-effect, and while I haven't really explored its value, I have a sense that some exists. At any rate, I find it interesting.

But really, Plake was designed to let you keep variations of a file in a single actual file on the hard drive, and then omit or include parts of it based on a target. And it does that really well. I use it in my own projects and it saves me a considerable amount of error-prone work.

I'm Trying To Quit... Commercial Software Pt. 2

noreply@blogger.com (JA) — Fri, 30 May 2008 15:00:00 +0000

In this experiment, FOSS is effectively graded on whether or not it can substitute all or most of my proprietary software needs, without me having to substantially change the way I use software. It's highly subjective, and human nature, like laziness and apathy, is very much a part of it, as you will see.

This is the second installment of my personal Free Open Source Software experiment. Read the first installment here.

Within a year of getting my new notebook, my wife's laptop gave up the ghost. It was a Dell Inspiron 8100, and frankly, we'd gotten our money's worth. I purchased a new laptop, a Gateway M6882, and we did the laptop shuffle again.

The Gateway came with Vista, but I wanted to run XP. I immediately discovered that XP was going to be difficult to manage. There was no floppy drive, XP didn't have the needed SATA controller, and there were only three hardware drivers available for XP on the Gateway site.

After thinking about it, I realized that regardless of my feelings for Vista, it's going to be inevitable, and I might as well get used to it. However, I'm resentful about my conclusion, and I'm sure I'm not the only one. As far as portents go, this is a bad one for Microsoft.

Ultimately, this ended up being a good thing. I'd been wanting an excuse to run Linux, and here it was. I decided to keep Vista, since I might need it, but repartition and dual-boot Linux.

Thus began the second phase of my "experiment". I would see just how little I'd have to use Vista, if Linux were available instead.

Linux

I started with Ubuntu 7.10 LTS, since it seemed like the distro with the most momentum. Installation was a breeze. I particularly like booting from the CD and getting to play around with the desktop before doing the install.

After installation, however, I began to bump into oddities and frustrations.

First, the M6882 is a widescreen with an optimal resolution of 1280x800. The Gnome desktop used the entire screen, but the top and bottom system bars only went to a width of 1024 pixels. I tried to change the resolution using the system config tools, but nothing worked. I had to hit the forums, and after some time (longer than I would have preferred), finally found a solution that involved editing the xorg.conf file. I still don't understand exactly what I changed, but it had something to do with TV out settings.

This gave a bad impression. The facts of life are that in the many many installs I've done of Windows, I've never had to do this much work to get the system to the correct screen resolution.

I still had one other hardware problem that was bothering me. The sound card didn't work. This took even longer to fix than the screen resolution, and was twice as painful.

I hit the forums again. I tried several suggestions with rather involved steps, with no success. I had a glimmer of hope when I found and downloaded the Linux drivers from the manufacturer's website. It was a source package, with some simple instructions for compiling and installing. But the install script first removed the existing sound libraries that the X server had been compiled against, using the fatal rm command. Then, the build failed. Unaware of what had happened, I gave up and at some point rebooted. The desktop failed to load the next time I tried to boot.

The manufacturer's Linux driver package had clobbered my non-working, but non-failing sound libraries without backing them up, or even checking that the build succeeded first. At this point I was pretty much hosed, and the easiest thing to do was to reinstall.

I reinstalled, fixed the screen resolution problem again, and still didn't have sound. I finally found a solution, on some guy's blog. There was no compiling required, just a bunch of funky steps to get a "backports" package installed, after which I had to re-run some updates I'd already done. After that, my sound worked fine. But, like the screen, this was far too much work to have to do for something I consider basic and essential to an OS.

The next hassle I had was that I changed my password, and suddenly was being prompted by the keyring manager every time I logged in. Again, my only resource was the forums. I'll spare all the details of resolving this problem, but I'll say this: the problem with forums as the help is that you don't know who you can believe. I'm not saying anyone would attempt to purposely mislead you (although they might), but they can and often do get things wrong, communicate the solution poorly, or miss a detail that is essential to your particular system.

In the keyring case, I followed one person's advice, which involved compiling from source, and began the descent down the dependency Inferno, only to find out that all I really needed was to run the following simple command:

rm ~/.gnome2/keyrings/login.keyring

Using the community forums as the help system is a problematic solution at best. With no monetary incentive, you get only the best someone is willing to offer at the time, you have no verification of the expertise of your source, and no one is responsible. You may get an excellent answer, a partial answer, the wrong answer, or no answer.

Never booting into Vista

After getting past the problems above, I began using Linux in earnest. As far as the basic things I need to do on a computer, e.g. programming, web surfing, email, FTP, document editing, spreadsheets, playing music, etc, Ubuntu was able to deliver.

But here's what I still need Vista for:

DVD playback. I couldn't play a DVD of 24 with Totem. I had installed GStreamer the ugly and also Mplayer. No dice. Mplayer looked like:

I also tried VLC. It got some images to the screen, errored out, and froze.

I didn't give up that easily. Next I installed Totem with the xine backend. When I played the DVD this time, I got the FBI warnings, but it complained about encryption when it came to the video, and also failed.

In Vista all I have is the Windows Media Center, which sucked in XP. It's been improved, and other than the audio being slightly lower than I would have liked (perhaps a hardware issue), I can play DVDs without a headache.

Photoshop. I know I could learn Gimp, but I already know Photoshop, and know it well. It had a steep learning curve, and has all the capabilities I need and then some, so switching doesn't appeal to me. I'd much rather just boot into the system where this app runs and use it there.

Doom9.net. I use a lot of the multimedia tools (e.g. BeSplit, MeGUI) available from this site. Most of these interfaces, while freeware, run on Windows.

Netflix. Sorry, but they have that Watch Instantly feature, which will not only just run on Windows, but also will only run on X number of installs of Windows. I don't like it, but like Vista, it's just the way things are.

Rooting for Linux

While I'm growing increasingly fond of Linux, and certainly rooting for it, it's got a ways to go. Hardware will be a weak point for some time to come. This isn't the fault of Linux, but instead the fault of economics. Money is the big incentivizer, and the OS that can bring in the most money will always get priority. My experience with the manufacturer's sound driver installation is a clear example of this.

Microsoft may not win any medals for its ideals, but sound drivers usually install without the user having to jump through hoops or inadvertently clobbering their system, and I can play most DVDs by just slapping it in the drive.

Linux also suffers in the support department. Again, this is because the model of Linux is essentially based on altruism. Really, it's an amazing feat that Linux works as well as it does, has the support it has, and is as advanced as it has become. I'm rapidly turning into a fan, and have optimism for the future.

Watch for my next installment, in which I begin to play around with Gimp, surprisingly, because of laziness, switch to openSUSE and am pretty happy with it, and have some trouble connecting to WIFI where Vista does not.

What's a Wiki?

noreply@blogger.com (JA) — Mon, 19 May 2008 17:41:00 +0000

Not a particularly hard question, and most people (whose primary exposure to the term is through Wikipedia), will pipe up: It's a website that lets anyone edit and make changes. And they'd be right, but there's more to it.

A Wiki was originally designed around the philosophy of incompleteness and interaction. The concept, created by Ward Cunningham was intended to foster collaboration [which] creates and develops new ideas.

But it's extremely difficult to know just exactly how your idea will be adapted and ultimately play out when presented to the world.

Wikipedia came along and decided to classify content using a Wiki, becoming the world's first collaborative encyclopedia. And it has stayed true to the ideas above. It's both incomplete and promotes interaction. But it doesn't use Wiki technology to develop new ideas. That's not why it was created.

Is Wikipedia any less a Wiki, then? Not really. While the original intention of Wiki may have been to foster the creation of new ideas, the functionality it provides to do that (i.e. ease-of-use, simple markup, natural collaboration) lends itself to other goals as well.

So then, a Wiki may be:

Content Creation Wiki
The original intention of a Wiki -- to collaborate and create new ideas. From c2.com:

Treat a page here as a half-finished piece of sidewalk art. Don't scuff it up. Don't rub it out. Don't write messages on it like "finish this you bum or I will scuff it" or "I disagree" or "me too".

Instead, see if you can head it toward completeness. If you can't do that now, leave it be. Maybe one day you will think of something to add. Or perhaps another will. We rely on each other to help new things come into being, like ants building nests.

Content Classification Wiki
Sites like Wikipedia, which classify existing knowledge to make it usable. These sites tend to be larger, edited more stringently, and try to present knowledge "authoritatively". [link]

Knowledge Base Wiki
I'm adding this one, since I'm increasingly seeing Wikis used this way. This type tends to be specific to organizations, and are either used to accumulate and distribute information about a specific product or service, or used internally to collaborate and share information, e.g. company policies, inter-departmental information, etc.

These Wiki "types" really only differ in their intention and audience. They all foster collaboration, are simple to use, and are generally ongoing, with no real finalization date.

Re: Why People Are Passionate About Perl

noreply@blogger.com (JA) — Tue, 13 May 2008 16:58:00 +0000

Here's my response to brian_d_foy's People Passionate About Perl meme.

I first starting using Perl to...
I began looking into Perl in the 90s -- when it was suffering less from perception issues -- as an alternative web development platform to ASP. ASP presented a low bar, and I was making web front-ends to databases in a very short time. Then, after satisfying initial needs, more demands began to be made on our web applications, and ASP's low bar began to be an inhibitor.

I reviewed Perl as an alternative, and (I'll be honest) after getting past the syntax, began to understand the power I was toying with. Then, I discovered CPAN. ASP never looked quite the same after that.

I kept using Perl because...
It's never given me a reason not to. Sorry, but I'm not loyal for loyalty's sake. If a tool like Perl can't make my life easier than tool X, then it's time to investigate tool X.

But Perl hasn't failed on this account. It's proven to be highly adaptable, and the energy of its community has fit it to new paradigms readily. For instance, is there a Ruby on Rails for Perl? Try Catalyst, or the newer Jifty.

As for pre-packaged functionality, I don't think there's a language that can compete. CPAN continues to grow and grow. In fact, if you want to contribute, the difficulty now is thinking up something that hasn't already been done. Try templates, for example.

This means one thing: If I need a tool to get something done, Perl is the easiest choice. It's powerful, flexible, and continues to edge into functionality that I haven't even begun to think about.

Oh yeah, and it also provides industry leading regular expressions via operator, for the absolutely most convenient and shortest possible way of using this very important technology.

I can't stop thinking about Perl...
Actually, I can stop thinking about Perl, and frequently do. That's because there are other things in my life besides Perl. However, I think in Perl when I think about crafting software, or anything abstract and computational. Its natural language model makes this easy.

And since web, Internet, and database are the spaces for the majority of my software ideas, thinking in Perl is a huge benefit for me, because so many others are thinking in Perl for the same spaces, answering questions I haven't thought to ask yet (CPAN again).

I'm still using Perl because...
This is mostly covered above. But here's one more.

Line count. I use Perl day-to-day to handle any number of tasks, of any size and importance. Perl itself reduces line count just in the power of its syntax. I'm not talking about merely writing obfuscated code. I'm talking out the power inherent in the language itself.

And now, back to CPAN.

Recently at work we needed to parse a handful of Excel spreadsheets that were formatted more or less the same. I handed this job off to a contractor who works for me. He created a C# project, and then left for the day. He wasn't able to come back the next day, so I took the project over. He had barely gotten started, but he already had five or six files involved, and a couple hundred lines of code.

I immediately thought we should be doing it in Perl. This was a one-off project, so why do a whole Visual Studio project? I Googled around and found this tutorial. I installed the modules from CPAN, adapted the samples to my needs, and about an hour and 80 lines of code later, I had the spreadsheets munged into SQL and ready to go.

Sorry, but I'll take the shorter route every time, if I can.

I get other people to use Perl by...
Well, I blog. Not exclusively about Perl, and not even explicitly to advocate Perl, but it is about Perl, because, like I said before, I think in Perl. It's going to leak out.

I have pointed people to Perl when it best suits their needs. A guy I work with wants to learn programming, and was looking at Python. I asked why he was interested in programming, and he admitted he just wanted to write a few scripts to download content off a website. I nodded and said, "You should use Perl."

Python may have this covered as well, but I showed him how in one line of code (via LWP::Simple) I could grab the text of a website. I also pointed him to all the modules -- you guessed it, available on CPAN -- that can rip apart HTML and extract just the things you need.

I also program in ... and ..., but I like Perl better since...
Although I know several other languages, I program primarily in C# and Perl.

Both languages work well for the domains in which I use them. I use C# to write Windows specific applications. C# just has better hooks into the system, with less weirdness.

I use Perl for pretty much everything else. And where they cross domains, i.e. web development (ASP.NET), I prefer Perl, because (1) it has more pluggable functionality, and for free, and (2) has a shorter development-to-production time. This is partly due to my proficiency in Perl, but also because there is less setup involved in new projects, and less OO wrapping.

If it's a simple app, I use a minimal amount of Perl. If it's more complex, I use the frameworks available, like CGI::Application and templates. With ASP.NET, you're pretty much bound to the framework -- with all its complexity -- even for simple projects.

Google Docs Finally Matter To Me

noreply@blogger.com (JA) — Fri, 09 May 2008 02:13:00 +0000

To be honest, online documents never really were a big sell for me. Frankly, it's an application space that's pretty boring, and ultimately, you sacrifice functionality. What functionality do you get with Google Docs that an "offline" word processor can't provide in spades? Well, just this: Your documents can be accessed and edited from anywhere [that you have a high-speed Internet connection]. That last part is mine, since you won't see that in any marketing phrase for an online word processor. But it's significant.

It may surprise you, but until recently I only had a dial-up connection at home. Because I live in a rural area outside the city, the only option for me was satellite, which I didn't find appealing due to the cost/performance ratio. At work, however, we have DS3, so I had no real Internet deficiency.

My work also provided me with a 4GB thumb drive -- and lanyard! -- so I had an extended sneakernet, and anything that was too painful to download from home (almost everything), I would download at work. Also, document synchronization between work and home was answered by simply keeping documents on the thumb drive. This guaranteed that wherever the location, I always had the up-to-date revision.

So because I had only one high-speed Internet connection, the single advantage Google Docs could provide over Word or OpenOffice Writer didn't exist.

And let's be honest. The interface, while well done for a web app, doesn't compare to a locally running application written for your platform. What is Google Docs, really? It's a word processor running under a web browser. What is Microsoft Word? It's just a word processor. Which program do you think is going to be better suited to the task of word processing, and capable of offering more power? The one that gets to focus its logic on word processing, or the one that also has to be a web browser? Google Docs only has an advantage as a web platform.

When DSL became available in my area, the game changed. I now have a fast, always on connection at the two places where I do most of my work: at home and at my office. My sneakernet has pretty much ceased to exist. If I need to transfer anything I simply use FTP, email, or VPN.

But all of the above methods are kind of clunky for synchronizing files. Our VPN only works with Microsoft clients, unfortunately, and I use Linux quite often when I'm home. FTP would work the best, but there are a lot of extra steps (or extra setup) when compared to just plugging in a thumb drive and clicking on the file you want to edit.

Many of the documents I work on are spec documents for software projects. I don't really need anything more than just basic word processing functionality: headings, emphasis, bulleted lists, tables. Google Docs does all this pretty well.

I recently started to develop a spec for a Perl library, and made this my first real try of Google Docs now that I have more than one reliable high-speed Internet connection. I started writing the spec about a half hour before I left work one day. On the way home, I had some new ideas, and wanted to add them while they were still fresh. This was the moment Google Docs finally began to matter to me. It was the easiest synchronized document edit I had made to date. I just logged in to my laptop when I got home, made my additions, and saved.

Since then, any document that I'll edit from more than one location goes directly to Google Docs.

Google Docs, or any online word processor, only has real value as a web platform. And a web platform only has value where there is a sufficiently high-speed Internet connection available. As that becomes more and more common, online word processing will begin to matter to more people.

Keeping A Digital Diary On A Trēo

noreply@blogger.com (JA) — Tue, 06 May 2008 17:13:00 +0000

About a year and a half ago, my wife and I took a trip to Mazatlan, Mexico. In my backpack, along with my laptop, I had stowed a Siemens SX56 PDA. This was the first time we had ever visited Mexico, so I decided I wanted to keep a day-by-day account.

The SX56, like most PDAs, has a microphone. I changed the recording settings to a low frequency -- 8 kHz 8 bit stereo (still good enough for voice recording) -- and recorded the events of our Mexico vacation. Since then I've maintained a personal audio diary on my PDA, trying to put something in for each day without being bogged down by boring minutia. Of which, sadly, there is enough.

The SX56 has only 32MB of storage, and part of that is used for system files. I found myself filling it up and having to dump to my laptop far too frequently. It turns out this was hardly an insurmountable problem.

I bought a Sandisk 256MB flash card, and switched the voice recorder to save to it automatically. This solved two annoyances at once: I didn't have to dump the voice recording files as frequently to my laptop, and I no longer needed to sync via cable, which is always a pain. I could just plug the flash card into my laptop move the files off with Explorer.

For a long time, this was a very workable solution. It still would be, in fact, but by a small stroke of fortune, I was able to upgrade to a Trēo. Here's a picture:

The journey of a digital photo: This picture was taken on my wife's Nokia cell phone, emailed from there to my Gmail account, download to my laptop, cropped using Gimp, and then FTP'd to my website.

Not very long ago, a guy I work with brought in a box of about thirty Trēo phones like the one I snagged above. He had gotten them from his old employer who no longer needed them since they had just gotten a new budget. (Aren't they the lucky ones...)

After playing with one for a while, I decided I'd take him up on his offer of having one for free. It had all the features of the SX56, and then some. Like twice the storage space on the phone itself. And a real keyboard and navigation button, instead of purely on-screen controls. And of course, my 256MB flash card plugs right in.

Also, it has a camera. This didn't seem that significant at first, but we were recently on a hike, and I was able to take a picture of my wife and daughter, and save it on the flash card along with my diary's audio files. So now my diary has taken on a whole new dimension: It will include real as well as audio imagery.

This isn't the first time I've tried to keep a diary. A couple times in the past I was inspired to do it, and each time, it fizzled. The reason my PDA diary hasn't, I think, is because it lends itself so well to the task. It's portable, by which I mean it has a battery and fits in your pocket, and it requires little effort -- just click and talk about the day's events.

Really, the hardest thing at this point is making sure that you only record things that are actually interesting. You don't want to bore your future self, after all.

ScratchPad MX - Save Stuff For Later

noreply@blogger.com (JA) — Sat, 03 May 2008 17:15:00 +0000

For the longest time, I've had a shortcut on my Start menu that launched a text document called scratch.txt. This way, with a few clicks, I could save something I might need later, or if I needed a place to temporarily stick some clipboard stuff, I could use it for that. But the problem was, I didn't need a full-blown editor (or even half-blown, like Notepad) to do this. I wanted something that was editor-like, but stripped down to and streamlined for just the functions I needed. A real scratch pad, not an editor acting like one.

Specifically, I wanted the following:

Key combo launch, so it would be available at a moment's notice.
A command to create a section (insert a divider of some kind) to keep stuff separate from each other.
Save and close with a single command, for when I need to save something quickly until I have time to think about it.
After a while, you'll accumulate an eclectic mix of stuff, so a way to jump from section to section. (Also, a way to search.)
Close without saving, for when I'm just using it to store something temporarily, like when I'm clipboarding heavily.

So that's what I came up with. I call it ScratchPad MX. Here's what it looks like:

Along the top, you can see the five commands available to the program. Pretty self-explanatory. Ctrl+f will jump through each section, which is defined by the line of "=" characters. If you type some text and highlight it, Ctrl+f will search down through the document for that text. (So actually, there are 5.5 commands.)

You can download the installer here. It will automatically install the program, and optionally, you can install a hotkey, WinKey+Space that you can use to bring ScratchPad MX up instantly.

ScratchPad MX is completely free. Also, this is version 0.01 (barely better than beta), so if there are problems or you think a feature might be useful, either leave me a comment, or email me at one of the addresses on my home page.

Google App Engine: [A Better] Many-to-many JOIN

noreply@blogger.com (JA) — Wed, 30 Apr 2008 15:00:00 +0000

(This is a follow-up to my original post GAE: Many-to-many JOIN. It probably wouldn't hurt to read that first, since this post sort of assumes you have.)

After getting some feedback on my original post, a simpler, more SQL analogous way to obtain the many-to-many behavior was pointed out to me.

I've created another sample (download it here), and will go over it below. Afterwards, I'll talk about why you shouldn't model your data this way, and instead should denormalize your data for optimization in the Datastore.

Here are the new data Models. (The full code listing is here.)

class Libraries(db.Model):
   notes = db.StringProperty()

class Books(db.Model):
   notes = db.StringProperty()

class Library(db.Model):
   name = db.StringProperty()
   address = db.StringProperty()
   city = db.StringProperty()
   libscol = db.ReferenceProperty(Libraries,
       collection_name='libscol')

   def books(self):
       return (x.book for x in self.librarybook_set)

class Book(db.Model):
   title = db.StringProperty()
   author = db.StringProperty()
   bookscol = db.ReferenceProperty(Books,
       collection_name='bookscol')

   def libraries(self):
       return (x.library for x in self.librarybook_set)

class LibraryBook(db.Model):
   library = db.ReferenceProperty(Library)
   book = db.ReferenceProperty(Book)

I still have the Books and Libraries models, as you can see. These are needed to collect the Library and Book entities so I can easily iterate over them and output. The Book model contains a reference to Books, via Book.bookscol, and Library to Libraries, via Library.libscol.

The LibraryBook model just contains references to the Library and Book models. This creates our "join". After we add libraries and books to the Datastore, we will link them to each other using LibraryBook entities.

When the page loads, we first create and store our data entities.

# Library collection
libs = Libraries()
libs.put()

# Book collection
books = Books()
books.put()

# Setup libraries
lib1 = Library(name='lib1', address='street a',
   city='city1', libscol=libs)
lib2 = Library(name='lib2', address='street b',
   city='city2', libscol=libs)
lib1.put()
lib2.put()

book1 = Book(title='book1', author='author one',
   bookscol=books)
book1.put()
book2 = Book(title='book2', author='author one',
   bookscol=books)
book2.put()
book3 = Book(title='book1', author='author two',
   bookscol=books)
book3.put()
book4 = Book(title='book2', author='author two',
   bookscol=books)
book4.put()
book5 = Book(title='book3', author='author two',
   bookscol=books)
book5.put()

l1 = LibraryBook(library=lib1, book=book1)
l2 = LibraryBook(library=lib1, book=book2)
l3 = LibraryBook(library=lib1, book=book4)
l4 = LibraryBook(library=lib2, book=book4)
l5 = LibraryBook(library=lib2, book=book5)
l6 = LibraryBook(library=lib2, book=book3)
l7 = LibraryBook(library=lib2, book=book1)
l1.put()
l2.put()
l3.put()
l4.put()
l5.put()
l6.put()
l7.put()

First, we create our Libraries and Books entities, libs and books. These will be passed into each Library and Book entity we create.

After we create our books and libraries, we generate a lot of LibraryBook entities, assigning a library and a book to each one. Each LibraryBook entity now links one library with one book. As you may have noticed, some books are assigned to both libraries, some are not.

Library contains a method called books(). It returns every book in the librarybook_set as an iterable data structure. Because LibraryBook holds a reference to Library, any Library entity (say, lib1), is given a back-reference to the collection of LibraryBook entities. If you do not define a collection_name, GAE automatically creates one by appending "_set" to the model name. This is where librarybook_set came from, in case you were wondering.

Given a library entity like lib1, the books() method allows us to easily return all the books at that library by simply assigning or iterating over lib1.books(). The Book model contains a method called libraries() which does just the opposite: allows you to get all the libraries where a given book resides.

Our data has been created and linked. Now we pass it in to the template.

template_values= {
 'lib': lib1.name,
 'books_at_lib': lib1.books(),
 'forbook': book1.title,
 'libs_by_book': book1.libraries(),
 'libs_books': libs.libscol.order('name'),
 'books_libs': books.bookscol.order('-author').order('title')
}

In this example, we not only display all libraries and all books (via libs_books and books_libs) the way we did in the previous post, but also output all books at a library (books_at_lib), and all libraries that contain a given book (libs_by_book).

Here's the template, if you want to take a look at it.

Denormalize your data

As I stated before, the GAE Datastore is not a relational database. Databases were designed for compactness and efficiency, and normalization is used, in part, as a way to minimize the size of your data on disk.

The Datastore has been built, first and foremost, with scalability in mind. Scalability means, in essence, "add more servers as needed, without re-writing your code". Specifically to the GAE Datastore, it means "disk space is cheap, stop worrying about it, and scale".

Consider modifying our LibraryBook model above to look like

class LibraryBook(db.Model):
  library = db.ReferenceProperty(Library)
  book = db.ReferenceProperty(Book)
  booktitle = db.StringProperty()
  libraryname = db.StringProperty()

Now, we are not only storing each book's title in the LibraryBook entity, but we are also storing it in the title property of the referenced Book entity. While this is obviously not space efficient, and certainly not the elegant, normalized way of storing relational data our brains are used to, it scales well and is fast.

It scales because the Datastore runs on who knows how many commodity computers in the background (without the knowledge of our application), and it's fast because we have the most commonly needed fields available immediately. If you need to poke further into the data, like to get the street address of the library, you would use the referenced models, and our JOIN then comes into play.

(Thanks, Ben the Indefatigable for illuminating this.)

Google App Engine: Many-to-many JOIN

noreply@blogger.com (JA) — Mon, 28 Apr 2008 20:15:00 +0000

Update: After reading this, you might want to check out GAE: [A Better] Many-to-many JOIN, which gives an improved way of doing this, plus goes into why you shouldn't normalize your data.

A public library has many books. In SQL-speak, this is a one-to-many relationship. (For the sake of the argument, I'll assume each library has only one copy of a given book). It follows then, that many libraries have many books. This is a many-to-many relationship. On the heels of my recent post GAE: One-to-many JOIN, here is an example showing how to do a many-to-many JOIN using the Google App Engine Datastore.

You can download this entire sample here.

A many-to-many SQL query for our library scenario would look something like

SELECT
    *
FROM
    library
INNER JOIN
    libraries_books
ON
    library.KEY=libraries_books.library_KEY
INNER JOIN
    books
ON
    libraries_books.book_KEY=books.KEY

To duplicate this functionality in the Datastore, we have to model our data as follows. (Full code listing here.)

# These are used for linking/ordering
class Books(db.Model):
    notes = db.StringProperty(required=False)

class Libraries(db.Model):
    notes = db.StringProperty(required=False)

# Data models
class Library(db.Model):
    name = db.StringProperty(required=True)
    address = db.StringProperty(required=True)
    city = db.StringProperty(required=True)
    library_list = db.ReferenceProperty(Libraries,
        required=True, collection_name='ref_libs')

class Book(db.Model):
    title = db.StringProperty(required=True)
    author = db.StringProperty(required=True)
    library = db.ReferenceProperty(Library,
        required=True, collection_name='books')
    book_list = db.ReferenceProperty(Books,
        required=True, collection_name='ref_books')

The Library and Book models share a one-to-many relationship. This is setup using the Book.library db.ReferenceProperty. Nothing really new here (if you read my one-to-many post, anyway).

We need some additional references to pull off the many-to-many relationships, however, plus a couple extra Models. (It's important to note that the db.ReferenceProperty in itself only allows for a one-to-many relationship. That's why we need more than one get the many-to-many behavior.) I've created the Libraries and Books models for this. You may notice that they have an optional, largely unnecessary property named notes. This can pretty much be ignored. We really just need these entities to exist in order to point to them from our Library and Book entities.

The Library model contains a reference to Libraries through a property named library_list. Book has a reference to Books via book_list. Having references to both Libraries and Books allows us to manipulate the sorting for each collection, as you will see below.

When the page loads in our browser, the first thing we do is create entities from our models, and give them some data.

# Library collection
libs = Libraries()
libs.put()

# Books collection
books = Books()
books.put()

# Setup libraries
lib1 = Library(name='lib1', address='street a', city='city1',
    library_list=libs)
lib2 = Library(name='lib2', address='street b', city='city2',
    library_list=libs)
lib1.put()
lib2.put()

# Books:
# Both libraries
book1 = Book(title='book1', author='author one',
    library=lib1, book_list=books)
book2 = Book(title='book1', author='author one',
    library=lib2, book_list=books)
# Only first library
book3 = Book(title='book2', author='author one',
    library=lib1, book_list=books)
# Both libraries
book4 = Book(title='book3', author='author two',
    library=lib1, book_list=books)
book5 = Book(title='book3', author='author two',
    library=lib2, book_list=books)
book1.put()
book2.put()
book3.put()
book4.put()
book5.put()

We declare our "link" entities, libs and books, first. Next we create two library instances, lib1 and lib2, and assign libs to library_list to create a one-to-many relationship from Library to Libraries.

A Book entity has two relationships to setup. A one-to-many relationship to a given Library entity, and a one-to-many relationship to the Books entity. These are established through the library and book_list properties, respectively.

After we store our data, we use the collections in our Library and Book models to create two objects that we will pass to our template.

libs_books = libs.ref_libs.order('name')
books_libs = books.ref_books.order('author').order('-title')

template_values = {
    'libs_books': libs_books,
    'books_libs': books_libs
}

Both libs_books and books_libs contain many-to-many relationships between libraries and books. But libs_books references books from libraries, allowing you to sort by library, and books_libs does the opposite, referencing libraries from books, letting you sort by books. This is certainly more clumsy and more work than our SQL counterpart, which just needs an ORDER BY clause to sort either way.

On to the template. To output books by library, we have to iterate over every library lib in libs_books, and then iterate over every book referenced to lib.

{% for lib in libs_books %}
    {% for book in lib.books %}
        <tr>
            <td>{{ lib.name }}</td>
            <td>{{ lib.address }}</td>
            <td>{{ lib.city }}</td>
            <td>{{ book.title }}</td>
            <td>{{ book.author }}</td>
        </tr>
    {% endfor %}
{% endfor %}

Because of the way references are setup in libs_books, we are able to order the output based on the libraries, as you can see in the first table below.

The second table above shows the output from books_libs, which we use to order by books. Here's how we generate the data in the template:

{% for book in books_libs %}
    <tr>
        <td>{{ book.title }}</td>
        <td>{{ book.author }}</td>
        <td>{{ book.library.name }}</td>
        <td>{{ book.library.address }}</td>
        <td>{{ book.library.city }}</td>
    </tr>
{% endfor %}

We don't have to use nested loops, and we simply use book.library as a normal reference (not a back-reference) to get the library associated to the given book. The reason we don't have to nest is because a Book entity has a many-to-one relationship with a Library entity, so each book is already attached to a Library. Library entities have a one-to-many relationship to Book entities, so every time you get lib, you have to find it's many, which requires the second loop.

There you have it. A first blush example, to be sure, but I think it conveys the core steps required to duplicate the behavior of a relational many-to-many JOIN.

Google App Engine: One-to-many JOIN

noreply@blogger.com (JA) — Sun, 27 Apr 2008 01:39:00 +0000

By now, no doubt, most developers have heard about the Google App Engine (GAE). And even if you didn't get one of the 10K free accounts, you might still have downloaded and started messing around with the SDK.

Google touts the platform's ease of development, and stepping through the samples reinforce that it is, in fact, quite easy. However, it doesn't take long to discover what will probably be the biggest hurdle for developers entrenched in the relational database paradigm: The Google Datastore. It's not a relational database, and it's not an OOP wrapper to a relational database. It's a web-specialized data storage mechanism, accessed through classes called Models, and objects called Entities.

I'm willing to bet that most of the developers playing with the SDK will first really "get" this when they move past the simple "one table" queries in the samples, and try to do a basic JOIN query. Although there is a SQLlike syntax called Gql -- as stated in the Docs -- there is no JOIN.

To get this functionality, you have to use db.ReferenceProperty to link one object to another. Here's a short demonstration of how it's done. I figure this is much needed, since there seems to be no good examples for it in the Google documentation. (The best information I could find was in the GAE discussion group.)

Below, I've listed example.py in its entirety (don't worry, it's short), and I'll refer to each pertinent section by the line numbers. (You can download the entire sample here. Put it under the SDK folder, and run it like any of the GAE samples.)

1  import os
2  import cgi
3  import wsgiref.handlers
4
5  from google.appengine.ext import webapp
6  from google.appengine.ext import db
7  from google.appengine.ext.webapp import template
8
9  class MainPage(webapp.RequestHandler):
10    def get(self):
11
12      url = EnteredUrl(url="http://domain.com/page.html")
13      url.put()
14
15      match1 = AffinityUrl(
16          url="http://domain.com/dir/page1.html",
17          affinity = .83,
18          entered_url=url
19      )
20      match1.put()
21
22      match2 = AffinityUrl(
23          url="http://domain.com/dir/page2.html",
24          affinity = .8301,
25          entered_url=url
26      )
27      match2.put()
28
29      matched_urls=url.matched_urls.order('-affinity')
30
31      aff_entries = AffinityUrl.all().order('url')
32
33      template_values = {
34          'url' : url.url,
35          'matched_urls': matched_urls,
36          'aff_entries': aff_entries
37        }
38
39      path = os.path.join(os.path.dirname(__file__), 'index.html')
40      self.response.out.write(template.render(path, template_values))
41
42  class EnteredUrl(db.Model):
43      url = db.StringProperty(required=True)
44
45  class AffinityUrl(db.Model):
46      url = db.StringProperty(required=True)
47      affinity = db.FloatProperty(required=True)
48      entered_url = db.ReferenceProperty(EnteredUrl,
49          required=True, collection_name='matched_urls')
50
51  def main():
52      application = webapp.WSGIApplication(
53                                         [('/', MainPage)],
54                                         debug=True)
55      wsgiref.handlers.CGIHandler().run(application)
56
57  if __name__ == "__main__":
58      main()

The above stores a URL someone has entered, and then stores other URLs that match it by some degree (the "affinity"). The affinity is a numeric score. This is a simple one-to-many relationship, and to get at the data using standard SQL, we'd write something like:

SELECT
    entered_url.url,
    affinity_url.url,
    affinity_url.affinity
FROM
    entered_url
JOIN
    affinity_url
ON
    entered_url.KEY=affinity_url.FOREIGN_KEY

Here are the steps using the GAE Datastore.

Lines 42-49.
First, let's define the data Model. EnteredUrl defines a single string property, url, for the obvious reason. AffinityUrl defines a string property for url, as well as a float affinity property, for storing the score.

Lines 48-49.
Also, AffinityUrl defines a db.ReferenceProperty named entered_url, which refers to an EnteredUrl object. This is the link between our two data objects, and how we effectively do a JOIN. The collection_name, matched_urls, is used to refer to the collection of AffinityUrl objects that will be linked.

Lines 12-13.
When the page is loaded in the browser we create an EnteredUrl entity named url, setting its url property to a string value.

Lines 15-27.
We setup two AffinityUrl objects, and assign them both a url and a numeric score. Additionally, we point entered_url to our EnteredUrl object, url. We have just linked one object (url) to many (match1, and match2).

Line 29.
This line queries the data in the one-to-many way, and stores it in an object, matched_urls, which I pass through to the template for iteration and output. This is where the collection name we defined in the db.ReferenceProperty attributes is used. Note that the collection name, matched_urls, is called like a method from url, since url is the object being referenced.

Line 31.
Additionally, for illustration, I query the AffinityUrl object data and save it in aff_entries. Just as in SQL, where you can JOIN tables, or query them individually, the App Engine allows you to do both. (Hopefully, you've realized by now that although they look and are accessed differently, these linked entities are behaving quite a lot like relational database tables.)

In the template, I output the data from matched_urls by getting each AffinityUrl object in the collection, and displaying that URL. Note that because of the .order('-affinity') call, we are displaying the URLs with the closest affinity at the top (descending order).

<table>
{% for affurl in matched_urls %}
<tr><td>{{ affurl.url }}</td></tr>
{% endfor %}
</table>

Load this up in your browser, and refresh a few times, and this is what you get:

You may have noticed from the code that I also pass all the data stored in the AffinityUrl model (line 31) to the template as well. This is output in the second table, above.

Because I've refreshed the page several times, I've generated and stored the match1 and match2 objects multiple times to the Datastore. This highlights something strikingly different about the Datastore and a SQL table. SQL statements like the one I give will display all the entries that match between EnteredUrl and AffinityUrl, even if entries in AffinityUrl are duplicated. As you can see, even though we have duplicate AffinityUrl entities stored, the reference from the EnteredUrl entity is smart enough to realize that they are duplicates, and only displays the ones that are unique. Update: please see the comments for a correction of the previous statements. The Datastore is creating new entities each time with a unique ID...

The Datastore takes a little getting used to, especially for those experienced in the standard relational data models. (Good ol' paradigm shift.) The GAE documentation feels unfinished or at least rushed, which is unfortunate. I personally think they should have concentrated more on giving good examples that demonstrate mapping relational concepts to Datastore concepts, since the majority of developers looking at the GAE will be old hands at the relational stuff.

I'm sure they'll get there eventually. In the meantime, I hope you found this tutorial useful.

I'm Trying To Quit... Commercial Software, Pt. 1

noreply@blogger.com (JA) — Tue, 15 Apr 2008 04:42:00 +0000

This experiment started out simply enough. It was 2007, and I got a new laptop. I had been running Quickbooks 2004 for our checking accounts, and Office 2003 for our meager office tools needs. I decided this software would stay on my old laptop (now my wife's), and I would try Free Open Source Software (FOSS) alternatives on the new one. I was bored with Office, and fed up with Quickbooks, anyway, so why not?

From there, the experiment broadened, and I decided to see if Linux/FOSS could keep me from ever having to boot into a proprietary system (Windows), or use proprietary software. I decided to keep notes, and now I seem to have enough material to start sharing the experience.

This is where it begins. I replace Quickbooks with GnuCash, and Microsoft Office with OpenOffice on my new laptop, which is running Windows XP.

GnuCash

Since I wasn't sure of anything, I didn't move our checking accounts out of Quickbooks. My wife and I were simply doing the laptop shuffle anyway, so it was just easier to leave everything where it was, and continue to maintain our registers on the old laptop.

However, we wanted to start a monthly budget, and I decided to let GnuCash step up and take a shot. Installing GnuCash was as easy as any other Windows application. Simply download the installer and run through the prompts. No sweat.

After doing a minimal amount of reading, and marginally more button punching and tab poking, I figured out that I would have to first create a register, and then apply a budget estimation to it.

So I setup a register called Monthly Budgeting. We decided on a monthly dollar amount, and I made this the initial deposit. Then, I began entering our receipts.

Here's what the register looks like:

So far, nothing surprising or mind-boggling. GnuCash felt a lot like Quicken. There's only so much variation a register is going to have, after all. This is good, because it means that the learning curve from one product to the next is minimal.

After finishing all my month's entries, I did some more poking around, and finally got the budget estimate working. Hint: Select the Budget tab, click "Options" to set your intervals etc, and then click "Estimate".

Here's our budget after a few months of keeping track. The budget outline for the Monthly Budgeting register is displayed horizontally, each month showing whether you are under budget (positive dollar amount), or over (negative) for that period.

The only problem I've had was with the backup. Quickbooks has an easy backup feature, and the backup is stored in a single file. I've been backing up GnuCash by copying all the files from its directory to a flash card.

This seems to work okay, but at one point GnuCash (or I, or both) got confused, and I had to restore from the backup directory, and in the end I lost about a month's worth of entries. The backup could be a little easier, I think.

GnuCash has worked out well. I've since added my business register to it, and it has all the standard features that you would at least find in Quicken. I'm not an accountant, so I can't really say whether GnuCash could replace Quickbooks for a business. I can say, however, that it seems like a pretty painless way to not pay for software for managing your personal check registers and budgets.

OpenOffice

This will be pretty short. I barely ever use MS Word for anything, but occasionally need Excel. My wife uses Word the most, but not in any way that OpenOffice (or even Wordpad) couldn't handle.

So far, Calc has been sufficient for my spreadsheet needs. There was barely a learning curve, and like I said, I don't make too many heavy demands on a spreadsheet. MS Access is another story, but for me that's more of something that I might use in development (say, of a .NET application, because it was convenient), so I'm not including it here.

Like GnuCash, I think OpenOffice ranks high enough in quality and design to work fine for a very large percentage of home users, and even for a lot of offices. As time progresses, whatever gaps there may be will only get narrower.

So...

As you might have figured out by now, this experiment is not a feature-by-feature scrutiny of competing products. I'm just using software the way I would normally, which is essentially, "I don't care about feature X, until I need feature X". I think most people work this way, unless they have a specific reason to become an expert. I'm not an accountant, doubt I will ever be an accountant, so I don't put a whole lot of time learning every arcane feature available in Quickbooks. I learn enough to do what I want, and won't go further until I need to.

In this experiment, FOSS is effectively graded on whether or not it can substitute all or most of my proprietary software needs, in the way in which I use software. It is highly subjective, and human nature, like laziness and apathy, is very much a part of it, as you will see.

(Next up: My old laptop dies, and we have to get another one. I decide to try Linux along with Vista, and see how little I actually have to use Vista.)

Insert or Update With a Single SQL Statement

noreply@blogger.com (JA) — Thu, 10 Apr 2008 14:25:00 +0000

Ever come across the situation while developing data-driven web applications when you needed to create a new record if one doesn't exist, but if one does exist, then you need to update it instead?

I certainly have, and I must admit with some shame that in the past I've handled it in the most obvious, and least elegant and efficient way, by

querying SQL for the existence of the record,
checking the result set in my code by looping and assigning a variable,
checking the variable for a value, and if one doesn't exist, then doing the insert.
Otherwise, doing the update.

There are a couple problems here. First, it's a lot more code than necessary. Second, it requires two calls to SQL instead of one.

You can eliminate this by making SQL do the conditional logic for you, via IF EXISTS. Here's the sample:

IF EXISTS(
 SELECT 1
 FROM MY_TABLE
 WHERE ITEM='somevalue' AND ENTERDATE='12/31/1999')
    --Update Statement
    UPDATE MY_TABLE
    SET ITEM='anothervalue'
    WHERE ITEM='somevalue' AND ENTERDATE='12/31/1999'
ELSE
    --Insert Statement
    INSERT INTO MY_TABLE
    (ITEM, ENTERDATE)
    VALUES
    ('somevalue', '12/31/1999')

EXISTS lets you run a query statement, and if a value is returned, it outputs true. Otherwise, it outputs false. Couple that to IF/ELSE, and you can see how useful this particular SQL clause is.

The query inside EXISTS returns 1 if the parameters in the WHERE clause match, and returns nothing otherwise. What we return really doesn't matter. We're interested mainly in the parameters. If the parameters match something, then we will update them. Otherwise (ELSE), we insert them into the table.

Pretty simple. We just add our code parameters to the above statement (if your language uses parameters, e.g. Perl or C#), and send it on its way. One SQL call, and a lot less logic.

Update: I should have been clearer. This is TSQL, and will not work, in say, MySQL. (Thanks anonymous commenter!)

Reverse Callback Templating

noreply@blogger.com (JA) — Wed, 19 Mar 2008 15:41:00 +0000

I've just had my first article ever published on Perl.com. It covers a template module I've written -- in Perl, obviously -- called Template::Recall.

Template systems provide a way to separate concerns, that is, design from logic. I won't cover it here, because that would be more than a tad redundant. If this topic interests you, here's the article link:

http://www.perl.com/pub/a/2008/03/14/reverse-callback-templating.html

Also, you might want to read this conversation on Perlmonks.com:

http://www.perlmonks.org/?node_id=674225

You'll see that template systems are a much debated topic. And if I may venture a personal observation, the Perl language has covered the topic more than any other language out there, and in much greater depth.

My Two Perls

noreply@blogger.com (JA) — Wed, 20 Feb 2008 15:11:00 +0000

Perl's greatest blessing and greatest curse, in my opinion, is CPAN. CPAN is an unbelievably rich repository of modules that do everything imaginable. I can't think of another language that has a resource like it. But using CPAN on the most widely used desktop platform available, Windows, presents some problems. Here is one developer's Perl on Windows saga.

Historically, I've always run ActiveState Perl. It's a great Windows distribution, and ActiveState has done a lot of work to make it very user friendly, especially by creating PPM, the Perl Package Manager. As opposed to the standard CPAN installation mechanism, which generally expects you to "make" your modules, sometimes compiling sources, PPM provides pre-compiled packages, so it's no hassle at all to install them. It's just a download/copy operation, really. The problem here is that if ActiveState's PPM repository doesn't have the module you want, you're back to compiling from source.

At some point (as I became nerdier, I guess), I decided to play around with compiling my own version of Perl and bundling it with a few important web modules from CPAN (i.e. CGI, CGI::Ajax, DBI, SOAP::Lite, Template Toolkit, etc), along with Apache/mod_perl, and MySQL. I decided to make this a distribution, and named it zangweb. It was intended to give a Windows developer everything he needs to start programming in Perl/Apache/MySQL with as little effort possible.

zangweb Perl replaced ActiveState on my machine for some time. I no longer had the convenience of PPM, so I just went ahead with the standard CPAN way of installing modules. For most modules, this wasn't too big of a headache. You just need to be sure to have a working development environment, one with nmake.exe available, and most modules installed without difficulty. Generally, I did something like

c:\>vcvars32
c:\>perl -MCPAN -e shell

and installed from the CPAN prompt.

However, modules like PerlMagick, or any others that had complex C/C++ builds and originally been developed for *nix, did not build easily. They took a lot of work, and while I thought it was kind of fun, from a hobbyist standpoint, I don't know if under other circumstances I would have wanted to go through all that trouble.

Nonetheless, zangweb worked well, and I was pretty content. Then Perl 5.10 was released, and it was available from ActiveState in short time. I wanted to try 5.10, naturally, and as usual, the path of least resistance was ActiveState. I downloaded it and ran it alongside zangweb Perl at work. On my own laptop, I decided I would try a different configuration: ActiveState Perl 5.10, and standard Apache and MySQL installations. Kind of as a comparison to see how valuable zangweb really was.

I realized the only thing that made zangweb more valuable was all the work I had done to get those web CPAN modules compiled and installed. Yet again, it boiled down to the modules, and the difficulty that came with installing them on Windows. For instance I want to have PerlMagick. I have it for zangweb. So far, ActiveState doesn't for v5.10:

http://ppm.activestate.com/BuildStatus/5.10-P.html

But you might get lucky, and find some kind soul who has created and bundled it in a PPM friendly package:

http://www.google.com/search?hl=en&q=filetype%3Appd ...

But I don't want to have to rely on the kindness of strangers to get my "must have" modules.

Recently, I wanted to do some charting in Perl. After looking around at the modules, I decided I wanted to use GD::Graph. This relies on libgd. At the time of this writing, they don't have a compiled binary for Windows for the latest revision. So now I've got compiling ahead of me once again.

After trying unsuccessfully to get it to compile natively on Windows, it dawned on me: Since CPAN is designed so much in the *nix way of doing things, why not make my second, "alternate Perl" run under an emulation of the Linux system? All the tools that are usually expected by these kinds of libraries, bash, configure, make, etc., are there, so surely I'd have a much easier time getting these modules on my machine this way.

No way to know until you try. I installed Cygwin, which came with Perl 5.8 already bundled. GD::Graph expects you to have libgd already compiled, so I went through the steps to do this, using my freshly installed Cygwin bash shell.

This is where the story gets remarkably pleasant.

I downloaded the libgd source, and after reading the README, downloaded the libraries it required, i.e. libpng, and freetype. These two compiled no problem. I jumped back over to the libgd source folder, did its configure and make steps, and after waiting a while for things to compile (something I'm not real fond of, I must admit), had a working version of libgd. The CPAN install of GD::Graph was a breeze after this, and soon I was charting in Perl, happy as could be.

Soon enough, I began to wonder why I wasn't just using Cygwin Perl as my main, and perhaps only, Perl distribution. I tried to think of anything I was doing with Perl that was only available to a Win32 distribution. (Yes, I know, that is kind of funny in retrospect.) Nothing came up.

The only thing I wondered about now was whether running Perl under emulation would be significantly slower than a natively compiled version. I know it should be slower. The more important question was would it be slow enough to matter?

The quickest, most basic way that I could think to check was to make Perl count. So I ran the following with each of my Perls:

ActiveState:

perl -e"$a=time; for($i=0;$i<=100000000;$i++){} print time-$a"  21

Cygwin:

perl -e'$a=time; for($i=0;$i<=100000000;$i++){} print time-$a'  19

Cygwin was faster by about 2 seconds. This satisfied me, initially. At least I knew that there wasn't an embarassing difference in performance. Curious, now, however, I found some good benchmark tests on the web, primarily for comparing the performance of different languages, but definitely useful for what I was trying to do. I downloaded the nsieve Perl code. This performs the Sieve of Eratosthenes, and is a way of finding primes.

Here are the results:

ActiveState:

perl C:\cygwin\home\nsieve.pl 7
Primes up to  1280000    98610
Primes up to   640000    52074
Primes up to   320000    27608

11

Cygwin:

perl nsieve.pl 7
Primes up to  1280000    98610
Primes up to   640000    52074
Primes up to   320000    27608

11

They both ran in 11 seconds. I'm reasonably satisfied that Cygwin, for most of my development purposes, will be fast enough.

So that leaves me with a nagging question. Why am I running two Perls? Unless there was a specific case where I need ActiveState -- performance or compatibility with some poorly designed app -- why not just run the Perl that works with CPAN?

Then My Two Perls can become My One CPAN-Compatible Perl. I like the sound of that, as a matter of fact. Because really, that's what it's been about all along.

High Level Languages Are Magic

noreply@blogger.com (JA) — Mon, 14 Jan 2008 04:30:00 +0000

After pondering the recent flap about how CS departments aren't providing a sufficient education by starting students in Java and ignoring lower level languages [link, link, and link], it seems to me that the problem can be boiled down to the simple fact that high level languages do too much work for you. They make it unnecessary to think about the low level things that cause the code work. It becomes easy to think of those things as "magic", and by and large dismiss them. Magic is an important productivity booster, but should be implemented only after understanding, to some degree, the little cogs that help it arrive.

High level languages do work hard for you, and I consider this an ultimate good, because I have a lot of work to do, and want to produce results as quickly as possible. One of the mantras of Perl is that given a context, it will simply Do The Right Thing. Java and C# make it unnecessary to think much (if at all) about pointers. This makes my life a whole lot easier. But it still helps to understand lower level concepts, for instance when considering the performance of various objects in a language, like StringBuilder in C#*.

I think magic is a danger even beyond CS departments. It's also inherent in productivity tools like Visual Studio, which will probably be learned on the job. If you only learn to use the magic, but don't understand that it isn't really magic, then you're headed for trouble.

I have a contractor who works for me developing ASP.NET applications. Out of college he didn't know C#/ASP.NET, but at a previous job had picked it up à la Visual Studio. I was just getting into ASP.NET myself when he hired on, and was a little confused about how the [auto] postback worked. I thought it would be quickest to ask someone with experience, so I did. But he didn't know, even though he was already producing fairly complex web applications for us. In his view, it was just a feature of ASP.NET, and beyond that was not important, as long as he could turn it on or off in the Properties of the various controls.

I had reasoned that it must be JavaScript, unless ASP.NET installed some sort of binary control on the sly. Sure enough, it turned out to be JavaScript. I decided then and there that our contractor was at a disadvantage because he understood web development primarily through Visual Studio, and this hindered him from realizing that ASP.NET was made to fit a set of (effectively lower level) standards, not the other way around.

We do a lot of programming in ASP.NET using Visual Studio, and it is a productivity booster. We're building more powerful applications in shorter time frames, and with less effort. Its magic is definitely appreciated. But seeing beneath the magic is what allows us to really understand and fix bugs, and build robust, maintainable applications.

Magic is for productivity. It's for those who have gotten the education, and the education is gotten by understanding the little cogs and how they relate to one another.

* This discussion, I might add, jumps right into the low level arguments of memory management, showing just how far removed you really are from those little cogs...

The Case for Flat-Threaded Discussions

noreply@blogger.com (JA) — Fri, 21 Dec 2007 16:32:00 +0000

As I stated in a previous entry, I've recently built and released an open source "conversation" system called Sylbi (currently in beta). This system was based on the idea that blogs with comments and forums differ very little, and there was no reason why you couldn't build a system that could be both a forum and a blogging platform.

Because Sylbi provides the ability to have discussions, that is, multiple people respond to each other's posts over time, it had to deal with how to display those conversations. The two most common ways for doing this are the flat and threaded models. For a detailed and intelligent commentary on the virtues of these methods, see this post from Coding Horror, and this one from Joel On Software.

As I began thinking about this problem, I decided that there is a third method for displaying conversations, one that I feel is preferable to the other two: threading without indention, or as I like to call it flat-threaded. Here is my conclusion, posted on the official "blog" for the Sylbi project. (You can read the full post here, which talks about this as well as the other unique features of Sylbi.)

It is my opinion that threading a conversation, that is, grouping replies to a post immediately below that post, provides the most logical organization method. Slashdot discussions are threaded, as are those on reddit. However, I think that indenting replies adds no real value, and instead actually makes the conversation more difficult to read. Sylbi threads conversations, but uses no indentation. So as you scan posts from top to bottom, post replies are clustered together, but you must use the content of the posts to determine the grouping. I refer to this as a "flat-threaded" conversation. Sylbi provides the means to quote previous posts, if this should be necessary.

Here's why I think this view works. Books are written from top to bottom. If an author refers to something that occurred in a previous chapter, you rely on your memory and comprehension to understand the reference. If the reference is subtle enough, an author may quote himself. Where a conversation is concerned, I think that memory and comprehension don't need to be aided by indentation, and where a reference may require it, you can easily provide a quote.

I am committed to "eating my own dogfood", and so am using Sylbi while I work on it. I have a live version running on my web hosting provider and use it to identify problems with my code as well as my assumptions.

One of my initial tests was of the flat-threaded view, and I created this conversation (which I unfortunately made a little difficult to read by using tons of self-references) and began using it to probe the concept. This was a discussion, so as I coded, I tested by adding to it, and eventually, this analogy fell out:

Consider a real conversation amongst a group. A topic is started by Alice, and Bob and Charlie discuss it with her for a length of time. Then, Bob touches on an individual point of Alice's initial topic, and a segue is created. Let's say that only Charlie and Bob discuss this point. Alice is silent. But she hasn't said everything she wants about the initial topic, so after they are finished, she brings them back to the topic, and they discuss it further. Bob's segue "held place" for additional comments by Charlie and Bob, and then the original topic was resumed. Viewed in a linear sense, this is exactly what a flat-threaded conversation does.

The holds place comment above is in reference to the (at least logical) "fairness" of grouping responses together. Because it is likely that a response to a post may come days after other posts have been made, and earlier posts are pushed down as this latecomer is inserted below the post it's a response to. For example:

Initial post (entry) E [day 1]
Response (to E) R1 [day 1]
Response (to R1) R3 [day 2]
Response (to E) R2 [day 1]

A response to a post becomes a subordinate post, as R3 is a subordinate to R1 above. R1 comes before R2, because it was posted earlier. So any responses to R1 get inserted directly below it, ahead of other, potentially earlier posts (R2). So R1 held place for R3, and it had the right to since it was made earlier. This is a sort of "first come, first serve for all my children" mentality. But it serves an important purpose: to keep direct responses together, which provides better cohesion, I think.

Of course, there is a caveat. The holds place idea is susceptible to gaming. For instance, if you want to have your entry appear higher up in the list of responses, you could respond to a higher level response, even if the content of your post is not particularly relevant to that one.

Taking our example above, let's say that it's days later, and there are over 100 responses. You want to post, but hate the idea of being all the way at the bottom of the list. So you pick the first response below the initial entry, and respond to it, but really, you just want to sound off on the original entry. Because you are responding to R1, the system inserts you at the bottom of the subordinate list for R1, which puts you higher in the list than other posts that followed the rules.

This is somewhat mitigated, however, by the fact that in 100 responses with no indention, it is difficult to be entirely clear which post is actually subordinate to which, and therefore where your post is going to appear vertically. It will be much more reasonable to simply respond to a post when you feel that the content of that post requires one.

On the other hand, this may simply be a risk involved with human communication, and a small one at that. Further on in the dogfooding conversation above, I observed that real human conversation is far from trouble free.

Alice starts a topic with Bob and Charlie. A segue is created, and Alice interjects that they are getting off the subject, and Bob and Charlie return from their tangent. Or they don't, and Alice's conversation is hijacked. I've also seen this conversation pattern (and been involved in it from probably all perspectives).

So I think that, basically, when talking about the "natural" flow of conversation and the mantra that trying to mimic this in a forum [is good], it should be noted that real conversation is not necessarily a smooth or clean or non-anarchic interaction. It can be, but it can also be an incredible mess, incredibly trite, or some mix of both.

Ultimately, I think the flat-threaded method provides a slightly better view of online conversations by trying to be as contextual as possible, and simplifying the presentation. However, just like real conversations, much depends on the humans.

Dormant Sticky Memory and Layered Comprehension

noreply@blogger.com (JA) — Sat, 15 Dec 2007 23:20:00 +0000

I recently finished reading Descartes: The Project of Pure Enquiry by Bernard Williams. As soon as I read the last page, I moved back to chapter 2, and started again from there. This is because I had retained and comprehended only about 50% of the book. Through the years, as I've learned better how to learn, immediately rereading has become an invaluable device for me, especially with a subject where I lack familiarity or educational background. (Like philosophy.)

If you had asked me on page 303 (the last one) to recall or explain anything from chapter 2, I would have been hard pressed to give you an answer. Just now, having finished reading the chapter again, I'd say that I grasped it nearly in full.

What I found really interesting, however, was how those things that I wouldn't have been able to recall at the end of the book jumped out from somewhere in the back of my mind the moment I read them again. For instance, there is an argument about "false lemmas" that uses an analogy about owning a Ford. After rereading the first few sentences, I could recall the full argument in most of its detail.

So there must be some aspect of memory that works like a hard drive. (There is: it's called long term memory.) It just dumbly writes the "file" there in one of its sectors, where it resides unknowingly until something recalls it and loads it into short term memory (RAM), where you can actively use it.

Here's a useful little graphic from Wikipedia (note: this model is criticized for being too simplistic, but it fits pretty well with how memory works upon personal reflection, so it's still a useful visualization, I think):

When I first read the book, I had very little stored on the subject of Descarte's Cogito ergo sum. Mr. William's book is a thorough analysis of the subject using modern logic, with the benefit of centuries of debate preceding him. In short, it was a pretty steep curve to dive into. This is why I think that on my first pass I retained and ultimately comprehended so little.

On the second pass, it was quite different. I had obviously retained more than I thought, but since it wasn't coupled with strong comprehension, it seems to have been just rather "dumbly" stored. I doubt that if I had never read the book again, I would have been able to explain the "false lemmas" argument. Perhaps I would have recalled hearing about it somewhere, but it would have been foggy.

But as I reread, my mind already had some notion of the concepts, and so comprehension occurred more rapidly and to a fuller extent than before. You might say that my comprehension came about in a layered manner. A hazy concept lay in memory, was fortified by reprocessing the original text, and then stored again (to disk!) as a much more useful item.

This makes me think of my early days learning to program, when there were plenty of concepts I was unclear about, and I was rereading all the time. I was playing around with QBASIC on a DOS computer, then tried my hand at Turbo Pascal. Languages ultimately without a future.

But I learned the "primitives" of programming from those languages: variables, looping, conditionals, routines, etc. This is a layer of comprehension and sticky memory still employed today. In fact, it's quite clear to me that despite the plethora of languages available, with all their different syntax and conceptual leanings, the actual number of concepts you need to understand really well are not that numerous. And once you've obtained and stored those layers, further comprehension occurs much faster.

For example, once you understand C pointers and how they work, all reference work in any language, whether Perl, C#, Java, Python, is easy to understand. The nuance presented by the language is just another, usually small, comprehension layer that must be added.

As new programming paradigms appear, I notice that I am able to grasp them much more quickly than I did the primitives from my early stages of instruction, even though those concepts are usually much more abstract and difficult. This is because, I think, like the second reading of my book, necessary, prior concepts are lying dormant in their sectors, ready to be loaded and rehearsed. Except it's more like the nth reading, where n is a pretty high number.

So if you're new to programming, are overwhelmed by concepts and language choices, or feel like you're learning at much too slow a pace, never fear: if you stick with it and do the work, you will soon notice your comprehension and retention accelerate.

Sometimes Faster From Scratch: An ASP.NET Hit Counter

noreply@blogger.com (JA) — Thu, 13 Dec 2007 18:03:00 +0000

Some users of a SharePoint site I maintain wanted to add a hit counter to their statically created HTML newsletters. After a little looking, I decided it would probably be best to use the FrontPage hit counter that is included with IIS/FrontPage Server Extensions. It was already there, after all, and should be easy to implement. Well, it was easy to implement, except for one major snag: it wouldn't fire unless you were authenticated.

Try as I might, I wasn't able to figure out how to get it to work for anonymous users. Google didn't turn up much. I even posted on the Microsoft newsgroups, and got no response. I spent a day and a half searching, trying this and that, and was no further along than when I had started. No, I take that back. I was further behind than when I had started, because I had exhausted most of the possibilities I had started out with.

Because I was sort of stuck, I did a cost-benefit analysis (which makes it sound much more scientific and rational than it actually was) of continuing to troubleshoot the problem. I came to the conclusion that I had already spent more time troubleshooting than it would have taken for me to develop a custom ASP.NET solution from scratch.

So I switched gears, and began searching for code samples on the web. Within minutes, I found one in Visual Basic that did most of what I wanted. Of course, I program in C#, so I merely used it as a reference. I improved it a little, too.

(You can download the VS2005 project here)

Here's how it works:

The project consists of a single Default.aspx page, and an Access database, hitcount.mdb, for tracking URLs and hits.

The Default.aspx page is passed an address, in the form

Default.aspx?url=some/path/on/a/server.htm

or even

Default.aspx?url=http://some/path/on/a/server.htm

In truth, you could just pass in some arbitrary text and it would use that as an identifier. Using a full URL, however, ensures that you don't get duplicates.

If the hit counter finds the URL in the database, it gets the count, increments it, converts it to a font (Arial), converts the font to an image, and returns as Content-Type "Image/Gif".

If it doesn't find the URL, it inserts it, and returns a count of 1. It's that simple.

The only thing you may need to change for it to work on your server is line 50:

string datapath = Server.MapPath("\\hitcount\\hitcount.mdb");

Set this to the directory on your web server where the database will live.

To use this in your static HTML pages, simply add a line like the following:

A note on security: there is no check in the code to prevent someone from "borrowing" the hitcounter and using it to store hits for their own pages. This might be a negligible concern, but you could add some code to check the IP address where the request is coming from if you are worried by this.