arbinger systems

As I stated in a previous entry, I've recently built and released an open source "conversation" system called Sylbi (currently in beta). This system was based on the idea that blogs with comments and forums differ very little, and there was no reason why you couldn't build a system that could be both a forum and a blogging platform.

Because Sylbi provides the ability to have discussions, that is, multiple people respond to each other's posts over time, it had to deal with how to display those conversations. The two most common ways for doing this are the flat and threaded models. For a detailed and intelligent commentary on the virtues of these methods, see this post from Coding Horror, and this one from Joel On Software.

As I began thinking about this problem, I decided that there is a third method for displaying conversations, one that I feel is preferable to the other two: threading without indention, or as I like to call it flat-threaded. Here is my conclusion, posted on the official "blog" for the Sylbi project. (You can read the full post here, which talks about this as well as the other unique features of Sylbi.)

It is my opinion that threading a conversation, that is, grouping replies to a post immediately below that post, provides the most logical organization method. Slashdot discussions are threaded, as are those on reddit. However, I think that indenting replies adds no real value, and instead actually makes the conversation more difficult to read. Sylbi threads conversations, but uses no indentation. So as you scan posts from top to bottom, post replies are clustered together, but you must use the content of the posts to determine the grouping. I refer to this as a "flat-threaded" conversation. Sylbi provides the means to quote previous posts, if this should be necessary.

Here's why I think this view works. Books are written from top to bottom. If an author refers to something that occurred in a previous chapter, you rely on your memory and comprehension to understand the reference. If the reference is subtle enough, an author may quote himself. Where a conversation is concerned, I think that memory and comprehension don't need to be aided by indentation, and where a reference may require it, you can easily provide a quote.

I am committed to "eating my own dogfood", and so am using Sylbi while I work on it. I have a live version running on my web hosting provider and use it to identify problems with my code as well as my assumptions.

One of my initial tests was of the flat-threaded view, and I created this conversation (which I unfortunately made a little difficult to read by using tons of self-references) and began using it to probe the concept. This was a discussion, so as I coded, I tested by adding to it, and eventually, this analogy fell out:

Consider a real conversation amongst a group. A topic is started by Alice, and Bob and Charlie discuss it with her for a length of time. Then, Bob touches on an individual point of Alice's initial topic, and a segue is created. Let's say that only Charlie and Bob discuss this point. Alice is silent. But she hasn't said everything she wants about the initial topic, so after they are finished, she brings them back to the topic, and they discuss it further. Bob's segue "held place" for additional comments by Charlie and Bob, and then the original topic was resumed. Viewed in a linear sense, this is exactly what a flat-threaded conversation does.

The holds place comment above is in reference to the (at least logical) "fairness" of grouping responses together. Because it is likely that a response to a post may come days after other posts have been made, and earlier posts are pushed down as this latecomer is inserted below the post it's a response to. For example:

• Initial post (entry) E [day 1]
• Response (to E) R1 [day 1]
• Response (to R1) R3 [day 2]
  
• Response (to E) R2 [day 1]
  

A response to a post becomes a subordinate post, as R3 is a subordinate to R1 above. R1 comes before R2, because it was posted earlier. So any responses to R1 get inserted directly below it, ahead of other, potentially earlier posts (R2). So R1 held place for R3, and it had the right to since it was made earlier. This is a sort of "first come, first serve for all my children" mentality. But it serves an important purpose: to keep direct responses together, which provides better cohesion, I think.

Of course, there is a caveat. The holds place idea is susceptible to gaming. For instance, if you want to have your entry appear higher up in the list of responses, you could respond to a higher level response, even if the content of your post is not particularly relevant to that one.

Taking our example above, let's say that it's days later, and there are over 100 responses. You want to post, but hate the idea of being all the way at the bottom of the list. So you pick the first response below the initial entry, and respond to it, but really, you just want to sound off on the original entry. Because you are responding to R1, the system inserts you at the bottom of the subordinate list for R1, which puts you higher in the list than other posts that followed the rules.

This is somewhat mitigated, however, by the fact that in 100 responses with no indention, it is difficult to be entirely clear which post is actually subordinate to which, and therefore where your post is going to appear vertically. It will be much more reasonable to simply respond to a post when you feel that the content of that post requires one.

On the other hand, this may simply be a risk involved with human communication, and a small one at that. Further on in the dogfooding conversation above, I observed that real human conversation is far from trouble free.

Alice starts a topic with Bob and Charlie. A segue is created, and Alice interjects that they are getting off the subject, and Bob and Charlie return from their tangent. Or they don't, and Alice's conversation is hijacked. I've also seen this conversation pattern (and been involved in it from probably all perspectives).

So I think that, basically, when talking about the "natural" flow of conversation and the mantra that trying to mimic this in a forum [is good], it should be noted that real conversation is not necessarily a smooth or clean or non-anarchic interaction. It can be, but it can also be an incredible mess, incredibly trite, or some mix of both.

Ultimately, I think the flat-threaded method provides a slightly better view of online conversations by trying to be as contextual as possible, and simplifying the presentation. However, just like real conversations, much depends on the humans.

I recently finished reading Descartes: The Project of Pure Enquiry by Bernard Williams. As soon as I read the last page, I moved back to chapter 2, and started again from there. This is because I had retained and comprehended only about 50% of the book. Through the years, as I've learned better how to learn, immediately rereading has become an invaluable device for me, especially with a subject where I lack familiarity or educational background. (Like philosophy.)

If you had asked me on page 303 (the last one) to recall or explain anything from chapter 2, I would have been hard pressed to give you an answer. Just now, having finished reading the chapter again, I'd say that I grasped it nearly in full.

What I found really interesting, however, was how those things that I wouldn't have been able to recall at the end of the book jumped out from somewhere in the back of my mind the moment I read them again. For instance, there is an argument about "false lemmas" that uses an analogy about owning a Ford. After rereading the first few sentences, I could recall the full argument in most of its detail.

So there must be some aspect of memory that works like a hard drive. (There is: it's called long term memory.) It just dumbly writes the "file" there in one of its sectors, where it resides unknowingly until something recalls it and loads it into short term memory (RAM), where you can actively use it.

Here's a useful little graphic from Wikipedia (note: this model is criticized for being too simplistic, but it fits pretty well with how memory works upon personal reflection, so it's still a useful visualization, I think):

When I first read the book, I had very little stored on the subject of Descarte's Cogito ergo sum. Mr. William's book is a thorough analysis of the subject using modern logic, with the benefit of centuries of debate preceding him. In short, it was a pretty steep curve to dive into. This is why I think that on my first pass I retained and ultimately comprehended so little.

On the second pass, it was quite different. I had obviously retained more than I thought, but since it wasn't coupled with strong comprehension, it seems to have been just rather "dumbly" stored. I doubt that if I had never read the book again, I would have been able to explain the "false lemmas" argument. Perhaps I would have recalled hearing about it somewhere, but it would have been foggy.

But as I reread, my mind already had some notion of the concepts, and so comprehension occurred more rapidly and to a fuller extent than before. You might say that my comprehension came about in a layered manner. A hazy concept lay in memory, was fortified by reprocessing the original text, and then stored again (to disk!) as a much more useful item.

This makes me think of my early days learning to program, when there were plenty of concepts I was unclear about, and I was rereading all the time. I was playing around with QBASIC on a DOS computer, then tried my hand at Turbo Pascal. Languages ultimately without a future.

But I learned the "primitives" of programming from those languages: variables, looping, conditionals, routines, etc. This is a layer of comprehension and sticky memory still employed today. In fact, it's quite clear to me that despite the plethora of languages available, with all their different syntax and conceptual leanings, the actual number of concepts you need to understand really well are not that numerous. And once you've obtained and stored those layers, further comprehension occurs much faster.

For example, once you understand C pointers and how they work, all reference work in any language, whether Perl, C#, Java, Python, is easy to understand. The nuance presented by the language is just another, usually small, comprehension layer that must be added.

As new programming paradigms appear, I notice that I am able to grasp them much more quickly than I did the primitives from my early stages of instruction, even though those concepts are usually much more abstract and difficult. This is because, I think, like the second reading of my book, necessary, prior concepts are lying dormant in their sectors, ready to be loaded and rehearsed. Except it's more like the nth reading, where n is a pretty high number.

So if you're new to programming, are overwhelmed by concepts and language choices, or feel like you're learning at much too slow a pace, never fear: if you stick with it and do the work, you will soon notice your comprehension and retention accelerate.

Some users of a SharePoint site I maintain wanted to add a hit counter to their statically created HTML newsletters. After a little looking, I decided it would probably be best to use the FrontPage hit counter that is included with IIS/FrontPage Server Extensions. It was already there, after all, and should be easy to implement. Well, it was easy to implement, except for one major snag: it wouldn't fire unless you were authenticated.

Try as I might, I wasn't able to figure out how to get it to work for anonymous users. Google didn't turn up much. I even posted on the Microsoft newsgroups, and got no response. I spent a day and a half searching, trying this and that, and was no further along than when I had started. No, I take that back. I was further behind than when I had started, because I had exhausted most of the possibilities I had started out with.

Because I was sort of stuck, I did a cost-benefit analysis (which makes it sound much more scientific and rational than it actually was) of continuing to troubleshoot the problem. I came to the conclusion that I had already spent more time troubleshooting than it would have taken for me to develop a custom ASP.NET solution from scratch.

So I switched gears, and began searching for code samples on the web. Within minutes, I found one in Visual Basic that did most of what I wanted. Of course, I program in C#, so I merely used it as a reference. I improved it a little, too.

(You can download the VS2005 project here)

Here's how it works:

The project consists of a single Default.aspx page, and an Access database, hitcount.mdb, for tracking URLs and hits.

The Default.aspx page is passed an address, in the form

Default.aspx?url=some/path/on/a/server.htm

or even

Default.aspx?url=http://some/path/on/a/server.htm

In truth, you could just pass in some arbitrary text and it would use that as an identifier. Using a full URL, however, ensures that you don't get duplicates.

If the hit counter finds the URL in the database, it gets the count, increments it, converts it to a font (Arial), converts the font to an image, and returns as Content-Type "Image/Gif".

If it doesn't find the URL, it inserts it, and returns a count of 1. It's that simple.

The only thing you may need to change for it to work on your server is line 50:

string datapath = Server.MapPath("\\hitcount\\hitcount.mdb");

Set this to the directory on your web server where the database will live.

To use this in your static HTML pages, simply add a line like the following:

<img src="Default.aspx?url=some/path/on/a/server.htm" />

A note on security: there is no check in the code to prevent someone from "borrowing" the hitcounter and using it to store hits for their own pages. This might be a negligible concern, but you could add some code to check the IP address where the request is coming from if you are worried by this.

Mr. Joel Spolsky recently published a talk he gave to the Yale Computer Science department, and in it proceeds to expound in his all or nothing manner why it sucks to be an "in-house" programmer. That is, a programmer who earns his or her pay by working for a non-software publishing company. The problem with his reasoning is that it seems to imply that there is no satisfaction at all to be gotten from being an in-house programmer. Life, in my experience anyway, fails to work in such an open or shut way. His arguments have some merit, but are just too far on the end of the spectrum. I work for a school district. Obviously software development is not the primary function; however, there are many reasons why my job doesn't suck.

Another more subtle implication falls out as well: The attitude that the only real satisfaction you are going to get from life will be your job. This may just be a result of the black or white rhetoric Mr. Spolsky uses. Certainly there is satisfaction to be gotten from your work, and maybe a large amount. But other things yield satisfaction as well.

You never get to do things the right way. You always have to do things the expedient way. This is true in some cases, but not all. There have certainly been projects that I've worked on that solved a one-time problem and were never used again. Or that weren't given enough thought and had a horde of changes to make later. Or were handed to me on the assumption that the users wanted it when in fact no research had been done to actually find out if they did, and it turned out that they didn't. But not in every case. We've also hit the bullseye and wowed the crowd, and not simply by luck, but by careful analysis. Especially as I settled in and learned how things operate, and what is ultimately important to the distirict. These are things you can only really learn over time. And usually, there has been time to go back and refactor those projects that gained traction. (Whether I've been too lazy when I had the chance is another story.)

My boss has made decisions about technology without really listening to my thoughts. For instance, he went full bore on SharePoint without me really buying into it. In the end, we do a lot of customization for SharePoint, and my opinion of it has changed. But usually, and especially in the technologies used for development, I get to have my way. If I want to use Perl, Ruby, Java, .NET, it's usually left up to me. The reason for this is that my boss and the people who use our software really don't care how we do our job, just that we do it. This may sound callous, but it's actually not. I don't particularly care how a teacher teaches, as long as my child learns. I sense that in a company that produces software, a manager will have a much deeper knowledge of underlying technologies, and as a result may be more difficult to sway.

You [don't] get to make beautiful things. This is certainly arguable. Some applications we've produced were quite attractive, and using them was so intuitive and easy that no training was required. You may think the function of the software was boring (generating reports for student test scores based on demographics), but the software itself was attractive. Software is for doing things, and usually those things, by themselves, are nothing you are going to consider beautiful or artistic. Because software is a tool, just like a car, or a portable MP3 player. Even if it's the most beautiful MP3 player in the word, it still has a function that can be mimicked by less beautiful players. The "beauty", if any exists, is in the way the tool looks and behaves. This can apply to in-house software as well as any other.

Management doesn't care about you. This is another cut-and-dry statement that just doesn't map to real life. My job in the school district falls under "support services", along with food service (school lunches), maintenance, business services, etc. Teaching is the district's primary function. Yet I have never run into the attitude that I simply don't matter. When principals and teachers are shown how they can look up their student's progress throughout the school year within seconds, they usually appreciate the support services I provide. And don't think management isn't paying attention to that.

Some management may be more callous than others, I realize, but in my experience, most people are not so simple minded that they don't understand that the primary function of an organization usually requires many secondary functions in order to operate, and they know that those functions are important.

Your work is your life. Although this isn't stated directly, Mr. Spolsky's talk certainly seems to imply it. That you better watch out: if you don't land a job in that 20%, not only will your job suck, but by association, so will your life. I have found quite a few things as fulfilling as work. Becoming a father, training a German Shorthair for bird hunting, mastering the snowboard (okay, "mastering" might be a bit strong of a term). To name a few. My faith is another, very important fulfilment, but I won't push that on you. (I will say that we do a lot of missionary work in the Philippines and South Africa, and have been able to do children's feeds and provide medical care in those places, and that certainly is fulfilling.)

What about software? What if this is an itch you have to scratch? I certainly agree on this one. It's doubtful, working in-house, that you are going to get to build software you care about deeply. But how many open source projects are there? Why not start your own? Or build shareware. Really, the opportunities are limitless. I've done several things outside of my "day job". I've created aumpel, a shareware MP3 converter, madxlib, an open source DLL for MP3 decoding, and I've just launched Sylbi, an open source conversation system built with Perl and MySQL. I don't list these to brag. I just find it silly to say that the only way you are going to build software you truly care about is by finding a job in a software publishing company. There are many ways. The Internet offers them in abundance.

Mr. Spolsky's claims, especially couched in the terms he chose, hardly do justice to the 80% out there doing in-house programming. Some are very happy with their situation, some may not be, some might just be complacent or view work as simply a means to an end. The important thing is that it doesn't just "suck" or "not-suck". Life really doesn't work that way.

Here's an article I wrote some time ago when I was first exploring Ajax: Creating JavaScript Objects From SQL Queries. There's a VS 2003 project available for download as well. The article demonstrates a way to convert the result set of a SQL SELECT query to JSON, or Javascript Object Notation. JSON is easy for Javascript code to parse and display. Much easier than XML, in my opinion, which is what Ajax applications were originally designed around.

An ASP.NET page, response.aspx, acts as a JSON "server" -- it receives a SELECT query, gets it from the server, translates it into JSON, and returns it. An Ajax-capable HTML page acts as the "client", and uses the JSON object to display data dynamically on the page.