Posts filed under 'Implementation'
So, this blog will be going into deep-freeze for a while (and I’ll regrettably have to close all comments shortly, because I don’t want to deal with the spam; email if you want to chat about what you find here). While I launch Fictionsuit, Cornucopt is going to take some time off to have an identity crisis. See, back when I first thought of the name Cornucopt, I actually envisioned it as an easier-to-install Ecore. With some push-button setup options for common social-software patterns, and in PHP. That’s about it. Oddly, as I retrace the development of Wicker, now rechristened Cornucopt, it has gotten closer and closer to Ecore. Let me give you a hazy sketch of what I think Cornucopt might look like if I get around to it after Fictionsuit, here in the post-Ruby on Rails world, where there are monorails through wide, green public parks with nary a trace of litter or cramped tennis courts.
Like Ecore, we start on the barest of wiki foundations. You have a bunch of things called nodes, or pages, or whatever, and you can add text and link them easily. Drupal does something like this, only you start with a front page and stuff like threaded comments enabled, for a more slash-cloney shape. Cornucopt’s starting structure will likely be similar to Drupal’s, only with non-threaded comments (or perhaps differently threaded) and tags enabled by default.
The interesting stuff about Ecore (or Drupal) lies in what you do from there, and how you do it. Ecore provides not only a framework for building out a site’s content, but for doing your customization and programming within the Ecore environment itself. You can define a node with a nodetype of “nodetype,” and then just start writing code into a textarea that defines what nodes of your new nodetype will do, and how they will display. You’ve got certain objects you can read or manipulate - a user object, a session object, the page’s text, and suchlike - and you just code up what you want. In theory, I think you have access to all of Perl.
So, even before I was all making out with Ruby on Rails, I was starting to think that Corny’s scheme of pagetypes was kind of a half-measure. I mean, if you want to make something new, you’ve got to write a bunch of PHP into a file, set all your permissions and stuff in there, throw the file up into a poorly named and placed directory (I was gonna get around to fixing that, honest), and then start using it and hope it works. I guess the last step is more or less immutable. But having tried to build things in Ecore, I could see that there was a better way, even though Ecore was pretty much the better way gone wrong. I wanted more things built for me, and more things that could hook together interestingly, and more things that were open to the rest of the net. Ecore doesn’t have a closed culture, exactly… but it has the Perl culture in spades. “There’s more than one way to do it, so we can’t possibly presume to do anything for you. Here’s a textarea. Good luck!”
The real reasons I decided to abandon Ecore were not the above, though. They were my unfamiliarity with Perl and the difficulty (impossibility, actually) of installing Ecore on shared hosting. Ironically, Rails isn’t very friendly with most hosting companies either. But this choice reflects my new, corrected-for-reality notion of who’s going to use Cornucopt and how. I’ve decided to commit to the fact that someone who really wants to customize a wiki is going to write code of some kind. I think they’ll find Ruby code friendlier to write than Perl code. I think Corny’s built-in options will go a decent-enough way to fulfilling other, less extensive needs that I don’t feel as bad about requiring code for the serious kung fu. And I think that Rails is hot enough that Rails hosting will spread fast.
Plenty of design hurdles remain, however. Funny enough, security of executable code isn’t a big worry; Ruby has some sandboxing built in, so I’m not really concerned about restricting them to selected objects and methods. Figuring out what methods people will need, though, might get interesting. Also, these built-in things I’m waving my hands about - what will they be, precisely? How will they be architected, and how the hell do I think I’m gonna make them all snap-together easy? And is it wise to build an application framework on top of another application framework like this?
All of these are questions I’ll have on the back burner until this fall at the earliest. First I’ve got to launch Fictionsuit, then build features I have foolishly already planned for “soon after launch,” then whatever comes after that. Then I need to see if a big generalizing machine to enable other people’s creativity is still something I want to build at all.
If not… well, all this work has been fun. I’ve learned a lot, including the ability to let go of projects without all that guilt. :-) But mostly I’ve learned about my work habits and ability to take a large project to fruition (or something like it) without having it just loom over me, ignored, like a heartbroken pet zombie. Much.
April 18th, 2005
It’s official: Fictionsuit is now a Ruby on Rails project rather than a Cornucopt project. Sometime after we are satisfied with the footing of Fictionsuit, Cornucopt itself will become a Ruby on Rails project, likely with some subtle shifts in focus as the new language and framework afford.
We may actually get Fictionsuit into beta in April due to this.
March 20th, 2005
So there’s this interview with Alan Kay, an old-school Apple guy and frequent denigrator of today’s computers. Jes was talking the other day about how Kay’s metaphor of contemporary apps being like the pyramids - brute-force stacks of blocks made possible only by large numbers of Egyptian slaves - was so depressing for an ambitious coder. What we need is not more pyramids, but cathedrals. But no one has discovered the arch yet.
When the architects of Everything2 attempted to productize their core code (along with some other stuff), they named their company Blockstackers Intergalactic. In the best case, programming feels pretty much just like that: stacking blocks, like a happy or at least distracted kid sitting on a shag rug in the middle of the family room. In the worst cases, it feels a little more like being a slave in Egypt. Big fucking blocks, those were.
So that was on my mind when I concluded my post yesterday. I think I’ve been slacking off on 0.2 milestones because what’s left to do - adding a feature to the admin pages, then adding another one, et cetera - feels very much like mindlessly putting one thing on top of another. Many of the distractions I’ve been employing lately have actually been themed around (hopefully) eliminating some of this kind of busywork. Funny enough, not long ago I linked to one of them - Ruby on Rails, a web framework that promises, and seems to deliver, dramatically faster development of database-backed web apps - as a joke about the other context of “on rails:” that of being bored and having nothing to do but mindlessly move forward. I am hoping that RoR promises the opposite, but there’s no time to waste exploring it now.
Must. Stack. Blocks.
February 17th, 2005
What’s to be done for 0.2, zoomed in a bit:
Better admin - about a third done, I’d say.
Refactored page object - we’ll include db schema and methods for children/hierarchy and permissions right there on Page by default, instead of relying on pagevars and pagetypes. Some other things might shift around as well. This is not started yet.
More permissions UI - If I get ambitious I might want to do something like, say, give the creator of a page the authority to delete comments (at least for some pagetypes). I’ll need that anyway for weblogs when we get around to those, and just generally need to think through how such pagetype features are going to interact with permissions. Help would be welcome on this and I’ll try to rustle some.
We’re behind schedule for Fictionsuit and I have spent too much time fretting about design issues. Need to start stacking the blocks again.
February 16th, 2005
Found this paper on something Drupal is preparing or possibly not preparing, called the Content Construction Kit. I don’t know enough about the technical underpinnings of Drupal to be able to say exactly what’s going on here, for example, but even just the beginning of the next page tells me that there is some useful thinking there for how to extend Cornucopt’s content model out to the next level.
See, it turns out to actually be pretty hard to deal with things like search. So far, searching CoCo has been limited to title searches, but once you do full text search, how do you give users and/or admins a manageable way to say what it’s appropriate to search and what isn’t? Another way of looking at search is from the perspective of application-wiki problems. How do you tell your in-page widget to do something to a given set of pages? And truthfully, even title search is complex - I’ve already special-cased Corny’s database to keep track of what’s a comment so that they don’t appear in title searches. It’s not as if I should have to tell the changes table what every pagetype is, and write special SQL code for them all. And search isn’t the only such problem - RSS feeds are starting to look similarly tough.
So I’m hoping something in this document is gonna save my ass. The hope is that I will end up with an easy way for admins and/or users to define new pagetypes without writing PHP, and in some way consistent with permissions and generally not naughty.
February 3rd, 2005
Preliminary tests on comments are going quite well on the test site, ho ho ho, and I should have some more integrated means of letting people play with them there shortly after new years. 0.2 will likely have a little more development of the adminny stuff, particularly user pages, comment deletion and usergroups, and then we’ll call it a point release. 0.3 will be weblogs, maybe some other hierarchic thingies, and probably wikified templates now that I’ve seen how easy that ought to be.
This maps pretty well to the needs of Operation: Sleeping Princess, except for some even more abstractified hierarchy that cuts to the core of Corny and will likely not be making it back into the source tree. We’re aiming for a beta of Op:SP in April.
It’s been a great year for the writing of unnecessary software! Here’s to more perverse notions and utterly compromised hackery leading the way to a better world in 2005.
December 27th, 2004
1) We now have in mind, and in the design process, the first major site that will be built on top of Cornucopt. Its needs are fairly specific, as the needs of most CoCo sites will be, so it will distort development in interesting ways.
2) Case in point: I’ll be developing hierarchy more fully for 0.2 instead of wikifying templates. (Work on the core pagetype architecture is complete for now, knock wood.)
November 20th, 2004
Dammit.
Although it looks like they don’t really do customization of the overall look of the page, plus they put script in the wiki pages, which always kinda gives me a rash. I mean, I guess they have some means of keeping non-admin users from accidentally hacking the script out, or they’re aimed at intranets and they don’t care. Also they seem to cost money.
From the beginning I have been aiming Cornucopt’s app-building features with the target of admins, not users, in mind. That is, the idea has always been that the human who sets up the server will know enough PHP to hurt himself set up the ways in which information will flow, and then other users will work within that. Maybe JotSpot’s approach is better, if they have a user-perms model that’s clean and easy. We shall see.
October 6th, 2004
As always, the more you do, the more there is to do. I have, technically, added “this is a small change” to Cornucopt as of yesterday; I was happily implementing all this stuff where the small change would overwrite the previous change, as long as the previous change was made by the same user. But I hit some sort of wall. The code was done, it looked like it’d work and everything… I don’t remember if it was thinking about the Recent Changes RSS that did it to me (I hate it when posts change slightly and my aggregator decides to tell me about them all over again!), or my doubts over what really caused the database explosion of a couple months ago, or what. I was just seized by dread and had to rewrite it.
So now all it does is keep your change out of Recent Changes. The non-bloat of the database will have to be the responsibility of a Kept Pages policy. That MeatballWiki link is an interesting read, but the concept can be a little tough to grasp. I’ll quote the salient bit:
We chose to version everything, but only keep revisions around for a limited timespan, say a week or a month. That is, destroy all revisions older than the timespan. The timespan will naturally depend on the amount of traffic to a site, the slower the traffic, the higher the timespan.
This scheme is not sufficient, though, as an attacker can destroy a page that hasn’t been touched in several months by merely editing it. After all, all its prior versions have been automatically erased, leaving you with only the latest, vandalized version.
Consequently, we add a little twist: timestamp the revisions not when they are created but when they are replaced. So, the current version of the page is not in the revision history, but when I make a change, the current version gets timestamped then and entered into the history. Then the new edit becomes the current version. […]
Equivalently, and more simply, you keep all previous revisions created during the timespan, plus one more.
That last sentence probably hews closest to how I’ll actually be coding it. But who the hell cares how I’ll actually be coding it.
Anyway. This policy seems the best compromise between posterity, security and efficiency. But it made me think of a lot of other things that are pretty important to have, which I haven’t worked on yet:
- IP banning (I have some GPL code pretty much ready to go for this, but there need to be admin pages to let you cope with it)
- Edit throttling (this is different from “shields up,” when you don’t take any edits at all - this is when one dork is flooding you with edits and you’d like to cool him off. Ideally you should be able to throttle individual users or IPs.)
- The fact that some users are going to want to keep every version of everything, which reopens the possibility that, to save bits, I should store things as diffs rather than full text, which reopens that technical can of worms
- Maybe I should just store small changes as diffs? I’d need to teach Version History and whatever I build to look at all changes to a page how to cope with diffs, but I think that’s it.
I also suspect there are more login bugs. I really don’t like those.
July 21st, 2004
And now, an exciting new chapter in the young life of Cornucopt: nobody can log in and no pages will display.
Last night, I innocently added a bug to the bug list, and the database apparently chose exactly that moment to hit the space limit assigned to it by my ISP. The pages table is now evidently corrupted. I can still pull data out of the changes table, so I was able to rescue some data immediately for the project some friends and I were embarking on (and which probably created all the data that filled the drive).
At least, that’s my best guess as to what happened; I am far from a DBA. But man, that changes table got freakin’ huge.
I had implemented (and have implemented) no “small change” functionality in Cornucopt. In many wikis, you can designate an edit as a “small change” to the page, like a typo correction or a layout tweak, and it will store that change lumped in with the last change, rather than add a new entry in the DB and in Recent Changes, annoying everyone. Good feature, just hadn’t gotten around to it.
It’s relatively simple to add such a “small changes” feature. It needn’t present a security risk of sneaking in a huge change (either by marking it as small, or by doing it slowly).
But is it really what I need? Does it explain how a database table that contained every full version of every page on a small, new wiki got to be 200 megabytes? When the pages in question are about a K at most?
I may have to bite the bullet and limit the number of changes that Cornucopt stores, to something like the last 20. I feel like I’m letting future generations down. For right now, though, I’m investigating all these problems and how other wikis handle them. And, of course, getting my actual work done.
May 17th, 2004
Previous Posts