Wednesday, November 08, 2006

Practically Web Hacking in Perl6

Audrey Tang recently announced that the Pugs project has been moved to new facilities, and the subsequent reissuing of my commit bit spurred me to fix a long-standing issue with the modules CGI and HTTP::Server::Simple (think of Ruby's webrick). Just a couple of small fixes allowed me to finally use these two modules together to create simple a standalone web application in Perl6:

use v6;
use HTTP::Server::Simple;
use CGI;

class WebApp is HTTP::Server::Simple {
method handler {
my $cgi =;
my $name = $cgi.param('name');
$.remote.say("<html><head><title>Web App</title></head><body>");
if $name {
$.remote.say("<p>Hello {$name}.</p>");
} else {
$.remote.say('<form method="get">Name: <input name="name" /></form>');
} port => 8080 ).run;

The above now works in Pugs, as of r14679 or so. The fixes required to make this work were trivial, but the result is enormously satisfying. I can tell this thing will require further investigation. More tests are certianly needed to ensure that these modules continue to interact properly. First thing's first, however. I think I'll enjoy some nice Perl6 web hacking before bed tonight.

Saturday, September 30, 2006

Four more years!

Boy oh boy, do I miss Bill Clinton. C'mon, who's up for a constitutional amendment allowing him to run again?

Seriously, the guy wasn't perfect, but he was smart, reasonable and likable. Plus he shares my fondness for plump young women. Remember when he refused to sign that bullshit budget and the resulting showdown actually shut down the federal government? Talk about leadership. Can you imagine George W. Bush doing anything like that?

I sure don't think friggin' John Kerry would have it in him. Probably Howard Dean was our best chance at outdoing the Clinton administration, but someone sure fucked that up. I have to admit that I'm about ready to start considering Hillary. I dunno. It's like: What choice have I got? I'm not going to vote Republican, and I'm too pragmatic for a protest vote. I know I'll toe the line like a good little Democrat because that's just what I did for (grr!) Kerry.

I'm a fairly young man, but I remember (barely!) when the words liberal and conservative both represented positive values that I wished to balance in my life and my government. I also remember when Newt Gingrich, Pat Robertson, and Rush Limbaugh started to twist the term "liberal" into a catch-all pejorative. Of course, the term conservative is now poisoned as well. I've always been mindful of the many true conservatives of this country who must be incensed at seeing the word associated with lying (watch how his eyes tick to the left with each one!), secrecy, outrageous spending and crass Constitutional blasphemy (Note: Bush lost the case mentioned in that piece).

Tuesday, September 12, 2006

Side Projects

I've been hacking a lot lately. Since the MacBook is in the shop due to random shutdowns, I decided to perform a long-anticipated (and dirt cheap!) upgrade to Cyclops, my aging workstation. I pretty much ended up replacing the whole machine, reusing only peripherals, power supply and hard drive. Cyclops is now a dual-core Athlon 64, which means faster compile times for indulging my favorite pastime: software research. I've been building the development versions of parrot, pugs and perl5, just to run the tests. I even tried using the v6-alpha compiler, but didn't have as much success. However, pugs is getting quite capable. I've built the version with perl5 support, so it's possible to load and use perl5 modules in perl6 code. I'm going to play around with this and try implementing a usable network server in perl6.

Using Ubuntu on a fast, modern system is a pleasure. It's cemented my resolve to switch to Ubuntu on the MacBook eventually, although I'm trying to give Mac OS X a chance. For the most part, Ubuntu makes it easy to do the stuff that's most important to me, but Mac OS X offers compelling features that I can't get from Ubuntu. Luckily I now have decent systems on which to run both, or will when the MacBook is fixed.

Just for fun I also created a templating module in perl5 (hasn't everyone?). It was fun to implement, I did some clever stuff with hierarchical data structures that I think fills a tiny niche for inline formatting of nested data. Hopefully I'll get to use it more often. It's assured that I'll be the only one.

Wednesday, August 02, 2006

Traveling Music with DAAP

I've accumulated a fairly large collection of digital music. I don't have a high-capacity music player, so I can't just carry it around with me everywhere. I suppose I could copy everything from my Ubuntu box to my MacBook, but that's a lot of hard drive space. Luckily, I have found a much better solution. Using the free DAAP server mt-daapd, my Ubuntu box can share my entire library over the wireless network. This means that when I'm at home I can listen to all of my music on any machine running iTunes, including my MacBook of course. So that's cool.

However, it gets even better than that! There are instructions available for tunneling a shared music connection over the Internet with SSH. Using SSH and Network Beacon I am able to listen to my music collection at work by streaming it from home! Amazing.

There are a couple of limitations to this approach, however. Since a large portion of my collection is ogg vorbis encoded, the DAAP server has to transcode it to WAV format before streaming or iTunes will not play it. This works fine over the local network since there's enough bandwidth, but this approach doesn't fly for streaming over the Internet. Ideally iTunes would support ogg vorbis natively (with the XiphQT plugin it'll play locally but not over a share), but I've been thinking it might be possible to transcode to MP3 if I really wanted to. This probably isn't feasible with my current setup, but it might give me the excuse I need to upgrade Cyclops. Dual-core Athlon here I come!

Friday, July 28, 2006

Back to Los Angeles

I'm now safely ensconced in my Westside apartment, after a crazy airport experience that resulted in Adam and I flying first class. There's nothing much to write about the rest of the conference, except that I've created an RSS feed just for my OSCON 2006 write-ups so that anyone interested can read them in order.

OSCON: Day three part 2

Next up was Chromatic's talk on Perl Hacks. I was looking forward to this one, especially after that disappointing "extreme Perl makeover" talk.

The presentation got off to a good start. Chromatic started with some perldoc command-line args, such as the -i (case-insentivie), -m (show source) and -l (display the path to the file). Here's a quick way to edit an installed module: vim $( perldoc -l Some::Module )

Whoa! You can put subroutine references in @INC! Dude. Apparently he picked this tip up from Nick Clark. You can return a filehandle, which Perl will use as the module contents. He then demonstrated a hack that uses this trick and Perl::Critic to review every module before it gets loaded. Impressive.

Next was an explanation of attributes. He showed us the Attribute::Handlers module. One useful application was Attribute::Method that allows you to access the invocant ("self") without declaring or shifting it.

Next up was the subject of calling C from Perl. He showed us a module called P5NCI that allows you to call functions in shared libraries without writing any XS glue code. Mark was sitting right up front and actually clapped for this one. But it got better! He also demonstrated some absurdly nice syntactic sugar to go along with it.

He went on to discuss a way to create private methods. The implementation involves using symbolic method calls to use names that the parser doesn't allow. Of course, it's exposed through extremely nice-looking attribute syntax.

The next hack was a great way to test well-encapsulated modules. The coolest part was that you could create a proxy that can test an object using live data while performing its usual tasks.

Yay! Perl 5.10 allows you to create your own pragmas. It's a simple matter of adding a key to the %^H (hints hash) during your import, and this allows you determine whether your pragma is active in a given scope. Many useful applications for this.

OSCON: Day three part 1

It's the last day of the conference. Adam and I checked out of the hotel and ran into Mark on the way out. We attended Damian Conway's keynote, which was quite funny, and had a little breakfast. After a quick stop at the Powell's bookstand for a book on programming Ruby, we split up for more talks.

My next talk was Peter Scott's "Extreme Perl Makeover" presentation. It was very similar to Mark's red flags talk, except that most of his examples were more heinous. His approach was to show Perl code written in a naive BASIC or C style, and change some 15-line block into one line of Perl using hashes or regexes. It was fairly contrived, especially since there are so few developers left who would write Perl in such a way. I got the feeling he created these examples specifically for the purpose of "fixing" them using a simple Perl idiom. Mark's talk contained many of the same lessons but communicated them in a much more effective way. I was not impressed with this talk. It seemed to be aimed at very inexperienced Perl developers. There was nothing "extreme" about this presentation. I guess we can't all be Audrey, eh?

Thursday, July 27, 2006

OSCON: Day two part 5

Next up was Damian Conway's general Perl presentation, "The Conway Channel." He talked about some of the problems that exist in Perl5 syntax and showed us some of the ways that Perl6 will help solve those annoyances. Of course, he also shows us a way to do many of these Perl6 tricks in Perl5 using CPAN modules. Some of the information presented was also covered by Damian and Larry's earlier Perl6 talk, but the focus was more on solving problems right now.

Much like I was during his earlier Perl6 talk, I found myself wondering what real problems these techniques are solving, other than creating more syntactic sugar. Don't get me wrong: Syntactic sugar is great, but as a working Perl developer I'm much more interested in real functionality, like that provided by the Moose module for instance.

The next example was more useful. Contextual::Return is a CPAN module that gives developers an easy to tell in which context their code is being called. I believe he presented the same or a similar module in his best practices book. There was one trick to this that I missed the first time, which is that values returned using this module are lazily evaluated, which allows even more cool tricks. This method also supports l-values (assignment), which was pretty impressive. Go go gadget Perl5!

OSCON: Day two part 4

After a break in the exhibit hall, during which I talked to people in the Oregon booth about getting Shopzilla to open a Portland office, we split up for more talks. I attended the "Live Perl Testing" talk, which featured a fellow writing tests live for the audience while another narrated. They started with a contrived example but moved on quickly to the bulk of the presentation, in which they took a script from Matt's Script Archive (a resource famous for it's terribly amateurish Perl code) and retrofitted it with a test suite. This is a very clever approach for illustrating how old, terrible code can be tested and subsequently refactored.

One tip they provided for testing existing code involves using the Test::Output module, which simply verifies that a piece of code emits some data to STDOUT. It's a quick way to verify some functionality of existing code, but obviously is not a long-term approach. However it can be an easy way to enable refactoring for some programs.

Next up was creating mock input, which they did by using the tie builtin:
tie *STDIN, 'IO::Scalar', \$input;

Things got weird for a second when they mocked a method in the class they were testing. I had to ask a couple of questions before I understood the purpose was to provide a way for other code to use that method internally without messing with the results. The difficulty (for me) came from the fact that their example was so contrived that it was technically a no-op in that context. Luckily they were able to explain this for me, and we moved on. It might have helped me if they had been more explicit by using a mock module to do it. Instead they used some Perl voodoo to accomplish the task, which struck me as a bad idea, but I think they were being intentionally pedantic to illustrate a point.

OSCON: Day two part 3

I wanted to get back to the hotel to pick up my phone but didn't get a chance. After Guido's talk I met up with Adam and David for lunch. Joining us was Mark Jason Dominus, a well-knowm Perl trainer. We walked to an Ethiopian restaurant and discussed all sorts of interesting technical stuff on the way there and back.

After lunch Adam I ducked into Damian Conway and Larry Wall's Perl6 language talk. We were a few minutes late, but what we saw was frankly a bit scary. We already knew that the main benefits of Perl6 over Perl5 was syntactic sugar, but this is insane. It's certainly expressive, but there were a couple of things that had no apparent application other than writing obfuscated code.

After that was Mark's talk about programming red flags. This was great stuff. It turns out Mark and I are both fans of the Daily WTF, and his first example of a "red flag" was truly heinous, definitely a WTF. However there was an important point hidden here, which is that even code that seems overly complex sometimes has relevant side-effects that a later maintainer must keep in mind. Before "fixing" broken code, you'd better be sure that you understand the context in which it's used. This has certainly come up during our refactoring of the Shopzilla codebase.

Another important point was that many experts are fond of saying "always do this" and "never do that." The point of red flags is not that they're always a bad idea, they just indicate that there is probably something wrong.

One easy-to-spot red flag is repeated code. He provided a large number of examples to underscore the point the when we see what we think are repeating patterns we begin to miss the details. Another one was a large if block without an else. Again, this isn't necessarily a bad thing, but it should cause one to rethink the approach. The bulk of the presentation dealt with an example program that he stepped through, fixing the red flags. The corrected version of the program code was 38% smaller and ran faster as well.

This was an excellent talk. I can see why Mark has been so successful as a trainer. He's articulate, humble and not at all dogmatic. What a pleasure to watch him work.

OSCON: Day two part 2

Next up was the Python 3000 talk by Guido van Rossum, the creator of Python. Python 3000 is the next-generation version of the Python language, really just Python 3.0. His approach is to leverage existing Python features while dropping deprecated or buggy design elements. All this while avoiding what he calls "an accidental paradigm shift."

He also doesn't want Python 3000 to get caught up in process and become "the next Perl 6." That got a nervous giggle from the audience. The first alpha will probably come along in early 2007, with the first release coming about a year later. Guido also expects a 3.1/3.2 release a few months after that, once wider use exposes more issues.

So what will change? Well new keywords can be added. Many builtins that formerly returned lists will return iterators. All strings will be unicode, supplemented with a new mutable "bytes" data type. The binary file I/O will be redesigned. '<>' will be dropped as an alias for '!=' (you mean there was MORE THAN ONE way to do it?).

Code migration was discussed as well. Since perfect mechanical translation is out of the question (too many changes are semantic, not syntactic). Guido expects that an automated tool could get most of the way, possibly augmented by a version of Python 2.x that complains about deprecated code.

Since Python 3000 is still a young idea, few things have been decided for certain. Guido doesn't want to change the look and feel of Python, so most of the work will go into fixing imperfect or deprecated features from the language and API. Besides unicode strings, another change will be from integer to float division. Importing will be absolute. These are all important features, but they're not very exciting. Unfortunately much more is known about what won't be in Python 3 rather than what will.

Some syntactic changes: Backticks will go. I like what's going to happen to list comprehensions. Basically they lose the square braces and become "generator comprehensions," but you can still wrap them in braces to coerce to a list. Clever. And lambda lives! Turns out they haven't yet found a suitable replacement after a year of looking, so Guido gave up ("It's as good as it gets, trust me!").

I have to admit that OSCON has cemented my excited about Perl and Ruby over Python. It's possible that having a BDFL is working against Python as a language.

OSCON: Day two

Ah, morning. Last night we partied with the Perl geeks in our hotel. I met a couple of people whose names I only know from books and mailing lists. After a good night's sleep we set off for breakfast in the exhibit hall.

After spending some time at the Shopzilla table, we split up to attend some talks. David and I went to Chris Nandor's talk about porting the Mac::Carbon Perl module to the new Intel Macs. It was a bit over my head so I didn't really retain much, but Chris is an old friend of David's and we wanted to represent, so to speak. Despite my extensive knowledge of the Perl language, I know very little about the internals. I'm sure Nick would have felt right at home. I did learn that the Macs have a different epoch (the "beginning of time") than Unix. I was a bit surprised.

Wednesday, July 26, 2006

OSCON: Day One wrap-up

After the Perl lightning talks we went back to the exhibit hall reception for some free beer. It was during this time that I was able to talk to a couple of prospective developers at the exhibit without feeling useless. Maybe it was the beer.

Besides the free beer, there wasn't too much interesting stuff going on in the exhibit hall. I got a tee-shirt and a few stickers. After the reception Thai and I walked back to my hotel. We've dropped off our stuff and now we're waiting for Adam to come back from dinner with David and his family. I heard from Nick that there's a Perl party of some sort, which is what I'm most interested in, but there will be lots to do tonight regardless I'm sure.

So far I've been pleasantly surprised at how much fun I've been having at this event. The talks have been fantastic, the people are friendly and knowledgeable, and there's usually free food. I'm looking forward to the next two days that much more.

OSCON: Day One part 6

The lightning talks were split up by a break, after which we watched a short dramatization of the Perl module selection and install process. You had to be there, AND be a huge Perl geek, but they got lots of laughs. It was pretty funny if you got the joke.

Next was Tim Bunce, who talked briefly about database interfaces for open-source languages. I should note that Tim is the primary architect behind DBI, which is the database library that Shopzilla (and almost everyone else) use for Perl database development. He told us about the migration from DBI to Perl6. I'm impressed at how much thought is going into this subject, that is, the move from Perl5 to Perl6. It's an important question, and I'm glad it's not being ignored. Not that it could be ignored, I suppose.

Next was a great talk about higher-order Perl. This was good stuff. It was almost like an old-school religious revival, with lots of hooting and cheering. It was pretty short though. I really need to pick up the book as well.

After that a first-time presenter got up and told us about a technique he uses for logging. It was pretty basic stuff, not the sort of exciting deep magic we Perl geeks love most. I'm just glad to see developers, even less high-level ones, sharing techniques with one another.

Folllowing that was a presentation about a templating system called Tiny Templater. The glut of available templating systems for Perl was something of a running joke during the lightning talks.

Patrick Michaud spoke a bit about Parrot, and why it's important to implement other languages on top of Parrot. The example he gave was his implementation of the APL programming language on Parrot. He had mentioned this during his earlier talk about PGC. He implemented the APL parser by porting the BNF grammar notation to PGC grammar. This is an important proof-of-concept for Parrot, and I enjoyed the presentation, if only for the realization of how terrifying APL is (you need a special keyboard!).

George Wood of Freescale (a chip manufacturer) talked to us a bit more about the state of Parrot and the languages that are currently available. He had lots of diagrams, but they were a bit dense.

OSCON: Day One part 5

Audrey's Jifty talk left us in great spirits, and also made us feel better about our existing Perl codebase. After a coffee break we wandered around the exhibit for a while. I made the mistake of standing directly behind the Shopzilla table, and almost immediately a fellow I didn't know came up and called me by name. This was a bit disorienting until I remembered that I was wearing a convention badge with my name on it. I answered a few of his questions and tried to talk up the experience of being a Shopzilla developer, but I didn't feel very convincing.

After that came the Perl lightning talks. These are a series of short presentations where individuals talk about interesting hacks or ideas relating to Perl. Audrey kicked it off by showing us a tool called ppencode. If you're not a Perl hacker it would have been pretty uninteresting, but the audience was roaring with laughter at her obscure jokes about the syntax of dynamic programming languages. Her slides were pretty slick too. Nice animation.

Another interesting presentation was for a tool called 'Ack' for searching source code trees. I had written a bash function at work for this purpose, so I'm glad to see there is a more robust tool for the job that also happens to use Perl regexes. Best of all, there is a thbbbt command-line argument! Get it? ack -tbbbbt!!!

There was more than one presenter who talked about Perl::Critic. The second was longer but more thorough. This module is basically like the old unix tool 'lint' for the Perl language, using standards from Damian Conway's Perl Best Practices.

Another presentation dealt with an innovative method of sending email that allows users to communicate without having to deal with spam filters.

After that was a talk about XML processing, specifically ad-hoc searches through XML content. Their solution was to create an extensible version of GNU Awk. Seems clever, but I don't use awk much, nor do I do much XML munging.

Next was a "Perl Success Story" from Colin Meyer of As he said at the beginning of the presentation, we already know that Perl is appropriate for enterprise development. He went on to show us how his application works, what it's made up of, and how much traffic they get. It was meant to be impressive, but the funny part was that our site gets much more traffic (we get in a day what they get in a month), the codebase is an order of magnitude larger and is also written in Perl. The Shopzilla contingent definitely giggled a bit over this fact.

The next presentation was from Chris Nandor, who is an old friend of David's. He showed us a small Perl hack he developed to assist his scoring of baseball games, using a fancy layout tool to render a view of the game. I can't say I really found this too interesting, since I'm not into baseball, although the way he modeled his API was pretty clever, plus he had a method called play_ball. Perl is among the more humorous of languages, and the lightning talks definitely demonstrated that.

OSCON: Day One part 4

I had been highly anticipating the next talk, which was a presentation on Jifty by Audrey Tang. I was not disappointed. Audrey is widely credited with re-energizing Perl6 development, and now I see why. She's brilliant and crazy, much like Larry Wall, Perl's creator, but with an degree of technical genius that not even he can match. The talk was quitedense, so I won't post many details. I will say that it was extremely impressive. You could hear members of the audience occasionally gasping and giggling in amazement. It's hard to overstate the excitement generated by this presentation, especially considering that this was all Perl5. Well done Audrey!

OSCON: Day One part 3

The next talk I attended was Tim Bray's presentation of "Atom as Universal Web Glue." Tim started by criticizing the popular term "user-generated content" (it's fairly insulting to us "content-generating users") and discussed the limitations that hold this idea back. Then he presented the Atom publishing protocol as a solution.

1. Starts with an URL: GET the "service document" from a publisher's site and extract "collections."
2. Retrieve those collections and see what you may already have.
3. POST an Atom entry to a collection. Get back it's URI in the returned HTTP header.
4. POST a picture or binary to a collection. Get back a URI for that data, then another for the metadata.
5. PUT or DELETE an existing atom entry.

That's all of it. Very simple indeed.

APP differs from other protocols in a few important ways. Unlike RPC, there is no API abstraction, just message passing. Using HTTP allows you to layer whatever securing you want over it (unlike WS-*). Finally the client doesn't need any knowledge of the server's namespace. This differs from web-based filesystems like WebDAV.

APP is already being used by Google for the GDATA interface. Even MS says that their next version of Word will support blog publishing with Atom. There are other implementations coming.

There are many interesting things that can be done with this technology. Tim was then able to show us a demo using curl and vi! Not only did he post textual entries, he was also able to easily post binary content as well. I think was an excellent way to underscore Atom's simplicity. It's also inspiring to realize you could create an Atom publishing client in shell script, if you were so inclined.

OSCON: Day One part 2

We attended a talk by Peter Scott called Mind Like Water: The Path to Perl Bliss. This was in the tradition of high-level Perl talks, and dealt more with the different "programming personalities" that compete for dominance in the Perl developer's head rather than technical issues.

Also included were some useful bits of advice distilled from many Perl luminaries, most of whom are here giving talks. A common theme was experimentation: "If you don't know, try it." This used to me more true in Perl than most languages, but I would say this is just as true for newer interpreted languages.

Another popular sentiment was that it's important to bring lessons learned from other disciplines and apply them to your development. There was also some discussion of good habits, and how we often mistake good habits for discipline.

Lunch followed. It wasn't that great, but it was free.

OSCON: Day One

Adam and I missed our plane last night and had to catch a later one, so we didn't get in until around 11. We met Thai at the redline and traveled to our hotels. Adam and I were staying at the Doubletree, but Thai was staying elsewhere so we split up at the platform. David had already arrived with his family earlier in the week, and Nick was supposed to have flown in earlier in the day. We were all here as representatives of Shopzilla, who is a sponsor of the event.

After a good night's sleep Adam and I got up and walked through town to the convention center. After a brief stop for registration and a cursory visit to the Shopzilla booth, I hustled off to my first talk, which was a presentation on Parrot grammar and rules given by Patrick Michaud. Parrot is a virtual-machine designed for powering dynamic languages. It was originally (and still is) created for Perl6.

I was surprised and impressed at how easy it is to write parsers using PGC, the Parser Grammar Compiler. I have heard many conflicting repors of Parrot's health, but this talk was quite encouraging. Patrick showed us an implementation of the BC grammar in PGC, and I found it to be very easy to understand. It's nice to know the next generation of Perl technology won't be beyond me. I was also excited to see that there's a Ruby/Parrot implementation coming along as well. He even showed us a bit of the fledgling Perl6 grammar.

Overall this was a highly encouraging talk. I had been following Perl6/Parrot development a bit, but I think I will have to download a copy of the Parrot source and muck around a bit. Plus Patrick will be discussing the Perl6 compiler tomorrow. Don't want to miss that.