Below are the 20 most recent journal entries recorded in Greg Hewgill's LiveJournal:

[ << Previous 20 ]

wednesday, november 25, 2009
9:00 pm	psil presentation and compiler details As promised last week, I've been working on making Psil work in a Python 3 environment. But first, a small but entirely relevant digression which includes a video of me giving a presentation: A few weeks ago, I attended Kiwi PyCon here in Christchurch. I had thought about maybe possibly presenting something when I signed up, but I didn't actually prepare anything ahead of time. Immediately after lunch on the first day, there was a "lightning talk" session where people get 5 minutes to present something, no questions from the audience. During lunch, I decided that my opportunity was right then, and that if I didn't try to present something then I'd probably be annoyed at myself for not having tried. Anyway, I hurriedly prepared a presentation (PDF, 8 slides) in the 30 minutes before the presentation, and wondered whether I would even be able to find 5 minutes worth of stuff to say. It turns out I did, because although it felt like I had zipped through my presentation in like 3 minutes, I actually took slightly over 5 minutes. There is a video recording of my (terrible) presentation that you can watch online (skip forward to about 52:00 using the slider). Right, now that you're back, I'll continue with what I was going to write about. The Psil compiler (as opposed to the interpreter) compiles Psil code into Python code, which is then executed directly by the Python runtime system. For example, here is a simple function and its use: Psil Python ---- ------ (define (sq x) (* x x)) def sq(x): return x * x (print (sq (+ 2 3))) print sq(2 + 3) Previously (before the current Python 3 migration), I did this by generating literal Python source code as shown in the right hand column above. But instead of generating the source code directly, I generated a Python AST (Abstract Syntax Tree) representation, then implemented a "de-parser" that creates equivalent Python source code from an AST. The original idea was to have Python compile and execte the AST directly (rather than going through the source code step), but I think the version of Python 2.x that I was using didn't support passing an AST to the compile() function. Since Psil now requires Python 3, the compile() function supports direct compilation of an AST that represents Python code. So now Psil can generate the AST, pass it to compile() without going through source code, and execute the resulting code object directly. One of the difficulties with this was that the internal Python AST changed completely from version 2 to 3, so I had to rework most of the compiler code. Fortunately, it turned out to be reasonably straightforward. I also maintained the de-parser because debugging actual source code is a lot easier than debugging with only an AST. Right now I've got this all working for simple examples. I still need to re-implement the "lambda lift" which promotes a Psil lambda expression (which may contain an arbitrary number of forms) to a Python function (because Python's lambdas can only contain a single expression). The next interesting thing I can do with an AST is annotate the nodes with source line and column numbers. This means that if a Python exception happens at runtime, the traceback will actually point to the location in the Psil source code that caused it! That's a pretty cool feature and I'm looking forward to playing with that. (Comment on this)
saturday, november 21, 2009
4:47 pm	thawte web-of-trust shut down, what next? Since Thawte has shut down their free Personal Email Certificates program, there doesn't appear to be a free community-driven replacement for S/MIME certificates. Thawte VeriSign offers personal email certificates for US$20 per year, but no thanks. It looks like we're back to the tried-and-true PGP for no-cost secure person-to-person communications. (3 Comments |Comment on this)
thursday, november 19, 2009
9:45 pm	python 2 to 3 upgrade and exception handling I've just (partially) upgraded Psil to Python 3. I had originally been developing it in Python 2, but there are a few particular things about Python 3 that make it a better choice. There's no particular reason it needs to run on Python 2, so I decided to not develop two parallel versions, and not try to run the same code in both Python 2 and 3, but just upgrade wholesale to Python 3 and not look back. There were in fact only a few simple changes that were required. The most obvious is the print() function, but I also ran into: The result of map() is an iterator not a list (this affected some unit test code) Renaming of raw_input() to input() Use of next(it) instead of it.next() (and my use of a local variable called next) Need to use f(*a) instead of apply(f, a) Lack of callable(x) (use hasattr(x, "__call__")) Exception syntax: except Exception as e instead of except Exception, e reduce() now lives in the functools module Some test code needed to handle Unicode strings properly A difference in default exception handling (more on that below) So I certainly got a pretty good coverage of the major differences. The problem I had with exception handling was related to my unorthodox method of handling tail recursion. When I ran a program that used a lot of tail recursion, the memory usage immediately and quickly went through the roof! Clearly there was a memory leak, but Python is generally supposed to handle that for me with its garbage collector. The clue to solving this lay in an obscure warning in the documentation for sys.exc_info: Warning: Assigning the traceback return value to a local variable in a function that is handling an exception will cause a circular reference. This will prevent anything referenced by a local variable in the same function or by the traceback from being garbage collected. I had read that warning, but I wasn't using the traceback of sys.exc_info at all so I thought that shouldn't be a problem. However, Python 3 now automatically includes a __traceback__ attribute of every exception (see PEP 3134). Due to the way I was calling a function referenced within the exception object itself, the presence of the traceback was creating a huge chain of unfreeable function and exception references. Fortunately, there was a simple solution: except TailCall as t: a = t a.__traceback__ = None Setting the __traceback__ attribute explicitly to None releases the reference to previous stack frames and my code no longer leaks memory. On the recommendation of the python-dev mailing list, I filed a documentation bug to clarify the warning quoted above. Finally, I said at first that this was a partial upgrade, because I haven't even addressed the compiler part of Psil (that compiles Psil code to native Python). The modules and interfaces that I was using previously are either gone or changed in Python 3, so a slightly different approach is needed. More on that in a later post. (2 Comments |Comment on this)
thursday, november 12, 2009
11:45 pm	look before you leap I started to investigate yesterday's idea of a shell implementation in Perl, and immediately got mired in the Byzantine shell quoting rules. The most basic thing about the shell turns out to be unnecessarily hard, and besides, it's already been done. Why did I think this was a good idea again? Somehow I had got it into my head that msysgit on Windows would only run from the msysgit shell. Since I don't actually use Git on Windows, it turns out I was mistaken. If you run any Git command from the regular Windows command prompt, it already works. The appropriate shell is invoked to execute the shell script (git-rebase is one such shell script, instead of being part of the native C implementation). Nothing to see here, move along... (2 Comments |Comment on this)
7:59 am	git on windows Git is mostly written in C, with some important parts in shell and Perl. This works well on Unix-like platforms, but unfortunately suffers on Windows because there is no native Windows (Bourne) shell. Git can be built under the Cygwin environment, but that is a heavyweight solution and Cygwin is not appropriate for general use. msysgit builds a native Windows executable plus uses the msys shell implementation, which provides a workable solution, but still requires Git operations to be done within the msys shell and not the native Windows command shell. I had a thought the other day, that one could implement a Bourne shell interpreter in a more widely available language such as Python. This would provide a more widely portable option for using Git, and on Windows in particular. Such a shell implementation would provide the minimum implementation necessary to run Git's shell scripts (and which would include minimal implementations of various utitilies such as grep, sed, and awk). That wouldn't solve the Perl script problem, so it seems obvious that a portable shell interpreter designed for use with Git should be written in Perl. A long time ago there was the Perl Power Tools project, which aimed to provide implementations of various Unix utilities in native Perl. Unfortunately, nobody ever submitted an implementation for sh. Is there any existing implementation of a Bourne-like shell in Perl? More generally, is this a viable plan to make Git work better on Windows? Update: See the next post to find out just how far I got with this. (3 Comments |Comment on this)
saturday, november 7, 2009
9:38 pm	lnk.nu and google app engine Last week, I was quietly working on my web server when all of a sudden the whole thing ground to a near-halt. It wasn't completely dead, because it would still ping and every few minutes I would receive another packet of characters (I was literally in the middle of refreshing a screen I was looking at). Not knowing what was happening and not having any way to find out, I went and did something else for a while. 45 minutes later, my server returned to normal operation as if nothing had happened. This did not appear to be just a network congestion problem, it was definitely something my server was busy doing. A bit of investigation showed that the culprit was in fact lnk.nu. Hundreds of machines all across Canada had all accessed the same short link at the same time, and completely pegged the PostgreSQL database processes, and also run my server out of memory. It's quite a testament to both FreeBSD and PostgreSQL that they survived at all. What I believe happened was that somebody had sent an email containing a lnk.nu link to a mailing list to which lots of people from Canada were subscribed. (The link in question happened to be a job opening at who.int.) Looking up the reverse PTR records for the machines that loaded the URL, there are names like "mail", "barracuda", "filtre", "antispam", "mx1", "incoming-smtp", "guardian", etc. It seems that they all accessed the link for purposes of virus checking, all at pretty much exactly the same time. This was not good for my poor server. I decided that it might be time to move lnk.nu to a different server. It's written in Python, so it's an ideal candidate for Google App Engine, and I've been looking for an excuse to play with GAE. So I downloaded the SDK, converted the code over to GAE (using Google's datastore instead of SQL), and made it work locally. This part was refreshingly easy and worked well. The next step is to set up the Google site so it responds to http://lnk.nu and handles the requests appropriately. Given that I've already got the code working locally, that should be straightforward. However, there is one gigantic caveat when using Google web site services (that I've actually already run into for another project): You cannot have Google's servers respond to a "naked" domain name that doesn't have a hostname. This means that having Google respond to http://lnk.nu is not possible. (There is in fact a good technical reason for the above restriction. When you set up a site with Google hosting, you add a CNAME record to the DNS for your hostname, ie. "www.example.com. CNAME ghs.google.com.". This lets Google completely manage the association between "ghs.google.com" and any particular IP address(es), which is critical for their load balancing setup. The caveat is that a record with a CNAME must not have any other DNS records associated with it, including an SOA record. The SOA record is required on a "naked" domain name like lnk.nu, so you can't add a CNAME there.) To work around this, I'll have to set up a hostname that Google can respond to, something like http://a.lnk.nu. Of course, that's a pretty lame name for a link shortener to use, so I'll still want the published link to be http://lnk.nu/blahblah. This means that I'll have to have some other, non-Google server respond to a http://lnk.nu/blahblah request with a redirect to http://a.lnk.nu/blahblah. This adds another level of indirection to the resolution process for a shortened link, which adds another browser round-trip, which might slow the whole experience down no matter how fast GAE hosting ends up being. It turns out that Namecheap (my registrar for lnk.nu) offers "URL redirection" where their server will respond to a particular hostname and redirect the browser somewhere else. It can also be configured to retain URL path information, so http://lnk.nu/blahblah would redirect to http://a.lnk.nu/blahblah. This would completely take my own server out of the loop, hopefully avoiding any more problems like those last week. (6 Comments |Comment on this)
friday, november 6, 2009
12:20 am	earthquake widget update Over the last couple of days, a few people have emailed me about my Earthquakes Widget which displays the most recent earthquakes on a small world map on your desktop (it's a Yahoo! Widget). It seems that it had stopped displaying earthquakes due to an error loading the data file from the earthquake.usgs.gov server. It turns out that it was due to an URL change on the usgs.gov server, and the widget framework's XMLHttpRequest object doesn't seem to automatically follow the redirect. So I changed the link and it all seems to work again. I sent a new widget to the people who had emailed me to make sure that it solves their problem, and if it's all okay then I'll submit it to Yahoo! again (they approve widgets manually, so it pays to make sure it's right before submitting). If you're looking for the latest version (1.5) before it gets submitted to Yahoo!, see the Earthquakes Widget page which has a direct download link. I've uploaded the quake-widget repository to GitHub. (Comment on this)
wednesday, october 28, 2009
8:51 pm	mandelbrot set viewer A couple of years ago I had the idea to build a Mandelbrot set fractal viewer in the spirit of Google Maps. Google Maps itself has a method to display your own map data by implementing a "tile" server, but it seems specifically oriented toward geographical data and doesn't have features like arbitrary zoom depth (which you'd clearly want for fractals). I looked around for other similar generic solutions and didn't find anything at the time. Back then, I implemented a drag-scrollable zoomable fractal viewer in Javascript from first principles. This involved doing all the crufty cross-browser compatibility work myself, which was really annoying and very nearly took all the fun out of the project. Recently, after playing with jQuery a bit, I thought it would be good to go back to the fractal viewer and reimplement it using jQuery. You can see the results of my efforts at my Mandelbrot set viewer. The controls are pretty much what you'd expect. Panning and dragging work as expected (but sometimes you have to wait for the tile server to keep up). The zoom control zooms in 2x for each step, with an effective upper bound limited only by the floating point precision of Javascript. Double-clicking zooms in 2x on a particular point on the plane. The currently-unlabeled button below the zoom control allows you to generate a larger (desktop-sized) snapshot of your current view. I was talking to Phil today who suggested I look at OpenLayers. Sure enough, in their Gallery there is already a Fractal Browser which does almost exactly the work that I've already done! Once again, somebody out there on the internet has already thought of the same idea. Nevertheless, I'm happy with my viewer (even though it's not everything I had envisioned yet) and I think I'll leave it as is and move along to something else. Finally, because this is Open Source Wednesday, you can find the code in ghewgill/mandelbrot on GitHub. Enjoy! (2 Comments |Comment on this)
saturday, october 24, 2009
9:38 pm	skiing around the world The ski season in New Zealand runs from Queen's Birthday (first Monday in June) through to Labour Day (last Monday in October, this weekend). Unfortunately, Amy and I didn't manage to go skiing at all this season. While we were in Iceland last April, we stopped in Akureyri and drove up to the local ski hill on the slopes of Hlíðarfjall (which is no more than about 10 minutes drive from town). We had a hot chocolate at the lodge, and seriously considered spending the day skiing. I said at the time that although it would be fun, I wouldn't be disappointed if we chose not to ski. We decided to press onward (since we only had a limited number of days left) and ended up spending our spare day in Stykkishólmur instead. However, I now think it would have been awesome to go skiing in Iceland. It was a gorgeous day (see above), not too cold, we had enough time, and Akureyri is a charming little town with an excellent hot pool. And all the red traffic lights in Akureyri are heart-shaped! At night, you could also see lights across the fjord Eyjafjörður that form a huge red heart. We didn't spend much time there, but I think Akureyri was one of my favourite places in Iceland. (Comment on this)
wednesday, october 21, 2009
11:09 pm	xearth tropical storm overlay Two new things for xearth this week. First, there is a new overlay for current tropical storm activity. The source data is the Tropical Storms, Worldwide data set from the University of Hawaii. I wrote a script to download the current tropical storm file (which is a flat ASCII file), some Python code to convert that to an SVG file, and then converted that to a transparent PNG for the overlay. The other new thing is xearth now supports multiple overlay files. Here's an example showing all map and overlay features (the Living Earth base map, and earthquakes, storms, and clouds overlays). I'm going to improve the presentation of the tropical storm map so the labels are more visible, especially when combined with the cloud overlay (white on white is not an ideal colour combination). (Comment on this)
tuesday, october 20, 2009
10:02 pm	python enhancement proposal The other day while writing some code in Python, I had an idea. The bit of code I was writing was shaped something like this: for a in results: if a.something: do_action(a) It occurred to me that if I were to write this as a list comprehension, then the "if" would combine with the "for" on the same line: [do_action(a) for a in results if a.something] What if I could write this instead: for a in results if a.something: do_action(a) This looks like a nice compact syntax for a reasonably common construct, which happens to also be nicely aligned with the list comprehension syntax. I thought I'd have a go at modifying the Python language to accept this, and perhaps even submit a PEP (Python Enhancement Proposal) for the core language. I read about the PEP process, found a related PEP to enhance the generator syntax (from 2009), which led me to a long thread discussing that PEP, from which I found a discussion about an "if-syntax for regular for-loops" (from 2008), and finally Guido's opinion on the matter (from 2006), which states that he doesn't like it and that it was proposed and rejected before. And as it turns out, after all that I agree with Guido. Lessons learned (none the hard way!): Ideas you think are new probably aren't. Do research before committing too much time or energy to a new idea. Open development processes are great because there is a permanent record of what has been discussed before. (5 Comments |Comment on this)
thursday, october 15, 2009
12:15 am	svg adventures (and earthquakes) Well, that was thoroughly frustrating. I wanted to render an SVG file to a PNG file (I'll get to why in a moment). So I did a quick search and found librsvg that includes a handy command-line program called rsvg that does exactly what I want. I seem to have already installed it on my laptop, and confirmed that it was what I needed. I then proceeded to install it on my FreeBSD server, so I installed graphics/librsvg from the Ports system. It turns out that the build pulled in a zillion Gnome-related libraries, so the build took quite a while. It installed ... but no rsvg command. It turns out that was the old version of the library. So I thought I'd update my Ports system to the latest version, but archived a copy of the previous one to save. While I was updating, I noticed it updated a library called librsvg2, which sounded promising. So I went to install that, but it complained that some dependent library was out of date and I was a bit hesitant to just update my whole system to the latest version of everything. So I reverted back to the previous Ports system I had installed (remember that backup I cleverly made?) and tried to install librsvg2. After a couple of hours of building more Gnome dependencies (why does an SVG renderer need audio libraries? or an object request broker?) somebody said "hey, why not just use the ImageMagick convert program?" I thought ImageMagick only handled raster graphics, but it turns out I was wrong. ImageMagick happily renders SVG files into PNG in exactly the form I wanted. Augh! Now my server (which is sort of low on disk space already) had a ton more useless crap installed that I don't need and don't really know what can be safely removed. Anyway, on to the interesting part. Xearth for Windows has an "earthquake" feature where recent earthquakes are displayed as yellow circles on the earth image, with a size proportional to the magnitude of the earthquake. It was a fairly quick and dirty implementation because it just draws a circle on the screen without regard to the map projection at all. With the new overlay support in xearth, I thought a good way to implement earthquakes would be to render the earthquake circles onto a mostly-transparent PNG file, and then overlay that over the map image. Here's what you get: If there are any earthquakes visible (there ought to be), they will appear as yellow circles. This is done by downloading the current earthquake list (magnitude 5 or greater) from the USGS, using XSLT to transform it into SVG, then converting it to PNG for use by xearth. The earthquake list is updated once an hour. (Comment on this)
sunday, october 11, 2009
12:40 pm	cherries Well, not yet cherries. About a month ago, we bought a dwarf cherry tree for our front yard. It was essentially just a stick with roots at the time, not looking much like a tree at all. But we planted it and fed it, and it seems to be quite happy. Just in the last week or two, it has sprouted a full complement of leaves and is looking resplendent in its foliage! Apparently these trees start producing fruit after about three years. (6 Comments |Comment on this)
friday, october 9, 2009
7:43 pm	christchurch bus system changes Just this morning, I was thinking how nice it was that the number 17 bus, which is the fastest one from the city centre to work, just happens to go right by my house which is nearly on the other side of town. I can leave the house at a predictable time, walk two minutes to the bus stop, get on, and not worry about having to change buses until I get off at the stop five minutes walk away from work. I happened to check the local bus info web site this afternoon, and found an announcement of some changing schedules. Fine, I thought, they do that from time to time. But this time everything has changed: New routes, deleted routes, changed schedules, changed route names and numbers. Just when I was starting to learn the system pretty well! Ultimately this change is a good thing because they seem to have improved most schedules to 20 minutes between buses (was 30). I'll have to see how the new schedule works out with having to change buses to get to work. But most days I still ride my bike anyway, unless the weather is unseasonably cold and rainy like today! (Comment on this)
thursday, october 8, 2009
11:02 pm	is spamassassin still state of the art? What is the state of the art in (open source) antispam content filtering software? Is SpamAssassin still it? There hasn't been a new SpamAssassin release at all in over a year, and the last major release was nearly 2.5 years ago. I'm not terribly happy with the performance of SpamAssassin these days (it blocks a lot of spam, but an annoying amount still slips through, 5-10 per day) and am wondering whether there is anything better out there now. I've read briefly about Dspam, but don't know much about it yet, has anybody used that? Note that I use SpamAssassin as a MUA delivery content filter (through procmail), and have all the usual MTA-based antispam knobs turned up to 11 in Postfix. (4 Comments |Comment on this)
wednesday, october 7, 2009
9:34 pm	cloud overlay in xearth The Xplanet project provides a nice cloud map which is updated every 3 hours and provides a worldwide view of the current cloud cover. I've added support to xearth for rendering the cloud map as an alpha-blended overlay over any other xearth map. Here's what it looks like in a worldwide context: You can see how it enhances an existing image with this side-by-side comparison: Finally, by tweaking the display parameters a bit you can view more localised cloud cover: The next big step for xearth is to merge my xearth for Windows code into the main code base. This will displace the X drawing code from being the default GUI for xearth, and put them both on an equal footing. Naturally, other platform ports will then follow. (But I probably won't have that done by next week!) (1 Comment |Comment on this)
thursday, october 1, 2009
12:02 am	xearth interactive page Just a bit of work on xearth this week. I've made an interactive page where you can experiment with different xearth options including the new "overlay" feature. The above image shows the current position of the earth as viewed from the sun. (2 Comments |Comment on this)
friday, september 25, 2009
7:14 am	travel photo slideshows Amy has been doing some great work collecting our best travel photos into video slideshows. Here are the two she's done so far plus some related videos: Islands of the World Tour 2009 Our 6-week trip to Canada, Texas, Frankfurt, Ireland, Iceland, London, and Tokyo earlier this year. Tossing snow into an ice vent in Iceland This very short video shows how snow melts immediately with a "poof" when you toss it onto a steam vent. Otago Rail Trail December 2008 Last Christmas we cycled the Otago Rail Trail, roughly 150 km over what was previously a railway line. Daniel Tammet - The Boy With The Incredible Brain Finally, in related nows: This TV clip shows a guy who learned Icelandic in a week, from scratch, and then appeared on national TV. (1 Comment |Comment on this)
wednesday, september 23, 2009
11:12 pm	xearth bmp output, true colour, and fonts Xearth has long had the ability to draw markers and labels on the image when it draws in a "live" context, that is in a GUI like X. However, it uses the native text drawing and font support to do this, so it never allowed the -label option when drawing to a bitmap file. I hacked support for labels into the Windows port a long time ago, using the Windows GDI support to draw text onto an in-memory device context. While this worked for bitmap files, it wasn't very flexible and was done at the wrong layer. I have just added support for drawing text to the base xearth code. This uses an in-memory representation of a bitmap font, and draws the text before the output functions even run. The text output works for all kinds of image file output now. While I was looking for a description of the internals of Windows .fon files (which are a convenient source of bitmap fonts), I came across Simon Tatham's Fonts Page which has some utilities for reading and writing FON files. Not only was there code to read FON files, but there was a public domain version of the X "fixed" 6x13 font which is a handy default font to use in xearth. As an aside, Simon Tatham is the author of the popular PuTTY terminal program, but also has some other cool stuff on his site. A really slick collection of puzzle games is his Portable Puzzle Collection, which are written in C for GTK, Windows, or OS X but with some clever wizardry can also run as Java applets. Anyway, another thing I added to xearth this week was BMP output file support. This is done without reference to the actual Windows header files, so that xearth on any platform can create BMP output files. I'll need this when I finally merge in the Windows port code. Finally, I added a -ncolors true option that enables 24-bit colour output for PNG and BMP files (GIF files are limited to 256 colours, and JPEG files are always true colour). Using the -overlayfile option automatically enables true colour output, because no assumptions are made about the colour palette of the overlay file. This implies that -gif cannot be combined with -overlayfile, but that's an inconsequential limitation. As always, you can find the latest xearth code in xearth on Github. (2 Comments |Comment on this)
sunday, september 20, 2009
1:25 am	the general public on twitter I was chatting with Phil the other day and he was working on some statistical analysis tools for the Twitter API. I was reminded of a little project that had popped into my head a while ago: How hard would it be to identify the language in which a twitter status update is written? With only 140 characters, some of which are going to be a URL or something, there won't be much info there. Is it possible? The Twitter API provides a method to get the most recent 20 updates from the general public on twitter. You can see this public timeline in your browser as well as getting it via XML or JSON or whatever. I took one look at that page and was immediately struck by: the wide range of languages represented (I imagine this varies by time of day) the high frequency of spelling errors, both active (because of the 140 character limit) and passive (because people can't or won't spell properly) the wide range of "words" used that simply aren't in any dictionary (mmmmm, XD, arrr, dat) (well, "arrr" is permissible because it's International Talk Like A Pirate Day today, of course) the low information content of most of the crap people post on twitter the (attempted) spam (who would actually read or click on your gratuitous message about affiliate marketing?) I was disheartened by what I saw, ready to give up on the project. Here's a typical gem: ne1 der.....2 b frndz wid me.. But you know what? Regardless of whether any of this could be considered correct, literate, crap, spam, or what have you, it's what people are actually writing. It's (mostly) human communication, even if it has a low information content. Doesn't it deserve analysis anyway? Real world problems are almost never the easy ones. (3 Comments |Comment on this)