Greg Hewgill (ghewgill) wrote,
Greg Hewgill
ghewgill

minilink performance improvement

My minilink/lnk.nu link shortener has been getting increasingly popular lately. There was a sharp jump up in activity in mid-October, which has now put the request rate at around 20k requests per day. There have been spikes up to 40k per day, too.

This increase in activity has taken a toll on my server, since minilink had been implemented as a simple Python CGI program. This was the original implementation I started with and I haven't needed to change it. However, CGI is only appropriate for low performance applications, and minilink is probably edging into medium performance at this point. A CGI program has to start up an entire Python runtime environment for every request.

Over the last two days, there have been big enough activity spikes that the load on my server goes through the roof and it can't really handle the requests anymore. This can cause things to crash and run out of memory and other impolite behaviours. This happened once last March, and twice in the last two days.

Today I did something I should have done long ago: I converted minilink from a CGI program to a WSGI program. WSGI improves performance because it keeps the Python runtime loaded in memory, and with Apache's mod_wsgi, the calls into Python code are much faster.

I also implemented a simple test suite so that I could refactor the implementation while ensuring that the behaviour stayed the same. That's something I should have done long ago, too.

You may see an increase in performance when using minilink starting today. If you see anything go wrong though, please let me know.
Tags: minilink, python
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 3 comments

Deleted comment

Looks like it went well. The request rate has maintained an average of about 20k-30k requests per day since mid-January, with no noticeable ill effects since I switched over to WSGI.
I probably shouldn't attach this to this post, as it's kinda unrelated, but...

Are y'all OK?!?!
Yes Amy and I are both safe at home, thanks for checking!