...making Linux just a little more fun!
Rick Moen [rick at linuxmafia.com]
I figured this will be of general interest. Ben and I (with Ben doing the real work) have been working, off and on, on a scriptable mechanism for checking a list of Internet domains for impending expirations, and then do some sort of appropriate notification. Ben's Perl script, "domain-check", for this purpose relies on the commodity open-source client for the TCP port 43 "WHOIS" information service, offered by most domain registries.
We've run into a fascinating tangle of complications, most of them (primarily) non-technical. It's been a lovely case study in all of the manifold ways that the interface between people and automated systems can go wrong.
Domain registrars (typically) don't want their customers' domains to expire: That leads to loss of business, after all -- though most are pleased to put expired domains in a "held" state for a while, in which one can ransom them back for significant money. Registrars have a few ways to help customers avoid accidental expiration: One of them is called "auto-renew". With this feature enabled for one's domain (now a default setting at many registrars), the registrar will attempt to charge a renewal fee to the customer's credit card of record, at some time prior to -- or exactly at -- the time of expiration.
Most registrars' implementation of that idea involves, say, three successive attempts to charge the customer's credit card, at, say, 30 days from expiration, 20 days, and then 10 days -- or something like that. At least, I recall seeing such a policy once or twice: I think Joker.com does that, for example.
But then there's Gandi.net, of France. Look at the first line of the domain-check report below:
----- Forwarded message from root <root@linuxmafia.com> -----
Date: Sun, 5 Aug 2007 08:46:12 -0700 To: rick@linuxmafia.com Subject: domain-check: Expired domains From: root <root@linuxmafia.com>According to 'whois', the following domains have expired:
laconiv.org (0 days ago) nic.edu (5 days ago) nic.cx (7 days ago) nic.hm (280 days ago)
----- End forwarded message -----
I've corresponded with Chaz Boston-Baden (a friend), owner of the laconiv.org domain (which was the site of the recent World Science Fiction Convention in Los Angeles, L.A.con IV): He thanked me for keeping laconiv.org on my monitoring list, and assured me that he's arranged with Gandi.net, his registrar, to auto-renew.
Here's the relevant snippet from WHOIS:
$ whois laconiv.org | more [Querying whois.publicinterestregistry.net] [whois.publicinterestregistry.net] [...] Domain ID:D99628485-LROR Domain Name:LACONIV.ORG Created On:06-Aug-2003 22:18:41 UTC Last Updated On:10-Sep-2005 01:00:04 UTC Expiration Date:06-Aug-2007 22:18:41 UTC Sponsoring Registrar:Gandi SAS (R42-LROR) Status:OK Registrant ID:0-622473-Gandi Registrant Name:SCIFI INC. c/o BADEN Registrant Organization:SCIFI INC. c/o BADEN
So, holy mama! Expiration is set for 10:18 PM UTC today, which is 3:18 PM Pacific Daylight Time -- and it's 3:26 PM PDT right this moment. (Probably some timing glitch.)
If one assumes that Chaz is correct, and Gandi is set to carry out the auto-renew action, then Gandi's cutting things awfully close -- and with vanishingly little opportunity for recovery if, e.g., the credit card charge is temporarily not possible, and so on.
If there are significant numbers of registrars who wait until the last few minutes of registration -- a Bond-flick-style countdown -- before running their autorenew scripts, then that makes it all the more difficult for third parties, running things like domain-check, to know whether domains are in danger of expiration, or not.
Ben Okopnik [ben at linuxgazette.net]
On Mon, Aug 06, 2007 at 03:32:50PM -0700, Rick Moen wrote:
> I figured this will be of general interest. Ben and I (with Ben doing > the real work) have been working, off and on, on a scriptable mechanism > for checking a list of Internet domains for impending expirations, and > then do some sort of appropriate notification.
Rick, meanwhile, has been doing yeoman duty testing and retesting my script against the hundreds of domains, and reporting the failures in enough detail that I can work on improving the parsing. Development would have gone nowhere without him.
I must say that hunting down all the odd little cases and writing the
regexes to parse all the weird little date formats (the human mind can
be a very twisty little thing) has been a fun experience. 
> Ben's Perl script, > "domain-check", for this purpose relies on the commodity open-source > client for the TCP port 43 "WHOIS" information service, offered by most > domain registries.
I've also discovered an interesting fact WRT the 'whois' client as compared to 'jwhois': the latter isn't hard to rewrite in Perl - it just creates a query (a text string formatted per RFC 954), pings port 43 on a server, and prints out the returned results. The former, however, is a bit more interesting: it does all of the above plus precedes that output with a block of information that contains even more useful data (often, that's the block I end up parsing.)
I haven't had the time to explore all the possible WHOIS queries in depth, so I don't know where they're getting the info - but it's a pretty nice feature of the 'whois' client.
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Rick Moen [rick at linuxmafia.com]
Quoting Ben Okopnik (ben@linuxgazette.net):
> Rick, meanwhile, has been doing yeoman duty testing and retesting my > script against the hundreds of domains, and reporting the failures in > enough detail that I can work on improving the parsing. Development > would have gone nowhere without him.
Eh, you flatter me.
> I must say that hunting down all the odd little cases and writing the > regexes to parse all the weird little date formats (the human mind can > be a very twisty little thing) has been a fun experience.
My friend Chaz, owner of laconiv.org, resolved part of the mystery about what was happening with that site. He said, quite understandably:
No, I don't have any of them set for auto-renew. I don't like having an auto-payment of any kind on my credit cards.So, I really did in this case help a friend avert an accidental domain expiration (and this wasn't a case of Gandi.net doing a last-minute autorenew) -- and got thanked, for a change.
You may recall that I had checked, yesterday, on the domain status and found that it was some number of minutes clearly past the indicated expiration time. I remember that the whois record still showed laconiv.org as in status "Active", rather than any variety of expired status. It would have been interesting to watch more closely and see when the DNS ceased working, and when the status was changed from "Active" -- bearing in mind, on the latter item, that there's always some lag between the time the TLD zone (in this case, .ORG) implements changes such as domain deactivation, and the time those changes propagate to the public WHOIS data.
However, I missed my chance to watch that unfolding disaster, and instead chose to help a friend. Damn, messed up again! ;->
Rick Moen [rick at linuxmafia.com]
Quoting Ben Okopnik (ben@linuxgazette.net):
> I've also discovered an interesting fact WRT the 'whois' client as > compared to 'jwhois': the latter isn't hard to rewrite in Perl - it just > creates a query (a text string formatted per RFC 954), pings port 43 on > a server, and prints out the returned results. The former, however, is a > bit more interesting: it does all of the above plus precedes that > output with a block of information that contains even more useful data > (often, that's the block I end up parsing.) > > I haven't had the time to explore all the possible WHOIS queries in > depth, so I don't know where they're getting the info - but it's a > pretty nice feature of the 'whois' client.
So, again mostly to fill everyone else in, and recap, the two WHOIS clients (cited above) each have drawbacks in being not able to deal intelligently with particular top-level domains (TLDs). Among other things, out of the box, each fails to know the correct TLD-specific WHOIS server to consult, on a bunch of domains.
IANA (the Internet Assigned Numbers Authority) maintains a supposedly authoritative list of where the WHOIS service (if any) resides for each of the 250 country-code TLDs (ccTLDs) and 20 generic TLDs (gTLDs), and post that data to public Web pages. In an ideal world, the two WHOIS client utilities' authors would at least bother to ensure that they benefit from that public information. Alas, they seem not to. Also, IANA's data are incomplete and sometimes wrong.
To be fair, each featherless biped on this planet probably only cares about at most a dozen TLDs at a time, and some of the TLDs strain even my internationalist instincts past the breaking point. (On the other hand, it's kinda cool to discover that Antarctica's domain, .aq, actually has at least one Web server, www.com.aq, even if the corresponding IP is probably somewhere in Texas.) However, in trying to make Ben's script be as universally useful as possible, we've ended up trying to "fix" or work around the two client programs' respective TLD blindnesses.
The "jwhois" client has the advantage of having a very flexible conffile format, in which among other things you can supply an explicit WHOIS server FQDN to consult for any TLD. Which suggests a subproject....
I've started a modest little list of test data at: http://linuxmafia.com/pub/linux/network/domain-check-testdata When complete, it will then serve as the basis for an optimal jwhois.conf file (optimal in the sense of returning real results from all TLDs that offer any useful data on a known TCP-43 WHOIS server -- which in turn Ben and I can then use to ensure that Ben's domain-check script works on all TLDs that provide domain expiration data in their WHOIS output.
Amazingly enough, most TLDs provide either no expiration data at all, or expiration data that obviously has no real-world consequence, since various domains shown as expired are still functional. It's a wacky world, out there.