Reports of my site's death are greatly exaggerated

UPDATE: The problem now, finally, seems to be fixed. Please comment on that post if you're still unable to see dansdata.com. And now, back to the original text of this post.

Some users of the Optus ISP here in Australia are having problems accessing dansdata.com. It's been happening for a while - here are people complaining about it in September, with the later reports only a few days ago.

I think all of the people with this problem have Optus cable Internet (as opposed to DSL or dial-up or satellite or carrier pigeon), though, fortunately, very far from all Optus cable users seem to have the problem. The nature of the problem is pleasingly clear: Dansdata.com has, from their point of view, been completely gone for weeks now, if not months.

Except it's not, of course. I may only put up one new article per decade on dansdata.com, but I have not died or been abducted by Zeta Reticulans or decided to reject technology and return to the land.

In the olden days of the late 1990s, the first diagnostic step when you wanted to see if a site was really down or if the problem was to do with your own Internet connection was to feed the site URL to Babelfish or one of the numerous dodgy proxy sites, and see if they could see it.

Now we've got more elegant solutions, in isitup.org and, if you prefer more verbose URLs, the very-similar-looking downforeveryoneorjustme.com. (I hope those two sites are actually run by different people - they seem, at least, to be on different servers - so they won't often ironically both go down at once.)

Anyway, I'm not certain about the exact nature of these problems, because a few people have e-mailed me about them, but when I ask them for details, they don't reply. I don't get a bounce message, either. This is exactly what you'd expect if some Optus router has decided that www.dansdata.com and mail.dansdata.com and everythingelse.dansdata.com are filthy spam servers all traffic from which is to be subjected to damnatio memoriae.

I've asked my Web hosts, SecureWebs, whether this is anything to do with them. It isn't. Well, it might be, very indirectly, since the server dansdata.com is on has occasionally been blocked on one or another of the many spam-server lists because of real or imagined misdeeds by other sites that share the server or nearby SecureWebs IP addresses. The Optus block could have been caused by that sort of thing, and then accidentally never cancelled. But Blogsome, who host this blog, stack rather more blogs per IP address than SecureWebs do sites, and the worst that's resulted from that to date has been a few days when bit.ly was warning people who clicked links from my Tweets that dansdata.blogsome.com might be bad.

I've also asked Optus, and they replied almost instantly to tell me that they could not replicate the problem, please send soil samples, et cetera.

So we need two things.

One: Some more detailed info about who using Optus can't see my site. This can easily be acquired by means I am about to explain in tedious detail.

Two: Complaints to Optus from the people who can't see my site, including the above info. Send the results to me as well - just posting them as a comment here will do very nicely - but you're much more likely to get action from a giant ISP on a weird problem like this if lots of people report it than if one person aggregates info and forwards it like a petition.

I could keep fiddling around trying to contact the Optus-using complainants from my addresses at other ISPs - I reckon my Optus account ought to be able to reach 'em. And I will. But I'll just point them to this blog post, so now that I've finally gotten around to writing it, so we can all try to figure it out together.

(I freely admit that I've known some people were having this problem for weeks now, but I was hoping the problem would just go away when someone at Optus hit a reset button or finally got rid of zzzzmust_delete_this_by_sep_9_09.cfg.)

The Whirlpool forum thread I mentioned earlier points to an excellent article on the Whirlpool wiki, "Is this site down?". The instructions there pretty much cover what you need to do, plus some other possibly-helpful stuff.

Basically, people who can't see dansdata.com need to ping and traceroute dansdata.com, and see what they get. Optus themselves turn out to have a Web-accessible Looking Glass server and a traceroute one too. Those can see my site, so if you can't, comparing and contrasting their results with your own could be helpful.

The easiest way to ping and traceroute from your computer is via the command line. In Windows, click Start, type "cmd", and in the resultant window just type

ping dansdata.com

and then

tracert dansdata.com

If your local DNS doesn't resolve dansdata.com to anything - "...could not find host dansdata.com", "unable to resolve target system name dansdata.com" - you can try bypassing the DNS and just going straight to the server's IP address, which is 64.85.21.19:

ping 64.85.21.19
tracert 64.85.21.19

(You can just type or paste 64.85.21.19 into your browser address bar to go to the site, by the way, if you actually can get to 64.85.21.19 from where you are. This advanced hacking technique has delivered precious, precious boobies to countless office workers and teenagers toiling under the yoke of sufficiently stupid site-blocking software.)

You can copy-and-paste the results from a Windows command-line window to somewhere else - like a comment and/or complaint message - by selecting the text, to do which you'll probably need to use the cumbersome Edit -> Mark option in the command-line window's lone menu.

If you want to be all fancy and bypass the Mark-ing, you can do this:

ping dansdata.com >>c:\dan_results.txt
tracert dansdata.com >>c:\dan_results.txt
ping 64.85.21.19 >>c:\dan_results.txt
tracert 64.85.21.19 >>c:\dan_results.txt

Presuming you have a C: drive, this will create a text file called dan_results.txt there and append the results of the commands to it, instead of just displaying them in the command-line window.

(If you used a single > instead of >>, each new output would overwrite the contents of the text file, instead of being tacked on at the end.)

Like all hip and happening ISPs, Optus only want you to contact them via some stupid Web form that redirects to a billion-character URL and that could be sending your message to screwyou@example.com for all you know. But with any luck a dozen or so people all suffering from the same disease will cause some action.

Now fly, my pretties! Fly!

20 Responses to “Reports of my site's death are greatly exaggerated”

  1. auraseer Says:

    I can see both sites just fine, but your 'damnatio memoriae' link is broken.

    [We're having some trouble replicating the problem at our end. Actually, that link looks to us like it's just fine, and always was. So, yyeeeeahh, we're gonna need to go ahead and ask you to provide some supporting evidence, and then submit it to us via the secret feedback page that you can only find with Gopher. You know, to protect us from hackers. -Dan]

  2. PaperDocket Says:

    Done and done. I've been having this issue ever since I switched from Optus ADSL to Optus Cable.

  3. fad Says:

    Optus Internet has been playing up for several weeks now, and it's not just their cable service.

    While I haven't seen DansData.com drop off the air, our business-class Sdsl link has been experiencing intermittent packet-loss issues that last from 15 minutes to a couple of hours. Most recently our connection just dropped off the net for 90 minutes.

    All of those weren't with our link but from deep inside Optus' network.

    We've had network techs from our hosting company talk to Optus. So far the responses have been along the lines of "problem? what problem?" and "we'll look into it".

    When the wait on the "Optus Business Service Assurance Support" number is over 40 minutes, after hours, and there is an apology and reference to a trouble-ticket repeated every couple of minutes, then you know they've got either a Heisenbug or major problems.

  4. Daniel Rutter Says:

    From an Optus cable user in Melbourne:

    By the way, you mentioned that the Optus ping and traceroute servers can see your site, however; when I checked the traceroute at http://traceroute.optusnet.com.au/?args=dansdata.com it timed out at the same place as my home PC, so it looks like the issue can be replicated. Unless it's only because I did it from home. Anyway, I've included those results in the attachment as well.

    ----------------------------------------------------------------------------------
    Results from my PC
    ----------------------------------------------------------------------------------

    Pinging dansdata.com [64.85.21.19] with 32 bytes of data:
    Request timed out.
    Request timed out.
    Request timed out.
    Request timed out.

    Ping statistics for 64.85.21.19:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

    Tracing route to dansdata.com [64.85.21.19]over a maximum of 30 hops:
    1 1 ms 1 ms 1 ms c210-49-81-238.rochd2.qld.optusnet.com.au [210.49.81.238]
    2 * * * Request timed out.
    3 7 ms 7 ms 9 ms meb4-ge2-0-4.gw.optusnet.com.au [198.142.160.13]
    4 24 ms 23 ms 24 ms sbr3-ge14-0-0-821.gw.optusnet.com.au [211.29.156.12]
    5 188 ms 186 ms 186 ms 203.208.191.73
    6 188 ms 186 ms 187 ms te-4-2.car2.SanJose1.Level3.net [4.79.42.229]
    7 188 ms 188 ms 200 ms vlan79.csw2.SanJose1.Level3.net [4.68.18.126]
    8 195 ms 187 ms 186 ms ae-72-72.ebr2.SanJose1.Level3.net [4.69.134.213]
    9 216 ms 206 ms 217 ms ae-7.ebr1.Seattle1.Level3.net [4.69.132.50]
    10 209 ms 207 ms 207 ms ae-13-51.car3.Seattle1.Level3.net [4.68.105.4]
    11 206 ms 207 ms 207 ms ge-0-4-sea-cr1.cet.com [4.71.152.174]
    12 215 ms 214 ms 217 ms ge-0-0-spk-cr1.cet.com [198.202.26.1]
    13 214 ms 215 ms 217 ms securewebs-gw.cet.com [198.202.27.27]
    14 218 ms 221 ms 217 ms fe-0-0-3-kf-br1.securewebs.com [64.85.23.2]
    15 * * * Request timed out.
    16 * * * Request timed out.
    17 * * * Request timed out.
    18 * * * Request timed out.
    19 * * * Request timed out.
    20 * * * Request timed out.
    21 * * * Request timed out.
    22 * * * Request timed out.
    23 * * * Request timed out.
    24 * * * Request timed out.
    25 * * * Request timed out.
    26 * * * Request timed out.
    27 * * * Request timed out.
    28 * * * Request timed out.
    29 * * * Request timed out.
    30 * * * Request timed out.

    Trace complete.

    ----------------------------------------------------------------------------------
    Results from http://traceroute.optusnet.com.au/?args=dansdata.com
    ----------------------------------------------------------------------------------

    $ traceroute dansdata.com
    traceroute to dansdata.com (64.85.21.19), 30 hops max, 38 byte packets
    1 mas4-vl300.gw.optusnet.com.au (211.29.132.2) 0.503 ms 0.332 ms 0.609 ms
    2 mas1-ge9-0.gw.optusnet.com.au (211.29.129.33) 0.231 ms 0.234 ms 0.235 ms
    3 mas3-ge2-0-0-811.gw.optusnet.com.au (211.29.155.11) 0.356 ms 0.358 ms 0.361 ms
    4 203.208.190.149 (203.208.190.149) 149.014 ms 149.123 ms 149.133 ms
    5 xe-9-2-0.edge3.LosAngeles1.Level3.net (4.78.195.201) 149.136 ms 149.137 ms 149.136 ms
    6 ae-62-60.ebr2.LosAngeles1.Level3.net (4.69.144.51) 149.637 ms ae-72-70.ebr2.LosAngeles1.Level3.net (4.69.144.115) 162.617 ms ae-82-80.ebr2.LosAngeles1.Level3.net (4.69.144.179) 149.522 ms
    7 ae-2.ebr2.SanJose1.Level3.net (4.69.132.13) 161.503 ms 161.750 ms 161.880 ms
    8 ae-7.ebr1.Seattle1.Level3.net (4.69.132.50) 178.741 ms 186.989 ms 180.119 ms
    9 ae-13-51.car3.Seattle1.Level3.net (4.68.105.4) 201.222 ms 178.736 ms 220.341 ms
    10 ge-0-4-sea-cr1.cet.com (4.71.152.174) 179.617 ms 179.111 ms 179.620 ms
    11 ge-0-0-spk-cr1.cet.com (198.202.26.1) 186.738 ms 186.479 ms 186.618 ms
    12 securewebs-gw.cet.com (198.202.27.27) 187.102 ms 187.110 ms 187.363 ms
    13 fe-0-0-3-kf-br1.securewebs.com (64.85.23.2) 189.864 ms 190.109 ms 189.739 ms
    14 * * *
    15 * * *
    16 * * *
    17 * * *
    18 * * *
    19 * * *
    20 * * *
    21 * * *
    22 * * *
    23 * * *
    24 * * *
    25 * * *
    26 * * *
    27 * * *
    28 * * *
    29 * * *
    30 * * *
    Singtel-Optus — 31/10/2009 03:06:15 +1100

    ----------------------------------------------------------------------------------
    Results from http://looking-glass.optus.net.au/cgi-bin/nph-looking-glass.pl
    ----------------------------------------------------------------------------------

    Performing trace to dansdata.com...

    1 FastEth8-1-0.sb1.optus.net.au (192.168.34.2) 1.395 ms 1.013 ms 0.876 ms
    2 203.208.148.89 (203.208.148.89) 161.569 ms 161.426 ms 161.603 ms
    3 te-4-2.car2.SanJose1.Level3.net (4.79.42.229) 161.637 ms 161.568 ms 161.648 ms
    4 vlan79.csw2.SanJose1.Level3.net (4.68.18.126) 168.607 ms 172.165 ms 162.344 ms
    5 ae-72-72.ebr2.SanJose1.Level3.net (4.69.134.213) 161.983 ms 170.865 ms 162.335 ms
    6 ae-7.ebr1.Seattle1.Level3.net (4.69.132.50) 179.451 ms 187.351 ms 179.781 ms
    7 ae-13-51.car3.Seattle1.Level3.net (4.68.105.4) 179.430 ms 179.597 ms 179.612 ms
    8 ge-0-4-sea-cr1.cet.com (4.71.152.174) 179.792 ms 179.350 ms 179.538 ms
    9 ge-0-0-spk-cr1.cet.com (198.202.26.1) 186.605 ms 186.611 ms 186.776 ms
    10 securewebs-gw.cet.com (198.202.27.27) 187.331 ms 186.456 ms 186.834 ms
    11 fe-0-0-3-kf-br1.securewebs.com (64.85.23.2) 189.615 ms 190.399 ms 188.976 ms
    12 * * *
    13 * * *
    14 * * *
    15 * * *
    16 * * *
    17 * * *
    18 * * *
    19 * * *
    20 * * *
    21 * * *
    22 * * *
    23 * * *
    24 * * *
    25 * * *
    26 * * *
    27 * * *
    28 * * *
    29 * * *
    30 * * *

    (Note that traceroute timeouts just by themselves don't necessarily mean anything. If you can't actually get to the site, though, they're significant.)

  5. Daniel Rutter Says:

    And here, for a change, is ping and traceroute data from a reader in Victoria (I presume, from the first few traceroute hops), who can't see dansdata.com from his iiNet account:

    Pinging dansdata.com [64.85.21.19] with 32 bytes of data:
    Request timed out.
    Request timed out.
    Request timed out.
    Request timed out.

    Ping statistics for 64.85.21.19:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

    Pinging 64.85.21.19 with 32 bytes of data:
    Request timed out.
    Request timed out.
    Request timed out.
    Request timed out.

    Ping statistics for 64.85.21.19:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

    Tracing route to dansdata.com [64.85.21.19]
    over a maximum of 30 hops:

    1 3 ms <1 ms <1 ms 192.168.0.1
    2 27 ms 27 ms 27 ms nexthop.vic.iinet.net.au [203.55.229.88]
    3 27 ms 27 ms 26 ms gi1-2.mel-pipe-bdr1.iinet.net.au [203.215.6.22]
    4 43 ms 42 ms 42 ms gi15-1-3.syd-ult-core1.iinet.net.au [203.215.20.12]
    5 43 ms 42 ms 43 ms te13-0-0.syd-mas-core1.iinet.net.au [203.215.20.149]
    6 40 ms 39 ms 39 ms 34023.sydp06.cu.reach.com [134.159.160.57]
    7 43 ms 42 ms 42 ms i-10-0.sydp-core02.bi.reach.com [202.84.220.149]
    8 192 ms 191 ms 191 ms i-14-0-0.wil-core02.bx.reach.com [202.84.140.237]
    9 192 ms 191 ms 191 ms i-1-1.tlot03.bi.reach.com [202.84.251.234]
    10 190 ms 191 ms 190 ms xe-11-2-0.edge1.losangeles6.level3.net [4.68.62.9]
    11 195 ms 198 ms 198 ms ae-72-70.ebr2.LosAngeles1.Level3.net [4.69.144.115]
    12 227 ms 233 ms 233 ms ae-2.ebr2.SanJose1.Level3.net [4.69.132.13]
    13 242 ms 248 ms 252 ms ae-7.ebr1.Seattle1.Level3.net [4.69.132.50]
    14 243 ms 242 ms 243 ms ae-13-51.car3.Seattle1.Level3.net [4.68.105.4]
    15 241 ms 240 ms 240 ms ge-0-4-sea-cr1.cet.com [4.71.152.174]
    16 251 ms 250 ms 250 ms ge-0-0-spk-cr1.cet.com [198.202.26.1]
    17 247 ms 247 ms 248 ms securewebs-gw.cet.com [198.202.27.27]
    18 253 ms 251 ms 253 ms fe-0-0-3-kf-br1.securewebs.com [64.85.23.2]
    19 * * * Request timed out.
    20 * * * Request timed out.
    21 * * * Request timed out.
    22 * * * Request timed out.
    23 * * * Request timed out.
    24 * * * Request timed out.
    25 * * * Request timed out.
    26 * * * Request timed out.
    27 * * * Request timed out.
    28 * * * Request timed out.
    29 * * * Request timed out.
    30 * * * Request timed out.

    Trace complete.
    DNS request timed out.
    timeout was 2 seconds.
    Server: UnKnown
    Address: 192.168.0.1

    Name: dansdata.com
    Address: 64.85.21.19

    Curiouser and curiouser.

    (I'm using iiNet ADSL myself here in NSW, and can see my site fine.)

  6. Bedlam Says:

    FWIW, I'm an iiNet user in Victoria and dansdata.com has been down since the last update around the 22nd. Although, I just checked now and it looks to be up again. And here I was thinking iiNet was specifically *not* supposed to be a URL-blocking nanny ISP.

    Although, given I haven't had problems with any other sites, it's probably as you suspect, a technical glitch with a router or some such.

  7. kamikrae-z Says:

    I'm also experiencing the block - Optus Cable in Sydney.

    I'll post my results but before I do... is it safe to post my IP? Is it ok if I blank out the first couple of lines for the tracert or is this not going to save me from 1337 h4>

  8. Joe in Australia Says:

    Here are my results (Optus Cable, Victoria, my IP is 211.28.130.226)

    Ping reports 100% packet loss.

    Traceroute reports the following:

    raceroute to dansdata.com (64.85.21.19), 64 hops max, 40 byte packets
    1 10.44.64.1 (10.44.64.1) 7.936 ms 11.499 ms 7.707 ms
    2 meb4-pos4-0-1.gw.optusnet.com.au (198.142.192.105) 7.713 ms 10.509 ms 7.469 ms
    3 sbr3-ge14-0-0-821.gw.optusnet.com.au (211.29.156.12) 23.215 ms 28.941 ms 24.228 ms
    4 203.208.192.121 (203.208.192.121) 237.879 ms 184.057 ms 183.397 ms
    5 te-4-2.car2.SanJose1.Level3.net (4.79.42.229) 180.621 ms 216.825 ms 181.351 ms
    6 vlan79.csw2.SanJose1.Level3.net (4.68.18.126) 180.153 ms 186.316 ms 178.849 ms
    7 ae-72-72.ebr2.SanJose1.Level3.net (4.69.134.213) 225.611 ms 196.830 ms 182.777 ms
    8 ae-7.ebr1.Seattle1.Level3.net (4.69.132.50) 206.834 ms 209.607 ms 210.150 ms
    9 ae-13-51.car3.Seattle1.Level3.net (4.68.105.4) 198.350 ms 202.765 ms 196.376 ms
    10 ge-0-4-sea-cr1.cet.com (4.71.152.174) 198.885 ms 203.271 ms 199.334 ms
    11 ge-0-0-spk-cr2.cet.com (198.202.26.4) 207.833 ms 208.134 ms 209.101 ms
    12 securewebs-gw.cet.com (198.202.27.27) 208.076 ms 206.329 ms 207.643 ms
    13 fe-0-0-3-kf-br1.securewebs.com (64.85.23.2) 212.368 ms 211.517 ms 212.078 ms
    14 * * *
    15 * * *
    16 * * *
    17 * * *

    And so on until I kill it. I hope this helps; I really thought that the site was down for good.

  9. a5nvxm1h Says:

    How come your reverse DNS is beechler.com?
    dnstools.com

  10. Chazzozz Says:

    No. 9 may be on to something. A forward lookup on beechler.com shows 208.109.181.29. Checking the SOA for both dansdata.com and beechler.com also gives differing results (ns1.dnswizards.com and ns51.domaincontrol.com, respectively).

    Looking at the mail side of things, mail.dansdata.com resolves to 64.85.10.4. Do a reverse lookup on that IP address and you get auricalcompany.com, for which a forward lookup resolves to 66.29.215.106.

    I wonder if Optus has a filter somewhere that's deciding your DNS records don't sufficiently match up, and that makes your site look suspicious? SecureWebs owns the entire Class B 64.85.x.x range so they should be dictating who gets to use IP addresses within it. Might be worth asking them for a "Please explain..." as to why other hosts which appear to be on different networks are using your IP addresses.

  11. raband Says:

    Hey Dan,

    I did originally get an email from someone at securewebs asking for tracert etc, but haven't heard anything since.

    For anyone missing out on their fix of dansdata (can still browse the blog though) they can get it if they go through a proxy e.g. http://proxy.org/

    As a side note we do still get the rss feeds notifying us of when your page updates - we just can't get to them. (if you wanted to find out if this is more widespread than just some Optus cable users then a new page with a temporary link as its title to the blog should pick up some other MIA's - maybe a poll or something?)

    It is also affecting other secureweb hosted sites I guess (can't log into their homepage) - would a list of some random sites or IP's hosted by them on different hardware help narrow down where the problem is? Not sure who is hosted by secureweb, but there may be someone who could be hit hard by missing out on a small percentage of Aussie cable users?

    Am pretty sure it's a wider problem for securewebs than for you, but you're the only point of reference I have ;)

  12. pc486 Says:

    FYI, tracert will not work for most people. I'm in the US with a major DSL provider and standard ICMP-based traceroute on Linux stops at:

    14 fe-0-0-3-kf-br1.securewebs.com (64.85.23.2) 68.745 ms 73.229 ms 73.709 ms
    15 * * *

    The much more reliable tcptraceroute (http://michael.toren.net/code/tcptraceroute/) gives a full trace ending with:

    14 fe-0-0-3-kf-br1.securewebs.com (64.85.23.2) 122.660 ms 116.401 ms 163.514 ms
    15 beechler.com (64.85.21.19) 162.387 ms 156.139 ms 158.885 ms

    Standard ICMP echo requests (ping) do appear to work normally:

    PING dansdata.com (64.85.21.19) 56(84) bytes of data.
    64 bytes from beechler.com (64.85.21.19): icmp_seq=1 ttl=114 time=83.9 ms

  13. Erik T Says:

    So are there any failures that don't end at fe-0-0-3-kf-br1.securewebs.com ?

  14. Malcolm Says:

    I think the suggestion that it's an MTU path discovery problem (as someone mentioned in the Whirlpool discussion thread) is worth looking into.

    I can do a traceroute -F up to a maximum packet size of 1492 successfully from my iiNet ADSL connection. I see reference to an MTU of 1500 for OptusNet Cable but that's beyond what i can test from here.

  15. OrgAdam Says:

    Wouldn't you know it! I HAVE been looking for a way to convert things into stuff!

    [The above refers to a spam-comment which I have now deleted. -Dan]

    As I'm a rep and on the road, my mobile broadband account isn't blocked the same way my http://www.spin.net.au account is... I get my fix that way.

  16. Daniel Rutter Says:

    UPDATE: Yeah, it's looking like SecureWebs were mistaken when they told me the problem was not at their end. I'm corresponding with them about this; watch this space!

  17. jaws_au Says:

    Yeah, I can't connect from work - which despite being in Brisbane, is actually via servers in Spain.

  18. JoaoBordignon Says:

    Dan, a tracert from Brazil, yesterday it was OK:

    C:\>tracert http://www.dansdata.com

    Rastreando a rota para http://www.dansdata.com [64.85.21.19]
    com no máximo 30 saltos:

    1 1 ms

  19. JoaoBordignon Says:

    Comment system didn't like the DOS text. Just the tree last hops so:
    ge-0-0-spk-cr2.cet.com [198.202.26.4]
    securewebs-gw.cet.com [198.202.27.27]
    fe-0-0-3-kf-br1.securewebs.com [64.85.23.2]
    Esgotado o tempo limite do pedido.
    Esgotado o tempo limite do pedido.

  20. Bern Says:

    Well, I could connect fine to Dan's Data when you first posted this the other day, but now I can't.

    On Bigpond Cable, in Brisbane. Can't remember how to get a tracert through Bigpond's L33t anti-haxx0r system (they block ICMP tracert, IIRC), so wont be posting one up, unless someone reminds me how.


Leave a Reply