Justin Ho

tightness is key

Server Status

all times Toronto time unless otherwise noted

Current status

  • No known issues.

Maintenance (recent)

  • 2013-03-21: Service degraded due to ISP issue, starting at 2:37pm Toronto time. Ticket opened with provider. (Service restored at 10:43pm)
  • 2013-03-16: Infrastructure maintenance may result in brief disconnections/interruptions over the next 24 hours.
  • 2013-02-17 2:27am: Some DNS changes are causing some domains not to resolve. I expect these problems to clear in the next 12-24 hours.
  • 2013-02-16 7:10pm: Antivirus and antispam software is being upgraded. Some mailboxes are offline for the next 10-15 minutes.
  • 2013-02-16 6:59pm: A mailbox server is being rebooted for security patches. Some mailboxes will be unavailable for the next 3 minutes or so.
  • 2013-02-16 10:01am: Exchange service upgrade complete. Downtime was 46 minutes across all users.
  • 2013-02-16 09:08am: Exchange services are being upgraded. Access to email may be intermittent for some users during this process. ETA for completion is 12:00PM noon, Toronto time. How long until service is restored?

Incident: 2012-05-18 -- 2012-05-29

  • 2012-05-18: Packet loss reported. Line problem identified, opened ticket with ISP.
  • 2012-05-18 2:01pm EST: Bell connectivity problem being worked on; connection lost.
  • 2012-05-18 4:09pm EST: Currently completely disconnected (again) due to a Bell problem. Backup link is being re-established, ETA 3-4 hours. All inbound and outbound mail is currently not functioning. Sent emails will be delayed until service is restored. We expect the main internet line to be re-established in approximately 24 hours (awaiting Bell response.)
  • 2012-05-29: Main link back online.

Incident: 2011-12-29 -- 2012-01-04

Summary:

  • A Bell Canada DSL line issue has taken the network offline. A ticket has been opened with the ISP and is awaiting resolution.
  • Partial access to some services is currently being established. Some services will begin to slowly come online via a cellular backup link.
  • Received emails during this period may have been returned to sender due to the extended outage -- please request emails to be resent once service is restored.

Overall status:

  • Note that most services are SLOWER than usual since they are running over a cellular link. Your patience is appreciated -- we're working on the problem.
  • Websites: Most are offline. In progress.
  • Outbound email: Now functional. If you have a domain name hosted you may need to add mx2.justinho.com as a MX DNS record if it is not already there. Email me if you need help.
  • Inbound email: Online.
  • Mobile push email: Mostly online, depending on client configuration. Not all phones will be able to connect. Add JUPITERSTATION as the domain or as a prefix to your login name (i.e. JUPITERSTATION\your-user-name)
  • Web access: Online at https://mail.justinho.com/owa . Authenicate using JUPITERSTATION\your-user-name.
  • Blackberry BES Push email: Online.

Log:

  • 2011-12-29 16:21PST -- ~5% packet loss being reported from monitoring, TekSavvy discovers lantern not mapped. Opened ticket with Bell.
  • 2011-12-31 07:55PST -- Site goes offline. Another ticket opened shortly thereafter.
  • 2011-12-31 11:20PST -- No update. Still waiting.
  • 2011-12-31 13:34PST -- Bell confirms that line was incorrectly modified.
  • 2011-12-31 22:48PST -- No update.
  • 2012-01-01 06:05PST -- Still no service.
  • 2012-01-01 14:23PST -- A backup cellular link is now being established to temporarily connect services. No ETA at this point, but hopefully things will be back online (but slow!) in the next few hours.
  • 2012-01-01 23:43PST -- We weren't able to provision service on a long weekend, but will get this backup link established in the future.
  • 2012-01-01 09:25PST -- A technician is onsite now to swap the modem to 'resolve' the first lantern not mapped problem. Site remains offline.
  • 2012-01-01 21:20PST -- The backup cellular link is now routing some traffic. We are gradually re-routing some services via this (slow) link to bring email and some websites online at this time.
  • 2012-01-03 06:05PST -- Ticket being routed to technician again.
  • 2012-01-03 07:30PST -- Bell ticket is being worked on by tech, expect fix between 11am - 3pm today
  • 2012-01-03 15:05PST -- Problem still not fixed. No update.
  • 2012-01-04 06:01PST -- Update from Bell: Line mapping incorrect on ATM link, being fixed, ETA 08:00PST (11:00EST).
  • 2012-01-04 07:56PST -- DSLAM reconfigured, ATM group believes everything should be working. Still no connectivity.
  • 2012-01-04 08:21PST -- Still no connectivity. Power cycled modem again, and called ISP again to escalate the issue. DSL was moved to gigE port on Bell's side, off ATM links.
  • 2012-01-04 09:25PST -- Service has been restored. DNS entries are now being updated to point back to the main DSL link -- expect full service restoration in the next 20-30 minutes while all routing entries are restored to normal configuration.