July 19 Routing problem
Around 1300 some customers experienced problems with network routing. Our
facilities provider at 151 Front St. disconnected a link 2 weeks ahead of schedule. By 1445 the issue was corrected.
July 11 Cybersurf dialup
The dialup information page has been updated with details on how IStop.com customers can use Cybersurf dialup.
July 9 New DNS IP addresses
Please use the following IP addresses for DNS servers:
209.195.118.108
66.11.167.161
July 5 Queensview colo outage
Queensview colo customers lost connectivity this morning due to a network
cutover to Cybersurf. Connectivity was restored at 2000.
July 1 New mail server IP address
When the mail server was moved yesterday, the IP address changed.
Some DNS servers still have the old IP address cached, but should
learn the new IP address before July 2. The new
IP address for mail.istop.com, pop.istop.com, and ns.istop.com
is 209.195.118.109.
June 30 mail server move
At 2230 tonight (June 30) the IStop.com mail server will be moved to
Cybersurf's network. This server handles POP/IMAP and webmail.
The server will be offline approximately 1.5 hours.
June 28 PPPoE login change
Bell has cut off the DSL service used by IStop.com. They chose to do so even though
they received a $300,000 deposit for the service in March 2005. In addition, they have
been paid well in excess of their General Tariff Item 5410 rates. Considering the
situation and the lack of support by the CRTC, IStop.com has made arrangements with
Cybersurf to continue service for our customers. PPPoE logins should be changed to
istop:userid@igs.net
Your password does not change. Staff at IStop.com and Cybersurf expect to have the new login working by 1900 today. We will issue another update on June 29 to
further assist our customers with this transition and provide more details on the actions of Bell.
June 27 Service outage
Shortly after 1800 most customers lost connectivity. The downtown Ottawa
POP connectivity was lost, so the incoming mail server and web server were
moved. The mail server was brought back online by 2300. The few
colocation customers in the Ottawa POP should retrieve their servers as we
do not expect to re-establish connectivity. We will have a further
update late in the day on June 28.
June 25 Queensview power outage
Likely due to storms in Ottawa, building power was lost at our Queensview
location shortly before 1530. This has affected office operations and
colocation customers.
June 22 packet loss affecting some Ottawa customers
Some Ottawa customers are experiencing packet loss. We have determined the problem to be in Bell's network between Ottawa and our Toronto POP. We expect the issue to be resolved within 1 day.
June 23 network maintenance
We just recieved the following maintenance notice from Cogent. They
provide the main internet backbone connectivity for IStop.com. Traffic
should reroute to our backup provider minimizing the impact to a few
minutes while our equipment reroutes the traffic.
Start time (local): 3:00 am 6/23/05
End time (local): 7:00 am 6/23/05
Work order number: WO 2617
Cogent customers receiving service at 151 Front Street, Toronto will be
affected by this maintenance while we upgrade equipment in your building.
During this timeframe, you will experience a brief interruption in service
while we complete the maintenance activities; the interruption is expected
to be less than 1 hour total - this may be several seperate outages. Due
to the complexity of the work, your downtime may be longer.
June 16 network outage
At 1530 Bell Canada experienced a failure in their network in Toronto. This affected the link between Toronto and our Ottawa POP, and about half our ADSL customers in the Ottawa area. Bell fixed the issue at around 2045.
June 10 Ottawa POP routing issues
Between 1345 and 1400 the Ottawa POP was unreachable due to a router configuration error.
May 27 Ontario PPPoE authentication problems
Some Ontario customers are having problems authenticating. A power glitch in southern Ontario shortly after 1600 has affected Bell equipment. Bell is aware of the problem and working to fix it.
Apr 28 Non-PPPoE connections down
Non-PPPoE customers are currently offline. A ticket has been opened with Bell to resolve the issue. We expect a resolution by 1500 today.
Apr 14 Ottawa-Toronto link failure
This morning high packet loss was measured on the link between Ottawa and
Toronto. By 1000 the link was down. The issue was determined to be a
problem in Bell's network. Bell's helpdesk indicated the issue was a Bell
switch in 151 Front St. The problem was resolved by 1555.
Mar 31 outgoing smtp problems
At 2200 the outgoing mail spool on smtp.istop.com was filled by spamming customers. This caused the server to reject new smtp connections. The responsible customers have been identified and suspended. Outgoing mail delivery may be slow while the spam is being purged from the spool.
Mar 16 emergency PPPoE router reboot
One of PPPoE routers is not accepting new sessions. It will rebooted at
1310. This will cause a PPPoE session reset for customers connecting
through this router.
Mar 11 network maintenance
A new ethernet switch will be installed in the Toronto POP between
0630 and 0830 on Saturday March 11. Customers may experience multiple
interruptions lasting 5-10 minutes during this interval.
Mar 4 Service interruption
At 1730 an apparent routing software bug caused a brief interruption in service. Related to this an emergency software upgrade is planned for tomorrow at 0500.
Mar 1 Ottawa POP congestion
At 0840 a traffic flood in our Ottawa POP caused network congestion. By 0900 the flood was identified and filtered, returning network performance to normal.
Feb 20 service interruption
At 0900 a brief service interruption was caused by an unexpected reset of a router in our Toronto POP.
Feb 16 mail server back online
At 1050 the mail server was brought online. The outage was caused by a security breach that corrupted some files. Services such as cgi and shell access to the mailserver will be disabled while we work to ensure the server is secured.
Feb 16 mail server offline
The incoming mail server stopped functioning properly at 0915. Work is ongoing to fix the problem. The outgoing (SMTP) mail server is not affected.
Feb 15 PPPoE router crash
One of our Cisco PPPoE routers in Toronto crashed around 1505 Eastern time. Most customers affected have been able to reconnect as we have a backup configured for the Toronto, Ottawa, and Montreal serving areas. We are working to restore service for the remaining customers. As of 1830 the router is back online. The crash was caused by a blown power supply.
Feb 3 morning service interruption
Due to a screw-up by one of our staff, the scheduled maintenance caused most customers to loose connectivity from 0400 until 0730.
Feb 3 maintenance
At 0400 a software upgrade will be performed on one of the routers in our Toronto POP. Customers may experience an interruption in traffic lasting a couple minutes.
Jan 18 Ottawa tunnel switch problems
At around 2250 one of Bell's PPPoE tunnel switches in Ottawa appears to have crashed. Connections through the other tunnel switch are running slow. We are attempting to open a trouble ticket with Bell to fix the issue. After finally reaching someone at the Bell wholesale helpdesk, we have learned that the tunnel switch did not crash. An OC48 card in Ottawa failed and was replaced. This caused the tunnel switch to drop all of its sessions. We expect the issue to be resolved by 0200.
Dec 30 Ottawa POP link down
Shortly before 1100 the Ottawa POP fiber link went down. A Bell technician replaced the fiber media converter power supply and service was restored by 1415.
Dec 22 mail server crash
At 1830 the mail server crashed. It should be back online by 1930. Upon further investigation the server did not crash; it was inaccessible due to a bad ethernet switch. As of 1930 it is back online.
Dec 22 Ottawa DOS attack
At 1750 network performance in our Ottawa POP deteriorated due to a DOS attack targeted at one of our co-location customers. By 1810 the attack was blocked.
Dec 21 mail server offline
As of 1015 the mail server is offline due to a sysadmin error. Work is underway to restore service. As of 1100 the mail server has been fixed.
Dec 19 mail server offline
This morning shortly after 0800 the server handling incoming mail crashed. One of the mirrored hard drives is not performing properly. The drive has been removed and the server will be brought back online, likely before 1300.
Dec 17 maintenance
Friday morning at 0200 a software upgrade will be performed on one of the routers handling PPPoE customer connections. The interruption to service should be less than 5 minutes.
Dec 11 - proxy.istop.com
proxy.istop.com is temporary unavailable.
Dec 7 - Ottawa POP outage
At 1130 a traffic flood coming from a colocation customer server in our
Ottawa POP was detected. A typo was made while attempting to filter the
flood. This blocked remote access to the Ottawa POP. By noon staff arrived
onsite to restore service and disconnect the server causing the flood.
Dec 6 - PPPoE session resets
This afternoon around 1515 one of our Cisco PPPoE routers crashed. The interruption to service was only a few minutes. We are investigating the cause of the crash.
Dec 4 - Ottawa problems
This morning Bell had an equipment failure with bas4-ottawa23. All ADSL subscribers routing through this equipment were not able to log on. This affected other ISPs and Sympatico as well as IStop.com. The problem was resolved by 1000 this morning. 1130 update - the problem was not completely fixed; some customers are having problems establishing a PPPoE session. Usually a connection can be established after 3 or 4 tries. Bell is aware of the problem and is working on it.
Dec 2 Service outage
We pleased to report that as of 1440 eastern 2004/12/03 we are in discussions with Bell and services will be reactivated shortly. 1500 update - all services appear to be restored. Customers will be issued credits or refunds for the service disruption.
Nov 30 Web server crash
The IStop.com web server crashed this afternoon and was down for about 2 hours. After investigating, the issue appears to be bad RAM. A hardware upgrade will be arranged for the server over the coming weeks.
Nov 25 More Ottawa session drops
It appears Bell has not resolved the problems from the weekend as Ottawa tsw2
restarted this morning around 0920. Tsw1 stayed up but did not pick up the
dropped sessions. As of 0950 tsw2 is up and appears to be functioning
properly. We continue to press Bell to permanantly resolve this issue.
Nov 20 More Ottawa session drops
Bell continues to have problems with equipment in Ottawa. Specifically the
tunnel switch boxes that route pppoe connections to the appropriate ISP
have been rebooting. Two tunnel switches serve Ottawa (tsw1 & tsw2).
Tsw2 rebooted around 1415. Tsw1 rebooted around 1535 and again around 1555.
Bell has not provided details on the problem, but it appears that the
boxes are getting overloaded, then crashing & rebooting.
Nov 20 Ottawa session drops
PPPoE sessions for many of our Ottawa customers were reset around 1430. This was due to a temporary problem with a piece of Bell equipment in Ottawa.
Nov 17 maintenance
At 0001 on the morning of Nov. 17th a new switch will be installed in the
Toronto POP. Some customers may notice a brief interruption as traffic
reroutes through the backup switch.
Nov 2 non-PPPoE packet loss
Starting yesterday many of our non-PPPoE customers started experiencing packet loss. The issue appears to be a problem in Bell's ATM network that is used to carry our customer traffic back to our POP in Toronto. We continue to work with Bell to troubleshoot the issue. The problem was resolved shortly before 1500 by disconnecting a customer that was flooding the network with spoofed traffic. This traffic seems to have caused problems for a Catalyst 5500 that connects to Bell's ATM network.
Nov 2 port blocking
Due to a large amount of virus & worm traffic, we have temporarily blocked port 135 and 139 traffic at the edge of our network. The block will be removed within 1 week.
Oct 31 network maintenance
Between 0200 and 0230 the routers handling customer PPPoE sessions will be reconfigured. Customers will be unable to establish a PPPoE connection for about 10 minutes during this window.
Oct 29 Queensview Colo outage
Between 0800 and 0900 Queensview colo was offline due to power problems.
Oct 25 MTU issues
We have determined the MTU issues are caused by a configuration problem with
Bell's connection to our network. We expect to have the issue resolved
within 24 hours. It is not affecting most customers - only applications
that use large UDP packets appear to be affected.
Oct 25 maintenance
At 0215 tomorrow (Oct 25) one of the 2 Cisco routers handling PPPoE
connections will be upgraded. This is intended to fix MTU problems that
some customers are having. Customers will experience downtime of up to 15
minutes.
Oct 22 maintenance
At 0230 Oct 22 a router in the Toronto POP will be reset.
Customer network traffic may be affected for a few minutes.
