C-Panel Shared Web Hosting
Questions and Answers briefing on recent service
problems
Q. I’ve experienced severe problems with my E-Mail service and
C-Panel Web site over the weekend of the 14th and throughout Monday
the 16th of July. What went wrong and what are you going to do to
prevent it happening again?
A. Zen’s shared hosting products rely
on a resilient storage area network or ‘SAN’ to store user data, including Web
site contents and E-Mail. The C-Panel servers run the C-Panel application
itself and access customer files stored remotely on the SAN. On the afternoon
of the 14th of July one of the disk shelves that store customer data
appeared to fail. A disk shelf is basically a large box of disks configured to
use RAID 5 and Zen utilises two of them, together with two specialised network
switches that control access to the disk shelves from within Zen’s service
network. The shelves copy or replicate data between them to form a mirror image
of the data. These storage elements are configured for 'two times' redundancy but on
this occasion customers with sessions using the failed shelf did not re-connect to the
other ‘mirror’ shelf automatically. This caused customer Web sites and E-Mail
to fail while the SAN equipment was re-initialised.
A decision
was then taken to replace the failed shelf but this introduced additional compatibility
problems and caused the SAN system to run slowly. Our engineers worked through
the night to keep the system running but some key hosting services, including
C-Panel E-Mail were suspended to prevent excessive load on the storage network
disabling key services and access to customer Web sites.
The
original failed disk shelf was found not to be faulty and has been restored to service and is
experiencing no compatibility problems with the storage network. An
investigation is underway with our equipment suppliers to try and isolate the
reasons why the system didn’t failover automatically and why the replacement
disk shelf failed to operate reliably.
Zen is
currently undertaking an upgrade and expansion project for the storage network.
The lessons learned from the problems experienced over the past weekend will be
carefully applied to the upgrade project. We need to improve our test plans,
diagnostic procedures and escalation processes to our suppliers to help ensure
the storage network operates reliably and in a resilient manner.
Q. I’m a C-Panel customer and I’ve been experiencing slow and
intermittent service. Why is this and what’s being done about it?
A. C-Panel services have been adversely affected by a number of
smaller issues that when combined together, have rendered some customer Web
sites slow and unresponsive and have caused the C-Panel E-Mail services to
fail. We have identified the issues and a plan has been put together to tackle
them. Some work has already been undertaken while other issues have only
recently been identified. The issues
are:
Tackling spamming and abusive
C-Panel accounts
Around 50%
of the workload of a typical C-Panel server is E-Mail. We have had problems
with a minority of accounts injecting large quantities of junk mail into the
C-Panel server E-Mail queues and this slows down the entire service.
Although
these accounts are quickly identified and removed, we need to strengthen our
order process and systems administration applications to make it more difficult
for abusive users to obtain and use C-Panel accounts.
A number of
new measures are being introduced, including a more robust order process and
the installation of new management scripts that will automatically identify
accounts that are sending junk mail. The script management tools go live this
week and improvements to our order processes are going live throughout August.
Improving E-Mail handling
To help
improve mail system performance, two additional servers have been installed
that will act as mail processors for C-Panel. The impact of these new servers
should be felt in the coming weeks as they offload most of the E-Mail
processing work from individual C-Panel servers.
Introducing better resource
management for every C-Panel server
On a shared
service, the actions of individual users can have a disproportionate affect on
the performance of the overall server if they inadvertently run scripts or
software containing bugs. Currently, the management of resources on the C-Panel
server has been shown to be inadequate and individual Web sites are sometimes able
to ‘run away’ with system resources, causing the server to fail. Zen’s Principal Engineer has been assigned the task of reviewing current resource
management practices and applying improvements where possible to prevent this
from happening.
Q. What practical improvements can I expect to see from the
product?
There are
two major changes to the product set that will benefit Zen hosting customers.
The first major change is the launch of a new Mail product, aimed at providing
business grade E-Mail services to micro and small businesses, being
competitively priced and bundled with a domain name. Zen won the 2007 ISPA
award for best business E-Mail for our core mail service and we are extending
and improving our core mail service to offer features such as a better Web mail
client with support for mobile devices. The crucial change is that we will be
treating E-Mail as a dedicated focused product, with inbuilt resiliency, and
not as peripheral function of a Web hosting package. All existing C-Panel
customers will be offered the new Mail product as a free upgrade following its
launch in early October 07. Future C-Panel sales will be focused on delivering a
pure Web hosting service with improved reliability and performance.
The second
major change will see a substantial upgrade to the C-Panel service itself, with
faster, more powerful server hardware being installed to replace the existing
server hardware currently in use. A new version of C-Panel (version 11) which
offers support for PHP5 and Apache version 2 will be installed on a new server
farm and following a period of testing, customers will be invited to migrate
onto it. There may be some coding changes needed to customer Web content that
uses PHP, but the performance gains and improved functionality offered by
version 11 will make the migration worthwhile and it will not be a forced,
abrupt event. The new server farm will be available from September 07.
We are also
committed to improving communications with our customers with more frequent
updates and better control over the release of information concerning service
maintenance and outages.
Q. Can you give me any assurances that the planned
improvements are going to happen?
A. Ensuring that all our hosting products are engineered and
supported to the highest possible standard is now the number one priority for
Zen. Extensive support and expertise from across the business is being focused
on accelerating the already planned improvements and to identify and tackle new
issues as they arise. I can guarantee that all hosting customers will
experience an improved and enhanced service going forwards, with tangible improvements
being felt over the coming three months.
Q. Who can I contact with any further concerns or feedback?
A. You can contact the Zen Web Hosting
team directly by sending an E-Mail to hostingfeedback@zen.co.uk
and we will deal with your enquiries as quickly as possible.
I would
like to take this opportunity to apologise to all those customers affected by
recent service problems, but would like to re-assure you that everything
possible is being done to improve the level of service experienced.
Thank you
for your patience.
Stuart
Birchall
Director
Zen Web
Hosting
--
Phil Long
Technical Support Manager - Zen Internet Ltd.