- Priority - High
- Affecting System - Spamexperts
-
We will be moving a primary node for spamexperts during this scheduled window. During this time, the spamexperts interface will not be available -- which includes quarantine management. During this time, mail is expected to flow through other nodes and deliver as normal. Updates will be provided once work is completed.
3/5 - 23:00 - this work continues within Spamexperts system admins' workflow. The node is on our hardware, however, they require complete management of it.
3/6 - 13:00 - We awaiting a wrap up email from spamexperts.
3/6/ - 15:00 - this work has been completed.
- Date - 03/05/2024 20:00 - 03/05/2024 23:00
- Last Updated - 03/08/2024 19:57
- Priority - High
-
We are aware of a data center outage within our Atlanta location. At this time, it appears to be a network issue that is data center wide. Further updates will be provided as information is available.
21:30 - the data center continues to work on this.
23:00 - services have been restored . Cause was a complete power outage at the building and it affected their redundant power switching equipment .
This issue started with a power outage earlier in the day and the data center was on backup power. When switching power back to mains, a surge happened and caused damages to different control boards for the power switching. The surge also caused one of the generators to go in to emergency shutdown. This event further caused Verizon and AT&T local outages. They are working on repairs for the control board and generator.
- Date - 02/27/2024 20:47
- Last Updated - 02/28/2024 15:09
- Priority - High
- Affecting Server - pk2-dal1
-
pk2-dal1 will have a few updates and likely a reboot during the scheduled window. Downtime is expected to be 5 to 10 minutes but probably less.
This work is in progress.
21:40 This work is done. A few minutes downtime.
- Date - 12/03/2021 21:00 - 12/03/2021 22:00
- Last Updated - 12/03/2021 21:43
- Priority - High
- Affecting Other - SMY1 data center in ATL; multiple servers
-
The data center for our current Atlanta operations has inform us of maintenance that will happen during the scheduled window. The building the data center is located within will be upgrading their power systems and during this time backup power within the data center will be used for 10 to 12 hours while the upgrades are done. No issues are expected to occur and backup power for the data center is completely separate from the buildings power supply. As this is work on critical systems, it is important to inform our clients that could be affected.
Servers affected:
pk1-smy1
pve-smy1
Client VPSes
11/2/2020 -- Due to recovery efforts from tropical storm Zeta, Georgia Power has suspended projects until that recovery is completed. As such, this work has been postponed and we will update this post once it is rescheduled.
12/8/2020 - This work has been rescheduled to 12/19/2020 .
- Date - 12/19/2020 07:00 - 09/21/2021 22:22
- Last Updated - 12/08/2020 17:15
- Priority - High
- Affecting Server - Zimbra Cluster 1
-
We will be performing upgrades on the Zimbra mail cluster during this scheduled window. This process will require the MTA service to be shutdown during this time. The MTA service handles delivering mail to and from the system. Any incoming mail will be queued at either spamexperts or the sending mail server and retried depending on that server's retry settings. This upgrade will implement additional features to the Zimbra service, which will be detailed after the upgrade is done.
If you have any questions on this work, please let us know.
20:30 ET: this work is starting.
21:00 ET: work is progressing as expected
22:00 ET: this work has been completed. Services are operating normally and only a few minutes of MTA downtime happened .
- Date - 07/07/2020 20:00 - 07/07/2020 22:59
- Last Updated - 10/12/2020 20:25
- Priority - High
- Affecting Server - pk-win1
-
Updates will be applied to pk-win1 . This will require a reboot or two. Downtime is expected to be 15 to 30 minutes.
9:06 PM ET - this work is starting.
9:10 PM ET -- server is rebooting for applying first run of updates
9:16 PM ET - server is still going through reboot processes, applying update configs, etc and at this point services are still up.
10:10 PM ET - server has rebooted and is going through second phase of updates.
10:12 PM ET - services are responding at this point. Couple minutes downtime so far. Second round of updates are being done now.
23:42 PM ET - this work has been completed.
- Date - 04/23/2020 21:00 - 04/23/2020 23:59
- Last Updated - 04/25/2020 00:37
- Priority - High
- Affecting Server - pk2-dal1
-
During this scheduled window, this server will receive updates on OS packages as well as a memory upgrade. As such, a shutdown of the server is required. Downtime is expected to be 15 minutes, though likely less.
10:30 PM ET -- This work is starting.
23:10 PM ET - Server is shutting down for memory upgrade.
23:20 PM ET - Server is back in operation at this time.
- Date - 06/05/2019 22:30 - 06/05/2019 23:30
- Last Updated - 06/05/2019 23:20
- Priority - Critical
- Affecting System - Network
-
We are aware of an issue with some connections to the data center and their network team is working on it at this time.
This issue has been resolved at this time.
- Date - 05/20/2019 01:49 - 05/20/2019 02:05
- Last Updated - 05/20/2019 02:17
- Priority - Critical
- Affecting Other - Datacenter network
-
We are currently aware of an outage of the network at the Dallas datacenter that we use. The datacenter networks admins are actively working on this.
This issue has been resolved.
- Date - 04/29/2019 00:49 - 05/20/2019 01:51
- Last Updated - 04/29/2019 01:42
- Priority - Critical
- Affecting Server - pk-win1
-
This server's disk requires expanding to provide it with more space. Downtime is expected to be 30 minutes to an hour due to the process requirements.
20:15 - updates are taking place currently
20:30 - updates are still applying at this time.
20:32 - server is shutting down at this time for the disk space increase process. This is expected to be about 30 minutes.
20:38 - current ETA on first part is 8 minutes
20:45 - first part has completed, second part has started and has an ETA of about 10 minutes
20:50 - server is booting at this time and will have further updates shortly
20:53 - services are operational at this time . Further items are needed before closing this out.
20:55 - This work is now completed. Work was done without error or unexpected events.
- Date - 12/16/2018 20:00 - 12/17/2018 00:00
- Last Updated - 12/16/2018 20:58
- Priority - High
- Affecting Server - pk2-dal1
-
During the maintenance window, pk2-dal1 will be rebooted to apply updates and other related items. Downtime expected to be less than 10 minutes.
9:15 PM Updates are taking place currently.
9:30 PM - server is rebooting at this time
9:33 PM - server rebooted and services are already back up . This work is finished. Downtime was less than 5 minutes.
- Date - 11/20/2018 21:00 - 11/20/2018 22:00
- Last Updated - 11/20/2018 21:34
- Priority - High
- Affecting System - DC Electrical
-
During the scheduled maintenance window, the datacenter will be performing maintenance on their power switching systems (ATS). This maintenance is to inspect and correct above normal thermal scans of the switching equipment. During this window, power will be switched to UPS (power backups) so that the power feed can be de-energized to safely work on the equipment. Their power is expected to be out for at least two different times during the window for 45 minutes. No service impact is expected from this work, however, there is a higher than normal risk of unexpected issues due to the nature of the work.
- Date - 10/16/2018 21:00 - 10/17/2018 06:00
- Last Updated - 11/20/2018 21:15
- Priority - High
- Affecting Server - pk2-dal1
-
During the scheduled window, this server will be rebooted to resolve a backup agent issue and apply some system updates.
9:25 - server is going down for reboot
9:30 - server is rebooted and services are coming back online. Downtime about 3 minutes. This finishes this work.
- Date - 10/10/2018 21:00 - 10/10/2018 22:00
- Last Updated - 10/11/2018 01:32
- Priority - High
- Affecting Server - pk-win1
-
During the scheduled window, pk-win1 will be receiving system updates and an increase in RAM. A reboot is required. Downtime is expeced to be 15 minutes.
21:00 this work is starting.
21:30 -- updates are still applying
21:47 - server is going down for RAM upgrade.
21:53 - server is applying update configuration options before shutting down.
21:57 - server is booting at this time and will need to apply second rund of update configs.
22:00 -- services are in operation. This work is complete. - Date - 09/12/2018 20:30 - 09/12/2018 22:30
- Last Updated - 09/12/2018 22:01
- Priority - High
- Affecting Server - Hsphere-Linux
-
During the schedule window, mysql3-dal1 will be updated to the latest version of MariaDB as well as any other system updates. A reboot is required for system updates. Downtime is expected to be about 10 to 15 minutes for mysql services.
20:18 - this work is starting.
A fresh backup is being taken with the mysql service turned off to get a proper backup.
First half of the mysql update is complee.
20:51 Server packages are being updated now. Mysql service is back online. Process took bit longer than expected
20:57 - server is rebooting.
20:59 - service is back in operation . This work is complete. - Date - 09/12/2018 20:00 - 09/12/2018 21:30
- Last Updated - 09/12/2018 21:00
- Priority - High
- Affecting Server - Hsphere-Linux
-
We are aware of an outage at the datacenter that is afecting multiple services. This is being investigated and this notice will be updated as more info is available.
Update: this issue is affecting the entire datacenter and not just our systems.
Unfortunately this is still ongoing an investigation on the datacenter's side.
For those using spamexperts, incoming mail should collect on the spamexperts instance we have in another datacenter and then delivered once the network is restored. For those not using spamexperts, mail should queue at the sending server until their next retry attempt and that timing depends on their settigs.
Network access to services is coming back up, however, awaiting a final confirmation from the datacenter that this is resolved.
Services have been stable for the last hour and half or so. This is resolved at this point. - Date - 06/15/2018 17:59 - 06/15/2018 19:30
- Last Updated - 06/15/2018 21:00
- Priority - High
- Affecting Other - Billing
-
Due to an issue with the PayPal billing module in use in our billing system, the paypal payments options are currently not available. This notice will be updated once this has been resolved.
Update: 19:15 ET - This is now resolved. - Date - 06/13/2018 16:23 - 06/13/2018 19:16
- Last Updated - 06/13/2018 19:16
- Priority - High
- Affecting System - Billing, client area, support
-
During the scheduled window, we will be moving the client area (https://u2-web.com/clientarea/ ) to a new sub domain and separating our site from the client area / support pages. During this time, updates to the client area system will also be applied. Those updates include a new design for the client area. The updates are also needed for upcoming services and other scheduled improvements.
During this switch over, the client area will be in maintenace mode. As such, if you require support, please use email or phone to contact us.
UPDATE: 3:15 PM -- this work has started.
UPDATE: 20:40 PM -- This work has been finished without issue or interruption to support. The client area is now at https://clients.u2-web.com/ . If you notice any issues, please let us know. - Date - 02/14/2018 15:00 - 02/14/2018 22:00
- Last Updated - 06/13/2018 16:27
- Priority - High
- Affecting System - mysql3-dal1
-
During this update window, mysql3-dal1 will hae the following updates done:
- System updates
- MariaDB update from 5.5 to MariaDB 10 .
- This effectively gives MySQL 5.6 functionality .
- Most scripts used should not be affected
Downtime: 20 to 30 minutes for database services
Update: 20:30 this work is starting.
Update: 20:45: MariaDB (Mysql) is updated and back in service. Downtime was about 10 minutes. Several sites checked to confirm operations. . Sysmte updates are appying at this time. - Date - 02/28/2018 20:00 - 02/28/2018 22:00
- Last Updated - 03/05/2018 16:47
- Priority - High
- Affecting System - spamexperts
-
During the scheduled window, the spamexperts appliances will be rebooted for updates to be applied. They will be rebooted one at a ttime until all finished and no impact should happen to mail delivery as they are setup in a failover setup. If you have any questions regarding this, please let us know.
Work has been completed. - Date - 08/08/2017 19:30 - 08/08/2017 21:30
- Last Updated - 08/08/2017 18:48
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
Due to a network issue on mail1, services are currently disrupted. The issue is being investiaged at this time.
4:51 PM ET - issue has been resolved. - Date - 03/02/2017 15:48
- Last Updated - 03/02/2017 15:52
- Priority - Critical
- Affecting Server - Hsphere-Windows
-
We are aware of service disruption with mail7-net, used by windows only accounts. Further updates will be posted once more info is available.
7:50 PM ET - server is under a file system check, which unfortunately takes a while as mail servers hae a lot of small files and all have to be checked.
10:30 PM ET - unfortunately the file system check is still going.
2:10 AM ET - services are coming up at this point, however, final checks are continuing.
2:30 AMT ET - there still some work on this server at this time.
3 AM ET - services have been functional for the last 20 minutes or so and this outage should be resolved at this point.
4 AM ET - unfortunately the server is experiencing addtional outages at this point. Techs are working on this as quick as possible.
4:30 AM ET - hardware faults are being diagnosed at this point
6:00 AM ET - hardware faults have been resolved, however, some system software faults are now coming up preventing a clean bootup process and that is being worked on currently
7:30 AM ET - server is back in operation at this point. We are sorry for the extended outage. - Date - 02/23/2017 16:51
- Last Updated - 02/24/2017 06:50
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
As previosuly announced, we will be removing PHP 5.2 from our shared linux servers and the default PHP version for the near future will be PHP 5.3 with that planned to be removed later this year or next as well.
This should not affect most sites, however, if you are using a database and after the removal you receive a database connection error, please make sure to update the database user's password using the long hash option. Please see https://u2-web.com/knowledgebase/146/Update-MySQL-User-Password-To-Use-Long-Hash.html for more information.
Update: This has been pushed to 10/3/2016 in order to prevent issues over the weekend.
10/3/2016 - 8 PM ET: php 5.2 has been removed from web2-dal1 web server. Auto checks for sites is in progrses at this time. After that is done, web1-dal1 will be worked on. To see which web server you are on, check the control panel -> FTP user / account -> web server.
10/3/2016- 8:30 PM: web2-dal1 Auto checks of sites returned a few sites with database connection errors and those were fixed. The checks only checked for keywords on the base of the site and not the whole site. Again, if you see database errors on your sites, blogs, etc please update the mysql's password hash.
Moving on to web1-dal1 and will update here once done.
10/3/2016 9:35 PM ET - PHP 5.2 has been removed from all shared linux web servers at this point. Default PHP version is now 5.3 for the near future. Sites have had a rough auto check for usual database connection errors and those found have been fixed. However, this does not include sub addresses such as /blog or scripts that do not provide an error. Again, if you are seeing database connection errors, please update the mysql password hash as described in https://u2-web.com/knowledgebase/146/Update-MySQL-User-Password-To-Use-Long-Hash.html
If you have any questions or issues, please contact support -- https://u2-web.com/contact/
This notice will be left open for at least a few days.
- Date - 10/01/2016 13:00 - 10/01/2016 15:00
- Last Updated - 02/24/2017 05:45
- Priority - High
- Affecting Server - Hsphere-Linux
-
During the scheduled window, services for mail1 will be moved to newly deployed hardware node and running new version of the operating system. This move is required to keep servcices updated and a in place upgrade is not possible.
Due to the volume of mail, mail files will be migrated eariy on the day of the window and then a resync afterwards. Due to this, you may see mail that was deleted / moved during this period come back.
Actual downtime is expected to be about 30 minutes or less, however, we have reserved an extended period in case of any issues.
Further updates will be provided as work begins.
4:10 PM ET - Prep work, including the mentioned message file copying, is starting at this time.
7:40 PM ET - mail message file copying is still underway is taking longer than expected. If the copying is not done by 10 PM ET, this work will be pushed to tomorrow night.
10:05 PM ET - message file copy is still going at this time . It will likely not completel until late tonight and as such actual switching will not be done until tomorrow night. No services are affected at this point.
7:25 PM ET 2/17/2017 - This work is being pushed until Saturday (tomorrow) due to other projects conflicting with the timing.
7:00 PM ET 2/18/2017 - The switch of servers is happening at this time. Again, downtime is expected to be less than 30 minutes but could extend further if issues come up.
7:30 PM ET - unfortunately some issues have come up for this work and it is being worked on. If not resolved soon, backpedaling will be done.
7:45 PM ET - due to issues in the final switch over, this work has been backpedaled to the previous server and services are up at this time. We will schedule another attempt later this week.
2/22/2017 - Will be attempting the switch to the new server tomorrow night. Issues during the last migration have been sorted and it should be a smooth switch though again extended window for the move is scheduled.
2/23/2017 - 7:10 PM ET - this migration is starting. Further updates will be done shortly.
7:40 PM ET - services have been up for about 20 minutes now without issue. Tests show mail is being delivered properly. Clean up work is under way at his point.
8:20 PM ET - mail1 is going down for a quick reboot.
8:26 PM ET _ mail1 is back in operation .
9:55 PM ET - Finaly mail message sync is under way at this time. After that, this work is completed. If you notice any issues, please let us know via support tickets.
10:10 PM ET - mail sync is completed. Old server has been shutdown. This work is now completed. - Date - 02/23/2017 19:00 - 02/23/2017 22:59
- Last Updated - 02/23/2017 21:08
- Priority - High
- Affecting Server - Hsphere-Linux
-
During the scheduled window we will be migrating mail2 to a new operating system install in order to keep it updated . A in place update to the new system is not possible due to the operating system version differences. During this process mail will be sync'd from the old / current system to the new one. Once the switch is done a furthe sync will be done to grab any mail that came in during the first sync. As such, you may see mail that you deleted on 10/18/2016 come back once the switch has been completed. Unfortunately there is no around that.
There is no action needed on customers part. Accounts can have a mix of the mail servers we have due to the clustering nature.
This process may require an hour downtime for a final sync. If that is required, any incoming mail will be queued at the sending mail server and attempted again per the sending mail server's settings (which can vary).
Update: 10/18/2016 - 11 AM ET - Preparations, including initial sync of mail files, has started for this work. As it can take several hours for the transfer of all the mail files, it is best to start now so that the switch can happen as scheduled. No downtime should happen during this time.
Update: 10/18/2016 11 PM ET -- Due to issues on final prep work before switch, this work is being postpoined until tomorrow night.
Update: 10/19/2016 8:30 PM ET - We are on track for the scheduled switch in 30 minutes.
Update: 10/19/2016 9 PM ET - mail2 is being taken down at this time for the switch to the new server.
Update: 10/19/2016 21:25 PM ET -- swithch as been completed. Downtime was around 5 minutes . A final sync of mail is in progress.
Update: 10/29/2016 21:40 PM ET - sync is done and this work is now complete. Tests show mail is being delivered as normal and mail is being sent out as expected. - Date - 10/19/2016 21:00 - 10/19/2016 23:59
- Last Updated - 10/19/2016 20:42
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
During the scheduled window, updates will be applied to the shared linux servers. A reboot of the server may be needed. Downtime is expected to be about 10 to 20 minutes per server.
Updates have been applied. A reboot was not eeded at this time. There was no downtime. - Date - 10/12/2016 21:00 - 10/13/2016 00:00
- Last Updated - 10/12/2016 20:48
- Priority - High
- Affecting Server - Hsphere-Linux
-
During the scheduled window, updates for Windows and other software will be installed. A reboot or two is expected to be needed due to the updates.
- Update Windows packages
- Update control panel packages
- Update Anti-virus software
Downtime is expected to be 10 to 20 minutes.
9:00 PM ET: Updates are being applied at this time.
9:37 PM ET: server is rebooting now.
9:44 PM ET: Server is booting up
9:46 PM ET: Update configurations are being applied. Web services are coming back online.
9:58 PM ET: This work has been completed. Downtime was about 10 minutes. - Date - 09/28/2016 21:00 - 09/28/2016 00:00
- Last Updated - 09/28/2016 20:58
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
The hsphere control panel is currently offline due to a database issue with it. It is currently running through a "garabage" collection process that requires the database to be offline. This process usually runs automatically, however, it appears to have not done so for some time. This process has been stalling all day and we have had to restart it multiple times. The process is required as to prevent data loss. Unfortunately, the process does not provide any progress report and so we can not provide an ETA on this.
Account services are not affected, beyond not being able to manage their settings.
We are looking at ways to either bypass this or speed it up.
We are sorry for the inconvenience this may cause.
9/21/2016 - 02:00 - we are able to bring the database online and the control panel is responding at this time. However, an online vacuum has been started to clean up the database further so that this issue does not happen again. As such, the performance of the control panel may be affected due to this process.
Again, we apologize for the control panel issues. Investigations on why this happened in the first place is underway in the background as well as other plans for the control panel in general. - Date - 09/20/2016 09:42
- Last Updated - 09/21/2016 00:53
- Priority - High
- Affecting Server - Hsphere-Linux
-
Updates will be applied during the scheduled window for PHP and any system updates for our shared web servers. A couple minutes downtime is expected while updates apply at the final stages.
- Date - 09/05/2016 20:00 - 09/05/2016 23:30
- Last Updated - 09/14/2016 16:58
- Priority - High
- Affecting Server - Hsphere-Windows
-
During the scheduled window, win1 will have updates applied to it and rebooted in order to apply those updates. This process may require up to 15 minutes downtime, however, could extend more depending on the updates.
22:15 -- updates are being applied.
22:25 -- Server is rebooting now
22:30 -- server is booting, applying update configs, etc.
22:35 -- server and services are in operation. Downtime was about 10 minutes. This concludes the work. - Date - 09/01/2016 22:00 - 09/01/2016 23:00
- Last Updated - 09/01/2016 21:37
- Priority - High
- Affecting Server - Hsphere-Linux
-
We will be doing updates on all shared linux web servers. Web services will need to be restarted and as such you may see errors connecting to sites one second and not the next A reboot of the server should not be needed.
Updates include:
- System updates
- Internal script updates
####################
10:20 PM ET: Updates are done. If you experience issues with Paypal TLS, auth.net TLS, add the file ".useApache24" to your domain's directory and your site will use a new set of web services. - Date - 06/15/2016 22:00 - 06/15/2016 23:59
- Last Updated - 06/15/2016 21:11
- Priority - Critical
- Affecting Server - Steven Test
-
We are aware of mail service issues on mail server "mail1". If you are having timeout or connection errors when checking mail, you are likely on this mail server.
14:17: server is currently going through a file system check, which may take a while to complete and unfortunately it is needed to protect against data corruption.
14:50: this issue has been resolved. Services are back up and running. Any mail sent to the server will be queued at the sending server and then retried at their next scheduled time. - Date - 02/24/2016 13:12 - 02/24/2016 13:51
- Last Updated - 02/24/2016 13:50
- Priority - High
- Affecting Other - Non-u2-web Related
-
For comcast internet subscribers only:
Comcast is currently experiencing outages across the country that is affecting routes to our systems as well as others around the internet. The behavior affects different services differently where one service works and one does not. The reason being that one address or even connection will take a different route. As such, if you are noticing any slowness in services and are using comcast, then most likely their outage is the reason.
Unfortunately this issue is related to comcast's systems and not our systems and as such we do not have any further information. - Date - 02/15/2016 12:22
- Last Updated - 02/16/2016 11:49
- Priority - High
- Affecting System - spamexperts
-
The base domain (which is a our new whilte label domain) that our spamexperts uses will be switched to a new set of nameservers. This switch should be seamless, however, we wanted to let you know of the change. Beyond your information, no action is needed.
22:30: this change has been done. - Date - 01/26/2016 20:09 - 01/27/2016 00:00
- Last Updated - 01/26/2016 21:30
- Priority - High
- Affecting Server - Hsphere-Linux
-
During the scheduled window, updates will be applied to the following servers:
web1-dal1
ns1
ns2
ns3
Work to include:
OS package updates
Control panel package updates
Configuration updates
Reboot of ns1, ns2, ns3 is expected. 5 to 20 minutes of downtime is expected for this work per server.
8/17/2015
10:00 PM this work has started with ns1, ns2, and ns3 in that order. Web1-dal1 will then be done last.
11:10 PM; ns1 is complete, ns2 is rebooting to apply updates. Ns3 will be next.
Web1-dal1 primary updates will be postponed until tomorrow night due to the nameserver updates taking longer then expected. Some web filter updates will be done, however.
12:05: ns3 had some issues coming backup, however, that is resolved at this point .
Again, we postponed work on web1-dal1 until tomororw except except for filter updates.
8/18/2015 - 9 PM -- Work on web1-dal1 is starting.
9:25 PM - this work is done. no reboot was needed as kernelcare was installed, which patches the kernel (the brains) and applies any needed update to make the effective kernel the latest.
- Date - 08/18/2015 21:00 - 08/18/2015 23:59
- Last Updated - 08/18/2015 20:33
- Priority - High
- Affecting Server - Hsphere-Windows
-
During the scheduled window, updates will be done on win1. This includes OS, configuration changes for PCI, and other such changes. A reboot will be needed to complete and as such 5 to 15 minutes is expected as downtime. If you have any questions, please let us know.
10:00 PM This work has started.
10:50 PM win1 is going down for reboot.
12:00 AM -- This work is done. Win1 was down a few minutes only. - Date - 08/17/2015 22:00 - 08/17/2015 23:59
- Last Updated - 08/17/2015 23:00
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
We are aware of issues with mail1, ns1, the control panel. Due to a hardware issue with the node that runs those services along with others, they may respond slowly or timeout. Techs are currently looking at the hardware and updates will be given here once more information is available. Unfortunately this will take time to evaluate.
9:45 AM - hardware is still under review from techs. We are preparing secondary plans.
10:30 AM - level 3 techs are working on the hardware currently and we should have further information shortly.
10:36 AM - disk / file system checks are going on with some of the servers on this node.
10:50 AM - the servers are trying to boot currently, however, there are some technical issues that are stopping them from booting cleanly. This is being worked on by our top techs.
11:15 AM - mail1 has been back in operational for last 10 minutes or os. Services are responding, however, due to mail clients all trying to send and receive mail as well as external mail servers trying to send in mail it may be slow for a bit.
12:00 PM - control panel server is going through a file system check currently. Mail1 is still having a large influx of traffic .
1:16 PM - Control panel server's file system check is still underway, however, we may need to restore part of the file system from last backups.
3:33 PM - control panel server' file system check completed, but we will need to restore files from backup as there was a lot of data corruption. That restore is progress at this point. This does not affect any other services.
6 PM - the control panel server has been up for sometime now, however, it is running slowly due to a self-repair / vaccuming that is taking place within the database .
6:30 PM: control panel server is back to normal. This concludes today's events. We apologize for the outage today.. As you may know, we do not like outages just as much as you. If you notice any problems, please let us know.
8/13/2015 - 9:45 PM - Due to continued load and IO issues, we have disabled STARTTLS on mail1 . StartTLS has been a problem before and continues to be. This was disabled yesterday and reenabled earlier today, however, the load and IO issues returned to mail1 . As such, unfortunately we can not continue to offer starttls . As such, please change your mail clients to use SSL/TLS with port 993 for IMAP and port 995 for POP3 and port 465 for SMTP / outgoing. If you need help changing these settings, please let us know. Using SSL/TLS instead of StartTLS better for connection security. - Date - 08/12/2015 07:17
- Last Updated - 08/13/2015 20:46
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
Investigations of repeated high loads and high input/output wait times on mail1 has shown that mail1 needs to have a file system check and repair done. This will require services to be stopped and/or the system to be rebooted in to a mode that allows a clean check and repair to be done. Unfortunately this will cause some downtime. This is required in order to prevent data corruption and to prevent further high load and IO waits, which effects performance of mail services. As such, this is scheduled as a critical / emergency maintenance window.
During this check, any mail sent in will be queued up at the sending mail servers and delivered at their next scheduled attempt.
If you have any questions, please let us know.
10:05 PM : This work is starting shortly.
10:45 PM -- the file system check is still running on the primary file system. Unfortunately, this process can take a long time, however, again it is needed to prevent further issues and resolve some issues that have come up lately.
11:15 PM - file system check is progressing.
11:45 PM -- Unfortunately the file system check is taking longer than usual. This indicates that sooner or later the server was going to crash and in this controlled setting we were able to take it down at night during low usages times instead of during the day during high usage. The checker does not provide a ETA and so any ETA we could provide at this point would be off . Updates will continue to be done here.
12:25 AM -- system check has been completed. Services are active at this time. Services might be slow for a bit while mail clients are checking in in mass. - Date - 08/06/2015 22:00 - 08/07/2015 12:52
- Last Updated - 08/06/2015 23:28
- Priority - High
- Affecting Server - Hsphere-Linux
-
We are aware of issues on mail1 with services giving either slow connections or timeouts.
10:35 AM: the server has gone down for a reboot and is currently under a file system check. Unfortunately that step takes a bit but is needed in order to ensure file system health.
10:51 AM: file system check is done and the server is continuing to boot at this time.
10:58 AM: server is returning to normal operations. We are watching it closer then usual . - Date - 05/30/2015 10:30
- Last Updated - 05/30/2015 10:59
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
During the scheduled window, we will be updating the following servers.
Update:
System packages
control panel packages
Recompiling of FTP for PCI reasons
Enabling pagespeed module by default, can be disabled per site by customers.
Misc updates
Downtime should be about 5 minutes, no reboot is needed at this time.
9:05 PM: this work is starting.
System packages
control panel packages
Recompiling of FTP for PCI reasons
Enabling pagespeed module by default, can be disabled per site by customers.
Misc updates
10:45 PM: This work is now done. Downtime was less then 5 minutes per server.
If you want to disable pagespeed, add a blank file with the name of ".disable-pagespeed" to the domain's web directory.
The above updates also resolved latest TLS issues of logjam. - Date - 05/28/2015 21:00 - 05/28/2015 23:00
- Last Updated - 05/28/2015 21:54
- Priority - Critical
- Affecting Server - Hsphere-Windows
-
Windows admins are currently looking in to an issue with win16 that is causing a service outage.
This has been resolved. Issue was cauesd by a DoS on the server. Outage was around 20 minutes. - Date - 05/10/2015 12:02 - 05/10/2015 12:48
- Last Updated - 05/10/2015 12:48
- Priority - High
- Affecting Server - Hsphere-Linux
-
We will be peforming updates on the folowing servesr during scheduled window.
- web1-dal1
- mail2
- Control panel packages
- System packages
- Adding KernelCare
- Web server specific: Updating nginx via a recompile in order to implement TLS 1.2
- During the nginx recompile, we will also add in pagespeed module
Downtime: 15 to 30 minutes to complete reboot per server
9:20 PM ET: This work is starting with mail2.
9:30 PM ET: Mail2 is going down for reboot.
9:32 PM ET: mail2 is back up and responding to service requests.
9:33 PM ET: Moving on to web1-dal1 updates.
9:43 PM ET: Web1-dal1 is going down for reboot .
9:50 PM ET: web1-dal1 is backup and services are responding. Moving on to recompiling nginx and other tasks.
10:20 PM ET: Updates are complete. Total downtime was around 10 minutes.
We're investigating time out errors produced by web1-dal1 . The issue is random of when it happens and there are no logs entries to help the investigation. As such this is taking a bit find the cause.
Server has not shown the timeout issue for a almost 3 hours now, however, as it has shown thie behavior before of working for a few hours and doing a timeout for 10 minutes we will continue to watch the server even more closely then usual.
This issue was resolved several hours ago. The issue was that a few customers PHP scripts were flooding PHP processes randomly and flooding error logs. This caused apache to not respond to requests due to the process forking. This issue was not seen before and took a while and sevearl staff members investigating it to find the issue. Unfortunately, due to the nature of apache, cloudlinux could not protect against this specific issue. To compound this, a new log monitoring system was also blocking the server IP, however, not giving any logs of that until extended logs were enabled.
We're sorry for the random timeout errors. - Date - 04/22/2015 21:00 - 04/22/2015 23:00
- Last Updated - 04/24/2015 02:09
- Priority - High
- Affecting Server - Hsphere-Linux
-
We will be peforming updates on the web server web2-dal1 during the set scheduled window. Updates will include:
- Control panel packages
- System packages
- Adding KernelCare
- Updating nginx via a recompile in order to implement TLS 1.2
- During the nginx recompile, we will also add in pagespeed module
Downtime: 15 to 30 minutes to complete reboot
10:10 PM : This work is under way.
10:20 PM: Server will be going down for a reboot shortly.
10:30 PM: Server is back online and packages are updated . Downtime was about 7 minutes. We'll move on to recompiling nginx.
11:30 PM: nginx has been recompiled, however, due to some additional items pagespeed is not yet implemented.
This work is done for now. - Date - 04/09/2015 22:00 - 04/09/2015 23:59
- Last Updated - 04/09/2015 22:44
- Priority - High
- Affecting Server - Hsphere-Linux
-
We will be performing updates on the mail1 server during the set window. Updates will include:
- Any control panel packages
- System packages
- Adding KernelCare
- Adding additional RAM to server
- Adding additional CPU resources to server
- Other misc changes for our side of things
Downtime: 5 to 15 minutes to complete needed reboots. During this time, mail will cache at the sending mail servers or within our anti-spam appliances if you're using them.
This work is under way.
Server is going down for a reboot now.
Server is back up at this point and services are responding normally. Downtime was less then 5 minutes.
This concludes this work. - Date - 04/07/2015 22:00 - 04/07/2015 23:59
- Last Updated - 04/07/2015 22:31
- Priority - Critical
- Affecting Other - Network
-
We are currently aware of a network routing issue going on at the dallas datacenter . it is being investigated by network admins. Some routes are working while others are not. You may see timeouts and the like when connecting to services.
Any further further info will be posted here .
UPDATE: it appears this is isolated to level3 routing.
UPDATE: Routes have been switched to another backbone and that should reoslve any timeouts . The issue is still being investigated by network admins at the datacenter. - Date - 02/25/2015 01:25 - 02/26/2015 03:46
- Last Updated - 02/25/2015 02:19
- Priority - High
- Affecting System - Mailstrainer
-
Maintenance on Sunday night (1/25/2015) will be done on the mailstrainer anti-spam system. This will include any updates needed and database optimizations. Expected downtime will be 20 minutes for each node in the mailstrainer system. Any mail that is sent to a node that is unreachable during that time will be queued at the sending mail server for redelivery to the available node.
- Date - 01/22/2015 21:14 - 04/07/2015 22:16
- Last Updated - 01/22/2015 21:18
- Priority - High
- Affecting Server - Hsphere-Linux
-
Over the next month, we will be migrating the following servers for our Linux customers to newly provisioned hardware:
web servers - web6 and web7-us mixed in among 2 new web servers -- COMPLETED
database servers - mysql3 mysql2-5 -- COMPLETED
mail servers - mail1 and mail2 - COMPLETED
nameservers: ns1, ns2, ns3 -- COMPLETED
We will also be moving Linux VPS servers and will be creating support tickets for each customer .
Customers using third party DNS servers will be contacted to make sure their IPs are updated.
All work done will be handled by us and the migration should be seamless for the most part and there is nothing on your side to do.
This migration starts other project offerings for us including a new anti-spam system to replace spamexperts as well as provide outgoing filtering to prevent spam from being sent from our servers and potentially causing them to be blacklisted.
10/6/2014: Mail1, ns2, ns3, and cp have been moved successfully.
10/6/2014 (night): ns1 has been migrated.
10/16/2014: accounts on web7-us have been migrated.
10/18/2014: mysql3 has been migrated.
11/13/2014: mysql2-5 is completed to new servers -- either mysql2-dal1 or mysql1-dal1 . mail2 is still in process of moving.
11/17/2014: web6 is completed.
11/18/2014: mail2 migration is completed.
11/20/2014: This migration has been completed. An email has been sent out about this completion.
We would like to highlight a few things that have come up during this migration.
-
FTP: If you are receiving errors when connecting to FTP via a ftp client, dreamweaver, web expressions, or other web designer:
-
Please make sure that the server address you're using is ftp.example.com where example.com is your domain name.
-
Make sure the server address is not "web6.u2-web.com" or "web7-us.u2-web.com".
-
-
Webmail: If you are receiving errors on connecting to the webmail interface:
-
Please make sure the address bar says "mail1.u2-web.com" or "mail2.u2-web.com" and not 96.125.178.57 or 96.125.178.55 . If the address bar says other numbers, then they should be fine.
-
If you have bookmarks, please make sure to update them to use "mail.example.com" where example.com is your domain name . Some bookmarks may use the IPs (a bunch of numbers) instead of the address and those numbers will not be updated automatically.
-
-
MySQL (databases): Most scripts were updated to the new database server names. If you are seeing connection errors in your scripts, please check the configuration for your script and update them to what is shown on the mysql page in the control panel.
-
SSLv3 - PODDLE: During this migration, the PODDLE SSLv3 vulnerability came out. The new servers were configured with SSLv3 disabled and as such this was not a concern. SSLv3 on the web browser side should not cause any problems with browsing of your SSL protected sites. Some very old browsers that go back to the early 2000s, may have problems. However, those browsers simply should not be used anyway. Most websites and services are or have turned off SSLv3 on their servers. If you are not using SSL on your site, this is not a concern for you.
-
Message from PayPal / Merchant Providers: If you received a message from Paypal or another merchant gateway about SSLv3, this relates to their side of the connection and how your scripts connect to them. Unless your script is specifically designed for SSL and not TLS usage, you should not have any problems. The web servers support TLS. You'll want to talk with your script developer to find out if there is a concern.
-
-
Dedicated IP usages: due to ICANN / ARIN (the agencies that regulate IPs) policies, dedicated IPs require to have a valid SSL or dedicated service. It has always been our policy to require an SSL in use with dedicated IPs, however, we have not enforced this unless it was an abuse of the IPs. Due to the shortages of IPv4 IPs ICANN / ARIN / our datacenters require that IPs be used per their policies and they are doing audits of IP usages. If they find our IP usages in violation, new IPs will not be provisioned to us. Some hosting providers have had their IPs taken away because of this.
-
With the above in mind, we will be conducting audits of IP usages to ensure that:
-
Each site using a dedicated IP is using a valid SSL.
-
That the site is actually hosted with us
-
-
Any site found not follow the dedicated IP requirements, will receive a support ticket (email) from us stating that we will be switching the IP to shared if an SSL is not used.
-
We will be offering SSLs to those that are required to switch at a discounted price
-
If you are unsure if about your IP usage type, you can check via the control panel (https://cp.u2-web.com/ ) ? Domain settings ? domain info ? the domain and it will say either “Dedicated IP” or shared IP and ask if you want to switch between them.
-
As always, if you have any questions on the above, please let us.
-
- Date - 09/22/2014 22:00 - 11/20/2014 21:37
- Last Updated - 11/20/2014 21:37
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
Tomorrow we will be moving the following services to new servers. Some downtime is expected, however, it should be short.
cp.u2-web.com (control panel linux) - done
mail1 - done
mail2 - done
Once the mail servers are moved, forwarding will be done to the new server locations.
.
- Date - 10/04/2014 11:00 - 11/20/2014 21:34
- Last Updated - 11/20/2014 21:34
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
We will be migrating mail2 this weekend to its new home. No downtime is expected. No changes in mail clients are needed as long as you're using mail2.u2-web.com or mail.example.com where example.com is your domain. Those using webmail should use mail2.u2-web.com and not the IP address .
If you have any questions on this, please let us know.
Due to scheduling conflicts, this migration was postponed. We'll update this further.
11/17/2014: This work will be done today.
Mail2 migration is in the final stages where DNS and other records need to be updated and propagated throughout the internet.
11/19/2014: Mail2 migration is completed. - Date - 10/11/2014 10:00 - 11/20/2014 21:34
- Last Updated - 11/20/2014 21:34
- Priority - High
- Affecting Server - Hsphere-Linux
-
Starting tomorrow and continuing until done, we will be migrating web service and mysql services for the following servers:
web6 -- COMPLETED
web7-us -- COMPLETED
mysql2-5 -- COMPLETED
mysql3 -- COMPLTED
MySQL configuration for scripts will be updated during the migration process.
Customers using external DNS services will be contacted to ensure they update DNS records. Customers using our nameservers do not need to take action on DNS records.
If you have questions regarding this or would like to start your move process before we do, please let us know.
All accounts on web7-us have been migrated and are either in a finished state or will be completed tonight.
We will be focusing on our attention on accounts that are using mysql3 and web6.
These moves will be done in alphabetical (a-z) order.
mysql3 has been migrated.
11/13/2014: mysql2-5 has been migrated to either mysql1-dal1 or mysql2-dal1 . If you notice errors in connection, please check the MySQL page under the control panel to confirm which mysql server you are now using.
11/13/2014: web6 is 95% done at this stage. The final accounts are in the last stage of moving and a few accounts require a bit more time for clean up actions.
11/17/2014: web6 is 100% done at this this stage. All accounts on this server are either on web1-dal1 or web2-dal1. This closes out this maintenance. - Date - 10/14/2014 17:23 - 11/17/2014 15:56
- Last Updated - 11/17/2014 15:56
- Priority - High
- Affecting Server - Hsphere-Linux
-
During the scheduled window, maintenance on mail1 will be performed that will require a reboot and possible downtime of 30 minutes, however, any mail that is attempted to send to this server should queue at the sending server. This work will include resizing the server's file storage and applying updates for server packages as well as control panel packages.
This work will start shortly.
9:10 PM This work is starting with updates to packages.
9:55 PM Packages have been updated. Moving on to the file system size increase.
10:15 PM it appears the server is not liking the updates and refuses to start . We are working on this issue.
This is still being worked on. The server is booting and mail services are responding, however, they are responding slowly.
Services have been up for while now with some interruptions. This issue come about due to packages that came over from the migrations we did, which conflicted with some low level system packages. Those have been removed properly and the system is back to normal. We are sorry for the extended downtime. - Date - 11/03/2014 21:00 - 11/03/2014 23:46
- Last Updated - 11/03/2014 23:46
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
We will be migrating ns2 to its new home during the scheduled time. Again, this move will update nameserver IPs and only resellers / VPS customers need to worry about changings. A few minutes downtime is expected, however, the other two nameservers will handle that.
Further information will follow below.
10:00 PM: We're about half way through on this work. No switch of services yet.
10:45 PM: Switching is in progress at this time.
10:55 PM: This switch is complete. Resellers, please update your ns2 IP to 65.99.251.229
10/28/2014: the old nameserver instance has been set to deny requests. Multiple notices have been sent to resellers / VPS customers to ensure updating of the Ip. - Date - 10/01/2014 21:30 - 11/02/2014 20:26
- Last Updated - 10/28/2014 14:54
- Priority - High
- Affecting Server - Hsphere-Linux
-
We will be migrating ns1 tonight. While ns1 is being migrated, ns2 and ns3 will functional and responding to any requests. An update will be done once this migration is complete.
9:05 PM ET: This work is currently in progress .
9:40 PM ET: This work has been completed. Resellers please update your IPs from 67.22.136.4 to 174.136.46.254 . We have updated nameservers where the domain is registered with us.
10/28/2014: the old nameserver instance has been set to deny requests. Multiple notices have been sent to resellers / VPS customers to ensure updating of the Ip. - Date - 10/06/2014 21:00 - 11/02/2014 20:26
- Last Updated - 10/28/2014 14:53
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
We will be doing a migration on ns3 as well as our site and support (ticket) system tomorrow night (9/26/2014). This will result in ns3's IP being changed. As such, resellers will need to update the IP for ns3 to the one that will be provided after the migration is done. The old IP will work for a couple weeks.
During this time, we will also migrate our site and internal items. This may cause support / client support systems to be offline for a short amount of time.
This work is currently progressing.
This work is complete. Resellers please update IP at registrar for ns3 to 74.116.40.89 . - Date - 09/26/2014 20:00 - 11/02/2014 20:26
- Last Updated - 09/27/2014 00:18
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
Hello,
We are currently aware of a problem on web6 that is causing some service outages . It is being worked on and should be resolved shortly.
UPDATE: 3 AM ET / 2 AM CT: this has been resolved for about 20 minutes. The server locked up unexpectedly and suddenly. This required a reboot of the server after trying troubleshooting steps first. The server is operation at this point and again has been for a while. - Date - 09/05/2014 01:35 - 09/05/2014 02:13
- Last Updated - 09/05/2014 02:13
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
Hello,
Mysql2-5 and mail2 services will be migrated during the alloted timeframe to net new hardware. The new hardware has faster RAM, drives, and CPU and is rated as an upgrade of hardware. This migration is required as the drives for this server is running low and if we do not migrate the services or take other action service reliability will be a concern. This migration is different from a previous migration attempt done last year and will be easier to back out of if issues do come up. Downtime is expected to be around 30 minutes for mysql and mail services. Data will be migrated to the new hardware over network and then IPs will be switched.
If you have any questions on this work, please let us know.
Update: This work is ongoing at this time. Further updates will follow.
Level 3 techs have advised that downtime might be a bit more due to sync and IP switches then previously mentioned.
This work is progress smoothly thus far.
Services have been operational for a bit now. Final tasks are underway.
This work has been completed. Please let us know if you notice any issues. Total downtime was around 38 minutes. - Date - 12/30/2013 22:00 - 12/30/2013 23:16
- Last Updated - 12/30/2013 23:18
- Priority - High
- Affecting Server - Hsphere-Linux
-
Between 11:00 PM and 11:30 PM tonight we'll be upgrading web7-us resources. This will take about 5 to 10 minutes downtime to complete.
- Date - 10/27/2013 22:00 - 10/27/2013 22:05
- Last Updated - 10/27/2013 21:48
- Priority - High
- Affecting Server - Hsphere-Windows
-
Win11 requires some maintenance on the storage and backup systems for the server and as a result requirs a reboot. This reboot will be done during the scheduled window.
This maintenance was completed as scheduled.
8/29/2013 2:30 am: Additional maintenance is required after this work. Updates will be posted here shortly.
8/29/2013 3:05 am: on site techs are working on the server at this time. The server is undergoing a chkdisk right now and unfortunately that takes time to complete to ensure disk operations. Additional updates will be given as more information is available.
4 AM: chkdisk is still running.
9 AM: Chkdisk is still running unfortunately. On windows the chkdisk process can take a while because of the nature of the file system.
12:03 PM chkdisk is still running the file repair / security ID repair functions, which requires a long process because of the number of files on a web server for content. We are sorry for the extended downtime this has caused.
4:23 PM: I am sorry for the delay of updates, though there really hasn't been new information to provide as the chkdsk is still running.
7 PM: The server has been up for about an hour now and final checks / fixes are being put in to place.
9:27 PM: admins are still going through sites at this time resolving any remaint issues found.
10:30 PM: There are still some clean up items, however, affected sites are opertional at this time. If you notice any issues, please let us know.
9/3/2013 8:05 PM: Due to this issue and to use newer tech, accounts on this server will be migrated to a server that was just provisioned . As such, the windows related issues with this server should not recur. Accounts affected by this move have been notified. - Date - 08/28/2013 23:00 - 09/06/2013 14:27
- Last Updated - 09/03/2013 20:22
- Priority - High
- Affecting Server - Hsphere-Linux
-
During the scheduled window, mysql2-5 and mail2 will require maintenance on the server hardware that will result in around 20 minutes of downtime to accomdate a shutdown, work, and start up. While mail2 is out, mail sent to the server will queue up at the sending server.
- Date - 08/16/2013 22:00 - 08/21/2013 14:30
- Last Updated - 08/13/2013 13:56
- Priority - High
- Affecting Server - Hsphere-Windows
-
During the specified window, win11 will have maintenance on it which will require a restart of the server. The expected downtime is less then 15 minutes.
Update: this work has been completed without further issue. - Date - 07/24/2013 23:00 - 07/25/2013 02:40
- Last Updated - 07/25/2013 02:39
- Priority - High
- Affecting System - Network
-
During the scheduled window, network engineers will be migrating upstream network connections to larger connections. Due to network redudancy, no impact should happen, however, less them opitium routing and/or packet loss may occur while routes are updated.
- Date - 05/30/2013 22:00 - 06/04/2013 17:27
- Last Updated - 05/29/2013 14:12
- Priority - High
- Affecting Server - Hsphere-Linux
-
We will be updating nginx, the frontline web server, on our linux servers to night to close off a zero day exploit that has been reported on security channels. A couple minutes of downtime for web service may happen while the update is done.
Update: This update has been completed. - Date - 05/08/2013 00:00 - 05/08/2013 00:07
- Last Updated - 05/08/2013 00:07
- Priority - High
- Affecting System - Backups
-
Hello,
We have been working behind the scenes to switch backup systems and now it is at the stage of switching over main systems to it. As such, if you try to restore files via the r1soft data restore, it may give an error. If that does happen and you need to restore files, please contact us as we will need to restore them for you. Once the migration is done, we can enable restoring via the control panel, however, we can't do that yet because it would bring in conflicts. If you do need to contact us to restore the files, please let us know:- Which files
- What date you would like restored from
The switch is for all customers currently. It does not affect your hosting services, just restoring abilities for the time being. There is nothing for you to do and we will let you know if that changes.
This process will take some time to complete and there is no ETA of completion due to the complexities involved.
Update: This change has been completed. - Date - 01/04/2013 22:47 - 04/18/2013 15:25
- Last Updated - 04/18/2013 15:24
- Priority - High
- Affecting Server - Hsphere-Linux
-
During the scheduled window, our linux servers will have updates applied to them. This may require 10 to 15 minutes of downtime to reboot each server being updated.
Updates were done without issue.
- Date - 04/13/2013 17:00 - 04/16/2013 03:00
- Last Updated - 04/16/2013 11:12
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
MSSQL5's MSSQL service will be updated to 2012 SP1 to address some issues in the current version. MSSQL service will be functional for most of the window and will go out for a service restart.
Completed without issue.
- Date - 03/27/2013 06:00 - 03/27/2013 07:30
- Last Updated - 03/27/2013 14:11
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
During the scheduled window, mysql2-5 and mail2 will be taken offline to start and complete hardware upgrades on the server. It is not expected to use the whole scheduled time and overall downtime should be under an hour.
Update: 2/1/2013 - 11:10 PM ET: this work will start shortly.
Update: 2/1/2013 - 11:34 PM ET: this work is currently in progress.
Update: 2/2/2013 - 12:20 PM ET: there are some minor config issues preventing this work from completing quickly and that is being worked on as quickly as possible.
Update: 2/2/2013 - 12:45 PM ET: this work has been completed, however, unfortunately due to the unexpected hardware config issue, this upgrade could not be completed at this time. We will schedule another window for this to be completed and post it here.
Update: 3/20/2013: the config issues mentioned have been resolved and tested on other chassis as working. As such, this work will be done this Friday (3/22/2013) at 11 PM ET. Onsite techs that will be doing this work have requested 4 hours for this work, though expect 20 to 30 minutes actual downtime.
Update: 3/22/2013 - 11:21 PM ET: this work has been in progress for a few minutes.
Update: 11:33 PM ET: work has been completed and server is performing a fsck during the boot process . That should be completed shortly.
Update: 12:19 AM ET: unfortunately the fsck is taking longer then expected and is running at this time. The fsck (file system check) is needed to confirm that data is intact on the drives of the server.
Update: 3:38 AM ET: due to issues with the drives of the server, the fsck took a while and that was aborted due to it taking longer then normal. On site techs are working on this quickly as possible.
Update: 4:26 AM ET: a new chassis has been provisioned and OS is installing. Unfortunately, we'll have to do a restore from backups of data on this server due to corruption while mounting the drives.
Update: 5:25 AM ET: the OS has installed and the restore has started a bit ago.
Update: 6:12 AM ET: the restore is progressing as fast as it can and at this point we are a cycling pattern of waiting.
Update: 7:46 AM ET: the restore is still going strong, though it will take a while due to the amount of data on this server.
Update: 9:35 AM ET: the restore is showing completed . A few things have to be done first to clean up from the restore and then a few tests have to be done. We appreciate your patience with this and apologize for the unexpected downtime even that happened because of the hardware work.
Update: 10:53 AM ET: on site techs are wrapping up hardware items to finish off the recovery of this maintenance and unforeseeable downtime.
Update: unfortunately the site techs ran in to driver conflicts at the last minute and they have been trying to resolve that.
update: 1:21 PM ET: Unfortunately the on site techs will not be able to resolve the conflicts any time so and as such it has been elected to drop back to original hardware via a backup restore since the drives would still be corrupt. This process will take sometime to complete.
5:07 PM ET: I'm sorry for the delay of this update as there was not much to report. The original hardware is operational and the restore is progressing. There was some issues with the backup agent running, however, that appears to have been resolved. Hopefully Murphy from Murphy's Law will leave us now.
Update: 7;22 pm et: the restore is complete, however, a couple of items need to be addressed post restore.
8:00 PM ET : The server has booted up and we're bringing services online. MySQL service is fighting come to back up
update: mail services are up now on mail2 that shares this node.
update: mysql service running, however, we do are doing a repair on databases as we see some with errors.
update: you may see mysql connection issues here and here while repairs and other items are ran .
update: 11:51 PM ET: mysql service has been going for a few hours now and is stabilizing its tables with auto repairs. innodb tables may show errors and that is being looked at by level 3 admins. Mail services were hit by all mail users trying to send mail and as such that slowed down mail delivery for incoming and outgoing. That has for the most part stabilized. We are sorry for this issue and have learned some unfortunately lessons because of it.
We'll post in a summary of what happened, why the maintenance was needed, and what we are going to do to prevent such issues for mysql in the future once we have all information together.
Update: 3/25/2013 2:06 PM ET: we have reports of some innodb errors on this mysql server and as such we will be working on that later tonight as it requires services restarts. During that time you may see mysql connection errors.
- Date - 03/22/2013 22:00 - 04/24/2013 15:01
- Last Updated - 03/25/2013 13:08
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
At the scheduled time, we will be doing updates on web6 which may require a restart of the web service. Though no downtime should happen, if it does it is expected to be less then 10 minutes.
Please let us know if you have any questions on this.
This has been rescheduled for tonight.
Update: 2/26/2013 11:58 PM: This is being wrapped up, but it is running a bit late and should be finished in 20 minutes.
Update: 2/27/2013 12:13 AM: This work has been completed. Please let us know if you notice any issues by contacting us.
- Date - 02/26/2013 23:00 - 02/26/2013 23:59
- Last Updated - 02/26/2013 23:15
- Priority - Critical
- Affecting Server - Hsphere-Windows
-
Hello,
Mail3-net (used by windows accounts) will be going under a migration starting in 24 hours or so to other servers in that cluster. This migration is to reduce the legacy hardware in use. Those using our nameservers will have their records updated automatically, however, customers using external DNS will need to update to the new addresses.
UPDATE: 1/17/2013: This migration is complete.
- Date - 12/02/2012 15:00 - 01/17/2013 00:00
- Last Updated - 01/17/2013 15:59
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
Admins are aware of intermittent connectivity problems with win4 and are working on it. This post will be updated as needed to provide further information/resolutions.
Update: this issue was resolved a bit ago.
- Date - 01/06/2013 00:11 - 01/06/2013 04:35
- Last Updated - 01/06/2013 04:35
- Priority - Medium
- Affecting Other - Phones
-
Hello,
Happy holidays and new years. During Christmas and new years, we are going to close the phone queues for support and sales/billing to allow for a relaxed schedule as is tradition here. Tickets and alerts will be answered normally.
Christmas Phone Closed Times: December 24 5 PM - Dec. 26 at noon.
New Years Phone Closed Times: December 31 5 PM - Jan 2 at noon. - Date - 12/24/2012 16:00 - 01/31/2013 16:03
- Last Updated - 12/26/2012 19:19
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
To continue with service resource optimizations, we will be moving mail2 to another server hardware tomorrow night (1/4/2012) after 9 PM ET. Some downtime may happen while the IPs are switched between servers. No changes are needed by customers.
Update: January 06, 2012, 12:06:13 AM » This work has been completed with no issues, however, please do let us know if you have any problems. - Date - 01/04/2012 21:00
- Last Updated - 12/24/2012 23:18
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
CP/web1 (which holds just our sites) and MySQL1 will have maintenance this weekend to have A+B power placed on them. Downtime will be 10 to 15 minutes per server.
January 7, 2012 between 23:00 ET and midnight.
Update: January 07, 2012, 10:26:39 PM » This work has been completed. - Date - 01/07/2012 23:00
- Last Updated - 12/24/2012 23:17
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
We are currently working with a customer on web6 that is causing some resource issues there. We will update this thread once that is resolved.
Update: February 01, 2012, 03:04:39 PM » This has been resolved for the time being. - Date - 02/01/2012 13:40
- Last Updated - 12/24/2012 23:15
- Priority - Critical
- Affecting Server - Steven Test
-
The following servers will be rebooted in the next couple of hours. A post was made earlier about this, however, it appears it was deleted by mistake.
mssql1
mssql2
mssql3
mssql4
Other servers may be following tomorrow night. - Date - 02/03/2012 23:12
- Last Updated - 12/24/2012 23:13
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
Web6 and cp/web1's service packages will be updated this Monday (2/20/2012) at 11 PM ET. Impact is expected to be 10 minutes or less for web services.
Please let us know if you have any questions.
Update: February 20, 2012, 11:44:29 PM » This work has been completed. - Date - 02/20/2012 23:00
- Last Updated - 12/24/2012 23:12
- Priority - High
- Affecting System - Phones
-
Hello,
We are currently experiencing an issue with our routing of phones to the departments. Our PBX provider is checking on this, however, there currently isn't a ETA to resolution. If you require support, please contact us via a ticket -- https://u2-web.com/contact/ .
Update: February 26, 2012, 01:54:06 PM » This issue is still ongoing. Our PBX provider is still looking in to this with. We'll update as more information comes in.
Update: February 27, 2012, 11:53:33 AM » This issue was resolved earlier. - Date - 02/26/2012 10:40
- Last Updated - 12/24/2012 23:11
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
Win14 will have some updates placed on it that require a reboot of the server. This will take place 3/15/2012 at 4 AM ET . - Date - 03/15/2012 04:00
- Last Updated - 12/24/2012 23:08
- Priority - Medium
- Affecting Server - Hsphere-Linux
-
Hello,
To ensure encrypted connections to the control panel services, we have blocked access to port 80 and 8080 for our control panel. This should not affect regular customers, however, legacy resellers and VPS clients with a reseller type control panel have had their CP addresses secured with self-signed certificate generated for them and configured for them. As such, those customers may receive a untrusted SSL warning for their unique login addresses and that can be ignored for this one case.
If you notice any issues, please ensure you're using https connections to the control panel service. If you still have issues with accessing the CP, please let us know. - Date - 03/25/2012 23:07
- Last Updated - 12/24/2012 23:07
- Priority - High
- Affecting Server - Hsphere-Linux
-
We will be adding a new nameserver to our linux hsphere cluster tonight to provide geographic redundancy on the DNS services on our linux hsphere cluster. This work should not be service impacting, however, we wanted to let our customers know of ongoing work in case oddities are noticed.
In order for customers to take advantage of the new nameserver, you will need to add the 3rd nameserver at your registrar. If we are your registrar, you can change this at http:///domains.u2-web.com. All nameservers that domains should be using are listed in the account portal --> domain settings --> domain info page. If you do not want to take advantage of this nameserver, no changes are needed, however, DNS reports will show errors.
Resellers and VPS customers will need to register this nameserver like they did for the other nameservers.
This new nameserver only affects linux plans.
Update: April 05, 2012, 12:14:54 AM » Tonight's work is done and we will continue adding in this nameserver tomorrow night.
Update: April 07, 2012, 01:08:08 PM » This work has been completed. All customers can use the ns3 server at any time and it is recommended to do so as well as ensure that nameservers at your registrar are set to those listed in the control panel. - Date - 04/04/2012 23:05
- Last Updated - 12/24/2012 23:06
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
We have changed a good portion of our mail servers and shared servers to use a spamexperts appliance for outbound filtering. We are now going to deploy this for incoming mail filtering now. There should not be any downtime for this change over and this change over will happen over the next few weeks/months. Customers do not need to do any changes on their side.
Update: January 31, 2012, 03:00:56 PM » We will be moving linux accounts using mailfoundry to use spamexperts this week. Each account will receive a notice of turning off mailfoundry and on of spamexperts. You may need to adjust spamexperts settings to match your custom settings in mailfoundry. No downtime should happen and any customers using external DNS will be contacted to update their MX records.
As some using mailfoundry have noticed, its filtering has let through more and more spam. Changing to spamexperts should help with those using to fight spam. In addition, this will bring some inbound mail redundancy because spamexperts is setup on two different servers in different locations.
Update: April 12, 2012, 08:46:17 PM » In continuing with this change, we will be moving the rest of our mail servers to use spamexperts for outgoing mail filtering . This change will take place within the next few weeks and we will let you know at that time. - Date - 01/21/2012 23:02
- Last Updated - 12/24/2012 23:04
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
We are aware of an outage with the mssql2 server and it is being worked on it to restore service. We will let you know once this is resolved.
Update: April 14, 2012, 07:47:29 AM » This issue is still being worked on and should be resolved shortly.
Update: April 14, 2012, 08:23:51 AM » Unfortunately the current server will not start up cleanly and so a bare metal restore to an identical hardware server has to be done in order to resolve this issue. This server has been quite reliable for a number of years .
We'll update this thread as progress is made .
Update: April 14, 2012, 01:07:13 PM » Due to hardware issues, the BMR had to be restarted and that will be done shortly.
Update: April 14, 2012, 02:04:33 PM » The BMR was started a bit a go and is a good deal done already.
Update: April 14, 2012, 03:01:30 PM » The server has finished restoring and has been in service for a bit. We are sorry for the extended downtime.
- Date - 04/14/2012 06:09
- Last Updated - 12/24/2012 23:02
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
Monitoring has detected errors in the memory modules of mysql1 server and as such the RAM module will need to be replaced. This will be done tonight at midnight (12:00 am 4/16/2012) and will impact the server for less then 15 minutes.
This maintenance will only affect mysql services on mysql1. Web and mail services will not be affected, however, scripts connecting to mysql1 service will be affected which can cause web sites to show errors.
Update: April 16, 2012, 12:14:09 AM » Unfortunately the server would not accept the replacement RAM module. We will need to reschedule this maintenance and we will update this thread once that is set.
Update: April 18, 2012, 11:23:49 PM » The chassis this server is running is is quite old and as we still need to take this server down to ensure stability from the RAM errors, instead of a RAM swap, a hardware swap will be done to a different model server that is similar enough for the OS. This will increase the outage for the maintenance, however, doing so was planned to be done at some point soon and we might as well get it over with now instead of having multiples spread out or stability issues from failing RAM.
Window: Friday April 20th from 11 PM to 3 AM (Eastern times)
Impact: 15 to 60 minutes
Work: Hardware swap
Update: April 20, 2012, 11:42:27 PM » This work is in progress. We will let you know once this is complete.
UPdate: April 21, 2012, 12:06:52 AM » This work has been completed. Total downtime was about 22 minutes. - Date - 04/16/2012 00:00
- Last Updated - 12/24/2012 22:59
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
Hello,
We are aware of some sites having problems on web6 and working resolutions right now.
Update: April 23, 2012, 10:22:32 PM » This issue has been resolved. A security layer caused the web server to not load some configurations for some users . - Date - 04/23/2012 21:38
- Last Updated - 12/24/2012 22:56
- Priority - Medium
- Affecting System - internal systems
-
Hello,
We will be moving our internal, mail, and site services to a new server tomorrow night. This move should not impact customers in any way, however, we wanted to notify customers in case of any issues that do come up. Services that customers use will not be moved, touched, or otherwise affected. This move is to put our systems in a separate datacenter then our customers in order to maintain continuity of support services in case of a datacenter failure. This has been an on going process for a while now and will be mostly completed once this is done.
Update: May 02, 2012, 12:28:12 AM » Due to other projects, this was pushed back until to night. Mail has been moved to the new server for our domain. Other items will be moved over the course of this week. Again, this is none customer affecting but we wanted to let our customers know of our work. - Date - 04/29/2012 22:00
- Last Updated - 12/24/2012 22:55
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
We are investigating an outage with mail4-net at this time. This server is used primarily by windows accounts.
Update: May 13, 2012, 05:28:50 AM » Sorry for the delay of updating. This server is currently going through a bare metal restore in order to correct the outage. Updates to come.
Update: May 13, 2012, 07:15:19 AM » The restore is going strong and progressing.
Update: May 13, 2012, 09:51:32 AM » The restore is over 60% now. Unfortunately the progress does not give an ETA and it varies on time because of the number of files to restore, how big they are, etc.
Update: May 13, 2012, 12:44:17 PM » Mail4 has been in service for a while now. Any mail sent to it will be queued at the sending servers and resent according to RFCs. - Date - 05/13/2012 00:40
- Last Updated - 12/24/2012 22:52
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
Some updates to win4 will be completed tonight at 3 AM ET and will require a reboot, which will require about 5 minutes of downtime. - Date - 05/23/2012 03:00
- Last Updated - 12/24/2012 22:49
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
The network components of MSSQL4 will be upgraded during this maintenance window. This will require about 10 to 15 minutes of downtime.
Date: 6/13/2012
Time: 2 AM EST
- Date - 06/13/2012 02:00
- Last Updated - 12/24/2012 22:48
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
We are currently aware of a outage on win4 and it is worked on at this time to resolve the issue. We'll update this thread once resolved.
Update: June 29, 2012, 05:32:18 AM » This issue was resolved a bit ago. If you notice any issues, please let us know. - Date - 06/29/2012 05:26
- Last Updated - 12/24/2012 22:46
- Priority - High
- Affecting System - Phones
-
Hello,
We are currently in the process of transferring phone providers, which should be seamless and should not impact customers in contacting our team. However, we wanted to let our customers know that this was in process just in case something did happen. Email support will not be affected of course and you can always contact us there via email or at https://u2-web.com/contact/
Update: July 20, 2012, 01:24:03 PM » Due to porting issues with our current UK freephone number, we have been forced to change to a new number. Our new freephone number is 0808 189-0856 . Please update any address book entries you have on us and please use this new number from now on.
This concludes this work. Please let us know if you have any questions. - Date - 05/31/2012 22:43
- Last Updated - 12/24/2012 22:45
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
Maintenance on mail4-net will take place Friday/Saturday 12 AM ET . Mail will be queued at the sending servers. This is expected to take 30 to 60 minutes. Mail4-net is primarily used by windows accounts.
Please let us know if you have any questions on this.
Update: July 20, 2012, 11:16:12 PM » This work is in progress.
Update: July 20, 2012, 11:35:48 PM » This work has been completed. - Date - 07/20/2012 23:59
- Last Updated - 12/24/2012 22:43
- Priority - Medium
- Affecting System - Phones
-
Our building is having floor work being done and unfortunately there is unexpected noise from this work. As such, phone support will be unavailable for the next few of hours due to the noise and poor call quality that could happen. If you need us, please submit a ticket.
Update: July 26, 2012, 11:23:02 PM » Work was completed without missing any calls. Some more work is expected tomorrow and we will post if the tools that caused a lot of noise will be used again.
Update: July 28, 2012, 12:17:55 PM » Friday's work was completed without issues to calls.
On Tuesday there is planned landscaping work right outside our office's windows and this work may involve loud machines/tools. We will let you know of any issues at that time.
- Date - 07/26/2012 16:40
- Last Updated - 12/23/2012 16:42
- Priority - Critical
- Affecting Server - Hsphere-Windows
-
Hello,
MSSQL4 will have hardware maintenance performed on it on July 26/27, 2012 at 12 AM ET . Expected outage is 15 to 30 minutes.
Update: July 26, 2012, 11:19:47 PM » This is currently under way.
Update: July 26, 2012, 11:48:35 PM » This work has been completed with less then 10 minutes downtime.
Update: August 08, 2012, 01:54:11 AM » MSSQL4 will have maintenance on it this Thursday/Friday night at midnight eastern. Downtime should be 15 to 30 minutes. - Date - 07/26/2012 23:59 - 07/27/2012 01:00
- Last Updated - 12/23/2012 16:40
- Priority - Medium
- Affecting Server - Hsphere-Linux
-
Hello,
We will be upgrading mysql2-5 to mysql 5.1 tomorrow, September 8 at 11 PM ET . This will require a bit of downtime for mysql services provided on this server to allow for the databases themselves to be upgraded. No other issues are expected. This thread will be updated after the upgrade is done. - Date - 09/08/2012 23:00
- Last Updated - 12/23/2012 16:37
- Priority - High
- Affecting System - DNS
-
Hello,
Bind (named, the nameserver software) will be updated tonight to close off CVE-2012-4244 . Each nameserver will have a few minutes downtime, however, sites should not be impacted due to the redundancy of DNS. - Date - 09/18/2012 22:35
- Last Updated - 12/23/2012 16:36
- Priority - High
- Affecting Server - Hsphere-Windows
-
Hello,
The RAM will be replaced on 9/24/2012 in mail3-net (windows accounts only use this) to prevent RAM failure. This will happen between 7 AM to 8 AM ET and will require about 15 minutes downtime for mail services to some accounts provided on this server. No other services will be affected. - Date - 09/24/2012 07:00 - 09/24/2012 08:30
- Last Updated - 12/23/2012 16:34
- Priority - Critical
- Affecting Server - Hsphere-Windows
-
Hello,
Win15 will be rebooted later tonight to apply configuration settings within different systems on it. This will require up to 15 minutes downtime for the reboot to complete. - Date - 10/01/2012 23:32
- Last Updated - 12/23/2012 16:33
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
As MySQL 4 has been end of life for a few years, we have decided to migrate all databases on mysql1 (MySQL 4 server) to mysql2-5 (a MySQL 5 server) and decommission mysql1. Most scripts should work with MySQL 5 without issue, though some really old scripts may not work, however, those should be upgraded in any case to get security fixes, bug fixes, and features. If you are not sure which MySQL server you are using, please login to your account portal (https://cp.u2-web.com / ) and go to Databases --> MySQL --> MySQL DBs and it will list the server under the host name area.
This migration will start October 3rd and continue until October 20 and at that point, we will decommission mysql1. Tickets will be created for customers that will be migrated before the migration starts. This migration only affects MySQL service and no other services will be moved during this migration.
The migration of MySQL services will be automated, ran at night (after 8 PM ET) and script configurations will be updated via that automated tool.
If you have any questions, concerns, or would like to move your account at a specific date, please let us know via a support ticket. If you are not sure if your script will work with MySQL 5 and would like our help to determine, please let us know. (https://u2-web.com/contact/support/ )
EDIT: pgsql1, which is housed on mysql1 has been discontinued at this time as it was not being used.
Update: September 20, 2012, 12:34:51 AM » Tickets have been opened for all customers that will be migrated. If you have not received a ticket and are on mysql1, please check the email address on file for the account and if that is out dated, please let us know so we can add your new email address to the migration ticket.
Update: October 09, 2012, 12:19:39 AM » This migration is running smoothly and none to very few issues have come up.
Update: October 14, 2012, 10:31:41 PM » mysql3 has been launched and will now be taking this migration and any new mysql services.
Update: October 18, 2012, 03:31:17 PM » This migration is now complete. Thanks to all our customers for your cooperation and continued business. Not many issues happened during this migration. This server is being reclaimed now. - Date - 12/23/2012 16:29
- Last Updated - 12/23/2012 16:32
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
Monitoring has detected a failing RAID backplane in the cp/mysql3 server. A maintenance window to replace this has been scheduled for tomorrow (October 23) at 7:30 AM ET. Downtime is expected to be 15 to 30 minutes to allow for replacement of the hardware. The replacement is needed to prevent complete RAID failure and/or data corruption.
Update: October 23, 2012, 07:34:26 AM » This maintenance has been completed.
Update: October 23, 2012, 08:54:33 AM » This server is currently experiencing an outage, which may be related to the maintenance. On-site techs are working on the server at this time.
Update: October 23, 2012, 09:42:26 AM » Techs are still actively working on this server to resolve the outage. We apologize for the unexpected downtime.
Update: October 23, 2012, 11:13:12 AM » Unfortunately, at this point it appears the replacement has caused data corruption. A bare metal restore from last night's backup is being prepped. I'll provide updates as more info is available.
Update: October 23, 2012, 12:22:55 PM » Bare metal restore is currently running. The bare metal restore does not provide an ETR and so providing one here would be a guess. With that in mind, this server doesn't have a lot of data on it and so it should not take to long to restore.
Update: October 23, 2012, 02:14:54 PM » The restore is 75% done.
Update: October 23, 2012, 03:33:41 PM » The restore is going slower then expected for this size server. Sometimes when it hits a large packet of small files, this can happen.
Update: October 23, 2012, 04:03:00 PM » Restore is finished on new hardware and server is booting up. We'll have to run through some checks and then we should be good to get back to normal services.
Update: October 23, 2012, 05:02:19 PM » Sorry for the delay of this update, the server is going through a second pass on a fsck (file system check).
Update: October 23, 2012, 07:46:58 PM » The fsck is still ongoing.
Update: October 23, 2012, 11:47:33 PM » First I would like to apologize to the customers affected by this extended outage. Most customers on mysql3 were moved from mysql1 recently and those customers are our long term and loyal customers. Although other customers were affected by the control panel outage also, it is a secondary service and not site affecting as mysql3 outage was -- our cluster setup really paid off as it kept other services (like mail) up. Thanks for your patience and cooperation during this outage -- it really helps to keep a stressful event for everyone, less stressful.
Today's events were basically murphy's law having a good day unfortunately. If it wasn't hardware related, it was a slow bare metal restore and then fsck (file system check). The bare metal restore was needed as the replacement backplane caused data corruption (potentially) and the risk of further outages and increasing data corruption was not worth it. The system then did the fsck, standard under an event like this, which then took a while to complete. After that a couple other hardware related items came up. Put all this together and you have a recipe for today's events.
Mysql3 and control panel services have been functioning since about midnight. I wanted to wait to post this message until we were sure they were stable and were running for a bit.
Please let us via a ticket (https://u2-web.com/support/contact/ ) know if you have any questions. - Date - 10/23/2012 07:30
- Last Updated - 12/23/2012 16:29
- Priority - Critical
- Affecting Server - Hsphere-Linux
-
Hello,
Currently mysql traffic travels over our public network card, which works fine and worked fine for a long time. In order to reduce our public network footprint, we will be making changes to web6 so that mysql traffic goes over our private network. No service impact will happen and no action is needed from customers. Going over the private network has several benefits and leads to our optimization goals.
This change will happen tomorrow tonight at 10 PM eastern.
Please let us know if you have any questions. - Date - 11/20/2012 22:00 - 11/20/2012 22:30
- Last Updated - 12/23/2012 16:21
- Priority - High
- Affecting Server - Hsphere-Linux
-
Hello,
On wednesday night at 10 PM eastern we will be switching web6's front end web server (the service that is first connected to when going to a site) from the usual apache to nginx. Nginx will then connect to apache if it needs to for php and such, while getting static files directly. (This is a reverse proxy setup) During this switch, you may notice a reset connection error for a couple minutes, however, we're expecting it to be smooth.
Changing to nginx will help give optimization options such as browser caching, file caching server side of recent most accessed files, gzipping of css, javascript, and html too all sites without additional configuration needed. This also helps with loading of static content (images, css, javascript files, etc) as nginx serves that content up faster then apache in most cases.
No changes are needed by customers for this to work. It is a server wide change.
Between now and then, you can test your site running via nginx by going to http://Your_domain.tld:8080/ -- make sure to keep the :8080 in the url at all times during the testing phase so that you will go to nginx and not apache. Some scripts, such as wordpress, may redirect you to a URL without the :8080 in the URL. After the switch, the :8080 is not needed and the behavior will be normal.
Impact: reset connections for a minute or two
Time: 11/21/2012 at 10 PM ET - Date - 11/21/2012 22:00 - 11/21/2012 22:30
- Last Updated - 12/23/2012 16:18
- Priority - Medium
- Affecting Server - Hsphere-Windows
-
Hello,
Mail packages for mail3-net, which is used by our windows accounts, will be upgraded in a couple hours. This may cause mail service interruptions for up to 15 minutes while the packages are being updated. Any mail attempted to be delivered should be automatically retried later by the sending server. - Date - 11/29/2012 16:15 - 11/29/2012 00:00
- Last Updated - 12/23/2012 16:16