Connectivity to the server is lost since Tuesday, 25 February 2014, 03:42 UTC.

I have already contacted with the network administrator and waiting for reply.


Thank you for your patience!


The support department's responses to my inquires about the issue are marked as italics.


UPDATE

Tuesday, 25 February 2014, 15:00 UTC


We apologize about the inconvenience that has caused. We are experiencing the issue with RAID in server. Our upstream engineers are working on server to retrieve the issue and the server will get back to online as quick as possible.

Thanks for your patience.


UPDATE #2

Tuesday, 25 February 2014, 22:01 UTC


We will have to rebuild the server from a bare metal restore and that we are in process of doing it. The server will get back to online as quick as possible.


UPDATE #3

Wednesday, 26 February 2014, 03:20 UTC


We need to rebuild the whole server from the backup, at this moment we are installing the OS. We will keep updating this announcement as soon as we get something new.


UPDATE #4

Wednesday, 26 February 2014, 13:05 UTC


We have restored the database and the server configurations, the only part left is the /home partition. 


UPDATE #5

Wednesday, 26 February 2014, 19:19 UTC


We are still restoring /home and that we don't have an ETA yet.


UPDATE #6

Wednesday, 26 February 2014, 21:20 UTC


We had to rebuild a RAID array and then restore about 500 GB of Data (we are still on the process of restore /home everything else has been restored)

We didn't lost any data, the backup is healty and it's being restore


UPDATE #7

Wednesday, 26 February 2014, 22:14 UTC


Unfortunately we can't give you an ETA, that we are not sure that we can accomplish. I think that shouldn't take more than 12 more hours, but as I said, I'm not sure and I don't want to give any information yet.


UPDATE #8

Wednesday, 26 February 2014, 23:39 UTC


We use RAID 10 in all our servers. The RAID card was OK, only 1 HDD failed we were working yesterday most of the time trying to bring the server back without the need of the restore, replacing the disk and rebuilding the array didn't helped to bring the server back, was trowing kernel panic message anyway. We tried to boot from a rescue CD and fix it from there and was impossible, so, I took the decission of build everything back and start the restore, and that's what we are doing since yesterday.


UPDATE #9

Thursday, 27 February 2014, 00:18 UTC


200 GB of the 500 GB are already restored


UPDATE #10

Thursday, 27 February 2014, 05:19 UTC


The first 18 hours were used to try to "recover" the server as it was, after that, I decided to quit that and start the from 0 and restore it from the latest backup (which was from 1 hour before the server fail), all the data of the server has been restored BUT /home, that's the biggest part and so far there are near 300 GB of data restored, the process is automatic, and we can't speed it up, since the files are copied from the backup server to the new server.


The backup server is on RAID 6, on the same datacenter than the server being restored, and in no way, can be restored in just 12 hours unless you are restoring only big files.


UPDATE #11

Thursday, 27 February 2014, 14:32 UTC


The restore has been completed, but there are some configuration issues with MySQL and PHP that currently are been solved.


UPDATE #12

Thursday, 27 February 2014, 20:14 UTC


Everything works fine, except of one website's MySQL database that appears corrupted, so that website is still non-functional:

Seems like the problem is related to the InnoDB, that DB is using it and the restore of that didn't worked very well when is done with R1soft.

We hired to an external consultor to work with that, he is doing it at this moment.


FIXED

Friday, 28 February 2014, 16:00 UTC


Server restoration completed.