From
this post at around 11:37am PDT (short version: an upgrade was attempted, didn't go well. back to the old data. nothing is broken.)
Hi everyone. We've found the problem and have a plan of action in place.
What was the problem?
The process of copying the data from the old database to the new database had issues with specific characters. Mainly these were older characters with large amounts of data associated with them. All of their data didn't copy correctly which caused them to not be able to load. The same behavior occured with other database record types, such as housing records, where their associated data was very large. This caused some house zones to fail to load as well. This is why some players received different error messages. You might have been attempting to log in a character that was fine, but he was camped in a house with a problem.
What is the solution?
Our database engineers will need a little more time to find out why the migration process had issues with these specific records. Once that issue is solved, we will perform another migration and test these characters specifically to ensure that problem is fixed. We don't have an ETA at the moment, but need to be prepared for this to take a couple of days.
What do we do in the meantime?
In order to get everyone back online again, we are going to revert back to using the old database which is still 100% intact. The data in the old database was frozen at the time we went down for maintenance. So this means anything you did early this morning will be rolled back. We're very sorry that this had to happen. Unfortunately we can't risk the time or data by copying backwards from the new server to the old server.
How do we keep this from happening in the future?
We are documenting our testing processes to be sure to include characters and houses that meet the critiera that caused this problem in the first place. In the future we will have specific test cases that will keep this problem from occuring again.
How does this affect the Venekor -> Nagafen Merge?
If we can repair the migration process quickly, we hope to not impact that timeline. However we must also plan for the possibility that the delayed database migration might cause us to delay the merge for a few days. Hopefully it will not come to that. We're all very excited about getting you on the new hardware and completing the merge as soon as possible, so we are doing everything we can to make that happen.
So again, I apologize for the rollback and the downtime associated with this maintenance. We're starting the process of reactivating the old database servers and hope to have you back up in an hour or two. As soon as we've got the issue with the migration process fixed we will post a revised timeline for the new maintenance.
