After months of ups and downs, plans built and destroyed, and countless hours invested, CRM 2016 is live. (Actually, it’s been live since Easter weekend. But keeping a blog up to date is obviously not one of my talents.) For the last installment in this series I’d like to go over how we went about the upgrade and highlight some successes. Before I get into that, though, there is one essential thing that needs to be understood about this entire project. From the moment we really got started, it was placed in the hands of Jesus Christ. From planning to implementation, and everything in between, it was submitted to Him. I don’t say that to be cheesy or legalistic, but rather to give credit where credit is due.
As I’ve mentioned before, we were attempting to go from our production Dynamics CRM 2011 environment all the way to the latest Dynamics CRM 2016. That environment has a database that exceeds 1.5TB and numerous integration points with systems like our dialer. During planning we reached out to a vendor, who shall remain nameless, for their advice. The overview of their plan looked something like this:
Following this approach, the migration would flow as such:
- Shut off the PROD 2011 servers.
- Back up the PROD 2011 organization database.
- Restore the PROD 2011 database to the temporary CRM 2013 environment.
- Import the PROD 2011 database to CRM 2013.
- Back up the upgraded CRM 2013 database.
- Restore the upgraded CRM 2013 database to the temporary CRM 2015 environment.
- Import the upgraded CRM 2013 database to CRM 2015.
- Back up the upgraded CRM 2015 database.
- Restore the upgraded CRM 2015 database to the temporary CRM 2016 environment.
- Import the upgraded CRM 15 database to CRM 2016.
- Back up the upgraded CRM 2016 database.
- Restore the upgraded CRM 2016 database to PROD 2016.
- Import the upgraded CRM 2016 database to PROD 2016.
We built out the “hop” environments as this plan suggested and began to test the theory. Aside from multiple technical challenges, some of which I’ve addressed in other posts, even when it worked the timeline was simply too long. End to end it took 1.5 weeks or more, and that’s not counting any additional steps such as importing our customizations or the work for our data warehouse. Taking down a tier zero application for one to two weeks was NOT an option for us.
The more I thought over the proposed plan, our past CRM experiences, and the options we had available to us, a new plan came into view. I wasn’t sure it would even work, but decided to do a “mock migration”, if you will, and test it out. The new approach looked like this:
The smaller database symbols represent the config database for each temporary environment. The temporary 2016 servers were eliminated altogether since our final (PROD) hop was 2016 anyway. Basically, I just made the 2011 organization database a central upgrade spot on a new SQL instance and pointed all the app servers there.
Following this new plan, the work flow became:
- Shut off the PROD 2011 servers.
- Remove the PROD 2011 organization database from our PROD Availability Group.
- Detach the PROD 2011 organization database and reattach it under the new PROD 2016 instance.
- Import the PROD 2011 database to CRM 2013.
- Remove the organization from CRM 2013.
- Import the upgraded CRM 2013 database to CRM 2015.
- Remove the organization from CRM 2015.
- Import the upgraded CRM 2015 database to CRM 2016 (using PROD App servers).
- Remove the other copy of the CRM 2011 database and sync the upgraded 2016 database to the other Availability Group node.
This had a profound affect on our timeline. Each of our backup and restore operations had cost us around six hours each, for a total cost of around 48 hours. By pointing all of the app nodes at one central SQL server, that was completely eliminated. It also allowed us to utilize the beefier PROD hardware. Instead of 1.5 weeks our tentative database upgrade timeline became 31-36 hours, very doable for a weekend maintenance window. And that was on slower storage that had been provisioned only for the mock migration tests. In the end our actual time for full migration was less than 24 hours. That includes not only the database upgrade portion but also importing customizations, changing our data warehouse structure, reconfiguring connected systems, and troubleshooting issues.
There was truly a fantastic team that worked on this project and I’m honored to have shared in the adventure with them. There is a great deal more work not detailed in these blog posts because I didn’t have direct insight into those portions. I’d like to call out a few of them at a high level though.
- Our development team spent months updating old code, much of which predated their employment here. They also developed innovative new ways of approaching old problems and bringing everything into compliance with CRM 2016. With the amount of customization we have, this is more than commendable.
- My co-upgrader, Matt Norris, put in many long hours and handled with stride any challenge thrown at him. He was new to CRM when we started but I now refer to him as “battle-hardened”.
- My fellow DBA, Bret Unbehagen, single handedly worked out the difference in table structures between CRM 2011 and 2016, determined how that would affect streaming data to our Oracle data warehouse, and created a process to mitigate those issues for the go-live weekend. I can’t even begin to describe to you how impressive this is because I don’t even fully understand it myself.
- Our networking and storage teams were indispensable in preparing for and carrying out the upgrade. I asked for terabytes upon terabytes of space for various tests and mock migrations, as well as innumerable firewall requests, and they delivered every time. The compute guys also gave me additional resources on the app server virtual machines, to which I credit much of our surprisingly small timeline.
It’s a blessing and a privilege to have been a part of such a successful effort. There is nothing more satisfying than doing good work to God’s glory.
If you have any questions about how we approached the upgrade, mock migrations, etc please feel free to ask. I’ll be glad to answer anything that doesn’t put my job at risk 😉
Technical Details
App Server Versions at Upgrade Time
- 2013: 6.1.0001.0132
- 2015: 7.0.0001.0129
- 2016: 8.1.0000.0359
SQL Server Version
- Microsoft SQL Server 2014 SP2
Leave a Reply