The Beginning
Not too long ago I Patched my very large SP2007 environment to the latest CU. (I can hear @ToddKlindt ‘s voice in my head right now) The environment had a large number of teamsites lets just say greater than 1500 and some of these teamsites were pretty large in size, pushing the 100GB content DB best practice limit. So I decided to apply a trick I’d learned from a different environment of removing the content databases from SharePoint, running the patches, then adding the content databases back once the actual servers were updated. This of course would upgrade the content database once it was re-added to the Web Application. Everything went great, or it seemed that way. The upgrade completed in about 3hrs for the entire environment. The sites I checked all were working correctly and well, the environment as a whole seemed to be very happy to be patched. I had no idea what I was to experience in the next couple days.
Excuse me sir?
The next week was going by smoothly until I was notified of a site collection that was giving a 404 error. Upon a quick inspection of the site collection list, I could see the site was listed, but the amount of disk space used was 0 mb. This was very strange to me, normally if a site was having an issue with the content database it would be listed in the site collection list, but would no return any information (site owner, content db, etc…) once it was hightlighted but this was once. My first thought was maybe something had happened to the default.aspx file and it wasn’t allowing it find the correct content db? (this was a flawed theory btw) So added a default.aspx page to the site collection got rid of the 404, but now the site was completely blank, as in it looked like a blank site template. No content anywhere. Just a blank template waiting for data. Around this time I had reports of more sites showing 404…
Houston we have a problem
At this point I did some major digging, and ran the enumsites command against the web application and piped the results into an xml file that I opened with Excel and sorted into a table. I figured one of the symptoms was the 0MB on storage used so I wanted to see if there were more sites having the same issues, and sure enough I discovered 4 additional sites with the storage size set as 0.2mb, and when I tried each link I would get the 404. At this point I began to think there was an issue with the content databases after patching. So I had the previous backups restored (you did take a backup before patching right?) and when I would addcontentdb back into the web app, same issue. It didn’t take me but going back a few weeks to realize that what ever was happening had been there a while.
Ah Ha, there’s the issue
It took some work, and a very brilliant team member of mine figured that we may have orphaned sites in the content database. Of course before we do patching we check for database corruption and run the databaserepair parameter, but (here’s hindsight) it appears that it only looks for orphaned items in the sites. We found the great post by @joeloleson http://www.sharepointjoel.com/Lists/Posts/Post.aspx?ID=291 that showed up how to delete orphaned sites in our content databases, but the next issue was to prove whether we had any or not. Amazingly enough someone had run the preupgradecheck utility against the environment previously and it showed us that there were orphaned sites in our databases, and sure enough it listed the exact 6 sites that we were having issues with. So after running enumallwebs per the blog post above and searching for the sites listed, I found the orphaned sites listed in the Site Map as True, and the sites that used to work listed in the Site Map as False. So what occured was somehow and orphan site was created, and then the correct site created and used. And when I removed the content database and re-added it, the orphaned site took it’s rightful place in the Site Map, and well there was the cause for our 404. Deleting these orphaned sites removing the databases and then re-adding them returned the correct sites into the Site Map resolving the issue.
What did we learn?
Orphaned sites are out there, and could be lurking at any time to show up and 404 some of your production sites. 🙂 Well maybe that’s not the moral. I believe the moral is the be proactive at watching your content databases for Orphaned Site Collections, especially in an environment where you have thousands of site collections. Happy Hunting!
(33)
