Magick Upgrading in SharePoint

Key to the Magick framework for SharePoint solutions is the concept of upgrades.

In previous articles, I’ve outlined how Magick supports and encourages certain principles that are important to building maintainable and upgradable SharePoint solutions. Now it is time to get a bit technical and look at how this actually happens.

There’s a bit of theory still, though.

Magick Solution Structure

Everything in Magick is by convention only, except for one single requirement.

By “convention only”, I mean that you are free to design your solutions in any way you like. You can use any development tool you like (WSPBuilder, VSTools for SP, STSDev, or Notepad), you can follow any patterns you like, and you can combine all of these in any way you like.

There’s one single requirement, though, and that is that you tag your features with a certain custom property.

Features in SharePoint support custom properties just fine, and they’re simply ignored if not used, so this requirement is a very basic and non-intrusive requirement.

The reason for this requirement is that these custom properties are how Magick determines what to do with those solutions.

There are two custom properties you need to add to your features. The first property, called UpgradeSolutionID tells Magick which upgrade path you want to take. The second property, called UpgradePriority, is a number that tells Magick in which order to run the upgrades.

An example feature may look like this:

<Feature xmlns=””
Title=”My Magick feature”
Description=”Some description”
<Property Key=”UpgradePriority” Value=”15″ />
<Property Key=”UpgradeSolutionId” Value=”Magick” />

I’ve highlighted the custom properties section, which you need to add to any features that you wish to add to Magick.

That’s it! If you learn how to add those custom properties (or simply copy them from here) you’re done. There’s no need to learn anything else, really, and there’s no need to make any other changes to any features.

How Magick Really Works

You may be surprised by the simplicity of the changes required to create Magick features, but don’t be deceived by this simplicity. Behind these properties lies an awesome power with virtually no side effects, and you can build some pretty complex multi-path upgrade scenarios.

I’ll go into detail on Magick and the 42 lines of the framework later, but for now, let me explain how Magick uses these properties to handle your upgrades.

In Magick, there is a single feature, called and Upgrade feature, which handles all the hard work for you. This is where those 42 lines reside.

When activated, the Magick Upgrade feature will go through all features in your farm and look for features that have those two custom properties added. Then, it will sort them in the order of the UpgradePriority property, and finally, it will activate all the features that haven’t been activated already.

This too may sound extremely simple, and it is, but again, this simplicity hides awesome power and incredible flexibility.

Let’s say that you want to add a certain list with an attached content type. Using Magick, you would then add a feature to create the list, a feature to create the content type, and finally a feature to connect the content type to the list.

Your features would then, for example, have the following UpgradePriority numbers:

  • 15 List creation (web scoped)
  • 25 Content type creation (site scoped)
  • 35 Connect content type to list (web scoped)

Upon activating the Magick Upgrade feature, Magick will activate these features in order, even if they have different scopes. You don’t need to activate the individual features yourself or ask your users to do so.

After activation, the Magick Upgrade feature automatically deactivates itself so that it is ready to be activated again. Of course, because Magick only activates features that aren’t activated, clicking it again won’t reactivate the existing features, so there’s no danger of getting two lists.

Next time you want to make changes to your sites, you create a new solution with new features and ensure that you have unique UpgradePriority numbers, and upon activating the Magick Upgrade feature, those features will be activated for existing sites. For new sites, all features will be activated, and you’ve ensure that all your sites have the same features activated.

Neat, eh?

Can I use this for existing features?

A possible, although not supported, scenario is to tag existing features with the Magick custom properties. For example, if you want the Publishing framework to be available, you can technically modify the feature.xml file of that feature and add the custom properties to have Magick add it for you.

Keep in mind, though, that modifying existing features, especially the out-of-the-box features that ship with SharePoint, is not supported.

You can, of course, tag any of your own features as much as you like, to ensure you at least get your features activated in the correct order.

Paths and Priorities

The two custom properties that Magick needs give you great flexibility in targeting different exactly what to upgrade. I’ll go into more detail on how you can customize these properties to create different upgrade paths.

In this context, and upgrade path means a group of upgrades that Magick will handle as a separate set of upgrades. With a few modifications, you can even get Magick to automatically pick the right path for you.

If you’re allergic to .NET development, this means you actually need to have multiple features to activate multiple paths.

For example, you may want to have one path that upgrades Team Sites and another path that upgrades your custom Project Sites.

To accomplish this, you need to tell Magick which path to upgrade. There are multiple ways to accomplish this, and I’ll dig into the details of how to do so once I show you the actual code.

It is also vital that you keep track of the UpgradePriority numbers you have used. This is really a matter of strategy, and I’ll share some of my experience in picking such priorities with you. In short, however, a good strategy allows you to inject upgrades into previous solutions. For example, you may find out that as part of your content type deployment, you really wanted to add a new form to the list first. With Magick, that’s as simple as deploying a new feature with an UpgradePriority that precedes a previous feature.

Like I said, I’ll go into more detail on strategies later, and next, I’ll also show you the framework code that makes Magick… well, Magick.


Found this article valuable? Want to show your appreciation? Here are some options:

a) Click on the banners anywhere on the site to visit my blog's sponsors. They are all hand-picked and are selected based on providing great products and services to the SharePoint community.

b) Donate Bitcoins! I love Bitcoins, and you can donate if you'd like by clicking the button below.

c) Spread the word! Below, you should find links to sharing this article on your favorite social media sites. I'm an attention junkie, so sharing is caring in my book!

Magick and Continuous Upgrades

In explaining what Magick does in SharePoint, I’ve previously briefly described how awesome Magick the Framework is and what SharePoint problems Magick solves.

However, that’s just the start of what Magick does.

In this article, I’ll explain how Magick allows you to implement agile approaches to SharePoint development by supporting continuous upgrades and evolutions of your SharePoint solutions.

Oh, and if I haven’t mentioned it already or you haven’t guessed: Magick works across all SharePoint versions, at least from SharePoint 2007 and onwards.

What are Continuous Upgrades?

In Agile development, a key principle is that you build and deploy functionality very rapidly. In fact, in many cases, you create functionality every week of a long project, and users get access to that functionality immediately.

This approach is a win-win for both the developers and the users. Users get to see rapid progress and get access to important functionality as soon as it is ready, rather than having to wait for a long time until an entire software development project is complete.

This also allows them to give feedback during development, for example to suggest further improvements, new features, or changes to features they don’t like.

For developers, the continuous feedback and rapid deployment means they can find issues sooner and can vastly improve the solution. It also means that the development sprints, as these short development cycles are often called, are done sooner so one gets more variety in development and see the results of ones work more quickly.

However, in SharePoint, this approach is often very difficult due to the somewhat static nature of how SharePoint works. If you recall from the previous article, many SharePoint components cannot be changed ever after they have been deployed once. How can you then make continuous improvements and deploy updates on a regular basis?

In a Word: Magick!

In Magick, we not only support the static nature of SharePoint solutions, we encourage it. In fact, in Magick, we build a solution once and we never touch it again, at least in a perfect world.

That may sound strange for a framework that claims to be agile and support rapid changes first and foremost. However, it is really quite simple.

Magick wants you to think about changes in your solutions, and not how something came to be. In other words, we look at how we need to move from point A to point B, not how we got to point A in the first place.

In the traditional approaches to SharePoint solutions, you often start out by deploying a solution that does something. This is what gets you to point A. Traditionally, when you need point A to change, many developers will look to change the solution that got us there. In technical terms, they modify the solution file and redeploy the solution.

This is problematic from a number of perspectives.

First, many components cannot change after they have been deployed. This is due in part to Microsoft policies, but more fundamentally, due to the way SharePoint solutions work. Remember that SharePoint solutions are recipes for how to build something, and changing the recipe does not change the already produced goods.

In Magick, however, we never change a solution after it has been deployed, with the exception being .NET code which can change if required, but also should be left as much alone as possible.

Instead, we seek to create solutions that look at the current state of affairs and what is required to move to where we want to be.

Forgive my slight technical departure here, but let’s look at a fairly simple example.

You have deployed a great new solution for project management which includes a great new list of project tasks. After a couple of weeks, users come back to you and say that they really like it, but would like different options for sorting and filtering the list.

In traditional SharePoint development, you’d simply go in and modify the original solution to support the new requirement. This, of course, breaks Microsoft’s policies on never changing any declarative components, but more importantly doesn’t always work. Users may already have modified the list, and if you redeploy those lists with different functionality, you may break what the users have worked hard to accomplish.

In Magick, on the other hand, you would instead deploy a new solution that will modify the list and add whatever functionality the users requested. Depending on what change you want to make and the toolset you have, this allows you to check whether the new change will actually make sense in a particular situation.

Of course, doing more advanced detection requires .NET code, but if you can use that, you can just check whether the current view is unchanged from when you originally deployed it, and only change that view if the users themselves have not made the changes. Otherwise, you could create a new view for them, or simply not make the changes at all.

Are these types of changes supported? Yes, very much so. In fact, you are really just automating the tasks that the users would do themselves through the web interface, including the decisions made to determine if you want to perform those tasks.

If you are allergic to .NET code, fine, you can still make certain changes, but mainly to add new components rather than change existing ones. Like I’ve said many times, if you really want to get stuff done, use third tier development and .NET code 🙂

Solution Mayhem!

The keen observer will at this point ask two very important questions.

First, if you build new solutions for every change, wouldn’t you end up with dozens of solution files rather than a single, easy to manage WSP?

Second, if you create new features all the time, won’t you create complexity in deploying new sites or upgrade existing ones? You’ll end up with tons of features that you need to activate in the right order!

These questions are exactly what Magick handles for you. In fact, it doesn’t handle anything else.

For the first question, remember that in Magick, we never touch a solution file ever again after it has been deployed. A lot of the reason why administrators see WSP files as problematic is that they need continuous management. In Magick, they are deployed once and then forgotten. There are no upgrades, there are no changes, so management of multiple files is actually easier than managing just one file.

For the second question, I’ll show you later how Magick handles upgrades, but suffice to say, users rarely, if ever, see or need to see any of the features you deploy. Magick handles the upgrades for you, including making sure everything happens in the correct order, for the sites that need to be upgraded only, and within the scope required.

I’ll detail that in a future article, though, and then I’ll show you all the code, the conventions, and the Magick behind it all 🙂


What Magick Solves in SharePoint

A little while back, I talked briefly about a SharePoint solution framework I built some years back, that I’ve now decided to dub Magick the Framework.

In this article, I want to talk about the problems that Magick solves and why it makes your SharePoint development life much easier. I’ll stay away as much as possible from technical lingo, but to understand what really goes on, you need to understand the basics of SharePoint solutions, features, and some Microsoft policies.

Let me elaborate…

What You Need to Know

First, you need to know that a SharePoint solution in this context is a WSP file, whether a farm solution or a sandbox solution.

A solution is basically a compressed file that contains files and instructions on how to deploy those files to a SharePoint farm. Think of them as software installation packages for SharePoint; like a setup.exe file is for Windows.

When third tier developers build things for SharePoint, they usually end up with a WSP file that they then hand over to a farm administrator for installation. The farm administrator will then install and deploy the solution and the components of the solution then becomes available to users of the farm.

These components are often of a specific type called features. A SharePoint feature can simplest be explained as being a plug-in to SharePoint that extends the default functionality of SharePoint somehow. Features are extremely powerful and flexible,  and can both deploy things like web parts, list and list templates, new sites, or they can perform specific actions when you activate them.

The last thing you need to know is something about Microsoft’s policies on modifying components after they have been deployed.

There are specific guidelines on what can be changed and what cannot be modified in certain situations. To make it very easy to understand and to avoid a lot of confusion, you can remember that for almost all things, you cannot make changes to any deployed solution, feature, template, or page after it has been deployed once.

Note: There are certainly exceptions to the rule, but if you remember to not change anything after deployment, you’re safe.

These policies exist for very good reasons. In SharePoint, features, templates, and solutions are recipes for what needs to be done. Just like when you are making a dish from a recipe, the result is not dependent on the recipe after it has been created. In other words, if you bake a loaf of bread and then change the recipe, the bread you already made doesn’t change; only future loaves fill be different.

However, there are exceptions to this as well. For example, list definitions or pages actually exist in the features and go away if you remove the features. This is by design, and if you utilize it the way it is meant to be utilized, you create a very powerful system for managing content.

The catch, though, is that when you modify an existing component, because it can partially be a recipe and partially a template, your recipe and template no longer matches what is on the site. In fact, the site itself can completely break down because the site expects certain components of certain functionality to be there.

In short, then, when you have deployed your solution once, you should never expect to be able to change that solution again, which places a huge burden on the design and architecture; if you make mistakes, you’re never going to be able to fix it.

Magick seeks to solve this challenge without breaking any of the Microsoft policies on changes. In fact, Magick allows you to be even more strict with your own policies than Microsoft demands, but without sacrificing any of the flexibility.

Key Principles of Magick

To accomplish this goal, Magick encourages certain patterns and principles in the way you develop your SharePoint solutions.

1. Use as little static code as possible

The first and by far the most important principle is that we want to limit the amount of code that cannot be changed or is static. We can’t break Microsoft’s policies about changing deployed content, so we want to minimize the chances of us wanting to do so.

The most visible aspect of this is how you design site templates or site definitions. In Magick, you want your site templates and site definitions to be as empty as possible. In fact, a site definition that contains absolutely nothing is perfect.

Of course, a completely empty site isn’t very useful, which brings us to the next principle of Magick.

2. Make all components atomic

This principle means that you break down your functionality into as small pieces as possible, and have each component do one thing and one thing only.

For example, although it is possible to have a single feature that both deploy a list definition or template and that creates a list instance from that definition, doing so means that you break the principle of atomic components. Instead, you should have one feature that deploys the definition or template and another feature that creates the list.

Of course, this means that you get a lot of work managing all those components or features. There are ways around this, however, and I’ll get to those later, but let me first bring up the third principle of Magick.

3. Deploy once and never touch again

This is the part where Magick encourages more restrictive policies than Microsoft. Rather than focusing on what you can change, deploy code in such a way that you never need to change it again.

This policy guarantees that we remain compliant with Microsoft’s requirements. If we deploy supported functionality and don’t touch it again, there’s no chance that we’ll touch anything that Microsoft prohibits us from modifying.

There is one exception to this principle. Regardless of what we define in our solutions, we can always modify and redeploy .NET components. Only declarative components like definitions, templates, and ASP.NET markup pages must remain static after deployment.

In fact, this principle makes it somewhat difficult to deploy upgrades to a site without using .NET feature receivers. Because we can never change a component, feature, or solution after we have deployed it once, we are limited to what we can do with new solutions to what we can accomplish by deploying new components.

.NET gives us far more options because we can run code to analyze the current setup and make decisions or even change or modify content based on what we see. Declarative (i.e. non-managed code) components have a lot fewer options for modifying content.

This also brings me to a fourth and optional principle and the solution to the management of all those features I mentioned.

4. (Optional) Use .NET feature receivers to install and activate functionality

In the first article, I mentioned that building .NET code is completely optional. That is absolutely true, and with the exception of installing the framework itself, there’s no hard requirement for you to ever deploy a single .NET component again.

However, if you don’t use .NET feature receivers, you miss out on a great way to ensure your solutions are deployed smoothly.

The most common example of this principle is in creating a Setup feature. A Setup feature in your solution is ultimately responsible for setting up (doh!) all the other components.

Let’s imagine, for example, that you want to create a project site based on the Team Site template. To do so, you may need to activate a range of list instance features, possibly some list template feature, certainly some content type features, and maybe some web part features.

Of course, because of the way SharePoint works, you need to activate these features in a particular order. You cannot activate a list instance before the template for creating that instance is activated, and you may not be able to activate that template until the content types are in place.

Doing so manually means that you need to go through a lot of hoops and users have a lot of chances to make mistakes. We don’t want to expose our users to that risk, so instead, we create a Setup feature.

Note: If you’re really allergic to .NET code, you can accomplish some of the basic functionality of a Setup feature using feature activation dependencies. However, Magick can still provide a better approach than using feature activation dependencies even without any .NET code at all.

How? Well, I just told you: it’s Magick!

The Setup feature will ensure that all the features are activated at the right time, in the right order, and at the right scope. Because we’re writing .NET code, we also have the opportunity to dynamically change the way we set up our components as well, and we can even catch errors and fix them automatically or at least present them to the users in a more user friendly manner than “Something went wrong” and a cryptic correlation token.

For the user, a Setup feature presents itself as a single and quite normal feature to activate to get everything set up the way the developers intended. This is in contrast to following a series of steps that must be done in the correct order or else chaos will ensue (and we don’t want Chaos Magick, do we?).

Magick takes away that complexity, even without .NET code, so as long as you follow the principles outlined here, and the very simple pattern that Magick requires, your users shouldn’t feel a thing when you want to set up a new solution or upgrade an existing solution for them.

With these principles in place, we can start looking at how Magick accomplishes its goal of solving the problems of installing, maintaining, and upgrading SharePoint solutions over time. I’ll get back to that in a later article, however, again as time permits.

For now, feel free to ask questions if you have them, and I can try to include the answers in future articles.