Disconnected SharePoint Workflows

In a previous article called SharePoint Workflows Aren’t Business Processes, I talked about why it is important to separate the idea of a SharePoint workflow from the business process that workflow supports. It seems to have resonated, but again, my students come to the rescue to point out where I need to elaborate further.

The question one of my students asked was how they could design their business entities or content types to support complete disconnection from the supporting workflow, and I had to think for a bit to come up with a good answer.

The Scenario

Note: I’ve changed the scenario to avoid disclosing details from the student scenario, but the idea and outline remains the same.

In our scenario, we’re looking at an order process with three stages. Order reception, order processing, and order completion. We want to design a system that can maintain this process without tying the process into the orders or the workflow into the process.

Our order entities can be very simple, and to simplify even further, I’ll even drop some of the necessary details, such as the customer relationship.

We need to maintain the following data:

  • Order Number (arbitrary identifier)
  • Order Date (Date/Time)
  • Order Sum (Currency)
  • Order Status (Received, Processing, Complete)
  • Received By (Person receiving the order)
  • Packaged By (Person packaging the order)
  • Sent By (Person shipping the order)

Again, this is simplified from a more complex business requirement, but should be sufficient to illustrate our approach.

Designing the Entities

Our first step is to normalize our data and come up with the business entities required to support this process.

Note: Data normalization is the process of structuring our data in a way that removes duplicate data and preserves data integrity.

You may be tempted in this scenario to use a workflow to represent the order process. By doing so, you can use the workflow status to represent the order status. SharePoint will even throw in a workflow status column in your default view when you start a new workflow.

Neat, eh?

No, not really.

You see, by doing that, we hard link the workflow we build into the order entity, and that order entity and its future now depends on that workflow operation like we think it will operate.

“Fine,” you say, “but I can just create a new workflow if I need to change the process radically”

If you do that, then you’ll break fundamental principles of virtually any development discipline. Your order will have two columns, one representing the old workflow and one representing the new. At the very least, this will confuse your users, but at worst, you create conflicting information. One of the workflows may say “Complete” while the other says “Processing”. Which is correct? You’ll have to tell your users, or hide the wrong workflow from sight.

“I’ll live with that,” you say, “we’re just a small company so telling the five people handling orders how to do this correctly is no issue”.

Well, you still have the issue of data history. If you remove a workflow that holds business data, you are essentially killing the history of those orders. Any orders process from that point in time will have a status but any orders processed prior to your new workflow will not.

Instead, we need to disconnect the workflow completely from the data.

First Stage of Separation

The first approach many take is to simply allow the workflow to update the Order Status. In a SharePoint Designer workflow, this is as simple as setting the value of a field in the current item.

The benefit of this approach is that you get rid of the direct relationship between the workflow status and the order status. If the workflow changes or goes away, you still maintain historical data. If you add a new workflow later, that new workflow can simply update the Order Status field and we’ve disassociated our workflow from our data.

This approach, while certainly better than trying the workflow directly to the Order Status field, is still not enough, though. Our goal isn’t just to separate the workflow from the data, we also want to separate out workflow from the process.

At this stage of separation, there is no distinction between the business process and the workflow. We still suffer from the same problems if our business process changes; we need to completely redesign our workflow to add a third intermediate order stage, for example.

What’s worse, if we cannot support in SharePoint workflows the requirements of the business process, we’re stuck. We either have to change the business process or add manual steps, both of which are very sub-optimal from the business perspective.

Finally, even though we do separate the workflow from the business data, we may still end up with business process information mixed in with the business data.

Let’s say that the order process requires an approval of some sort, maybe even a few approval stages (order approved, shipping verified, etc). In order to store such approval states outside the workflow, we need to add a column to the business data to store the state.

However, by adding one or more approval state columns, we have essentially polluted our business data. Whether an order is approved is irrelevant to the order itself; the approval is part of the process, not the order.

If we later decide we no longer need the shipping verification, for example, we now have a ton of orders containing irrelevant shipping verification data.

If we didn’t think that we needed a shipping verification in the beginning but later decided to introduce it, well, all orders prior to that decision will not have the correct shipping verification values and our reports will not be correct.

This certainly isn’t convenient in a flexible process where we want to ensure that our workflows adapt to changes in the business process.

Second Stage of Separation

The answer is to introduce a new stage of separation.

We’ve determined that we need to separate the business process from the workflow, and the business data from the process. Neither of these links are beneficial from the perspective of ever-changing business processes.

To accomplish this, we need to create an entirely new entity; the order process entity.

This entity will represent the order process, regardless of how the business data looks and regardless of what our workflow does.

In fact, this layer of separation allows us to have multiple and independent workflows, even from different automation technologies. We can combine SharePoint Designer workflows with event receivers with SharePoint 2013 type workflows with manual processing.

The business process entity will be responsible for working with the business data by updating information as required. Our workflows will never touch that business data at all, except to read information relevant to the workflow.

In our relatively simple order process, our order entity may look something like this:

  • Order ID (Lookup or reference to an order identifier)
  • Order Process Stage (Received, Processing, Complete)
  • Order Approval State (Approved, Declined, Pending)

The order process entity will be the only gateway to update or modify our orders. Our workflow will update information in the order process, not in the orders.

This stage of separation requires a bit more work, though, because we need to get the business process to update the business data. However, this is a one-time operation that allows us much more flexibility in how we later work with our business process, so the tradeoff is definitely worth it.

By isolating our access through such an entity, we accomplish several goals.

  1. We avoid storing superfluous data in the order (such as ever-changing states)
  2. We have a single interface with which we can modify order data, accessible by any method or technology
  3. We ensure that changes in the order data does not break our workflows

This approach is very common in regular software development, where developers are accustomed to building separate business layers and data layers, and access the data through the business layer only. Generally, it is considered a much safer approach as it prevents error in our interfaces from ruining our business data.

Note: Of course, this statement is only true if our business logic layer does not break the data 🙂

Further, this approach allows us much more freedom in terms of controlling our process. Let’s say we have one order that falls somehow outside the regular process for example if a customer picks up the shipment directly. In this case, an employee needs to manually change the process for this order only.

If we maintain a hard workflow-to-data model, in which the state of the order is stored in the order, we need to modify the order and possibly break the workflow currently running on that order.

With the two-stage separation model, however, the user can update the process entity without fear of breaking the underlying data or other processes that run.

Of course, this doesn’t apply to manual updates only. Perhaps we want to change our workflow engine to utilize the new SharePoint 2013 engine or we’ve purchased an external engine like Nintex Workflow.

Well, guess what, as long as our new workflow operates on the business process entity only, that’s perfectly fine. We can swap out parts of the business automation system without killing data or ruining reports.

Finally, this separation also allows us to change the business process without ruining data. This happens because there aren’t any links between the business data and the business process either; the process is simply making changes to the data but has no other ties to that data.

If we change or even remove completely the business process, that’s fine, our underlying data remains unchanged.

Conclusion

I hate writing these, but I’d like to summarize a few points.

Separation of data, business logic, and interface is a key principle in modern software development. This model explains how to accomplish this in SharePoint workflows as well, by creating a complete disconnection between the software we use to manipulate data (the workflow), the business logic or process, and the underlying data.

For years, I’ve been trying to get novice and professional SharePoint developers to understand that software development principles apply to SharePoint as well. Equally, bad software development introduces the same risks in SharePoint as it does on any other platform.

The multi-tier model described here will be well-known to experienced software developers, for example those familiar with MVC patterns. However, for many SharePoint developers, who have little or no other software development background, this may seem like a lot of extra hassle without any tangible benefits.

If you think this, however, I would like to take this opportunity to call you an idiot.

.b

Found this article valuable? Want to show your appreciation? Here are some options:

a) Click on the banners anywhere on the site to visit my blog's sponsors. They are all hand-picked and are selected based on providing great products and services to the SharePoint community.

b) Donate Bitcoins! I love Bitcoins, and you can donate if you'd like by clicking the button below.

c) Spread the word! Below, you should find links to sharing this article on your favorite social media sites. I'm an attention junkie, so sharing is caring in my book!

Pin It

SharePoint Workflows Aren’t Business Processes

Richard Harbridge recently posted an article on SharePoint workflow, explaining rather brilliantly what they are, how they work, how you should think about them, and so on.

He asked for comments on Facebook, and I responded with one thing I feel he is forgetting, that is vital to securing SharePoint success.

You see, unlike that may be the attitude of many, a SharePoint workflow is not a business process. A SharePoint workflow is there to support a business process, but it isn’t and should never be the process itself.

Let me elaborate what that means.

The Lock of Doom

A lock of doom, well, I’m not really sure it is a term, but it sounded ominous, is when you have something that is so important you can’t live without it but so complex that you don’t dare change it. A lot of legacy software systems are like this; they were built as atomic units and 20 years down the line, the original developers are gone and nobody knows how to maintain the system. You’re locked in and you can’t change.

A key feature in well-designed software solutions is the ability to create loosely coupled components that work together to achieve a goal. If any component fails, needs to be modified, or replaced with different functionality, that’s fine because we can work on a single component only, leaving the others to go about their business.

I’m not going into too many technical details here, but think of it like a mobile phone charger. The process at which we’re looking is to get power from somewhere into your phone.

To get your Android or Windows phone to work, because you’re an idiot if you use iPhone, you need three loosely coupled components. You need

  1. The power grid
  2. The charger
  3. The phone

Each of these components have specific tasks that they do successfully (another reason why I can’t include the iPhone in the list). The power grid delivers electricity to your socket, the charger converts that electricity into a format that the phone can use, and the phone eats that electricity like it’s candy.

You can replace any of these components, however. If your charger breaks, you can get another one or a different one. The power grid doesn’t even need to be the power grid, it can easily be a USB port on a PC. In fact, with these components, provided again you stay away from that god-awful iPhone thingy, you can charge other things too, like your MP3 player (no, not iPod), your tablet (no, not iPad), or your headset.

The interchangeability of the components means you have reuse and adaptability.

Imagine what the world would look like if all electrical devices had to be built into the power grid, like one giant atomic super-entity. It would be really impractical; you couldn’t put away your vacuum cleaner, for example, and you’d have to have the world’s longest cable to have a mobile phone. And, if something broke, you’d have to fix the entire grid. This would really be a lock of doom, because you would have to know about every single future device when you designed the power grid. Luckily, we have come up with far better solutions.

The process, however, is simple; you want to get electricity from wherever it is produced into your phone. The workflow and the components used to drive this process are the power grid, the charger, and the phone, but these are not the process.

SharePoint Workflows

Far too often, I see workflows that try to be the super entity of the former example. They try to implement everything into one workflow, often redesigning the process to accommodate the workflow rather than the other way around. “Sorry,” they say, “You can’t do it in that way because SharePoint doesn’t support it”.

The thing is, the workflow is not the process. The workflow is there to support and facilitate successful execution of a process, but like the charger isn’t the electricity or the charging process, a workflow isn’t the process.

Your workflows, then, should work to manipulate data to the extent it makes sense, but do so in a way that does not lock you into using that workflow. In other words, you can use a workflow to store data or even the status of a process.

This idea is actually already built into several aspects of workflows. For example, when a workflow assigns a task to someone, it does so by creating an external item in a task list. That task list is completely independent of the workflow, so the workflow effectively disconnects itself from the result of the workflow. The same happens with the workflow history; it is an external list that’s technically independent of the workflow.

Workflow designers need to think in the same fashion and understand that if you try to map a complete process in a single workflow, you will create a lock of doom situation, where you end up with a massive beast that you cannot touch from fear of breaking something.

It’s perfectly acceptable to create smaller sub-workflows that can run as part of a bigger process. For example, you may have an order execution process consisting of several ‘stages’. Each stage can be its own workflow, focused solely on supporting the process at that stage. On completion of an order entry stage, for example, you may trigger or update another workflow which will create and mange the subsequent stages of the process.

Note: Note my use of the words process and workflow; they mean different things.

I discussed workflows with one of my students a few weeks ago, and we discussed a somewhat complex business process involving many different stages and multiple levels of branching. The process itself was several pages of Visio screenshots, spanning at least two departments, and crossing several site collection boundaries.

Initially, the student was overwhelmed with the task, but as we broke it down into smaller components, we were actually able to build the entire process in a couple of hours. Then, when something needs to change, it is merely a matter of updating one small workflow, and the rest of the process goes on as if nothing happened.

A final point to make with this is that if you ensure the output of your workflow is not store in the workflow itself, you also ensure that if you decide to replace the entire workflow, no data is lost.

For example, if you have a report that shows the status of your latest 200 orders, well, if you get that status from the workflow and the workflow goes away, you suddenly have no data on the status.

Instead, if you update a status column or something like that as part of the workflow, then you can effectively remove the entire workflow, replace it with something else, and move on as if nothing had happened; no data loss, no broken reports, no redesign of taxonomy.

And that makes me a happy bear.

.b

Pin It

SharePoint Domain Names for Sale

I own a number of SharePoint-related domain names that I’m not going to use, and I’d like to offer these to the SharePoint community first, before I go into the open market with them and auction them off to whoever wants them.

Over the years, I’ve had great plans for stuff I want to do in SharePoint. Some of the ideas have turned into USP Journal issues, others into products, frameworks, or services, others have just been sitting there in the back of my mind, waiting for a day when I suddenly find myself with plenty of time and nothing else to do.

I come up with ideas all the time, often in rapid succession. However, I have the attention span of… Ooh, squirrel!

What I mean is that I rarely have time or the attention span to write down the ideas, come up with reasonable business plans, or anything like that. However, I always have easy access to register domain names, so that’s what I do as a reminder of the ideas I have.

By now, I have over a hundred domain names, many of which I’m not going to use, and I’d like to offer these domain names to the SharePoint community first before I try to sell them elsewhere. I figure you guys and gals will probably treat them better than I have.

If you like to get one of these domains, here’s your chance.

At or around May 3, I’ll start setting up an auction solution and start selling off whatever domains remain. Bidding will start at US$500 per domain with a Buy Now price US$1,500.

If you like to pick up one of these domains right now, however, you can do so for US$750. I’ll even give you a discount for three domains for a total price of US$1,500.00.

Update: Domains are now available through SEDO for $1,390 each. Contact me with offers above $1,000.

Terms of Sale

Before you decide, though, please read the following carefully:

To buy a domain, simply comment in the form below or send me an email at furuknap<[at]>gmail.com with the name you want to buy and your name and email address. I will not publish such comments, but I will contact you soon after to verify your information and start the transfer.

After I have received payment, I will transfer the domain to you and mark the domain as sold. If you have a GoDaddy account, this usually takes just a few minutes.

For settlement, I prefer Bitcoins or Litecoins, but I can accept bank payments to a US bank (Wells Fargo). With Bitcoin or Litecoin, payment confirmation happens within minutes, whereas with a bank transfer, it may take days.

If you do not have Bitcoins or Litecoins, and you don’t know how to pay to a US bank (and it can be tricky), you can pay through PayPal. However, there are some caveats with paying with PayPal.

If I do not know you already and you are not a highly reputable person in the SharePoint community, I will hold the domain name for 90 days after the payment has been received. This is due to PayPal’s policy of siding with the buyer in sales of electronic goods in case of a charge back for credit card based purchases. If I transfer the domain to you and you decide to scam me by filing a fraudulent chargeback, I have no recourse for getting the domain back.

I decide alone if you are known to me or a highly reputable person in the SharePoint community.

During those 90 days, I can either manage the domain for you or provide you with a login to manage the domain yourself. Only after 90 days will I transfer ownership of the domain to you.

I really want to trust everyone, but sadly I’ve been burned by people up to no good.

All sales are final. No refunds will be given for any reason, except if I am unable for whatever reason to transfer the domain to you.

Are we clear on these terms? Good, here is

The List

Domain names marked pending have received a purchase request but have yet to clear payment. Please check back in a few days to see if the domain is marked as sold.

Note that I have highlighted some interesting names for your attention only; highlighting is not an indication of any status. A sold domain is marked as (SOLD) and a pending domain is marked as (PENDING)

BAYAREASHAREPOINT.COM
BEGINNINGSHAREPOINTFOUNDATION.COM
BUILDINGSHAREPOINTAPPS.COM
DEVELOPINGSHAREPOINTAPPS.COM
HOWTOSHAREPOINT.COM
LEARNINGSHAREPOINT2010.COM
LEARNSHAREPOINT2010ONLINE.COM
LEARNSHAREPOINT2013.COM
LEARNSP.NET
MANAGINGSHAREPOINT.COM
MICROSOFTSHAREPOINT2010.COM
MYSPLAB.COM
MYSPLAB.NET
OFFICE-FOR-EDUCATION.COM
OFFICEFOREDUCATION.COM
OFFICEFOREDUCATION2013.COM
OFFICEPREVIEW.COM
PLANNINGFORSHAREPOINT.COM
SHAREPOINT-ACADEMY.COM
SHAREPOINT-DESIGN.COM
SHAREPOINT-FOUNDATION.COM
SHAREPOINT-U.COM
SHAREPOINT-U.NET
SHAREPOINT-U.ORG
SHAREPOINT-UNIVERSITY.COM
SHAREPOINT-UNIVERSITY.NET
SHAREPOINT2010RIBBON.COM
SHAREPOINT2010SOCIALFEATURES.COM
SHAREPOINT2010WORKFLOWDEVELOPMENT.COM
SHAREPOINT2013DEVELOPMENT.COM
SHAREPOINTADMINS.COM
SHAREPOINTBOOKS.COM
SHAREPOINTBUSINESSCONNECTIVITYSERVICES.COM
SHAREPOINTCERTIFICATION.COM
SHAREPOINTCHALLENGE.COM
SHAREPOINTCHALLENGES.COM
SHAREPOINTDESIGNER2013.COM
SHAREPOINTDESIGNERTRAINING.COM
SHAREPOINTFORDUMMIES.COM
SHAREPOINTFORUMS.COM
SHAREPOINTFOUNDATION.NET
SHAREPOINTFOUNDATION2013.COM
SHAREPOINTFOUNDATIONADMINISTRATION.COM
SHAREPOINTFOUNDATIONDEVELOPER.COM
SHAREPOINTFOUNDATIONDEVELOPMENT.COM
SHAREPOINTFOUNDATIONFORDUMMIES.COM
SHAREPOINTFOUNDATIONPOWERSHELL.COM
SHAREPOINTFOUNDATIONSEARCH.COM
SHAREPOINTFOUNDATIONTRAINING.COM
SHAREPOINTFOUNDATIONWEBPARTS.COM
SHAREPOINTFOUNDATIONWORKFLOW.COM
SHAREPOINTGROUPS.COM
SHAREPOINTGROUPS.NET
SHAREPOINTKNOWLEDGE.NET
SHAREPOINTMIGRATIONS.COM
SHAREPOINTMOBILITY.COM
SHAREPOINTMYSITE.COM
SHAREPOINTPROGRAMMING.COM
SHAREPOINTSOCIALFEATURES.COM
SHAREPOINTTRAINING.ORG
SHAREPOINTTRAININGREVIEW.COM
SHAREPOINTTUTORIALS.ORG
SHAREPOINTUPDATES.COM
SHAREPOINTVISIOWORKFLOWS.COM
SHAREPOINTVISUALSTUDIO2010WORKFLOW.COM
SHAREPOINTVISUALSTUDIOWORKFLOWS.COM
SHAREPOINTWEBPART.NET
SHAREPOINTWHITEPAPER.COM
SHAREPOINTWHITEPAPERS.COM
SHAREPOINTWORKFLOWS.COM
SHAREPOINTWORKFLOWTRAINING.COM
SILVERLIGHTINSHAREPOINT.COM
SP2013.NET
UPGRADESHAREPOINT.COM
VISUALSTUDIO2010SHAREPOINT.COM
WHATSNEWINSP2013.COM
WORKFLOWINSHAREPOINT.COM

Well, what are you waiting for? Fill in the comment field below or send me an email to request the domain you want! I don’t have all day, you know.

Pin It