So, I’ve been hanging out a bit in the SharePoint subreddit and I was asked what companies do if they need custom forms and cannot do custom development.

The normal answer would be to use InfoPath, but let’s face it, even with a very sympathetic sales rep, you’re still looking at thousands of dollars in license fees alone, not to mention that you’d still need to develop, test, and maintain those forms.

There are some third party alternatives like Nintex Forms, but it is still going to be thousands of dollars worth of licenses. The free and open-source alternatives I’ve seen haven’t impressed me so far.

I’ve made it a goal for 2013 to get developers to think more about the user experience of their solutions. The default user experience, especially when working with form data, leaves a lot to be desired, which is probably why InfoPath was so popular.

Your Wet Dream Come True!

Building custom forms in SharePoint shouldn’t be hard, though. In my SPInvoice solution, I have a form that looks like an invoice, with support for adding invoice rows and all, and it took me less than a day to get it running, and that included having to learn a bit about SPServices and JavaScript.

The method used in SPInvoice, though, isn’t complicated even if the results are stunning. In fact, give me a few minutes of your time to show you this video, and you’ll see what I mean.

In SPInvoice, I’m using far more complex techniques to accomplish some goals that may not be required for most forms. The underlying technique of getting the form to work, however, doesn’t need custom WSP development at all. It does require HTML, a bit of JavaScript including jQuery and SPServices, and you’ll need to setup a content type and a list to hold the submitted form data.

All of the techniques shown here rely on SharePoint Designer and a bit of know-how only; no custom development, no external dependencies, no programming required (unless you count copy-pasting some jQuery code programming).

For any PDF form, for example, or Word, or other applications that produce forms, you should be able to convert those forms into HTML using their respective applications and then tweak the HTML code to make it work with this method.

In fact, I’ll claim that with a bit of HTML5 know-how, you could easily replace a lot of the InfoPath functionality, and have the benefit of saving thousands of dollars and be able to put your forms on the public web if you so desire, even on a completely separate web server running your favorite brand of OS. With that, you can add nifty jQuery UI features or other web frameworks to your heart’s content. No longer are you limited by what the damned SharePoint Designer Design View can or cannot do.

Want to know the best part? This works in all version of SharePoint, from 2007 through 2013, because, well, it’s just HTML and JavaScript.

No stupid App framework to get in your way either… You can host your forms anywhere, set up SPServices to connect to your SharePoint backend, and viola, you have separated your web front end from the limitations of SharePoint.

Who said you couldn’t do SharePoint 2007 development on Linux?

Here’s the video, feel very free to post comments, questions, and so on below.

Oh, and here’s the sample code, but I must stress that this code is far from optimal so use it to learn, not in production.

.b

I’m in a bit of a sad place right now because I’ve realized some of my fears are coming true. Since February 2012, I’ve been following SharePoint 2013 with great interest. I’ve had great hopes that SharePoint 2013 would turn out to be the one version to rule them all; that would bring peace to the collaborative world and unify all the business platform kingdoms under one benevolent ruler.

I’ll break the news to you right now. That isn’t what SharePoint 2013 has done or will do.

And that’s a good thing because if SharePoint 2013 had succeeded, it would have killed SharePoint.

Why SharePoint 2013 Will Fail

Despite the massive hype around the new versions of virtually everything Microsoft, SharePoint 2013 has failed to live up to the expectations in a vast number of areas. The Apps platform hasn’t caught on, the move to cloud first isn’t comfortable for a lot of businesses and is riddled with issues, and the community doesn’t see the major benefits that the new version offers.

Combine this with the uncertainty of features, with SharePoint Online changing and taking choice away from users (You will use Yammer or you will be in pain! We’ve killed what you loved in SharePoint 2013 social because we can! We’ll roll out new features to your employees any day we damn well please!), and quite frankly, a bit of a roadmap nightmare for SharePoint developers in all tiers, and SharePoint 2013 has ended up in a very bad place.

These issues come on top of the issues that any new version of software has, with undiscovered bugs, new features or ways of working that users need to understand, lack of vendor support, and so on. These problems aren’t unique in any way to SharePoint, but they add to the burden that SharePoint 2013 must carry and hopefully resolve.

Microsoft isn’t helping with the situation, but looks like they treat this as some sort of experiment. As SharePoint users start to love and use the new and improved social features, Microsoft announces that it will take them away and replace them with Yammer, forcibly with SharePoint online shortly and likely at least in the next version of SharePoint on-premises too.

Developers and vendors are left bewildered. When SharePoint 2010 came out, we were asked to rush to the sandbox development method, and a lot of vendors and professionals went down that path. Three years later and Microsoft announces that it was all just a joke and that sandbox development really isn’t cool at all.

This time, we’re going to rush to the promise of HTML+JavaScript Apps, which apparently is a new thing. We should do all our development in Apps now, despite Apps technically not offering anything new that cannot already be done in earlier versions or even in SharePoint 2013 without the bonds of the App engine.

Am I the only one that has a trust issue with these kinds of reversals and promises about the glorious future of a new development method? Look at how Microsoft is handling the Windows 8 Start button and boot-to-desktop situation; they can’t seem to make up their minds either way.

Further, and this may or may not be a good thing, Microsoft has left thousands of SharePoint Designer users in the dark by removing Design view from the program. These users have no incentive to upgrade or push their organizations to upgrade. Staying with SharePoint 2010 means they still have a job; upgrading to SharePoint 2013 means they need to retrain significantly, and with training budgets apparently at a all-time low, they’ll probably need to pay for that training themselves.

Ask yourself, would you really sacrifice evenings and weekends for the next maybe six months to learn something completely new that offers no real benefit to you, or would you prefer to just stay with what you already know and hope the issues goes away in the future? I commend those that choose the former, but fully understand those that simply don’t see learning SharePoint as a goal in their lives.

At the same time, many businesses realize that the investments they have made in SharePoint 2010 based solutions still work very well! SharePoint 2013 doesn’t offer any significant improvements in solving business problems, and that’s what businesses want. Technically, sure, SharePoint 2013 may have benefits, but Mark in HR doesn’t really care because his vacation tracker solution works just the same today as tomorrow. It may very well break if he upgrades to 2013, though, and in the very best of situations, he’ll have to learn how to do his work with a new interface.

I’ve previously written that I believe SharePoint 2010 may be the Windows XP of SharePoint, a platform so good that it becomes virtually impossible to get users to upgrade. I’m even more convinced now.

Why SharePoint Will Succeed

You may think this is all doom and gloom, but it’s actually quite the opposite. You see, SharePoint is still alive and kicking, and it’s kicking hard. No, it won’t be the Facebook of the enterprise, and no, it won’t compete or should even try to compete with Google Docs. It won’t be the App platform (and I mean App in the 2013 sense, not in the generic sense) that Microsoft wants.

What SharePoint will be, though, and this is why it will succeed, is an efficient money-making platform for organizations. That’s right, SharePoint gives organizations money, and organizations like that, just as much as they dislike money being taken away from them.

SharePoint’s main advantage is its ability to solve people’s problems. For organizations, this can be as simple as replacing antiquated file servers and providing better search, but it can be as complex as you can imagine.

SharePoint does this with an efficiency that no other platforms can match. In some of the private presentations I do, I’ve set up vacation tracking for HR departments, I’ve set up time sheet reporting with HTML app access for road warriors, I’ve set up CRM systems that track and suggest sales opportunities, and I’ve usually done this to a fairly complete concept within the space of roughly 75 minutes. If I have 90 minutes, I usually end up adding some workflow automation too, to get every single jaw in the room down to floor level.

This is where SharePoint truly shines; not in its ability to be everything else, but in its ability to actually help people with the problems they have, right now, that are causing them pain every day. Throw in a couple of days of getting used to working with SharePoint, and people start solving their own problems, without the need for external consultants. Sure, they’ll more often than not fall on their faces, but they’ll learn, they’ll evolve, and they’ll be happier, with fewer problems than they had without SharePoint.

In more complex scenarios, SharePoint developers can come in and provide guidance. If those those SharePoint developers know how to tie their own shoelaces, they’ll know how to build solutions that take changing needs into account so that the solution can grow and adapt with the organization. They’ll protect the vital parts of a critical solution and allow flexibility of experimentation in more casual applications. That’s not because those developers are so brilliant, it is because SharePoint has the ability to support all those scenarios and thus support a much wider range of solutions than any other platform.

Notice anything in particular about this description?

It doesn’t mention SharePoint version once.

That’s right, campers, SharePoint isn’t about versions any more than astronomy is about telescopes, to paraphrase the late Edsger Dijkstra. You can do all these things with virtually any version of SharePoint. That is good news for new users because they can jump on the SharePoint 2013 wagon and be perfectly comfortable for years to come, but it’s bad news for Microsoft because existing users have very few or no reason to upgrade.

Just a few months ago, Microsoft stated that 40% of all SharePoint installations were still SharePoint 2007 or older! That’s a statement from its users; we have no need for a new interface, we have a need to solve our problems, and we can do that with our existing version.

I’ve said this before and I don’t mind saying it again: Business problems do not change when Microsoft decides to ship a new version of their software.

.b

In a previous article called SharePoint Workflows Aren’t Business Processes, I talked about why it is important to separate the idea of a SharePoint workflow from the business process that workflow supports. It seems to have resonated, but again, my students come to the rescue to point out where I need to elaborate further.

The question one of my students asked was how they could design their business entities or content types to support complete disconnection from the supporting workflow, and I had to think for a bit to come up with a good answer.

The Scenario

Note: I’ve changed the scenario to avoid disclosing details from the student scenario, but the idea and outline remains the same.

In our scenario, we’re looking at an order process with three stages. Order reception, order processing, and order completion. We want to design a system that can maintain this process without tying the process into the orders or the workflow into the process.

Our order entities can be very simple, and to simplify even further, I’ll even drop some of the necessary details, such as the customer relationship.

We need to maintain the following data:

• Order Number (arbitrary identifier)
• Order Date (Date/Time)
• Order Sum (Currency)
• Order Status (Received, Processing, Complete)
• Received By (Person receiving the order)
• Packaged By (Person packaging the order)
• Sent By (Person shipping the order)

Again, this is simplified from a more complex business requirement, but should be sufficient to illustrate our approach.

Designing the Entities

Our first step is to normalize our data and come up with the business entities required to support this process.

Note: Data normalization is the process of structuring our data in a way that removes duplicate data and preserves data integrity.

You may be tempted in this scenario to use a workflow to represent the order process. By doing so, you can use the workflow status to represent the order status. SharePoint will even throw in a workflow status column in your default view when you start a new workflow.

Neat, eh?

No, not really.

You see, by doing that, we hard link the workflow we build into the order entity, and that order entity and its future now depends on that workflow operation like we think it will operate.

“Fine,” you say, “but I can just create a new workflow if I need to change the process radically”

If you do that, then you’ll break fundamental principles of virtually any development discipline. Your order will have two columns, one representing the old workflow and one representing the new. At the very least, this will confuse your users, but at worst, you create conflicting information. One of the workflows may say “Complete” while the other says “Processing”. Which is correct? You’ll have to tell your users, or hide the wrong workflow from sight.

“I’ll live with that,” you say, “we’re just a small company so telling the five people handling orders how to do this correctly is no issue”.

Well, you still have the issue of data history. If you remove a workflow that holds business data, you are essentially killing the history of those orders. Any orders process from that point in time will have a status but any orders processed prior to your new workflow will not.

Instead, we need to disconnect the workflow completely from the data.

First Stage of Separation

The first approach many take is to simply allow the workflow to update the Order Status. In a SharePoint Designer workflow, this is as simple as setting the value of a field in the current item.

The benefit of this approach is that you get rid of the direct relationship between the workflow status and the order status. If the workflow changes or goes away, you still maintain historical data. If you add a new workflow later, that new workflow can simply update the Order Status field and we’ve disassociated our workflow from our data.

This approach, while certainly better than trying the workflow directly to the Order Status field, is still not enough, though. Our goal isn’t just to separate the workflow from the data, we also want to separate out workflow from the process.

At this stage of separation, there is no distinction between the business process and the workflow. We still suffer from the same problems if our business process changes; we need to completely redesign our workflow to add a third intermediate order stage, for example.

What’s worse, if we cannot support in SharePoint workflows the requirements of the business process, we’re stuck. We either have to change the business process or add manual steps, both of which are very sub-optimal from the business perspective.

Finally, even though we do separate the workflow from the business data, we may still end up with business process information mixed in with the business data.

Let’s say that the order process requires an approval of some sort, maybe even a few approval stages (order approved, shipping verified, etc). In order to store such approval states outside the workflow, we need to add a column to the business data to store the state.

However, by adding one or more approval state columns, we have essentially polluted our business data. Whether an order is approved is irrelevant to the order itself; the approval is part of the process, not the order.

If we later decide we no longer need the shipping verification, for example, we now have a ton of orders containing irrelevant shipping verification data.

If we didn’t think that we needed a shipping verification in the beginning but later decided to introduce it, well, all orders prior to that decision will not have the correct shipping verification values and our reports will not be correct.

This certainly isn’t convenient in a flexible process where we want to ensure that our workflows adapt to changes in the business process.

Second Stage of Separation

The answer is to introduce a new stage of separation.

We’ve determined that we need to separate the business process from the workflow, and the business data from the process. Neither of these links are beneficial from the perspective of ever-changing business processes.

To accomplish this, we need to create an entirely new entity; the order process entity.

This entity will represent the order process, regardless of how the business data looks and regardless of what our workflow does.

In fact, this layer of separation allows us to have multiple and independent workflows, even from different automation technologies. We can combine SharePoint Designer workflows with event receivers with SharePoint 2013 type workflows with manual processing.

The business process entity will be responsible for working with the business data by updating information as required. Our workflows will never touch that business data at all, except to read information relevant to the workflow.

In our relatively simple order process, our order entity may look something like this:

• Order ID (Lookup or reference to an order identifier)
• Order Process Stage (Received, Processing, Complete)
• Order Approval State (Approved, Declined, Pending)

The order process entity will be the only gateway to update or modify our orders. Our workflow will update information in the order process, not in the orders.

This stage of separation requires a bit more work, though, because we need to get the business process to update the business data. However, this is a one-time operation that allows us much more flexibility in how we later work with our business process, so the tradeoff is definitely worth it.

By isolating our access through such an entity, we accomplish several goals.

1. We avoid storing superfluous data in the order (such as ever-changing states)
2. We have a single interface with which we can modify order data, accessible by any method or technology
3. We ensure that changes in the order data does not break our workflows

This approach is very common in regular software development, where developers are accustomed to building separate business layers and data layers, and access the data through the business layer only. Generally, it is considered a much safer approach as it prevents error in our interfaces from ruining our business data.

Note: Of course, this statement is only true if our business logic layer does not break the data

Further, this approach allows us much more freedom in terms of controlling our process. Let’s say we have one order that falls somehow outside the regular process for example if a customer picks up the shipment directly. In this case, an employee needs to manually change the process for this order only.

If we maintain a hard workflow-to-data model, in which the state of the order is stored in the order, we need to modify the order and possibly break the workflow currently running on that order.

With the two-stage separation model, however, the user can update the process entity without fear of breaking the underlying data or other processes that run.

Of course, this doesn’t apply to manual updates only. Perhaps we want to change our workflow engine to utilize the new SharePoint 2013 engine or we’ve purchased an external engine like Nintex Workflow.

Well, guess what, as long as our new workflow operates on the business process entity only, that’s perfectly fine. We can swap out parts of the business automation system without killing data or ruining reports.

Finally, this separation also allows us to change the business process without ruining data. This happens because there aren’t any links between the business data and the business process either; the process is simply making changes to the data but has no other ties to that data.

If we change or even remove completely the business process, that’s fine, our underlying data remains unchanged.

Conclusion

I hate writing these, but I’d like to summarize a few points.

Separation of data, business logic, and interface is a key principle in modern software development. This model explains how to accomplish this in SharePoint workflows as well, by creating a complete disconnection between the software we use to manipulate data (the workflow), the business logic or process, and the underlying data.

For years, I’ve been trying to get novice and professional SharePoint developers to understand that software development principles apply to SharePoint as well. Equally, bad software development introduces the same risks in SharePoint as it does on any other platform.

The multi-tier model described here will be well-known to experienced software developers, for example those familiar with MVC patterns. However, for many SharePoint developers, who have little or no other software development background, this may seem like a lot of extra hassle without any tangible benefits.

If you think this, however, I would like to take this opportunity to call you an idiot.

.b