Open Source Community, Here I Come

This won’t be a very long post but just wanted to follow up and let everyone know that I will be speaking at the All Things Open conference in Raleigh.  The news came as a total shock to be honest.  Now the incredible task in front of me is to pull together a presentation that is worthy of the honor.  I’ve certainly been rolling thoughts around for the topic for many months but now I have to pull it all together.

Wish me luck and if you are so inclined please come out and support me.  I’ll need all I can get.

-Andy

Taking a new approach, engaging the Open Source Community

It’s no secret that I am constantly frustrated at what I feel is the lack of drive or community from within the manufacturing space to do new and open things. With that in mind I’m starting to approach this problem in a different way. Elon Musk knew it would be impossible to change the Auto Industry or the Space Exploration/Rocket industry from the inside. Holy cow, just stop and think about that for a second. Anyway, because of this issue he decided to change it by doing something from the outside. When his solution/product exceeded what was available from traditional vendors he got traction. Now, at least in the public’s eye, he is the darling of the automotive and space exploration world. When it comes to high speed city to city transport he’s not trying to design a new locomotive engine or optimize steel tracks. He’s coming at it from a totally different angle.

With that in mind I am going to try to approach this open source thing from a new angle also, from the outside.  My starting point will be attending the All Things Open Conference in Raleigh during the month of October.  I learned about this conference from a new friend, whom I’ve never actually met, Dan Thyer at Logical Advantage in Charlotte.  Usually the really cool and innovative conferences are somewhere in the Northeast or the West Coast.  This is my chance to get in there and mix it up with some serious heavyweights.  Take a look at the site, scroll down a bit and be amazed.  These are the biggest of the big, and the most powerful people and companies in the market today.

But silly as I am I’m taking it one step further.  I have actually submitted a talk.  The deadline for submissions in April 30 so I’m hoping I’ll learn the fate of my talk in the next month or so.  The working title is “Dear Open Source, Please Help.  Love, Manufacturing”.  The TL DR is that we want developers and leaders from the open source community at large to come over to our house and play, get to know us better.  If they like us maybe they will hang around for a while and help build a cool tree house or something.  There are a lot of brilliant people in manufacturing today.  The problem is that for every 1 Elon Musk we’ve got 1000’s of insular and overprotective companies that don’t see the value of pushing our entire community forward.  The best possible outcome, if by some miracle I am actually selected to speak, is to have a handful of people talk to me after the conference to say “that sounds like fun, where should we start?”.

For me this is much more of a moonshot than Deploy Camp was.  Luckily it won’t take near the up front commitment to see if it actually goes.

I’m not saying I would like a publicity campaign but if you want to mention something on the twitters like, “I would love to hear the aaOpenSource people speak at the All Things Open Conference” I wouldn’t be offended :-).   The twitter handle is @AllThingsOpen and the hashtag is   Just sayin….

Here’s hoping for the best.

-Andy

aaMQTT – A System Platform to MQTT bridge

wsp_mqtt3

I feel like this latest project is a culmination of many months of poking, prodding, and exploring.  Put simply, in about 500 lines of code I have developed a fully functional bridge to allow you to transport data bi-directionally between Wonderware System Platform and any MQTT broker.  With this concept System Platform can now take an imposing seat at the table with respect to platforms for hosting, viewing, and logging IOT data from the billions of devices that are coming our way in the coming years.  Can System Platform handle 500,000 or 1,000,000 points?  The answer is absolutely.  We know Historian 2014R2 can handle over 1 million tags.  And we also know that there have been multiple projects with over 1 million I/O. Continue reading

System Platform was a PAAS before it was cool (Part 3)

This will be the final post in a 3 part series discussing why System Platform was a PAAS long before anyone was even thinking about PAAS.

You can find Part 1 and Part 2 at the associated links.

To quickly recap here are some of the distinguishing features of a modern PAAS

  • Managing the run-time process for logic execution (Part 1)
  • Communication between clients and servers or between servers (Part 1)
  • Service discovery (Part 1)
  • Managing fail overs and redundancy (Part 2)
  • Monitoring of software environments for performance (Part 2)
  • Code deployment
  • Diagnostic logging

So let’s jump into it. Continue reading

System Platform was a PAAS before it was cool (Part 2)

In Part 1 of this series I opened a discussion of how I think System Platform meets many of the classic definitions of what modern IT practitioners call a PAAS (Platform As a Service).

To recap, here are some of the features of a good PAAS

  • Managing the run-time process for logic execution
  • Communication between clients and servers or between servers
  • Service discovery
  • Managing fail overs and redundancy
  • Monitoring of software environments for performance
  • Code deployment
  • Diagnostic logging

We’ve already covered up through Service Discovery in the previous post.

Managing fail overs and redundancy

Honestly it can’t get much easier than the model System Platform has chosen.  If you want a host to participate in redundancy, set up a dedicated network connection on the host and configure a single address on the Platform.  Done.  On your App Engine it’s a single check box to enable redundancy.  After that assign your App Engine to a primary host and a backup host.  Deploy.  Done and Done.  After that fail-overs happen automatically with no need for the user or engineer to lift a finger.  A well managed system will check the check-boxes to enable alarming so they know what’s going on but that’s just gravy on top of the run-time.  The only thing missing is the ability for an App Engine to restart on one of many different hosts.  Currently you must assign the backup to one and only one host.  I’d like the ability to create a cluster that allows the app engines to be deployed and fail-over to any host in the cluster.  Take a look at the clustering and HA techniques employed by VSphere to get a clearer picture of what this might look like.  It would certainly be more expensive in terms of network traffic to communicate the checkpoint to multiple machines but with modern compute and network capabilities I hardly think it would be an issue.  I remember the days when the recommendation was to use a dedicated crossover cable between servers for the RMC.  So glad we are past that.

Monitoring of software environments for performance

Well this is just low hanging fruit for those of us from the System Platform space.  If you want to know what the current load or throughput or timing is for any logical component in the system just start browsing in object viewer and you will be quickly overwhelmed by the diagnostic data available.  What if you want to log this data for longer term study or incident review?  That’s easy enough.  Check a few check-boxes and you are done.  What if you want to raise an alert when a threshold is exceeded?  No problem, a few more check-boxes.  Finally maybe you want some complex logic that considers three different configurable thresholds and a few discrete values before raising an alert.  No problem, create an attribute with alarm and a script, maybe at a template level, and now you have completely customized alerting logic.  I honestly feel bad for people running standard IT platforms who have to setup third party solutions or complex logging systems just to keep a long term record of the machine performance.  It’s just too easy for us in the System Platform world.

I’ll end it here for today and finish up the series tomorrow with a detailed discussion of the final two points.

– andy

System Platform was a PAAS before it was cool

First I will preface the next few articles by pointing out that I am an avid consumer of all things cloud, IAAS, PAAS, and XAAS but by no means am I am expert of heaven forbid, “thought leader”.

When the original System Platform architecture was released way back in the early 2000’s, frankly I’m not sure when because I didn’t get on board until the 2.1 days, it was revolutionary.  It was so revolutionary it has literally taken almost 10-15 years for the general IT space to catch up.  I’m sure if you read my writing and rantings you know that there are plenty of areas where I think the software can be improved but that’s mostly around the engineering experience. In terms of run-time architecture it is simply unsurpassed still to this day.

Continue reading

How do we attract young talent? Be more Open

Just this morning I read an interesting article in Control Magazine, Attracting good young engineering talent takes more than a good pay package.   The basic gist of the article is that you need to provide young engineers with interesting challenges and a company culture that aligns with their values.  I think these are definitely good points and are on the right track.  I would like to add to this list with another item in line with my recent postings, be more Open!

When you left college what office productivity software did you use at your first job?  Microsoft Office of course.  Why?  Because that’s what you used in college.  These days this early indoctrination is reaching down into the elementary school level as they are learning how to create PowerPoint presentations as early as 3rd grade.  Did your first employer make you use something different like Corel Office or some other package?  Did you hate it?  Did you consider leaving the company because they were so far behind the times?

I think one way to get more kids interested in automation and manufacturing in general is to make the software we use more accessible to college students and even primary school students in high school and middle school.  A few years ago I was doing some work on the weekends and my kids saw me drawing some HMI graphics.  We spent the next two hours designing a basic graphic with some scripting, that they helped write, to take a square and move it up, down, left, right, and spin.  They loved it!  By the way, we also spend time doing Lego Mindstorm EV3 which is a great gateway to PLC programming.  The only problem is that as a practical matter they would never get the chance to do something like that because the software I used starts out north of $10K. No student, or young engineer with an interest, will ever get access on their own for exploration and learning purposes.

Yes I know you can call up the companies and distributors and possibly get a free or discounted license if you meet requirements and sign up for academic programs.  The problem is that young students these days don’t want to go through that hassle.  They want to go on to github or some other completely open site.  Download the bits and start working.  Maybe if they make something that produces value, i.e. running code on a machine that actually makes a little money, they will think about getting support and professional services.  Initial exploration, however, is always free and relatively easy.

I feel like this is an incredible hurdle to getting young minds interested in manufacturing and automation.  In the words of the famous New York mayoral candidate a few years ago “the rent is too damn high!”.  The price/barrier to entry for our world is simply too damn high.  Making our tools accessible via an open source approach with supplier value being derived from support and professional services as opposed to ridiculous licensing costs just for installation and exploration is where me must go as as industry.  All you need to do is look at Microsoft.  5 years ago they were the kings of closed source.  Today they are open sourcing technologies as fast as they can.  Why?  Because they are/were getting trounced by open alternatives to their core platforms.  If Microsoft can do it and figure out a revenue model why not our suppliers?

 

-andy

Why I think Manufacturing Software should embrace Open Source

osi_symbolI have rewritten this blog post about  4 times now.  Each time changing my approach or tone all around what is essentially the same commentary.  You can see from the title that I’ve settled on discussing why I think the manufacturing software space should embrace open source technologies.

Quality

Many uninformed decision makers might equate open source with a rag tag bunch of programmers hacking away at some useful tools on their off hours.  While this concept holds true for many projects, there are also many very serious projects with large teams from all of the major players in the software world; Google, Microsoft, Twitter, etc.  These teams are very structured with strong governance, a clear vision, and merciless quality control.  When there is no inherent profit motive to add cruft or push a product out before it is ready the calculus of when and how to release changes dramatically.  A very well respected annual survey (link) routinely polls decision makers on why they choose open source software.  For the last few years the leading reason for choosing open source has been software quality, not cost as some may think.  I have a specific personal experience with a situation where using an open source package would have made a real difference on a project.  I was using a component provided by the software manufacturer but there was a small bug in that the software did not properly recognize when all of the parameters were successfully downloaded.  I spent somewhere around 60-80 hours coding a workaround.  I worked for days with an engineer who was literally checking out code from their code repo and sending me dll’s to try.  We fixed the issue but the next step was to have the fix blessed as an official hotfix before I could install it and be supported.  For whatever reasons, political or technical, this engineer was told that even though he had solved the issue they would not be issuing a hotfix.  This was a core function and yet they would not fix it.  If I was using open source software I could have made the choice myself to recompile the application with the corrected code and run that code in my application.  If I didn’t know how to do that I could have hired a talented developer on a very short term contract to do this for me.  Unfortunately I was never given this choice.  Open source software means every user is a potential developer who can study problems and propose fixes.  All you really need to do is look at the operating system wars.  While Windows certainly has the lion’s share of desktops, when it comes to mission critical systems by and large it is Linux, the flag bearer for open source, that is installed and trusted.  The list of open source projects that are considered good enough for mission critical (from the software sense, not true safety) applications is simply too long to document here.  Consider this.  About 70% of the worlds top million web servers run on Apache or Nginx, two popular open source web servers. (Source)

Value for the Customer

Yes, at it’s core open source software is “free”.  You are “free” to download the source and compile the code for yourself.  Then if you need help there is typically a pretty decent community of fellow enthusiasts or power users who are more than happy to share.  Typically, though, a facility owner won’t feel comfortable depending on Stack Exchange to run his critical business functions.  As an ironic aside that same business owner probably isn’t worried about the 3 person integration company that just put in a custom inventory tracking system last year that is the lynchpin of their operations.  While the software quality may be excellent the concept of depending on just a handful of people to support you is one that should give any responsible person great pause.  Returning to the value concept, most if not all mature organizations that use open source software typically buy a support contract for that software.  You also typically have a choice of who to buy support from.  The difference between this and the traditional software model is that you are paying for services on an ongoing basis instead of making a large investment up front and then having to pay every year just to stay current with the software.  While I am not familiar with all of the different models I could also see one model where you pay a modest retainer type fee to a professional services company and then basically pay by the hour for support.  I have had numerous customers in the past that either simply let their support contract lapse or seriously considered moving to a different software package because of the incredible amount of money, typically in the 15-20% of installed software cost, for sometimes nothing more than an assurance they can call up tech support with a question.  Unfortunately many times the support received is less than stellar.  One particularly funny antecdote involves my time at my previous employer.  An employee from another office called up the front line tech support with an issue they were having.  The first words out of the tech support person’s mouth were “Have you talked to Andy Robinson about this?”.  While this is funny it represents a fundamental truth that for many software companies, tech support is simply a line item cost deducting from overall profits.  While the first line of defense will always be writing quality software there  will always be a class of users who burden the technical support organization with simple questions that really could be handled at the community level instead of having to pay thousands of dollars for remedial training.  The models for supporting open source software as a commercial product are very well established with companies like Red Hat, from my hometown of Raleigh, leading the way. You can purchase Red Hat Enterprise Linux (RHEL) which is considered ultra stable suitable for any application or you can choose to work on the leading edge distribution known as Fedora.

Value for the Integrator

I feel like this is one of the more unique angles as to why open source is good for the manufacturing software world.  Having been in the integration business for a while I can give you a pretty rough breakdown of a typical project with engineering, hardware, and software.  Typically the price to the customer is about 50% engineering with about 15% contingency.  Hardware and Software are usually split down the middle 50/50 for the rest with about 10% or so markup.  So take a roughly $100,000 project.  You charge the customer 50K for engineering services.  In a well run organization you will aim to price your services at about 3X the cost of an engineer.  1x pays the engineer.  1x pays overhead and burden (insurance, benefits, etc), and 1x goes to profit.  So by that math say we make about 22K on a project.  17K from services and 5K from markup on the materials.  Honestly you rarely do that well b/c someone always overruns hours or under specifies hardware, but you get the idea.  Now imagine if we used open source software and we didn’t have to pay 25K for the software.  What if you split the difference and took 12K for additional profit and dropped your price to the customer by 13K.  We can now charge the customer less and improve our profits by almost 70%!  Those are economics that benefit everyone.  To make things even better the integrator doesn’t have to charge the customer up front for support costs passed through from the supplier.

Ecosystem

NPM (node.js) 138,807

Nuget(.Net) 34,801

GoDoc(go) 63,202

Those are module counts for each respective language in the most common repository users will search for functions and code to solve issues with their application.  Some are simple utility functions and some are massive application frameworks.  By comparison I suspect if you added up all of the available community codesets for DeltaV, Rockwell, and Wonderware you may be lucky to break 250.  And most of those are really old and not maintained.  The ecosystems are what make a programmer so productive with these languages.  Today if you have a particular problem you are looking to solve with your manufacturing software your only hope is that someone in your company has already solved it or your local distributor has some dusty utility laying around and they are willing to share because you are a good customer.  Outside of those two options you are probably solving a problem that 30 other people have solved in isolation before you.  No wonder innovation is so slow in our industry.  I suspect we spend 30% of our time solving problems that 30 other people have solved before us.  This was and still is a driving factor behind aaOpenSource.

Talent

My final consideration of why the manufacturing software space should embrace open source technologies is talent and more specifically recruiting that talent.  If you look to the big tech companies today; Google, Facebook, Twitter, and even Microsoft you will find a prominent role in their recruiting efforts involves promoting their participation in open source projects.  It is a simple fact that today’s most talented developers want to work on open source projects.  Maybe it starts as a labor of love but there are now very real and very lucrative career paths working on open source projects for large and small firms.  It is not uncommon to see a small open source project with 3 or 4 developers land multi-million dollar VC funding to continue their open source project.  That’s a lot of cold beer and pizza.  It is also a painful truth that the top technical minds these days have a declining interest in going into manufacturing as a career.  Working at Netflix on the Chaos Monkey or at Joyent on Node.JS is much more exciting and satisfying.  Finally you can’t discount the publicity effect.  When you work for a large closed source organization you can talk about the code you wrote and how good it was but realistically you can’t show this to any prospective employer.  By contrast, every commit to a project on Github displays your name prominently.  Many software companies are just as interested in your contributions on Github as they are your credentials and certifications.  You can tell a lot about a developer by studying how they write code.  If manufacturing software companies continue to cling to and archaic closed source model they will assuredly lose out on the best talent.  And when they lose, the end customer loses as well.

My original versions of this post took a much harsher and adversarial tone.  In this post I am saying all of the same things but I hope it is presented in a more helpful and constructive framework.  What are your thoughts?  Any other benefits you see from bringing manufacturing software into the open source world?

Comments and complaints welcome.

–  andy

We need more composers, not luthiers

composerdownload

 

Over the course of the past month I have been consuming everything I can find with respect to IOT and the imminent wave that’s about to transform all of us in ways we simply can not imagine.  In many talks I have given over the last year I have harped on a very specific concept that some call the network effect.  I think Scott Mcnealy of Sun Microsystems coined the term or the idea many years ago.

It is most simply presented in the form of a math equation:

If(A, B, and C are all > 1
then A x B x C > A+B+C

Put simply if everything has a little value we can derive more value from multiplying these things together instead of simply adding them together.  Considering the concept of a network this means that if A can talk to B and C, and B can talk to A and C, and C can talk to A and B then we have dramatically increased the value of A, B, and C together as opposed to each of them presenting value on their own by a simple summation of the parts. lutheieriStock_000016864591XSmall

Just today I saw a Tedx Toronto talk with a speaker who started with a discussion of a smart blanket the monitored your sleep patterns.

So that’s an interesting application and not exactly revolutionary.  There are a number of sleep aides out there today.  The blanket can watch your sleep patterns, a very well understood field of science, and optimize the best time to wake you up.  But when should it wake you?  What if it knew you mad a meeting at 9 am at the office.  And it also knew that the weather was bad today so that means it takes on average 7.3 minutes more to get to the office.  To top if off looking at traffic feeds we see there is an accident so now we can calculate that it will take 12.2 minutes more than usual.  But that’s right now, what about in 1 hour when I leave the house.  Maybe another cloud based analytics platform can crunch the data and tell us that based on 15 other factors you should leave the house around 8:11 to make our 9 am meeting.  The problem is that the blanket itself probably won’t have that intelligence.  But a smart orchestration platform certainly could perform the composite or mashup.  So that’s great we know that we need to leave the house at 8:11 and we know that it takes us an hour to get ready so set off that alarm at 7:11.  But the alarm is startling and not very nice.  What if we had control over the curtains or the opacity of our curtains to slowly let in the sunlight to wake us up in a way that humans are supposed to wake up, with sunlight, not a loud alarm.

The speaker goes on to extend the analogy but I feel like the key takeaway is this.  The technology to perform all of these device level functions is either here now or not very far off.  The pace of innovation in hardware has far outstripped our ability to innovate at the software layer.  If you want a vivid example of this check out a product i just backed on Kickstarter from Phasor systems.  The different applications with this $20 piece of hardware are simply amazing and I get anxious thinking about all of the totally obvious use cases that I haven’t even considered.

So what does this have to do with  composers and luthiers?  For reference, a luthier is someone who makes stringed instruments.  I think we can draw a simple analogy between software/hardware stacks and the pieces that come together to make a symphony.

Instrument Maker –> Musician –> Composer
Hardware Maker –> Software Writer –> Orchestration architect

All three layers are absolutely essential but most people are moved in a much more dramatic fashion when they here a beautifully conducted orchestra as opposed to simply hearing each of the musicians as individuals.  Have you ever heard all of the individual pieces of a major arrangement separately?  On their own they are typically quite nice and technical but typically don’t have the same soul stirring power as when you hear all of brass, and percussion, and woodwinds roar in perfect time and harmony.

One of the founding fathers of MQTT, Arlen Nipper, discusses in a TedX Talk from 2012 about how IOT is just getting started but we need app developers to realize the value.  He briefly mentions at about the 11:30 mark how many times in his career he has been starting up a system with a customer and they have that aha moment.  “So if I know X and I can send it to Y then I can do Z.  This can fundamentally change my whole business!”

This really hit home with me as I have lived this life for the last 10 years.  Almost weekly in discussions with customers I would see the light come on when I say “yeah, we can collect that data for you and send it over here”.  To them that was so much more valuable than my ability to control the tank level +-2%.  While I would always try to understand their business and what brought true value for them it was simply impossible to really know any one market segment or unique value proposition too deeply.  Every now and then I would provide some unique insight but most of the time the customer always knew what they could do if they had the data, they just never thought it was feasible, or never developed the idea because they assumed it wasn’t possible.

I think you are going to see me return to this theme many times in coming blog posts.  All of the pieces are in place right now.  Cheap computing, cheap networking, well defined protocols.  The only limiting factor at this point is our imagination.  What a great time to be an engineer!  Or just a thinker.

-andy

Why I think Open Source software will eat the Industrial SCADA

This is the first of what I think will be 2 posts regarding open source software and Industrial SCADA systems.  In these posts I will address, at a very high level, the how and the why.

First, the how.

I have been involved in the industrial automation world going back to my days as a co-op sitting at a Provox console watching FST’s run on a funda filter cleaning up washwater for million dollar resin beds.  That was 1998.  My first year or so of permanent employment revolved more around process engineering but after I moved to our facility in Houston I was all in on the controls world.  Starting up a $350MM new facility on what was at the time the largest DeltaV installation in the world was an amazing opportunity that I only really appreciated years later.  Fast forward a few years and I left that operating company to join a wonderful integrator in North Carolina, Avid Solutions.  While at Avid I had the opportunity to work on systems as diverse as DeltaV, ABB Bailey Net 90, Rockwell, and Wonderware, just to name the top few.  While a majority of my time was spent with Wonderware software I think I have a pretty good basis to judge a wider range of offerings.

One thing you learn or accept very quickly is that setting up industrial automation software systems is just hard.  If you have every tried to setup a Factory Talk environment you know exactly what I mean.  Wonderware System Platform is better but you need to get your SQL Server installed and then configure the web server if you want to use Information Server, don’t forget your special network accounts and make sure you aren’t running a DC on any platform because….. My problem was that after a while I got so good at it, everything seemed simple and easy.  At some point you convince yourself that this is really complex stuff because it has to be so reliable and runs such critical functions.  If it was too easy well that meant it wasn’t really reliable and ready to use in “my” environment.  Maybe the kiddos down the street can play with it but it’s not good enough for me. People from outside industrial automation certainly couldn’t understand what it takes to do what we do.

I had always known about famous open source projects like Linux and Apache.  I even setup and played with a few Linux VM’s.  The problem was that it was always way too complicated to even get started.  And oh goodness running an Apache webserver.  I tried one time and fell on my face hard.  I liken it to a race car driver in an F1 car. You know the capabilities of the car are out of this world.  But if you take an average person and put them in a F1 car I suspect you will quickly form an opinion that the car is unwieldy, difficult to control, and doesn’t perform very well.  However, put a professional driver behind the wheel and you quickly change your opinion.  You realize that it takes serious expertise to harness the power and technology.  I always thought that about open source software.  It may be really powerful but unless you are an expert it is simply not worth the effort.

But then something, or some things, changed in my mindset.  If I had to pin it on one thing I think it might be the discovery of node.js in late 2013.  For the unitiated, node.js is basically a way you can run javascript, yes javascript, as a server side process to serve has the backend for an application.   Most things will talk HTTP to it but you can just as easily have a conversation in TCP if you like.  Here is a link to the main node.js site with an introduction.   Then you see it.  If I want to run a web server and serve up a page the hello world is this.

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(1337, '127.0.0.1');
console.log('Server running at http://127.0.0.1:1337/');

To run the server you simply type this at your command line

node example.js

That’s it. Go to your web browser and go the URL and you see a web page that simply says “Hello World”.  That’s impossible.  It can’t be that easy. Don’t I have to run a huge web server like IIS or some other complex project with some JAVA or some other resource hog.  Nope.  Install Node with click click next and roll.  So the boo-birds will chime in and say “well that’s neat and all but that’ s not for serious production use.  It’s just scripting.  How reliable can that be?  I’ll put it succintly.  The back end systems supporting Wal-Mart Mobile are all written in Node.JS.  Yes, I said that correctly.  Here is a nice article from Tech Crunch about the staggering scale of traffic Wal-Mart mobile handled this past year.  Do you think your requirements can even sniff the roundoff errors in the kind of load and speed these guys are handling?  If you want a deeper dive from 2013 here is a great NodeUp episode with the team from Wal-Mart labs.  http://nodeup.com/fiftysix

Ok, so there is some open source technology out there that could support might needs in terms of a run-time environment but the protocols and applications are just so complex.  I thought the same.  Until I kept sniffing around and discovered MQTT.  What is MQTT you ask?  It is a standard protocol developed by IBM many years ago specifically for an oil pipeline project to convey sensor data over high latency, low bandwidth network.  There is plenty to read about at MQTT.org it but basically think OPC but designed for a much lighter weight implementation and instead of having client-server you have client-broker with a publish-subscribe model.  I won’t go into the details here but if you do some brief reading you can very quickly understand how a data source (like a sensor or a PLC) can publish data to the broker who then in turn sends that data out to all clients who have subscribed to it.  The concepts are fairly basic and fairly universal across message buses in general.  This is amazingly similar to what most if not all SCADA systems operate.  The SCADA I/O server can be thought of as the broker but it also acts as a client to many devices in the field.  The only problem is that if you want to get data in our out of any SCADA system you have two options.  Use their proprietary software tools for visualization, alarm and event handling, and historization or use an OPC server/bridge to get data in and out.  The final option, if you SCADA offers it, is to write a custom package using toolkits from the vendor to read and write the data.  We can do this with the MXAccess toolkit for Wonderware System platform.  It’s fairly easy to use and works pretty well.  But all the work and you can only access data from Wonderware System platform.

So this MQTT things sounds like crazy complicated with all the brokers and publishers and subscribers.. well hang on.

Step 1.  Get yourself a broker.  Thing of this as the grand central station for your data.  All data goes up through the broker.  The most popular one is mosquitto.  It must be complex to run right?  Nope.  Download from here and run a single command

mosquitto.exe

Good, now you have a broker running.

Step 2.  Write a client that connects to the broker and send/receives data.  Because most of my life is in C# I’ll post the relevant snippets.

// Setup your client  MqttClient 
client = new MqttClient(IPAddress.Parse("192.168.10.53")); 
// Send Data to the broker to anyone else can see it 
client.Publish("sensor/temp", Encoding.UTF8.GetBytes(temp); 
// Setup a subscription to receive data with 
// specific quality of service guarantees

string[] topic = { "sensor/temp", "sensor/humidity" };

byte[] qosLevels = { MqttMsgBase.QOS_LEVEL_AT_MOST_ONCE, MqttMsgBase.QOS_LEVEL_AT_MOST_ONCE };
byte[] grantedQos = client.Subscribe(topic, qosLevels);
//Finally setup to handle any data you receive from your subscriptions
client.MqttMsgPublishReceived += client_MqttMsgPublishReceived;
void client_MqttMsgPublishReceived(object sender, MqttMsgPublishEventArgs e)
{
// access data bytes through e.Message
}

In literally less than 20 lines of code you have the foundation for a complete pub/sub model with guaranteed delivery of messages.

OK, so we can move data around in an efficient manner, but what about visualization?  I have one word..ok maybe acronym.. for you, HTML5.  With a combination of a javascript client library for MQTT and HTML5 canvas there is essentially no limit to what can be accomplished for visualizations.  Taking a look at existing libraries like D3 and you get a pretty good idea that we aren’t talking about your mom and dad’s ugly dial gauges anymore.  And D3 is only one library out of many that are available today and in use by many thousand around the world.

So I hear all of this and I’m still skeptical because any good SCADA needs a historian.  And in industrial automation, well, we have lots of data and it’s moving really really fast.  We all know off the shelf SQL servers databases are resource hogs, very expensive, and can’t really handle my situation.  For the skeptics, I give to you.. InfluxDB.  They call InfluxDB an application that is designed for time series data.  In other words… a historian.  It can handle non time-series data, like logs, alarms, and events, but their differentiation is the tooling around time series data.  Do yourself a favor and spend 15 minutes looking over their site.  Tell me that aside from the data ingest what does it not do that your current historian is doing for you?  99% of what you do today is just graph some points and do some occasional min/max/average summaries.  So that seems fine but I’m sure it can’t scale to what I need it for.  I’ve got like 50,000 tags and about 1/4 of them change every few seconds.  While I can’t comment specifically about performance as I have not tested it directly, take a look at this write up from the middle of last year discussing their performance testing for underlying datastores.  After you read that you start to get a small sense for what kind of scale they intend for this solution to operate at.  Also a post from the lead developer in late 2013 indicated that they were seeing somewhere around 20K to 70K points per second on a single node.  And one of the more recent posts indicates they are look at the 2M points PER SECOND type of performance for a cluster.

So what have I done here?  I have just laid out the foundational elements for a 100% open source SCADA system using standards based communication methods.  Yes, OPC is a standard.  But I ask you to go off and write a client or a server.  You will very quickly realize it is really complex and at some point you will probably be ponying up some cash to the OPC foundation for something.  Doesn’t sound terribly open an inviting to me.  A standard way to communicate, yes.  Open, no.

So what’s missing, why hasn’t anyone done this yet? First, we are still missing the tools to make it configuration and not programming.  While those who work on control systems like to call themselves (me included) programmers we are really configurers.  We configure scripts and code inside of a pre-built environment.  We don’t worry about scheduling loops, multitasking, authentication mechanisms etc.  We point, click, and import csv files.  No company or person has built the tooling necessary to make this underlying tech accessible to configurers.  If you want a great case study on tools making all the difference go read up on Docker.  Basically think of a lightweight VM inside a VM.  At it’s core all Docker has done is create a nicer interface to some underlying tech, namely LXC, that has been around for a while.  I think right now we are at the pre-Docker stage.  The technology is in place, we just need tools to configure, not program.

So what has changed or is changing to make me thing this could happen.  One loaded acronym, IOT.  I used an expression a while back that went something like this; “the barbarians are at the gate and they are bringing their fancy development tools and source control with them”.  What I meant by this is that the rest of the world.. and it’s a really big world.. has all of a sudden gotten the bug for connecting their physical world to the digital world.  Don’t worry I wont, and really can’t, wax poetic about what all this means in a big picture.  But on a small scale the people who only cared about lines and pictures on a screen before are looking over at us and saying hey, I can read a temperature over TCP/IP and it’s really easy.. I can do something with that.  I think I’ll make a Nest thermostat and sell it for $3B+ to Google.  You want a good summary of how I feel right now with respect to all of this technology.  Watch this video about people basically recreating a Nest thermostat for almost nothing.  Someone is going to find an opportunity and seize it.

I feel incredibly strongly that we are doing to see a massive influx of new interested parties gather around the walls of our little fortresses of solitude.  We’ve told outsiders for years they are welcome to come on in, but they have to do things they way we’ve always done them.  No wonder the pace of innovation in our industry sucks.  Yes we run our systems on Windows instead of VAX and the IDE’s are a little better but at the end of the day we really aren’t that far along as compared to the rest of the software world.  Again, go read up on the histories of node.js and Docker.  They literally did not exist a few years ago and now they are deployed on a massive scale across thousands of applications, many of them mission critical; again see Wal-Mart.  Please don’t try to convince me that a technology has to be around and in use for 5-10 years before it’s “good enough”.

There will always be applications where new technologies are not welcome and sometimes for good reason.  Nuclear is the extreme example.  I was chatting with a project manager for a very large multi-national automation company here in Taiwan about a year ago.  At the time there was great debate about whether or not to build a 4th nuclear power plant.  Basically what he was telling me was that if the plant was actually built due to timelines and technology lockins they would probably startup a 10 year old version of the DCS in the beginning.  This is an extreme corner case.  The vast majority of our applications don’t have the sort of redundancy and fail-safe requirements that prevent us from using even remotely modern tech.

So that’s the “how”.  There are lots of holes in my argument.  I accept that.  Instead of showing you a finished product instead I am showing you just a few examples of foundational elements.  What I have shown above is by no means the best or even a complete solution.  Instead I want to inspire you to look at these pieces and go find others.  The open source world is moving so fast that 6 months from now the best solution for a particular piece might be a completely different application that is not on anyone’s radar right now.

In my next post I will discuss the “why” of open source for Industrial SCADA.