Elastic analytics concepts

What do we mean by elastic analytics? Let's define it as designing your analytics processes so that scale is not a concern. You want your focus to be on the analytics and not on the underlying technology. You want to avoid constraining your analytics capability so it will fit within some set hardware limitations. Focus instead on the potential value of your analytics versus the limit of what can be done with existing hardware.

You also want your analytics to be able to scale. It should go from supporting 100 IoT devices to 1 million IoT devices without requiring any fundamental changes. All that should happen is that the costs increase as demand increases.

This reduces complexity and increases maintainability. This translates into lower costs, which enables you to do more analytics. More analytics increases the probability of finding value. Finding more value enables even more analytics. Virtuous circle!

Some core elastic analytics concepts:

• Separate compute from storage: We are used to thinking about resources as we think about laptop specifications. You buy one device that has 16 GB memory and 500 GB hard drive because you think that will meet 90% of your needs, and it is the top of your budget. Cloud infrastructure abstracts this away. Doing analytics in the cloud is like renting a magic laptop where you can change 4 GB memory into 16 GB by snapping your fingers. Your rental bill increases for only the time you have it at 16 GB. You snap your fingers again and drop it back down to 4 GB to save some money. Your hard drive can grow and shrink independently of the memory specification. You are not stuck having to choose a good balance between them. You can match compute needs with requirements.
• Build for scale from the start: Use software, services, and programming code that can scale from 1 to 1 million without changes. Each analytic process you put in production has continuing maintenance efforts that will build up over time as you add more and more processes. Make it easy on yourself later on. You do not want to have to stop what you are doing to re-architect a process you built a year ago because it hit the limits of scale.
• Make your bottleneck wetware not hardware: By wetware, we mean brain power. My laptop doesn't have enough memory to run the job should never be the problem. It should always be I haven't figured it out yet, but I have several possibilities in test as we speak.
• Manage to a spend budget, not to available hardware: Use as many cloud resources as you need as long as it fits within your spend budget. There is no need to limit analytics to fit within a set number of servers when you run analytics in the cloud. The traditional enterprise architecture purchases hardware ahead of time, which incurs a capital expense. Your finance guy does not (usually) like capital expense. You should not like it either, as it means a ceiling has just been set on what you can do (at least in the near term). Managing to spend means keeping an eye on costs, not on resource limitations. Expand when needed and make sure to contract quickly to keep costs down.
• Experiment, experiment, and experiment: Create resources, try things out, and kill them off if they do not work. Then, try something else. Iterate to the right answer. Scale out resources to run experiments. Stretch when you need to. Bring it back down when you are done.

If elastic analytics is done correctly, you will find your biggest limitations are time and wetware, not hardware and capital.