Introduction

In general, designing and creating a computer system is a balancing act. We're constantly trying to add features and capabilities while keeping the code simple and the system's performance reasonable. In this respect, data analysis systems are no different. In fact, they may be worse. Often data is only partially consistent, and we need to employ a variety of strategies to extract usable data before we can even begin its analysis. Each added strategy adds a little more complexity, weight, and bloat to the code, each of which makes it a little harder to maintain. This can get out of hand.

Clojure has a number of libraries to help us manage our systems' complexity. One of the most powerful of these is concurrent programming. This allows us to conceptualize our programs differently and in ways that can help manage the complexity. Instead of having monolithic blocks of code that do many things and have direct, tight dependencies, we can structure our program more modularly by composing many independent modules together, each of them doing one thing. These communicate using simple, well-defined protocols, but they all work independently and concurrently (that is, at the same time).

Clojure also has a concurrent message processing system (its agents) built on top of the STM. Agents contain state information and we send them function messages to update that state concurrently. Together, the STM and agents provide a way to structure programs to make them maintainable and easy to understand.

Both of these work well because all native Clojure data structures are immutable. They cannot be changed. Because it's working with immutable data, the STM can provide guarantees about the consistency and safety of its transactions, even in a highly concurrent environment. These guarantees are good for us because they help us think and reason about our data and our program, and they help us manage the complexity of the systems we're building.

Note that concurrent describes how a program is structured to work that will hopefully result in some speedup. Each thread may be doing different things, but concurrency is often just a good way to organize your program. It separates out and decouples the different parts of your program that are engaged in different tasks but still are coordinating with each other or interacting in some way. If you're doing the same thing over and over, where everything is independent of all the other things, and you want to do it faster, that's parallelism. We'll look at recipes related to about that in Chapter 4, Improving Performance with Parallel Programming.