Flexible Consistency

Flexible ConsistencyDuring last week’s PASS Summit day 2 keynote, Rimma Nehme discussed some off the architecture behind Cosmos DB, the globally-distributed flexible database system rolled out earlier this year. Dr. Nehme’s presentations are almost always buzz-worthy, with a great mix of technical depth and big-picture vision. Take 90 minutes to watch the recording; you won’t be disappointed.

Among the many notable points in this presentation was a journey through the concept of database consistency. Database consistency, also referred to as transactional consistency, is the attribute of database systems describing the coupling of different parts of a single transaction.

The simplest example of this is the data tasks that go into a banking transfer. In the database, money doesn’t “move” from one account to another; rather, the amount of money in one account is decreased by adding a credit line item, and the same amount is added to the other account by inserting a debit line item. Although these are two separate operations, they should be treated as one, with the whole operation succeeding or failing as one transaction. In a transactionally consistent system, the two parts of the transaction would not be visible unless both of them were complete (so you wouldn’t be able to query the accounts at the moment of transfer to see that money had been removed from one but not yet added to the other).

During Dr. Nehme’s keynote, she pointed out that most of us look at database consistency as a binary choice: either you have full transactional consistency or you don’t. However, data workloads aren’t always that simple. In Cosmos DB, there are actually five different consistency models to choose from.

Flexible Consistency

For those of us who have spent most of our careers working with ACID-compliant operations, having this kind of flexibility in database consistency requires a shift in thinking. There are many business cases where such transactional consistency is not critical, and the rise of NoSQL-type systems has demonstrated that, for some workloads, transactional consistency is not worth the latency it requires.

Some of us old-school data folks (this guy included) have in the past poked fun at other consistency models, using the term “eventually consistent” as a punch line rather than a real consideration. However, it’s no longer safe to assume that every database system must have absolute transactional consistency. As workloads become larger and more widely distributed, the use of less-restrictive consistency settings will be essential for both performance and availability.

This certainly doesn’t mean that transactionally consistent systems are going away. It just means that the classic RDBMS way of maintaining consistency is no longer the only way to do things.

Cloud Adoption Concerns (And How To Overcome Them)

Cloud Adoption ConcernsCloud technology has become very pervasive, with 89 percent of organizations globally identifying it as part of their IT strategies while another 75 percent say they have already implemented at least one cloud application. Some of the key benefits enjoyed by organizations that adopt cloud technology include lower costs, faster time-to-market, improved reliability, and better business intelligence insights, among others.

In spite of the rise in usage, there is still a significant amount of concern and push-back for any potential new cloud-based solution. These include security concerns, integration challenges, industry standards compliance issues, information governance, and difficulty in measuring the real economic value of cloud solutions. In many cases, however, organizations often think these challenges are much bigger than they really are.

Cloud Adoption Concerns (And How To Overcome Them)

Here are five of the most common cloud adoption challenges and how to overcome them:

1. Security Concerns

Problem: Although security is becoming less of a concern for cloud adoption, there are still those who believe that data stored in the cloud is inherently less secure than on-prem data storage.

Solution: Your approach to security for your cloud workloads should not be that different from how you approach security for your on-premises workloads. Develop overarching security policies that are based on your particular needs within your organization, then use these as yardsticks to evaluate different cloud solution providers. Major cloud providers such as Azure and AWS invest heavily in securing client data, and thus the level of security on their platforms tends to be significantly better than what a small or mid-sized organization can develop internally.

2. Integration with Existing Systems

Problem: It’s rare to migrate all of an organization’s infrastructure to the cloud at once, so there is naturally some concern over how migrated cloud applications will integrate with existing on-prem systems.

Solution: While cloud vendors were initially known for pushing an “all-cloud” approach, the industry has shifted to now support and adopt a hybrid on-prem/cloud model as the default architecture. As part of that shift, the major cloud vendors have made the process of integrating from on-prem to cloud (and cloud to cloud) much easier, through APIs, specialized integration frameworks, and hybrid authentication architectures.

3. Cost Concerns

Problem: Making a long-term commitment for services (rather than one-time hardware purchases) means that we’re locked in to a monthly subscription fee. It’s harder to predict what we’ll spend.

Solution: Organizations need to understand that transition and implementation costs are not unique to the cloud. You need to treat your cloud implementation just like you would a normal IT project regardless of whether it’s SaaS, IaaS or PaaS.

Remember that running your own data center isn’t a one-time capital investment. Servers and supporting equipment have ongoing costs as well, including electricity, cooling, physical security, and labor costs to install, maintain, and upgrade the hardware. Cloud models typically involve smaller one-time capital expenditures than on-premises models, but this model also allows you to scale up your architecture needs (and costs) as your business to grows. The cloud pricing model means you’re paying what you need today, without overbuying to anticipate what you might need tomorrow.

4. Concerns About Loss of Control

Problem: Organizations with on-prem systems often feel they are giving up too much control to cloud providers, surrendering the ability to define granular settings for system resources, networking, and security.

Solution: The cloud isn’t just one thing. One of the best analogies of the different cloud offerings is the pizza-as-a-service model. You can make your own pizza at home, buy a take-and-bake pizza, have a pie delivered, or go out to eat at a pizza place. These different services, much like cloud architectures, allow you to pick the level of service that meets your needs. For workloads that require the organization to have more granular control over how the application runs, the IaaS model makes it possible to migrate to a virtual machine in the cloud without giving up the ability to configure the machine environment, storage, etc. For other workloads requiring less hands-on configuration, using a managed PaaS model makes more sense.

5. Legal and Compliance Concerns

Problem: A common challenge to cloud adoption is that it may not meet compliance or regulatory demands.

Solution: The major cloud vendors are certified for use for most any regulated domain of data. Further, most vendors allow you to specify the geographic region(s) in which your application and data will be stored, thus eliminating the concern of violating geographic boundaries of compliance or regulation.

Conclusion

With cloud architecture becoming more and more common, most of the classic objections to migrating to the cloud no longer hold water. Security, integration, and compliance used to be blockers to cloud adoption, but the platforms have matured to the point that these are no longer barriers.

[POSTPONED] Event: Is Your Data-Driven Organization Ready for Azure?

Ready for Azure?

UPDATE 8/29/2017:

Due to the ongoing weather situation in Houston and its likely long-term impact, we have decided to postpone this event. Our thoughts go out to our friends in Houston and the surrounding areas who have been impacted by Hurricane Harvey.

 

Is your organization exploring a move to the cloud? If so, you’ll be interested in this half-day event we are cohosting in Houston in September.

Along with our friends at Straight Path Solutions and Opifex Solutions, we’ll walk you through the benefits of using Azure for data storage and computing. In this seminar, we’ll help you to answer the following questions:

  • Are we ready to move to the cloud?
  • Which of our workloads is suitable for the cloud?
  • Why choose Azure over the other cloud providers?
  • How can I integrate on-premises data with cloud data and applications?

This seminar is ideally suited for data architects, managers, and CxOs who are cloud-curious and are interested in learning how Azure can help save time and money.

Admittance to this event is free, but does require advance registration. To reserve your seat, visit https://azurehouston2017.eventbrite.com. We look forward to seeing you there!

Benefits of Cloud-Based Database Applications

Depositphotos_26627079_x_thumbI still remember the week I realized that the cloud would most likely live up to its hype. It was a couple of years ago at the PASS Summit, and Microsoft had shifted their marketing message from being all-in on the cloud to a more pragmatic hybrid approach. In many of my offline interactions that week, I heard story after story of successes in moving some workloads to the cloud, resulting in significant cost savings and greater flexibility. That was the week that turned this skeptic into a believer that the cloud is the real deal, and it’s here to stay.

Although not every application is a perfect fit for cloud deployment, the fact is that cloud-based services are rapidly becoming the standard for new development. Platform as a Service (PaaS) vendors are also making it very easy to move existing on-premises applications to the cloud through myriad tools for data and metadata migration.

Benefits of Cloud-Based Database Applications

As the PaaS offerings from vendors such as Amazon and Microsoft have matured over the last few years, the benefits of moving workloads to the cloud have become clearer.

Decreased cost. Let’s say you decide to run your own standalone Microsoft SQL Server machine. Assuming you’re buying modest hardware, you’re going to be out $5,000 for the box, not to mention the periphery (network equipment, UPS units, etc.) required to set up a server and keep it connected and running. Then there’s SQL Server licensing, which for Standard Edition only was around $4,000 per core as of this writing. Assuming your modest server is a quad-core machine, you’re at $16,000 just for SQL Server licensing. If you plan to have redundant hardware to account for high availability or disaster recovery, expect to double (or more) these fixed costs. Also, keep in mind that servers don’t live forever; plan to repeat this purchase every 3-5 years. In contrast, the recurring cost model of PaaS applications reduces the steep capital expenditure into a more finely tuned pay-as-you-go economy, often resulting in lower total cost.

Time to market. Think about the things that will slow down your development or deployment timeline. It takes time – sometimes weeks or months – just to get approval for a capital expenditure for the hardware and software. After that, the new machine has to be built by the manufacturer, delivered, and set up in your data center. Install the operating system, configure it on your network, and then you can finally install SQL Server. In the absolute best case, this is a weeks-long process; in reality, going from zero to on-premises SQL Server will take a month, perhaps much longer. Building a PaaS database can be completed in minutes rather than weeks or months; in fact, earlier today I built a SQL Database in Azure in about the same length of time it took me to type out this paragraph. If time is a factor (and really, when is it not?), the speed at which a new cloud database can be created is a very compelling argument for using a PaaS architecture.

Managed services. If you’re hosting your own database server for a 24×7 application, your organization is on the hook for handling things like operating system hiccups, hardware failures, network snafus, and power outages. Keeping resources and staff to handle these issues takes time, training, and most importantly, money. Add to all of that the ongoing environmental costs of running a data center: cooling, power, backup power apparatus, and physical security, among others. By using a cloud platform to develop and run your database applications, you’ll shift the responsibility of data center management over to your cloud vendor, saving a great deal of effort and money. You’ll also rest a bit easier, since software updates and service SLAs will also be handled by the PaaS host.

Easy scalability. With your on-premises database server, what happens in six months when you discover that the hardware can no longer keep up with demand? You could upgrade the server hardware (assuming it’s got headroom for that) or just buy a new larger server. Similarly, I’ve seen a few cases of buyer’s remorse where the needs were overstated and the server hardware was much more robust (and therefore, more expensive) than it needed to be. Overbuying or underbuying hardware is always costly. Conversely, when building a database application in the cloud, it is relatively easy to set – and later change, if needed – the resources allocated to it. As the processing and storage needs change, the resources can be scaled up or down in seconds with just a few mouse clicks.

Conclusion

Platform as a Service offerings have evolved and improved significantly in the past couple of years. These cloud services make a strong argument for cost savings, time, and flexibility for both new development and migrated applications. As the volume of customers migrating to the cloud continues to increase, prices will continue to fall, furthering the cost benefit advantage of cloud-based applications over their on-premises counterparts.

Data Warehouse: On-Premises or Cloud?

I’ve been fielding this question a lot these days: “We’re building a data warehouse – should we build it here or in the cloud?” It’s a fair question, but it’s not the question that should be asked. The more appropriate question is this: “What part of our data warehouse solution should be in the cloud, and how does it work together with our on-premises data?”

Data Warehouse: On-Premises or Cloud?I shared a few of my thoughts on this topic a few weeks ago in a podcast interview with Carlos Chacon, when we discussed whether or not the on-premises data warehouse was dead. Without spoiling all of the details of that conversation, my short answer is that the on-premises data warehouse is alive and well but is no longer the only DW option.

As recently as three years ago, the cloud was still relatively new and not yet widely in use in most organizations. At the same time, companies selling cloud services were in the midst of a massive marketing effort to direct customers to the cloud. Microsoft famously declared themselves to be all-in on cloud well before the market was ready to follow. Many IT leaders and technologists bristled at the thought of being forced into the cloud at the expense of tried-and-true on-premises solutions.

However, in the past couple of years the message from cloud providers has softened. No more is it “cloud or bust”. Rather, cloud services companies – and Microsoft in particular – have reshaped the message to one in which the cloud is just one piece of a heterogeneous architecture that may include on-prem, PaaS, IaaS, and SaaS solutions. At the same time, consumers are realizing the value of cloud-based solutions for some of their architecture. Although I rarely have a client that wants to build an all-cloud infrastructure, most everyone I work with is at least exploring if not actively using cloud services for a portion of their data systems.

Cloud services are here to stay. No, the cloud absolutely will not take over on-premises data storage and processing. Rather, cloud offerings will be one more option for managing data and the code around it. So the question is not whether you should be in the cloud – the answer is yes (or it soon will be). The more practical question is how to best leverage cloud services as part of a hybrid strategy to minimize development time and total cost of ownership.

 

This post originally appeared in the Data Geek Newsletter.