Why Most Big Data Projects Fail, and How to Make Yours Succeed

Big data is on the minds of just about everyone, with IT departments large and small grappling with exponentially growing volumes of both structured and unstructured data. But, despite big data’s place as a mainstream IT phenomenon, the bulk of big data projects still fail, as organizations struggle to find ways to capture, manage, make sense of, and ultimately, derive value from their data and information. Taming big data and being able to get the business insight you need is a daunting task all by itself; but, if everyone isn’t on the same page when it comes to defining the scope of the project, and all the right pieces aren’t in place, the project is destined to fail.

These are what I see as the main causes of failure for big data projects:
· Lack of alignment. Business and IT groups are not aligned on the business problem they need to solve, but, instead, are tackling it from a technology perspective. Lack of true commitment from business stakeholders also makes alignment harder to achieve.
· Lack of access. Access to data often is restrictive, and team members don’t have access to the data sets they need to find answers that will make the project successful.
· Lack of knowledge. Many of the technologies, approaches and disciplines around big data are new, so people lack the knowledge about how to actually work with the data and accomplish a business result.

Of all of these pitfalls, the first – lack of alignment on the business questions you’re trying to answer – is the most important. The whole idea is that you are exploring and searching for what you don’t know; so, to achieve success, it’s critical to define the project in terms of exactly what the business is trying to accomplish, and what questions need to be answered. Although it’s the most important factor in the success of a big data project, alignment is challenging to achieve. Not only does big data mean different things to different people, but, a host of external factors can influence changes in business requirements and priorities faster than IT can keep up. If IT and the business aren’t aligned on the scope of the project, it can shoot off in too many directions, involving too many people, and shifting the focus from answering specific business questions to managing the technology needed to achieve everybody’s piece of the pie.

Another challenging impact on business/IT alignment comes from unwillingness to change. Too often, if a big data project suggests an action or change that feels too foreign to business stakeholders, they may be reticent to accept it, dismissing it as a faulty process, analysis or dataset. In response, analyst groups may steer results for a future project in a direction they think the business will agree to and act upon, resulting in recommended actions that may create a sub-optimal business outcome.

The second reason big data projects fail – lack of access to data – goes back to a fundamental IT premise: silos. There are data silos for sales, marketing, HR and others, each restricted and guarded to meet compliance. There are good reasons why data silos exist but, if the data you need is not available to you, you’re limited before you even start to solve the problem. To get past that, big data projects have to start with executive-level ownership. Without all the relevant data from the business, it’s impossible to see the relationships and patterns that will answer the business questions, so buy-in has to come from the top. Somebody at the top has to say, “This team is looking to solve a specific business issue and is important enough to have access to any data they need.” Without access to the right data, the project will be at a standstill. Period.

The third pitfall – lack of knowledge – is about having the right people who have the right skills to execute the project. You need people who understand machine learning and natural language processing to name just two, but, because this technology is so new to the ‘mainstream’, IT teams frequently lack those who understand how to use it for analysis purposes. While hiring a data scientist is one possibility for addressing this knowledge shortfall, it’s not feasible for many organizations. This new role combines the mindsets of scientist and investigative researcher with the skills of a programmer, but it’s an expensive position to fill and the required skill set is neither commonly found, nor easy to create.

How to Succeed in Your Big Data Project
Consider a practical approach. To start with, don’t call it a “Big Data Project.” Call it something like, “a project to learn more about our customers and why they shop in a particular store.” The project answers important business questions, and big data is the source for the answers. Here are some best practices to help your project realize a successful outcome:
· Start with a list of questions and business problems you want to answer
Don’t tackle something huge. Start with a small project, make it about a specific issue you need to address, and stick with it. Make a list of the questions you need to answer, and don’t lose sight of your objective by getting stuck on making the technology work. Keep your team from becoming too broad or all-encompassing, so you can avoid scope creep and its project failure button: a constantly changing set of requirements flowing from the business to IT. Make sure all stakeholders agree on the objective, and keep everyone focused on driving it to completion.
· Get endorsement from the top before you start
Once you have identified the business problem you want to solve, the business team must have endorsement from the top down to get access to all the data needed to complete the project successfully. Get company leadership on board with granting the team access to all relevant business data, so they can find the patterns and relationships that will answer the business questions. They must be given access – controlled, of course – but with authority.
· Make sure your team has the knowledge necessary to execute the project
Ideally, you will have someone on your team who understands machine learning, has the skills and mindset of a data scientist, and can actually work with the data to produce the needed business result. But, if not, you may be able to solve the problem with your existing system. This is a good time to again step back and consider the business questions you need to answer. You may be able to get the answers you need without machine learning or NLP, just by granting access to the right people inside the organization.
Choose a problem that creates value for the business, have the fortitude to stick with it, and you’re already on the right path. And remember, a successful project is so regardless of its scope. Don’t set yourself up to fail by biting off a chunk that’s too big. After all, it’s better to succeed in a small project, than to fail over and over on a grand scale.

Darin Bartik is executive director of product management for Dell Software’s Information Management solutions.