Organisations slow to wake up to changing data integration demands

A quarter of organisations are already hitting the functional limits of their existing data integration platforms, but only one in ten expect to be using Hadoop as their primary enterprise data management platform in 12 months’ time. Such slow progress by plateauing organisations points to a lack of confidence in identifying alternatives to traditional tools – but those organisations risk falling behind the competition as they fail to maximise the value of their data, according to Nejde Manuelian, Director of EMEA Operations at Syncsort.

Based on experiences with thousands of customers around the world, Syncsort has identified five stages in the “Big Data Continuum”, a yardstick against which organisations can measure the maturity of their data integration strategies.

29 per cent of organisations questioned as part of Syncsort’s EMEA Big Data and Hadoop Trends survey are languishing in the initial “Data Awakening” stage, primarily still hand-coding using SQL, JCL, or basic ETL tools. The same proportion are identified as “Advanced Traditional”, having standardised on a data integration platform, while a further 23 per cent are at “Plateau”, experiencing architectural challenges, spiralling costs, and diminishing returns as their current tools hit the limits of their capabilities. 17 per cent are “Dynamic”, prototyping and experimenting with Hadoop, and only 2 per cent are currently at the “Evolved” stage, using Hadoop as their enterprise data management platform.

While significant numbers expect to move out of the Data Awakening phase over the next 12 months – only 12 per cent of organisations expect to be there this time next year – overall progress along the Big Data Continuum is not anticipated to be swift. Only 11 per cent expect to be at the Evolved stage in 12 months’ time, and the figures for Advanced Traditional and Plateauing show little change, at 29 per cent and 20 per cent respectively.
“The slow pace at which organisations seem to be moving along the Big Data Continuum is almost certainly a product of their uncertainty and lack of confidence when it comes to identifying alternative tools and knowing how to get the best out of them,” said Manuelian. “This is backed up by the fact that 29 per cent of organisations anticipate they will be at only an experimental stage with Hadoop this time next year – there is a lot of work to be done, both on the platform itself and in terms of educating organisations, before the majority will feel comfortable taking the plunge.”

Smaller organisations (those with fewer than 500 employees) were more polarised at each end of the Continuum: Almost half (47 per cent) are still in the Data Awakening phase now, but one in five (19 per cent) expect to be in the Evolved category within 12 months.

“It is not surprising that smaller organisations are lagging behind their larger counterparts – they have fewer and more thinly spread resources, they often do not have the same level of in-house IT expertise, and they need to be absolutely confident in the viability of any change to their working practices,” Manuelian continued.

“But, equally, it is perhaps not a surprise that these smaller organisations are projected to exceed the overall figure for the Evolved phase one year on – they generally have to take more innovative steps to earn and safeguard competitive advantage, and have the flexibility to do so, unlike many larger organisations,” he concluded.