What are the challenges and Application in Big Data
1. The Uncertainty of Data Management:
One disruptive facet of big data management is the use of a wide range of innovative data management tools and frameworks whose designs are dedicated to supporting operational and analytical processing. The NoSQL (not only SQL) frameworks are used that differentiate it from traditional relational database management systems and are also largely designed to fulfill performance demands of big data applications such as managing a large amount of data and quick response times.
2. Talent Gap in Big Data:
3. Getting Data into Big Data Structure:
4. Syncing Across Data Sources:
5. Extracting Information from the Data in Big Data Integration:
6. Miscellaneous Challenges:
7. Dealing with Data Growth The most obvious challenge associated with big data is simply storing and analyzing all that information.
8. Generating Insights in a Timely Manner
Of course, organizations don’t just want to store their big data they want to use that big data to achieve business goals. According to the New Vantage Partners survey, the most common goals associated with big data projects included the following: Decreasing expenses through operational cost efficiencies Establishing a data-driven culture Creating new avenues for innovation and disruption Accelerating the speed with which new capabilities and services are deployed Launching new product and service offerings
All of those goals can help organizations become more competitive but only if they can extract insights from their big data and then act on those insights quickly. PwC’s Global Data and Analytics Survey 2016 found, “Everyone wants decision-making to be faster, especially in banking, insurance, and healthcare.”
To achieve that speed, some organizations are looking for a new generation of ETL and analytics tools that dramatically reduce the time it takes to generate reports. They are investing in software with real-time analytics capabilities that allow them to respond to developments in the marketplace immediately.
9. Recruiting and Retaining Big Data Talent
But in order to develop, manage, and run those applications that generate insights, organizations need professionals with big data skills. That has driven up demand for big data experts and big data salaries have increased dramatically as a result. The 2017 Robert Half Technology Salary Guide reported that big data engineers were earning between $135,000 and $196,000 on average, while data scientist salaries ranged from $116,000 to $163, 500. Even business intelligence analysts were very well paid, making $118,000 to $138,750 per year.
In order to deal with talent shortages, organizations have a couple of options. First, many are increasing their budgets and their recruitment and retention efforts. Second, they are offering more training opportunities to their current staff members in an attempt to develop the talent they need from within. Third, many organizations are looking to technology.
They are buying analytics solutions with self-service and/or machine learning capabilities. Designed to be used by professionals without a data science degree, these tools may help organizations achieve their big data goals even if they do not have a lot of big data experts on staff.
10. Integrating Disparate Data Sources
The variety associated with big data leads to challenges in data integration. Big data comes from a lot of different places enterprise applications, social media streams, email systems, employee-created documents, etc. Combining all that data and reconciling it so that it can be used to create reports can be incredibly difficult. Vendors offer a variety of ETL and data integration tools designed to make the process easier, but many enterprises say that they have not solved the data integration problem yet.
In response, many enterprises are turning to new technology solutions. In the IDG report, 89 percent of those surveyed said that their companies planned to invest in new big data tools in the next 12 to 18 months. When asked which kind of tools they were planning to purchase, integration technology was second on the list, behind data analytics software.
11. Validating Data
Closely related to the idea of data integration is the idea of data validation. Often organizations are getting similar pieces of data from different systems, and the data in those different systems doesn’t always agree. For example, the ecommerce system may show daily sales at a certain level while the Enterprise Resource Planning (ERP) system has a slightly different number. A hospital’s Electronic Health Record (EHR) system may have one address for a patient, while a partner pharmacy has a different address on record.
The process of getting those records to agree, as well as making sure the records are accurate, usable, and secure, is called data governance. In the At Scale 2016 Big Data Maturity Survey, the fastest-growing area of concern cited by respondents was data governance.
Solving data governance challenges is very complex and usually requires a combination of policy changes and technology. Organizations often set up a group of people to oversee data governance and write a set of policies and procedures. They may also invest in data management solutions designed to simplify data governance and help ensure the accuracy of big data stores and the insights derived from them.
12. Securing Big Data
Security is also a big concern for organizations with big data stores. After all, some big data stores can be attractive targets for hackers or Advanced Persistent Threats (APTs).
However, most organizations seem to believe that their existing data security methods are sufficient for their big data needs as well. In the IDG survey, less than half of those surveyed (39 percent) said that they were using additional security measures for their big data repositories or analyses. Among those who do use additional measures, the most popular include identity and access control (59 percent), data encryption (52 percent), and data segregation (42 percent).
13. Organizational Resistance It is not only the technological aspects of big data that can be challenging people can be an issue too.
In the New Vantage Partners survey, 85.5 percent of those surveyed said that their firms were committed to creating a data-driven culture, but only 37.1 percent said they had been successful with those efforts. When asked about the impediments to that culture shift, respondents pointed to three big obstacles within their organizations: (I) Insufficient organizational alignment (4.6 percent) (II) Lack of middle management adoption and understanding (41.0 percent) (III) Business resistance or lack of understanding (41.0 percent)
In order for organizations to capitalize on the opportunities offered by big data, they are going to have to do some things differently. And that sort of change can be tremendously difficult for large organizations.