Big Data for Governance - Implications for Policy, Practice and Research

A recent IDC forecast shows that the Big Data technology and services market will grow at a 26.4% compound annual growth rate to $41.5 billion through 2018, or about six times the growth rate of the overall information technology market. Additionally, by 2020 IDC believes that line of business buyers will help drive analytics beyond its historical sweet spot of relational (performance management) to the double-digit growth rates of real-time intelligence and exploration/discovery of the unstructured worlds.

This predicted growth is expected to have significant impact on all organizations, be it small, medium or large, which include exchanges, banks, brokers, insurers, data vendors and technology and services suppliers. This also extends beyond the organization with the increasing focus on rules and regulations designed to protect a firm’s employees, customers and shareholders as well as the economic wellbeing of the state in which the organization resides. This pervasive use and commercialization of big data analytical technologies is likey to have far reaching implications in meeting regulatory obligations and governance related activities. 

Certain disruptive technologies such as complex event processing (CEP) engines, machine learning, and predictive analytics using emerging big-data technologies such as Hadoop, in-memory, or NoSQL illustrate a trend in how firms are approaching technology selection to meet regulatory compliance requirements. A distinguishing factor between big data analytics and regular analytics is the performative nature of Big Data and how it goes beyond merely representing the world but actively shapes it.

Analytics and Performativity

Regulators are staying on top of the big data tools and technologies and are leveraging the tools and technologies to search through the vast amount of organizational data both structured and unstructured to prove a negative. This forces the organizations to use the latest and most effective forms of analytics and thus avoid regulatory sanctions and stay compliant.  Analytical outputs may provide a basis for strategic decision making by regulators, who may refine and adapt regulatory obligations accordingly and then require firms to use related forms of analytics to test for compliance. Compliance analytics are not simply reporting on practices but also shaping them through accelerated decision making changing strategic planning from a long term top down exercise to a bottom up reflexive exercise. Due to the 'automation bias' or the underlying privileged nature of the visualization algorithms, compliance analytics may not be neutral in the data and information they provide and the responses they elicit.

Technologies which implement surveillance and monitoring capabilities may also create self-disciplined behaviours through a pervasive suspicion that individuals are being currently observed or may have to account for their actions in the future. The complexity and heterogeneity of underlying data and related analytics provides a further layer of technical complexity to banking matters and so adds further opacity to understanding controls, behaviours and misdeeds. 

 Design decisions are embedded within technologies shaped by underlying analytics and further underpinned by data. Thus, changes to part of the systems may cause a cascading effect on the outcome. Data accuracy may also act to unduly influence outcomes. This underscores the need to understand big data analytics at the level of micro practice and from the bottom up. 

Information Control and Privacy

The collection and storage of Big Data, raises concerns over privacy. In some cases, the uses of Big Data can run afoul of existing privacy laws. In all cases, organizations risk backlash from customers and others who object to how their personal data is collected and used. This can present a challenge for organizations seeking to tap into Big Data’s extraordinary potential, especially in industries with rigorous privacy laws such as financial services and healthcare. Some wonder if these laws, which were not developed with Big Data in mind, sufficiently address both privacy concerns and the need to access large quantities of data to reach the full potential of the new technologies.

The challenges to privacy arise because technologies collect so much data and analyze them so efficiently that it is possible to learn far more than most people had predicted or can predict . These challenges are compounded by limitations on traditional technologies used to protect privacy. The degree of awareness and control can determine information privacy concerns; however, the degree may depend on personal privacy risk tolerance. In order to be perceived as being ethical, an organization must ensure that individuals are aware that their data is being collected, and they have control of how their data is used. As data privacy regulations impose increasing levels of administration and sanctions, we expect policy makers at the global level to be placed under increased pressure to mitigate regulatory conflicts and multijurisdictional tensions between data privacy and financial services’ regulations.

Technologies such as social media or cloud computing facilitate data sharing across borders, yet legislative frameworks are moving in the opposite direction towards greater controls designed to prevent movement of data under the banner of protecting privacy. This creates a tension which could be somewhat mediated through policy makers’ deeper understanding of data and analytics at a more micro level and thereby appreciate how technical architectures and analytics are entangled with laws and regulations. 

The imminent introduction of data protection laws will further require organizations to account for how they manage information, requiring much more responsibility from data controllers. Firms are likely to be required to understand the privacy impact of new projects and correspondingly assess and document perceived levels of intrusiveness. 

Implementing an Information Governance Strategy

The believability of analytical results when there is limited visibility into trustworthiness of the data sources is one of the foremost concern that an end user will have.  A common challenge associated with adoption of any new technology is walking the fine line between speculative application development, assessing pilot projects as successful, and transitioning those successful pilots into the mainstream. The enormous speeds and amount of data processed with Big Data technologies can cause the slightest discrepancy between expectation and performance to exacerbate quality issues. This may be further compounded by Metadata complications when conceiving of definitions for unstructured and semi-structured data.  

This necessitates the organizations to work towards developing an enterprise wide information governance strategy with related policies. The governance strategy shall encompass continued development & maturation of processes and tools for data quality assurance, data standardization, and data cleansing. The management of meta-data and its preservation, so that it can be evidenced to regulators and courts, should lso be considered when formulating strategies and tactics. The policies should be high-level enough to be relevant across the organization while allowing each function to interpret them according to their own circumstances. 

Outside of regulations expressly for Big Data, lifecycle management concerns for Big Data are fairly similar to those for conventional data. One of the biggest differences, of course, is in providing needed resources for data storage considering the rate at which the data grows. Different departments will have various lengths of time in which they will need access to data, which factors into how long data is kept. Lifecycle principles are inherently related to data quality issues as well, since such data is only truly accurate once it has been cleaned and tested for quality. As with conventional data, lifecycle management for Big Data is also industry specific and must adhere to external regulations as such.

Security issues must be part of an Information Governance strategy whichwill require current awareness of regulatory and legal data securityobligations so that a data security approach can be developed based on repeatable and defensible best practices. 

Common & Practical Problems of Requirements Elicitation

Requirement elicitation is an important and challenging phase of any software project. This holds good for both product and project development activities, but the approach, techniques might vary. A well specified requirement has been found to considerably improve success rates of projects. Though various methods and techniques have evolved over the last couple of decades to better produce a good requirements specification, many struggle to get it done well.

This could be mainly because that requirement elicitation is just not science, it is an art too. It is more an art because it is highly human intensive and much depends on the skills of the people involved in the process. More so, as which method or technique to use and the way the document is structured and written depends on the abilities of the person driving this activity. Based on my experience in the be-spoken project development and product development activities, I have listed down some of the most common and practical problems with this activity as below:

1. Preconceived Notions

The requirements of every customer even in the same business domain, would be different. For example, requirement of a bank X would not be the same as that of bank Y. Each enterprise would have different business processes to differentiate their abilities or value deliveries from their competitors. The teams involved in requirement elicitation shall start with a clean slate for every project and thus should not try to bias the elicitation work with their previous project experience in mind. Ignoring this principle would result in misaligned requirement specification and thus ending up delivering a deficient product. As this is a human intensive process, it is quite common for the customer representatives too to easily miss out on such things.

This is quite a common problem with the product companies. Irrespective of whether the client contracts for the product with customization or a project, the vendors would prefer to reuse their existing code assets. As such, the business analysts engaged in the requirement elicitation tend to scope the customer requirements in such a way that it fits within the existing product architecture and related constraints. Even in case of a product based contract, the requirement elicitation or the gap study shall focus shall be unbiased and then it is the Solution Architects who will come in to come up with solutions to bridge the gaps. In the process, the customer will have the option to decide to dilute his requirement in favor of an existing work around.

The business analysts shall master the art of unlearning and relearning to handle this area well.

2. The Design Mix-up

The next common problem is to mix up the requirement elicitation with the solution design. This happens on both the sides i.e, the vendors and the customers. The business analysts from the vendor side often would start visualizing the solution design with a specific use case and would start suggesting deviations or work around to the use case. Similarly on the customer front, the users may start talking on the system perspective. For example, customers when narrating the requirements might talk about a drop down list, check boxes, etc. Ideally such details should be left to the design teams and where appropriate, the customer might want to review those designs or might specify the design guidelines to be followed or specify usability requirements for the vendor to conform to.

There is another school of thought that visualizing or thinking of solution early on would eliminate feasibility issues down the line. While this is partly true, the problem arise when such design constraints hide the underlying actual business requirement, which could lead to mis-interpretations later on.

3. Poor Planning

The requirement elicitation has to be a planned process with proper entry and exit criteria for each of the sub processes. There are many frameworks and techniques to perform this activity. Irrespective of the methods or techniques, the elicitation process shall comprise of the following activities: Identifying the Stake Holders; Define Use Case specifications; Generate scenarios;Organize walk throughs / interviews; Document Requirements and Validate Requirements. It is quire possible that each of these activities might have to be performed in multiple iterations. Poor planning of these activities might result in ambiguous or deficient requirements.

A related key issue is the exit planning. i.e. when to consider the requirement elicitation as complete. Depending on other project constraints, the exit criteria has to be carefully identified and further planning should be around that. For instance, if time is a key constraint, just for the sake of meeting the timeline, the elicitation activities should not be hurried up and thus ending up with an imperfect specification. Instead, in such cases, the scope can be divided into broader sub components and agree with the customer to defer some such components to a later phase based on priorities. Agile approach could also be thought of to solve this situation. i.e. start eliciting the requirements as specific user stories are taken up in respective sprints. A careful consideration of all the project constraints and priorities is a must in choosing a solution and there by coming up with the best course of action.

4. Volatility

In one of the projects we were handed off with a four hundred page requirements specification document was an year long work of the internal business analysts of the customer. But it was no surprise, that the actual business requirements were far different than it was documented as the business practices and processes  have changed a lot during this very same period. This has been a common problem that the industry has been battling with and Agile approach is emerging as a solution to this problem. This volatile nature of the business requirements requires the solutions to be delivered quicker to reap the time to market advantage.

Another aspect of volatility is that the requirements as elicited from different users / departments could be different and at times conflicting too. In some cases such differences could be misstatements or misunderstanding or in some cases it could be genuine, in which case the different requirements shall be specified appropriately and let the design teams come up with solutions to meet all such differences.

5. Undiscovered Ruins

It is the human nature to answer just the questions that were asked. Thus the business analysts shall master the art of asking appropriate follow up questions based on the responses from the customer representatives. That is where the elicitation is important. i.e. the business analysts shall provoke the customer to fully reveal what is required of the system. In the process it is very much common that certain needs might go undiscovered, but would show up later on as a deficiency. This problem can be partly addressed by identifying the right stakeholders for the purpose and then to get those validated by different stakeholders, who would look at these with a different perspective, which might bring out gaps if any.