Sunday, August 7, 2016

Distributed Ledger - Strengths That Warrants Its Adoption

Blockchain is the most talked about technology today that is likely to have a pervasive impact on all industry segments, more specifically in the Banking and Financial Services. Blockchain packs the principles of cryptography, game theory and peer-to-peer networking. Blockchain, once the formal name for the tracking database underlying the cyptocurrency bitcoin, is now used broadly to refer to any distributed ledger that uses software algorithms to record transactions with reliability and anonymity. An increasingly interesting aspect of blockchain use is the concept of smart contracts – whereby business rules implied by a contract are embedded in the blockchain and executed with the transaction.

Built on the peer-to-peer technology, blockchain uses advanced encryption to guarantee the provenance of every transaction. The secure and resilient architecture that protects the distributed ledger is on of its key advantage. The other benefits of block chain include reduction in cost, complexity and time in addition to offering trusted record keeping and discoverability. Blockchain has the potential to make trading processes more efficient, improve regulatory control and could also displace traditional trusted third-party functions. Blockchain holds the potential for all participants in a business network to share a system of record. This distributed, shared ledger will provide consensus, provenance, immutability and finality around the transfer of assets within business networks.

The Banking and Financial Services industries world over are seriously looking at this technology. The Central Banks in many countries including India have formed committees to evluate the adoption of the blockchain technology, which is expected to address some of the problems that the industry is wanting to overcome over many years. For the financial services sector blockchain offers the opportunity to overhaul existing banking infrastructure, speed settlements and streamline stock exchanges. While many institutions understand its potential, they are still trying to work out whether blockchain technology offers a cost-cutting opportunity or represents a margin-eroding threat that could put them out of business.

Like the Cloud Computing, there three categories of blockchain, public, private, and hybrid. A public block chain is a fully decentralized “trustless” system open to everyone and where the ledger is updated by anonymous users. A private blockchain finds its use within a bank or an institution, where the organization controls the entire system. Hybrid is a combination of both public and private implementations, which is open to a controlled group of trusted and vetted users that update, preserve, and maintain the network collectively. Blockchain exploration has propelled banks in multiple directions, from examining fully decentralized systems that embed bitcoin or other virtual tokens to function, to ones where only authorized and vetted users are granted ac-cess to a network. 

The technology is being commercialised by several industry groups and are coming out with the use cases that this technology will be suitable for across different industry vertical. With the surge in funding for the FinTech innovations, the block chain technology may find its retail and institutional adoption in about 3 to 5 years, while some expect that this will take even longer. Some have invested in in-house development, while others have partenered with others in their pursuit to adopt the blockchain as part of their main stream business technology. 

Listed here are some of the key strengths that drives the adoption of the technology worldover.


With the frequency at which data breaches are happening, users are seeking to have control over sensitive data. Blockchain by its nature puts users in total control. Applied to payments, blockchain allows users to retain control of their information and enable access to information about only one act of transaction. Participants are able to trust the authenticity of the data on the ledger without recourse to a central body. Transactions are digitally signed; the maintenance and validation of the distributed ledger is performed by a network of communicating nodes running dedicated software which replicate the ledger amongst the participants in a peer-to-peer network, guaranteeing the ledger’s integrity. They will also want the ability to roll back transactions in instances of fraud or error – which can be done on blockchain by adding a compensating record, as long as there are permission mechanisms to allow this – and a framework for dispute resolution.


The cryptographic connection between each block and the next forms one link of the chain. This link ensures the  maintenance of trace for the information flow across the chain and thus enabling the articipants or regulators to trace information flows back through the entire chain. The distributed ledger is immutable as entries can be added to, but not deleted from. This information potentially includes, but is not limited to, ownership, transaction history, and data lineage of information stored on the shared ledger.  If provenance is tracked on a blockchain belonging collectively to participants, no individual entity or small group of entities can corrupt the chain of custody, and end users can have more confidence in the answers they receive.


Operates seamlessly and removes dependency on a central infrastructure for service availability. Distributed processing allows participants to seamlessly operate in case of failure of any participants. Data on the ledger is pervasive and persistent, creating a reliable distributed storage so that transaction data can be recovered from the distributed ledger in case of local system failure, allowing the system to have very strong built-in data resiliency. Distributed ledger-based systems would be more resilient to systematic operational risk because the system as a whole is not dependent on a centralised third party. With many contributors, and thus back-ups, the ledger has multiple copies which should make it more resilient than a centralised database. 


Use cases that centre on increasing efficiency by removing the need for reconciliation between parties seem to be particularly attractive. Blockchain provides the benefits of ledgers without suffering from the problem of concentration. Instead, each entity runs a “node” holding a copy of the ledger and maintains full control over its own assets. Transactions propagate between nodes in a peer-to-peer fashion, with the blockchain ensuring that consensus is maintained. Reconciling or matching and verifying data points through manual or even electronic means would be eliminated, or at least reduced, because everyone in the network accessing the distributed ledger would be working off the exact same data on the ledger. In the case of syndicated loans, This is more so, since information is mutualised and all participants are working from the same data set in real time or near-real time. .


When a blockchain transaction takes place, a number of networked computers, process the algorithm and confirm one another’s calculation. The record of such transactions thus continually expands and is shared in real time by thousands of people. Billions of people around the world lack access to banks and currency exchange. Blockchain-based distributed ledgers could change this. Just as the smartphone gave people without telephone lines access to communication, information, and electronic commerce, these technologies can provide a person the legitimacy needed to open a bank account or borrow money — without having to prove ownership of real estate or meeting other qualifications that are challenging in many countries.

Efficiency Gains

Removal of slow, manual and exception steps in existing end-to-end processes will lead to significant efficiency gains. Blockchain also removes the need for a clearing house or financial establishment to act as intermediary facilitating quick, secure, and inexpensive value exchanges. Blockchain ensures the most effective alignment between usage and cost due to its transparency, accuract and the significantly lower cost of cryptocurrency transaction. Distributed ledger technology has the potential to reduce duplicative recordkeeping, eliminate reconciliation, minimise error rates and facilitate faster settlement. In turn, faster settlement means less risk in the financial system and lower capital requirements

Sunday, April 10, 2016

Economics of Software Resiliency

Resilience is a design feature that facilitates the software to recover from occurrence of an disruptive event. As it is evident, this is kind of automated recovery from disastrous events after occurrence of such events. Yes, given an option, we would want the software that we build or buy has the resilience within it. Obviously, the resilience comes with a cost and the economies of benefit should be seen before deciding on what level of resilience is required. There is a need to balance the cost and effectiveness of the recovery or resilience capabilities against the events that cause disruption or downtime. These costs may be reduced or rather optimized if the expectation of failure or compromise is lowered through preventative measures, deterrence, or avoidance.

There is a trade-off between protective measures and investments in survivability, i.e., the cost of preventing the event versus recovering from the event. Another key factor that influences this decision is that cost of such event if it occurs. This suggests that a number of combinations need to be evaluated, depending on the resiliency of the primary systems, the criticality of the application, and the options as to backup systems and facilities.

This analysis in a sense will be identical to the risk management process. The following elements form part of this process:

Identify problems

The events that could lead to failure of the software are numerous. Developers know that exception handling is an important best practices one should adhere to while designing and developing a software system. Most modern programming languages provide support for catching and handling of exceptions.  This will at a low level help in identifying the exceptions encountered by a particular application component in the run-time. There may be certain events, which can not be handled from within the component, which require an external component to monitor and handle the same. Leave alone the exception handling ability of the programming language, the architects designing the system shall identify and document such exceptions and accordingly design a solution to get over such exception, so that the system becomes more resilient and reliable. The following would primarily bring out possible problems or exceptions that need to be handled to make the system more resilient:

  • Dependency on Hardware / Software resources - Whenever the designed system need to access a hardware resource, for example a specified folder in the local disk drive, expect a situation of the folder not being there, the application context doesn't have enough permissions to perform its actions, disk space being exhausted, etc. This equally applies to software resources like, an operating system, a third party software component, etc.
  • Dependency on external Devices / Servers / Services / Protocols - Access to external devices like printers, scanners, etc., or other services exposed for use by the application system, like an SMTP service for sending emails, database access, a web service over HTTPS protocol, etc. could also cause problems, like the remote device not being reachable, or a protocol mismatch, request or response data inconsistency, access permissions etc. 
  • Data inconsistency - In complex application systems, certain scenarios could lead to a situation of inconsistent internal data which may lead to the application getting into a dead-lock or never ending loop. Such a situation may have cascading effect as such components will consume considerable system resources quickly and leading to a total system crash. This is a typical situation in web applications as each external request is executed in separate threads and when each such thread get into a 'hung' state, over a period, the request queue will soon surpass the installed capacity. 

Cost of Prevention / recovery

The cost of prevention depends on the available solutions to overcome or handle such exceptions. For instance, if the issue is about the SMTP service being unavailable, then the solution could be to have an alternate redundant, always active SMTP service running out of a totally different network environment, so that the system can switch over to such alternate service if it encounters issues with the primary one. While the cost of implementing the handling of multiple SMTP services and a fail-over algorithm may not be significant, but maintaining redundant SMTP service could have significant cost impact. Thus with respect to each such event that may have an impact on the software resilience, the total cost for a pro-active solution vis-a-vis a reactive solution should be assessed.

Time to Recover & Impact of Event

While the cost of prevention / recovery as assessed above will be an indicator of how expensive the solution is, the Time to Recover and the Impact of such an event happening will indicate the cost of not having the event handled or worked around. Simple issues like a database dead-lock may be reactively handled by the DBAs who will be monitoring for such issues and will act immediately when such an event arise. But issues like, the network link to an external service failing, may mean an extended system unavailability and thus impacting the business. So, it is critical to assess the time to recover and the impact that such an event may have, if not handled instantly.

Depending on the above metric, the software architect may suggest an cost-effective solution to handle each such events. The level of resiliency that is appropriate for an organization depends on how critical the system in question is for the business, and the impact of the lack of resilience for the business. The organization understands that the resiliency has its own cost-benefit. The architects should have this in mind and design solutions to suit the specific organization.

The following are some of the best practices that the architects and the developers should follow while designing and building the software systems:
  • Avoid usage of proprietary protocols and software that makes migration or graceful degradation very difficult.
  • Identify and handle single points of failure. Of course, building redundancy has cost.
  • Loosely couple the service integrations, so that inter-dependence of services is managed appropriately.
  • Identify and overcome weak architecture / designs within the software modules or components.
  • Anticipate failure of every function and design for fall-back-scenarios, graceful degradation when appropriate.
  • Design to protect state in multi‐threaded and distributed execution environments.
  • Expect exceptions and implement safe use of inheritance and polymorphism 
  • Manage and handle the bounds of various software and hardware resources.
  • Manage allocated resources by using it only when needed.
  • Be aware of timeouts of various services and protocols and handle it appropriately

Sunday, March 20, 2016

Big Data for Governance - Implications for Policy, Practice and Research

A recent IDC forecast shows that the Big Data technology and services market will grow at a 26.4% compound annual growth rate to $41.5 billion through 2018, or about six times the growth rate of the overall information technology market. Additionally, by 2020 IDC believes that line of business buyers will help drive analytics beyond its historical sweet spot of relational (performance management) to the double-digit growth rates of real-time intelligence and exploration/discovery of the unstructured worlds.

This predicted growth is expected to have significant impact on all organizations, be it small, medium or large, which include exchanges, banks, brokers, insurers, data vendors and technology and services suppliers. This also extends beyond the organization with the increasing focus on rules and regulations designed to protect a firm’s employees, customers and shareholders as well as the economic wellbeing of the state in which the organization resides. This pervasive use and commercialization of big data analytical technologies is likey to have far reaching implications in meeting regulatory obligations and governance related activities. 

Certain disruptive technologies such as complex event processing (CEP) engines, machine learning, and predictive analytics using emerging big-data technologies such as Hadoop, in-memory, or NoSQL illustrate a trend in how firms are approaching technology selection to meet regulatory compliance requirements. A distinguishing factor between big data analytics and regular analytics is the performative nature of Big Data and how it goes beyond merely representing the world but actively shapes it.

Analytics and Performativity

Regulators are staying on top of the big data tools and technologies and are leveraging the tools and technologies to search through the vast amount of organizational data both structured and unstructured to prove a negative. This forces the organizations to use the latest and most effective forms of analytics and thus avoid regulatory sanctions and stay compliant.  Analytical outputs may provide a basis for strategic decision making by regulators, who may refine and adapt regulatory obligations accordingly and then require firms to use related forms of analytics to test for compliance. Compliance analytics are not simply reporting on practices but also shaping them through accelerated decision making changing strategic planning from a long term top down exercise to a bottom up reflexive exercise. Due to the 'automation bias' or the underlying privileged nature of the visualization algorithms, compliance analytics may not be neutral in the data and information they provide and the responses they elicit.

Technologies which implement surveillance and monitoring capabilities may also create self-disciplined behaviours through a pervasive suspicion that individuals are being currently observed or may have to account for their actions in the future. The complexity and heterogeneity of underlying data and related analytics provides a further layer of technical complexity to banking matters and so adds further opacity to understanding controls, behaviours and misdeeds. 

 Design decisions are embedded within technologies shaped by underlying analytics and further underpinned by data. Thus, changes to part of the systems may cause a cascading effect on the outcome. Data accuracy may also act to unduly influence outcomes. This underscores the need to understand big data analytics at the level of micro practice and from the bottom up. 

Information Control and Privacy

The collection and storage of Big Data, raises concerns over privacy. In some cases, the uses of Big Data can run afoul of existing privacy laws. In all cases, organizations risk backlash from customers and others who object to how their personal data is collected and used. This can present a challenge for organizations seeking to tap into Big Data’s extraordinary potential, especially in industries with rigorous privacy laws such as financial services and healthcare. Some wonder if these laws, which were not developed with Big Data in mind, sufficiently address both privacy concerns and the need to access large quantities of data to reach the full potential of the new technologies.

The challenges to privacy arise because technologies collect so much data and analyze them so efficiently that it is possible to learn far more than most people had predicted or can predict . These challenges are compounded by limitations on traditional technologies used to protect privacy. The degree of awareness and control can determine information privacy concerns; however, the degree may depend on personal privacy risk tolerance. In order to be perceived as being ethical, an organization must ensure that individuals are aware that their data is being collected, and they have control of how their data is used. As data privacy regulations impose increasing levels of administration and sanctions, we expect policy makers at the global level to be placed under increased pressure to mitigate regulatory conflicts and multijurisdictional tensions between data privacy and financial services’ regulations.

Technologies such as social media or cloud computing facilitate data sharing across borders, yet legislative frameworks are moving in the opposite direction towards greater controls designed to prevent movement of data under the banner of protecting privacy. This creates a tension which could be somewhat mediated through policy makers’ deeper understanding of data and analytics at a more micro level and thereby appreciate how technical architectures and analytics are entangled with laws and regulations. 

The imminent introduction of data protection laws will further require organizations to account for how they manage information, requiring much more responsibility from data controllers. Firms are likely to be required to understand the privacy impact of new projects and correspondingly assess and document perceived levels of intrusiveness. 

Implementing an Information Governance Strategy

The believability of analytical results when there is limited visibility into trustworthiness of the data sources is one of the foremost concern that an end user will have.  A common challenge associated with adoption of any new technology is walking the fine line between speculative application development, assessing pilot projects as successful, and transitioning those successful pilots into the mainstream. The enormous speeds and amount of data processed with Big Data technologies can cause the slightest discrepancy between expectation and performance to exacerbate quality issues. This may be further compounded by Metadata complications when conceiving of definitions for unstructured and semi-structured data.  

This necessitates the organizations to work towards developing an enterprise wide information governance strategy with related policies. The governance strategy shall encompass continued development & maturation of processes and tools for data quality assurance, data standardization, and data cleansing. The management of meta-data and its preservation, so that it can be evidenced to regulators and courts, should lso be considered when formulating strategies and tactics. The policies should be high-level enough to be relevant across the organization while allowing each function to interpret them according to their own circumstances. 

Outside of regulations expressly for Big Data, lifecycle management concerns for Big Data are fairly similar to those for conventional data. One of the biggest differences, of course, is in providing needed resources for data storage considering the rate at which the data grows. Different departments will have various lengths of time in which they will need access to data, which factors into how long data is kept. Lifecycle principles are inherently related to data quality issues as well, since such data is only truly accurate once it has been cleaned and tested for quality. As with conventional data, lifecycle management for Big Data is also industry specific and must adhere to external regulations as such.

Security issues must be part of an Information Governance strategy whichwill require current awareness of regulatory and legal data securityobligations so that a data security approach can be developed based on repeatable and defensible best practices.