Sunday, December 25, 2016

The Mobile Phone Is Your Private Property

This morning, when I was on my morning walk, a person came out of a construction site and was requeting me to lend my phone to make a phone call. I was not comfortable lending my phone primarily for three reasons: First he is a stranger to me; Second, he seem to be working in the construction site and he should have sought help from those around in his workplace as they would be more comfortable helping him; Third, my mobile is my private identity and would not want a stranger to use impersonate me. I did not lend my phone on that occasion.

How about you? Would you mind lending your phone for such requests? I understand, the answer will be "it depends." Thank's to "Selfie" feature, seeking help from a stranger to take a snap on the mobile phone is not required any more. Any ways, I thought it would be useful to list out the concerns, so that one can decide how safe is to part with one's smart phone. These apply for stolen / lost mobile phones as well.

Your Phone Contains Sensitive Information

You have your email configured on your mobile and typically, it does not expect you to login every time you use your mail app on your mobile. So lending a phone may allow the stranger gaining access to your emails and depending the duration it remains with such stranger, the impact of such compromise could be larger. Similarly, all your social media accounts do not expect any additional authentication. It is needless to say that what a smart or malicious stranger could do with access to your social media accounts. Exposing all the intimate details of our lives because of a lost, stolen or hacked phone is a serious issue.

Banking / Payment Applications

"There is an App for everything". Yes, every bank and the investment advisors are rolling out their own Apps with pre-stored credentials for the mobile savvy customers. Mobile users, find it convenient to use such an App, without having to login every time. However, the issue of how many such Apps will you install on your mobile phone is an issue to be discussed in a separate blog. For the purpose this blog let us consider the prevailing App culture. Driven by the Digital economy, there are humpteen number of Payment / eWallet Apps out in the store. The user convenience always wins over the security requirements and as such most such Apps doesn't requie a login to initiate a payment. This could be a potential risk one should be aware of and be careful about.

Personal & Corporate Information

If you are working for an organization, it is most likely that you would have setup your corporate email account as well on your smart phone and there you go, you are putting your organization's data / information at risk. Your organization would have a BYOD policy and procedure, stating what precautions you should take on the corporate data that you use or access using your smart phone. If you are an senior level executive, it is likely that you will have access to your organizational applications configured on your mobile. This includes compromise of your or your organization's cloud storage if any configured on the phone.

Illegitimate Calls / Messages

In addition to your device, your mobile phone number (SIM) is very well linked to your identity. As such any calls or message that such a stranger sends using your phone will be logged against your identity and you are responsible and answerable for consequences if any that may arise out of such calls or messages. Even if the activity is legitimate, it may be possible that the other person might call or message you back in future with or without any specific intent.

AVAST did a research in February 2016 and according to them, their researchers were able to recover the following files from the 20 phones that were sold:

  • More than 1,200 photos
  • More than 200 photos with adult content
  • 149 photos of children
  • More than 300 emails and text messages
  • More than 260 Google searches, including 170 searches for adult content
  • Two previous owners’ identities
  • Three invoices
  • One working contract
  • One adult video

Given the ever evolving capabilities of the smart phones, the devices are increasingly becoming one's identity and as such should be handled with care and caution, or else one has to face the consequences that may arise as a result of such compromise.

Sunday, November 13, 2016

A Software Product Vs Project

In short, a software Project is all about to execute a Statement of Work of an internal or external customer, where what customer required is right irrespective of what is ideal or what the end user would expect. Though some projects are scoped in such a way that certain aspects of non-functional requirements are left to the choice of the project teams.

Product development isn’t about implementing what the customer wanted to. In product development, the product manager owns and comes up with the product requirements. A large product or product suite, typically comprise of many projects and will evolve over time.

Unlike a project the product will be improved continuously without an end date based on feedback from end users and the product team prioritizes what needs to be built next based on its perceived value for its target users or customers.

A project on the other hand is funded with specific goals, a business case in mind and with finite expected value and cost.

Here is an attempt to bring out the differences between a software project and product and such differences are categorised as below:

The Mindset:

Projects are many a times started off with main focus on to deliver on time, under budget, within scope and with a temporary team. All these constraints are set in stone and any deviation is viewed seriously, which may impact the course of the project depending on the methodology adopted. So, the mindset of the project team will be with primary focus on the project parameters that determine the success of delivery and may not be the success of the product that the project may form part of. This is more so as the resources keep changing and the resources with no or little knowledge on the business domain may still deliver the project, but the product may be crappy.

Products tend to have a longer lifetime than projects and mostly built with more focus on the outcome instead of the output. Product teams are given the freedom and responsibility to think of a strategy they believe will result in the best product within a boundary of product framework. This leads to less waste and more creativity being introduced into the product development process, allowing room for embracing changes continously.


The product roadmap is key for the success of the prodct and as such, the product manager shall align the product vision and strategy with that of the business. A Project Manager, on the other hand, is responsible for executing on a predefined objective.

A Project Managers function is to create a plan, that the project will follow, and then to drive the people involved in the project to follow that plan with as little change as possible. If deviations from the planned execution are beyond an accepted threshold, the Project Manager must escalate and explain the situation to the stakeholders, who in turn will either accept the deviation or may choose to fail the project.

A product manager with the focus on constantly evaluating the viability of the product, will typically follow an agile approach with shorter sprints of developments, so the product evolves incrementally, delivering values at every stage.


With the primary focus of the project team being on delivering on time and within budget, the team does not have enough room to be creative enough. This brings down the motivation because the teams lose a sense of purpose and the autonomy in how to operate.

On the other hand, as typically, the resources stay longer with the product teams, they get aligned to the product strategy and the vision and thus they are given the freedom to bring in their thinking and creativity into the product, process and methodology. The feedback and collaboration with stakeholders enables the right environment, where the resources reach a higher potential and operate autonomously, resulting in better problem solving, higher ownership of outcomes, and faster time to market.


Product management software and project management software are entirely different tools — each designed for a different type of role, to help address different business needs. Product management software helps product managers organize, develop, and communicate the product strategy, while project management software helps project managers in track the execution and incidentally manage the resource allocation, risk and issue management.


Product scope is defined as "The features and functions that characterize a product, service, or result". Whereas the project scope is defined as "The work performed to deliver a product, service, or result with the specified features and functions".

The Product Scope defines all the capabilities of a product from the User point of view. The Product is the end result of your project and characterizes by the Product Scope. Thus, the Product Scope description includes features of a product, how the product will look like using these features, and how will it work. Product Scope also describe the ways of measuring the product performance.

The Project Scope on the other hand is an agreement of the work which is needed to deliver the product, service, or result. To develop a product features, you establish a project which has a schedule, budget, and resource allocation. In other words, the work you do to construct your product is the Project Scope.

Design & Architecture:

The product owner or manger is responsible for defining the architecture and design of the product, which should take the following into consideration:
  • Business Idea & Strategy
  • Identifying and Creating a product feature
  • Aligning with Market Trends
  • Define Product Performance Indicators
  • Prioritize the implementation of features and bugs
Though a project may include the product architecture and design as part of the scope, the focus of the project team will be more on the following:
  • Defining the project scheduling, taking into account the deliverables at various milestones.
  • Monitoring the budget
  • Planning and managing resources
  • Problem and issue management
  • Risk management
  • Managing the scope creep.

Saturday, October 1, 2016

DNS Security Extensions - Complexities To Be Aware Of

The Domain Name System (DNS) primarily offers a distributed database storing typed values by name.  The DNS acts like a phone book for the Internet, translating IP addresses into human-readable addresses. Obviously, as close to 100% of the internet requests are by the domain names, requiring the DNS servers resolve the domain names into IP addresses. This results in a very high load on the DNS servers located across the world. In order to support such a high frequency of requests, DNS employs a tree-wise hierarchy in both name and database structure. 

However, the wide-open nature of DNS leaves it susceptible to DNS hijacking and DNS cache poisoning attacks to redirect users to a different address than where they intended to go. This means that despite entering the correct web address, the user might be taken to a different website.DNS Secrutity Extension (DNSSEC) was brought in as the answer to the above problem.

DNSSEC is designed to protect Internet resolvers (clients) from forged DNS in order to prevent DNS tampering. DNSSEC offers protection against spoofing of DNS data by providing origin authentication, ensuring data integrity and authentication of non-existence by using public-key cryptography. It digitally signs the information published by the DNS with a set of cryptographic keys, making it harder to fake, and thus more secure.

The DNSSEC brings in certain additional records to be added to the DNS. The new record types are: RRSIG (for digital signature), DNSKEY (the public key), DS (Delegation Signer), and NSEC (pointer to next secure record). The new message header bits are: AD (for authenticated data) and CD (checking disabled). A DNSSEC validating resolver uses these records and public key (asymmetric) cryptography to prove the integrity of the DNS data. 

A hash of the public DNSKEY is stored in a DS record. This is stored in the parent zone. The validating resolver retrieves from the parent the DS record and its corresponding signature (RRSIG) and public key (DNSKEY); a hash of that public key is available from its parent. This becomes a chain of trust — also called an authentication chain. The validating resolver is configured with a trust anchor — this is the starting point which refers to a signed zone. The trust anchor is a DNSKEY or DS record and should be securely retrieved from a trusted source.

The successful implementation DNSSEC depends on the deployment of the same at all levels of the DNS architecture and the adoption by all involved in the DNS resolution process. One big step was given in July 2010 when the DNS root zone was signed. Since then, resolvers are enabled to configure the root zone as a trusted anchor which allows the validation of the complete chain of trust for the first time.  The introduction and use of DNSSEC has been controversial for over a decade due to its cost and complexity. However, its usage and adoption is steadily growing and in 2014, DNS overseer ICANN determined that all new generic top-level domains would have to use DNSSEC.

Implementing DNSSEC is not always unproblematic. Some faults in DNS are only visible in DNSSEC – and then only when validating making the debugging the DNSSEC difficult. DNS software that apply only to DNSSEC has many issues to be plugged, leading to disruptions in service.
Interoperability amongst the DNS software is another issue that is adding to the problems. Above all, attackers can abuse improperly configured DNSSEC domains to launch denial-of-service attacks. The following are some such major complexities that one should be aware of.

Zone Content Exposure

DNS is split into smaller pieces called zones. A zone typically starts at a domain name, and contains all records pertaining to the subdomains. Each zone is managed by a single manager. For example, is a zone containing all DNS records for and its subdomains (e.g., Unlinke DNS, with DNSSEC the requests will be at the signed zone level. As such, enabling DNSSEC may expose otherwise obscured zone content. Subdomains are sometimes used as login portals or other services that the site owner wants to keep private. A site owner may not want to reveal that “” exists in order to protect that site from attackers.

Non-Existent Domains

Unlike standard DNS, where the server returns an unsigned NXDOMAIN (Non-Existent Domain) response when a subdomain does not exist, DNSSEC guarantees that every answer is signed. For statically signed zones, there are, by definition, a fixed number of records. Since each NSEC record points to the next, this results in a finite ‘ring’ of NSEC records that covers all the subdomains. This technique may unveils internal records if zone is not configured properly.The information that can be obtained can help us to map network hosts by enumerating the contents of a zone.

The NSEC3-walking attack

DNSSEC has undergone revisions on multiple occasions and NSEC3 is the current replacement for NSEC. "NSEC3 walking" is an easy privacy-violating attack against the current version of DNSSEC. After a few rounds of requests to a DNSSEC server, the attacker can collect a list of hashes of existing names. The attacker can then guess a name, hash the guess, check whether the hash is in the list, and repeat.  Compared to normal DNS, current DNSSEC (with NSEC3) makes privacy violations thousands of times faster for casual attackers, or millions of times faster for serious attackers. It also makes the privacy violations practically silent: the attackers are guessing names in secret, rather than flooding the legitimate servers with guesses. NSEC3 is advertised as being much better than NSEC. 

Key Management

DNSSEC was designed to operate in various modes, each providing different security, performance and convenience tradeoffs. Live signing solves the zone content exposure problem in exchange for less secure key management. The most common DNSSEC mode is offline signing of static zones. This allows the signing system to be highly protected from external threats by keeping the private keys on a machine that is not connected to the network. This operating model works well when the DNS information does not change often.

Key management for DNSSEC is similar to key management for TLS and has similar challenges. Enterprises that decide to manage DNSSEC internally need to generate and manage two sets of cryptographic keys – the Key Signing Key (KSK), critical in establishing the chain of trust, and the Zone Signing Key (ZSK), used to sign the domain name’s zone. Both types of keys need to be changed periodically in order to maintain their integrity. The more frequently a key is changed, the less material an attacker has to help him perform the cryptanalysis that would be required to reverse-engineer the private key.  

An attacker could decide to launch a Denial of Service (DoS) attack at the time of key rollover. That is why it is recommended to introduce some "jitter" into the rollover plan by introducing a small random element to the schedule. Instead of rolling the ZSK every 90 days like clockwork, a time within a 10-day window either side may be picked, so that it is not predictable.

Reflection/Amplification Threat

DNSSEC works over UDP, and the answers to DNS queries can be very long, containing multiple DNSKEY and RRSIG records. This is an attractive target for attackers since it allows them to ‘amplify’ their reflection attacks. If a small volume of spoofed UDP DNSSEC requests is sent to nameservers, the victim will receive a large volume of reflected traffic. Sometimes this is enough to overwhelm the victim’s server, and cause a denial of service. Specifically, an attacker sends a corrupted network packet to a certain server that then reflects it back to the victim. Using flaws in DNSSEC, it is possible to use that extra-large response as a way to amplify the number of packets sent – anywhere up to 100 times. That makes it an extremely effective tool in efforts to take servers offline.

The problem isn't with DNSSEC or its functionality, but rather how it's administered and deployed. DNSSEC is the best way to combat DNS hijacking, but the complexity of the signatures increases the possibility of administrators making mistakes. DNS is already susceptible to amplification attacks because there aren't a lot of ways to weed out fake traffic sources.

"DNSSEC prevents the manipulation of DNS record responses where a malicious actor could potentially send users to its own site. This extra security offered by DNSSEC comes at a price as attackers can leverage the larger domain sizes for DNS amplification attacks," Akamai said in a report.

Sunday, August 7, 2016

Distributed Ledger - Strengths That Warrants Its Adoption

Blockchain is the most talked about technology today that is likely to have a pervasive impact on all industry segments, more specifically in the Banking and Financial Services. Blockchain packs the principles of cryptography, game theory and peer-to-peer networking. Blockchain, once the formal name for the tracking database underlying the cyptocurrency bitcoin, is now used broadly to refer to any distributed ledger that uses software algorithms to record transactions with reliability and anonymity. An increasingly interesting aspect of blockchain use is the concept of smart contracts – whereby business rules implied by a contract are embedded in the blockchain and executed with the transaction.

Built on the peer-to-peer technology, blockchain uses advanced encryption to guarantee the provenance of every transaction. The secure and resilient architecture that protects the distributed ledger is on of its key advantage. The other benefits of block chain include reduction in cost, complexity and time in addition to offering trusted record keeping and discoverability. Blockchain has the potential to make trading processes more efficient, improve regulatory control and could also displace traditional trusted third-party functions. Blockchain holds the potential for all participants in a business network to share a system of record. This distributed, shared ledger will provide consensus, provenance, immutability and finality around the transfer of assets within business networks.

The Banking and Financial Services industries world over are seriously looking at this technology. The Central Banks in many countries including India have formed committees to evluate the adoption of the blockchain technology, which is expected to address some of the problems that the industry is wanting to overcome over many years. For the financial services sector blockchain offers the opportunity to overhaul existing banking infrastructure, speed settlements and streamline stock exchanges. While many institutions understand its potential, they are still trying to work out whether blockchain technology offers a cost-cutting opportunity or represents a margin-eroding threat that could put them out of business.

Like the Cloud Computing, there three categories of blockchain, public, private, and hybrid. A public block chain is a fully decentralized “trustless” system open to everyone and where the ledger is updated by anonymous users. A private blockchain finds its use within a bank or an institution, where the organization controls the entire system. Hybrid is a combination of both public and private implementations, which is open to a controlled group of trusted and vetted users that update, preserve, and maintain the network collectively. Blockchain exploration has propelled banks in multiple directions, from examining fully decentralized systems that embed bitcoin or other virtual tokens to function, to ones where only authorized and vetted users are granted ac-cess to a network. 

The technology is being commercialised by several industry groups and are coming out with the use cases that this technology will be suitable for across different industry vertical. With the surge in funding for the FinTech innovations, the block chain technology may find its retail and institutional adoption in about 3 to 5 years, while some expect that this will take even longer. Some have invested in in-house development, while others have partenered with others in their pursuit to adopt the blockchain as part of their main stream business technology. 

Listed here are some of the key strengths that drives the adoption of the technology worldover.


With the frequency at which data breaches are happening, users are seeking to have control over sensitive data. Blockchain by its nature puts users in total control. Applied to payments, blockchain allows users to retain control of their information and enable access to information about only one act of transaction. Participants are able to trust the authenticity of the data on the ledger without recourse to a central body. Transactions are digitally signed; the maintenance and validation of the distributed ledger is performed by a network of communicating nodes running dedicated software which replicate the ledger amongst the participants in a peer-to-peer network, guaranteeing the ledger’s integrity. They will also want the ability to roll back transactions in instances of fraud or error – which can be done on blockchain by adding a compensating record, as long as there are permission mechanisms to allow this – and a framework for dispute resolution.


The cryptographic connection between each block and the next forms one link of the chain. This link ensures the  maintenance of trace for the information flow across the chain and thus enabling the articipants or regulators to trace information flows back through the entire chain. The distributed ledger is immutable as entries can be added to, but not deleted from. This information potentially includes, but is not limited to, ownership, transaction history, and data lineage of information stored on the shared ledger.  If provenance is tracked on a blockchain belonging collectively to participants, no individual entity or small group of entities can corrupt the chain of custody, and end users can have more confidence in the answers they receive.


Operates seamlessly and removes dependency on a central infrastructure for service availability. Distributed processing allows participants to seamlessly operate in case of failure of any participants. Data on the ledger is pervasive and persistent, creating a reliable distributed storage so that transaction data can be recovered from the distributed ledger in case of local system failure, allowing the system to have very strong built-in data resiliency. Distributed ledger-based systems would be more resilient to systematic operational risk because the system as a whole is not dependent on a centralised third party. With many contributors, and thus back-ups, the ledger has multiple copies which should make it more resilient than a centralised database. 


Use cases that centre on increasing efficiency by removing the need for reconciliation between parties seem to be particularly attractive. Blockchain provides the benefits of ledgers without suffering from the problem of concentration. Instead, each entity runs a “node” holding a copy of the ledger and maintains full control over its own assets. Transactions propagate between nodes in a peer-to-peer fashion, with the blockchain ensuring that consensus is maintained. Reconciling or matching and verifying data points through manual or even electronic means would be eliminated, or at least reduced, because everyone in the network accessing the distributed ledger would be working off the exact same data on the ledger. In the case of syndicated loans, This is more so, since information is mutualised and all participants are working from the same data set in real time or near-real time. .


When a blockchain transaction takes place, a number of networked computers, process the algorithm and confirm one another’s calculation. The record of such transactions thus continually expands and is shared in real time by thousands of people. Billions of people around the world lack access to banks and currency exchange. Blockchain-based distributed ledgers could change this. Just as the smartphone gave people without telephone lines access to communication, information, and electronic commerce, these technologies can provide a person the legitimacy needed to open a bank account or borrow money — without having to prove ownership of real estate or meeting other qualifications that are challenging in many countries.

Efficiency Gains

Removal of slow, manual and exception steps in existing end-to-end processes will lead to significant efficiency gains. Blockchain also removes the need for a clearing house or financial establishment to act as intermediary facilitating quick, secure, and inexpensive value exchanges. Blockchain ensures the most effective alignment between usage and cost due to its transparency, accuract and the significantly lower cost of cryptocurrency transaction. Distributed ledger technology has the potential to reduce duplicative recordkeeping, eliminate reconciliation, minimise error rates and facilitate faster settlement. In turn, faster settlement means less risk in the financial system and lower capital requirements

Sunday, April 10, 2016

Economics of Software Resiliency

Resilience is a design feature that facilitates the software to recover from occurrence of an disruptive event. As it is evident, this is kind of automated recovery from disastrous events after occurrence of such events. Yes, given an option, we would want the software that we build or buy has the resilience within it. Obviously, the resilience comes with a cost and the economies of benefit should be seen before deciding on what level of resilience is required. There is a need to balance the cost and effectiveness of the recovery or resilience capabilities against the events that cause disruption or downtime. These costs may be reduced or rather optimized if the expectation of failure or compromise is lowered through preventative measures, deterrence, or avoidance.

There is a trade-off between protective measures and investments in survivability, i.e., the cost of preventing the event versus recovering from the event. Another key factor that influences this decision is that cost of such event if it occurs. This suggests that a number of combinations need to be evaluated, depending on the resiliency of the primary systems, the criticality of the application, and the options as to backup systems and facilities.

This analysis in a sense will be identical to the risk management process. The following elements form part of this process:

Identify problems

The events that could lead to failure of the software are numerous. Developers know that exception handling is an important best practices one should adhere to while designing and developing a software system. Most modern programming languages provide support for catching and handling of exceptions.  This will at a low level help in identifying the exceptions encountered by a particular application component in the run-time. There may be certain events, which can not be handled from within the component, which require an external component to monitor and handle the same. Leave alone the exception handling ability of the programming language, the architects designing the system shall identify and document such exceptions and accordingly design a solution to get over such exception, so that the system becomes more resilient and reliable. The following would primarily bring out possible problems or exceptions that need to be handled to make the system more resilient:

  • Dependency on Hardware / Software resources - Whenever the designed system need to access a hardware resource, for example a specified folder in the local disk drive, expect a situation of the folder not being there, the application context doesn't have enough permissions to perform its actions, disk space being exhausted, etc. This equally applies to software resources like, an operating system, a third party software component, etc.
  • Dependency on external Devices / Servers / Services / Protocols - Access to external devices like printers, scanners, etc., or other services exposed for use by the application system, like an SMTP service for sending emails, database access, a web service over HTTPS protocol, etc. could also cause problems, like the remote device not being reachable, or a protocol mismatch, request or response data inconsistency, access permissions etc. 
  • Data inconsistency - In complex application systems, certain scenarios could lead to a situation of inconsistent internal data which may lead to the application getting into a dead-lock or never ending loop. Such a situation may have cascading effect as such components will consume considerable system resources quickly and leading to a total system crash. This is a typical situation in web applications as each external request is executed in separate threads and when each such thread get into a 'hung' state, over a period, the request queue will soon surpass the installed capacity. 

Cost of Prevention / recovery

The cost of prevention depends on the available solutions to overcome or handle such exceptions. For instance, if the issue is about the SMTP service being unavailable, then the solution could be to have an alternate redundant, always active SMTP service running out of a totally different network environment, so that the system can switch over to such alternate service if it encounters issues with the primary one. While the cost of implementing the handling of multiple SMTP services and a fail-over algorithm may not be significant, but maintaining redundant SMTP service could have significant cost impact. Thus with respect to each such event that may have an impact on the software resilience, the total cost for a pro-active solution vis-a-vis a reactive solution should be assessed.

Time to Recover & Impact of Event

While the cost of prevention / recovery as assessed above will be an indicator of how expensive the solution is, the Time to Recover and the Impact of such an event happening will indicate the cost of not having the event handled or worked around. Simple issues like a database dead-lock may be reactively handled by the DBAs who will be monitoring for such issues and will act immediately when such an event arise. But issues like, the network link to an external service failing, may mean an extended system unavailability and thus impacting the business. So, it is critical to assess the time to recover and the impact that such an event may have, if not handled instantly.

Depending on the above metric, the software architect may suggest an cost-effective solution to handle each such events. The level of resiliency that is appropriate for an organization depends on how critical the system in question is for the business, and the impact of the lack of resilience for the business. The organization understands that the resiliency has its own cost-benefit. The architects should have this in mind and design solutions to suit the specific organization.

The following are some of the best practices that the architects and the developers should follow while designing and building the software systems:
  • Avoid usage of proprietary protocols and software that makes migration or graceful degradation very difficult.
  • Identify and handle single points of failure. Of course, building redundancy has cost.
  • Loosely couple the service integrations, so that inter-dependence of services is managed appropriately.
  • Identify and overcome weak architecture / designs within the software modules or components.
  • Anticipate failure of every function and design for fall-back-scenarios, graceful degradation when appropriate.
  • Design to protect state in multi‐threaded and distributed execution environments.
  • Expect exceptions and implement safe use of inheritance and polymorphism 
  • Manage and handle the bounds of various software and hardware resources.
  • Manage allocated resources by using it only when needed.
  • Be aware of timeouts of various services and protocols and handle it appropriately

Sunday, March 20, 2016

Big Data for Governance - Implications for Policy, Practice and Research

A recent IDC forecast shows that the Big Data technology and services market will grow at a 26.4% compound annual growth rate to $41.5 billion through 2018, or about six times the growth rate of the overall information technology market. Additionally, by 2020 IDC believes that line of business buyers will help drive analytics beyond its historical sweet spot of relational (performance management) to the double-digit growth rates of real-time intelligence and exploration/discovery of the unstructured worlds.

This predicted growth is expected to have significant impact on all organizations, be it small, medium or large, which include exchanges, banks, brokers, insurers, data vendors and technology and services suppliers. This also extends beyond the organization with the increasing focus on rules and regulations designed to protect a firm’s employees, customers and shareholders as well as the economic wellbeing of the state in which the organization resides. This pervasive use and commercialization of big data analytical technologies is likey to have far reaching implications in meeting regulatory obligations and governance related activities. 

Certain disruptive technologies such as complex event processing (CEP) engines, machine learning, and predictive analytics using emerging big-data technologies such as Hadoop, in-memory, or NoSQL illustrate a trend in how firms are approaching technology selection to meet regulatory compliance requirements. A distinguishing factor between big data analytics and regular analytics is the performative nature of Big Data and how it goes beyond merely representing the world but actively shapes it.

Analytics and Performativity

Regulators are staying on top of the big data tools and technologies and are leveraging the tools and technologies to search through the vast amount of organizational data both structured and unstructured to prove a negative. This forces the organizations to use the latest and most effective forms of analytics and thus avoid regulatory sanctions and stay compliant.  Analytical outputs may provide a basis for strategic decision making by regulators, who may refine and adapt regulatory obligations accordingly and then require firms to use related forms of analytics to test for compliance. Compliance analytics are not simply reporting on practices but also shaping them through accelerated decision making changing strategic planning from a long term top down exercise to a bottom up reflexive exercise. Due to the 'automation bias' or the underlying privileged nature of the visualization algorithms, compliance analytics may not be neutral in the data and information they provide and the responses they elicit.

Technologies which implement surveillance and monitoring capabilities may also create self-disciplined behaviours through a pervasive suspicion that individuals are being currently observed or may have to account for their actions in the future. The complexity and heterogeneity of underlying data and related analytics provides a further layer of technical complexity to banking matters and so adds further opacity to understanding controls, behaviours and misdeeds. 

 Design decisions are embedded within technologies shaped by underlying analytics and further underpinned by data. Thus, changes to part of the systems may cause a cascading effect on the outcome. Data accuracy may also act to unduly influence outcomes. This underscores the need to understand big data analytics at the level of micro practice and from the bottom up. 

Information Control and Privacy

The collection and storage of Big Data, raises concerns over privacy. In some cases, the uses of Big Data can run afoul of existing privacy laws. In all cases, organizations risk backlash from customers and others who object to how their personal data is collected and used. This can present a challenge for organizations seeking to tap into Big Data’s extraordinary potential, especially in industries with rigorous privacy laws such as financial services and healthcare. Some wonder if these laws, which were not developed with Big Data in mind, sufficiently address both privacy concerns and the need to access large quantities of data to reach the full potential of the new technologies.

The challenges to privacy arise because technologies collect so much data and analyze them so efficiently that it is possible to learn far more than most people had predicted or can predict . These challenges are compounded by limitations on traditional technologies used to protect privacy. The degree of awareness and control can determine information privacy concerns; however, the degree may depend on personal privacy risk tolerance. In order to be perceived as being ethical, an organization must ensure that individuals are aware that their data is being collected, and they have control of how their data is used. As data privacy regulations impose increasing levels of administration and sanctions, we expect policy makers at the global level to be placed under increased pressure to mitigate regulatory conflicts and multijurisdictional tensions between data privacy and financial services’ regulations.

Technologies such as social media or cloud computing facilitate data sharing across borders, yet legislative frameworks are moving in the opposite direction towards greater controls designed to prevent movement of data under the banner of protecting privacy. This creates a tension which could be somewhat mediated through policy makers’ deeper understanding of data and analytics at a more micro level and thereby appreciate how technical architectures and analytics are entangled with laws and regulations. 

The imminent introduction of data protection laws will further require organizations to account for how they manage information, requiring much more responsibility from data controllers. Firms are likely to be required to understand the privacy impact of new projects and correspondingly assess and document perceived levels of intrusiveness. 

Implementing an Information Governance Strategy

The believability of analytical results when there is limited visibility into trustworthiness of the data sources is one of the foremost concern that an end user will have.  A common challenge associated with adoption of any new technology is walking the fine line between speculative application development, assessing pilot projects as successful, and transitioning those successful pilots into the mainstream. The enormous speeds and amount of data processed with Big Data technologies can cause the slightest discrepancy between expectation and performance to exacerbate quality issues. This may be further compounded by Metadata complications when conceiving of definitions for unstructured and semi-structured data.  

This necessitates the organizations to work towards developing an enterprise wide information governance strategy with related policies. The governance strategy shall encompass continued development & maturation of processes and tools for data quality assurance, data standardization, and data cleansing. The management of meta-data and its preservation, so that it can be evidenced to regulators and courts, should lso be considered when formulating strategies and tactics. The policies should be high-level enough to be relevant across the organization while allowing each function to interpret them according to their own circumstances. 

Outside of regulations expressly for Big Data, lifecycle management concerns for Big Data are fairly similar to those for conventional data. One of the biggest differences, of course, is in providing needed resources for data storage considering the rate at which the data grows. Different departments will have various lengths of time in which they will need access to data, which factors into how long data is kept. Lifecycle principles are inherently related to data quality issues as well, since such data is only truly accurate once it has been cleaned and tested for quality. As with conventional data, lifecycle management for Big Data is also industry specific and must adhere to external regulations as such.

Security issues must be part of an Information Governance strategy whichwill require current awareness of regulatory and legal data securityobligations so that a data security approach can be developed based on repeatable and defensible best practices. 

Sunday, January 3, 2016

Enterprise Architecture - Guiding Principles

Enterprise Architecture (EA) artifacts must be developed with a clear understanding of how the EA will be used and who will use it. The EA may be used as a tool for evaluating design alternatives and selecting optimal solutions, as a guide providng insights into how practices will be streamlined or improved through automation or as a plan for needed investments and an understanding of what costs savings will be achieved through consolidation. Throughout, the people involved in the development and maintenance of an EA Framework shall consistently follow certain guiding principles, so that the EA contributes to the vision and mission of the enterprise. That makes the guiding principles of most important and mostly the first step in developing EA.

Enterprise architecture principles serve as a Framework for decision making by providing guidance about the preferred outcomes of a decision in a given context. This acts as a mechanism for harmonizing decision making across organization functions & departments in addition to guiding the selection and evolution of information systems to be as consistent and cost effective as possible. Alignment with enterprise architecture principles should be a goal for any initiative and will result in fewer obstacles, surprises and course corrections later in the project.

The usefulness of principles is in their general orientation and perspective; they do not prescribe specific actions. A given principle applies in some contexts but not all contexts. Different principles may conflict with each other, such as the principle of accessibility and the principle of security. Therefore, applying principles in the development of EA requires deliberation and often tradeoffs. The selection of principles to apply to a given EA is based on a combination of the general environment of the enterprise and the specifics of the goals and purpose of the EA. The application of appropriate principles facilitates grounding, balance, and positioning of an EA. Deviating from the principles may result in unnecessary and avoidable long-term costs and risks.

Typically there will be a set of overarching general principles and specific principles with respect to Business Architecture, Application & Systems, Data & Information, Security, etc. The following are some of the generic guiding principles that could be applicable to all enterprises.

Maximize Value

Architectures are designed to provide long-term benefits to the enterprise. Decisions must balance multiple criteria based on business needs. Every strategic decision must be assessed from a cost, risk and benefit perspective. Maximizing the benefit to the enterprise requires that information system decisions adhere to enterprise-wide drivers and priorities. Achieving maximum enterprise-wide benefits will require changes in the way information systems are planned and managed. Technology alone will not bring about change. To maximize utility, some functions or departments may have to concede their preferences for the benefit of the entire enterprise.

Business Continuity

As system operations become more pervasive, the enterprise become more dependent on them. This calls for ensuring reliability and scalability to suit the current and perceived future use of such systems throughout their design and use. Business premises throughout the enterprise must be provided with the capability to continue their business functions regardless of external events. Hardware failure, natural disasters, and data corruption should not be allowed to disrupt or stop enterprise activities. The enterprise business functions must be capable of operating on alternative information delivery mechanisms. Applications and systems must be assessed for criticality and impact on the enterprise's mission in order to determine the level of continuity that is required as well as on the need for an appropriate recovery plan.

Applications & Systems Architecture

Applications and Systems should be scalable to support use by different size organizations and to handle decline or growth in business levels. While the unexpected surge or decline in the volumes are to be handled, support for horizontal scaling is also essential. Enterprise applications should be easy to support, maintain, and modify. Enterprise applications that are easy to support, maintain, and modify lower the cost of support, and improve the user experience. Applications and Systems shall have the following characteristics: Flexibility, Extensibility, Availability, Interoperability, Maintainability, Manageability and Scalability

Legal and Regulatory Compliance

Information system management processes must comply with all relevant contracts, laws, regulations and policies. Enterprise policy is to abide by laws, policies, and regulations. This will not preclude business process improvements that lead to changes in policies and regulations. The enterprise must be mindful to comply with laws, regulations, and external policies regarding the collection, retention, and management of data.Education and access to the rules. Efficiency, need, and common sense are not the only drivers. Changes in the law and changes in regulations may drive changes in our processes or applications. Staff need to be educated about the importance of regulatory compliance and their responsibility to maintain it. Where existing information systems are non-compliant they must be strategically brought into compliance.

Leverage investments

All systems shall leverage existing and planned components, enterprise software, management systems, infrastructure, and standards. It is impossible to accurately predict everything upfront. A try before you buy approach validates investment plans, designs and technologies. Prototypes enable users to provide early feedback about the design of the solution. If the enterprise capability is incomplete or deficient, efforts will be made to address the deficiency as against duplicating or investing further in building such new capabilities. This will allow us to achieve maximum utility from existing investments.

Risk Based Approach to Security

Following a risk-based approach provides the enterprise with an opportunity to: Identify threats to projects, initiatives, data and the ongoing operation of information systems; Effectively allocate and use resources to manage those risks; Avoid unwarranted speculation, misinterpretation and inappropriate use; and Improve stakeholder confidence and trust. Information systems, data and technologies must be protected from unauthorized access and manipulation. Enterprise information must be safe-guarded against inadvertent or unauthorized alteration, sabotage, disaster or disclosure. The cost and level of safeguards and security controls must be appropriate and proportional to the value of the information assets and the severity, probability and extent of harm

Continuous Improvement

The rate of change and improvement in the worldwide information technology market has led to extremely high expectations regarding quality, availability and accessibility. As a result, ICT must deliver projects and service-level agreements (SLAs) on progressively shorter deadlines and information systems with increasingly higher quality in an effective cost-control manner. This demand requires an operating model that continuously reviews and improves upon current practices and processes. Routine tasks that can be automated should be, but only where the benefit justifies the cost. The complexity of the process, the potential time savings and the potential for error reduction should be factored into the benefit. Processes and tasks must be analyzed and understood to determine the opportunity for improvement and automation. Service outages, errors and problems need to be analyzed to understand and improve upon deficiencies in existing processes and practises. Manual integration, where data is copied from one information system to another by hand, should give way to automated processes that are repeatable, timely and less prone to error.

Responsive Change Management

Changes to the enterprise information environment are implemented in a timely manner. If people are to be expected to work within the enterprise information environment, that information environment must be responsive to their needs. Processes may need to be developed to manage priorities and expectations. This principle will, at times conflict with other principles. When this occurs, the business need must be considered but initiatives must also be balanced with other enterprise architecture principles. Without this balanced perspective short-term considerations, supposedly convenient exceptions and inconsistencies, will rapidly undermine the management of information systems.

Technology Independence

Business architecture describes the business model independent of its supporting technology and provides the foundation for the analysis of opportunities for automation. Eliminate technology constraints when defining business architecture and ensure automated processes are described at the business process level for analysis and design. Enterprise functions and IT organizations must have a common vision of both a unit’s business functions and the role of technology in them. They have joint responsibility for defining the IT needs and ensuring that the solutions delivered by the development teams meet expectations and provide the projected benefits. Independence of applications from the supporting technology allows applications to be developed, upgraded and operated under the best cost-to-benefit ratio. Otherwise technology, which is subject to continual obsolescence and vendor dependence, becomes the driver rather than the user requirements themselves.

Data is a Shared Resource

Timely access to accurate data is essential to improving the quality and efficiency of enterprise decision making. It is less costly to maintain timely, accurate data and share it from a single application than it is to maintain duplicate data in multiple applications with multiple rules and disparate management practices. The speed of data collection, creation, transfer and assimilation is driven by the ability of the enterprise to efficiently share these islands of data across the organizations. A shared data environment will result in improved decision making and support activities as we will rely on fewer sources (ultimately one) of accurate and timely managed data. Data sharing will require a significant cultural change. This principle of data sharing will need to be balanced with the principle of data security. Under no circumstance will the data sharing principle cause confidential data to be compromised.

The above is not an exhaustive list. The set of principles actually depends on the enterprise's vision and mission and as the EA is aligned to such vision and mission, the principles should also be formulated with alignment in mind. While the above principles are generic and may be used by all enterprises, it is important to state the principle in a structured manner. The principle shall be supported with a rationale, so that the users can understand, why this principle exist and to what extent the same can be traded-off when a conflict arise.