Building Secure Applications

The awareness on security has increased manifold and thanks to the frequent news headlines on security breaches and compromises leaving organizations with heavy damage and / or loss. The board has started realizing that information security is one of the primary causes of most of the IT Risks that their organization has to mitigate or to be ready to face. But the application design and development processes has not seen significant change to ensure that the delivered application is secure and would help the organization bring down the risk exposure.

 

Most of the software engineers and architects involved in application design and development still have very less security awareness and they believe that that security is something that the IT Infrastructure team will take care of. On the other hand, the IT infrastructure team believes in the investment they would have made in robust security tools and expect that they have done their part to protect the organization and pass on any application specific security issues back to the development teams. Then, whose responsibility it is to ensure building a secure application?

 

This is where an independent, enterprise wide security consulting team needs to extend the security consulting to various project teams within the organization. But still, this will not solve the problem completely, as it is for the development teams to be security aware and design and build security applications. 

 

Here is what the security consulting team can offer to the project teams:

 

Pre-project security reviews: This helps the decision makers to be aware of what security threat landscape that they are going to be dealing with upon implementation of the project. This could even have an impact on the project costing as additional cost might have to be incurred to mitigate the security risks that this project may bring on and in most cases it is not just one time investment and it has to be recurring operating expenses as well. This helps the organization to make informed decisions considering the extent of impact on the organization’s risk exposure.

 

 

Design level security reviews: Like in case of software quality, earlier one spot the security concerns, it would cheaper and easier to factor in the mitigation plans. At this stage, the security consulting teams can help the project teams in the following ways:

• A high level evaluation of the security concerns that the design could expose and suggest possible security controls to mitigate such concerns.
• Equip the project stake holders with necessary information associated with the identified security concerns, so that they can take better risk management decisions.
• Extend guidance to the project teams on the choice of controls and solutions that might best address the security concerns.
• Perform research and exploration on any the technology or feature is quite new and innovative which could potentially open a new thread landscape
• Offer training to the design and development teams about the most common security vulnerabilities and the attack patterns, so that they design and build counter measures early on.

Implementation Security Reviews: Security consulting team offers to perform the reviews and verifications in the later phase of the application development, so that vulnerabilities if any exist may not slip through to production. Typically the services offered by the security consulting team are one or more of the following:

• Ensure that the security concerns identified in the design review is fully addressed.
• Perform security specific reviews on the code and this could be on a sampling basis, depending on the acceptable risk level of the application.
• Use automated tools that will examine the code and / or the packaged component or application.
• Review the security specific test cases as created by the software testing team and suggest additional test cases to ensure better coverage in the security testing.

Having a security consulting team will not just be enough to build secure applications. The Software Engineering Process which defines and describes the approach and methodologies used for project execution should also mandate for practising and consuming the services of the Security Consulting teams and their review reports should be identified as entry and exit criteria for various phases of the application development.

 

Security education is also required to ensure that the project team is aware why security is so important for the organization. The following measures towards security education would help the teams to design and develop secure applications:

• Include a module of security education in the employee induction program, so that every new employee is oriented towards the security needs of the organization.
• Offer in-depth security training to architects and select software developers and ensure that every project team has at least couple of such trained resources. 
• Ensure that the coding guidelines document also factors writing secure code.
• Hold periodic training sessions where in addition to security related presentations, recent security review findings and or defects found during the security testing can be taken up for discussion and in the end come up with solutions to prevent such issues creeping in.
• If the organization publishes a periodic news bulletin, ensure that security is a mandated section in it and use it to publish security related updates.

As it can be observed, security is not just one person or department’s responsibility and it should be the outcome of collaborated efforts of concerned teams with the common goal to build a secure application. The security consulting team can also be external to the organization, i.e. this function can be outsourced to security consuling firms who would be specializing in the area of application security practice.

---

Software Architecture Reviews

Review is a powerful technique that contributes to software quality. Various artifacts of the software development lifecycle are subject to review to ensure that any deficiencies could be spotted early on and addressed sooner, before letting it slip through further phases and in turn consuming more efforts than expected down the line. One such important review is the review of the software architecture. If you are asked to review the architecture of a software, it could be due to one of the following reasons.

1. Possibly, you are a Senior Architect and is expected to complement your fellow Architect by reviewing his work and thereby helping him and in turn the organization to get the best possible software Design. Some or most organizations mandate this need as part of their engineering process. When this review is done effectively, the benefits are huge, as this review occurs early in the development life cycle. 
2. One or more of the custom built application(s) used in the organization are suspected to have certain serious reliability / performance issue and you are engaged to come up with an analysis and a plan to set it right. If this situation arise, then it is very much evident that the first one did not happen or it wasn’t done well. In some cases, such situation arise when the stake holders knowingly compromise on certain software quality attributes initially and then surprised to see its impact down the line as it hits back. 
3. You are possibly looking out to license a product and are evaluating its suitability to your organization. In this, case you will probably have a checklist of items created based on the IT policies and framework of your organization and this is highly dependent on the information revealed by the product vendor. 

Though there could be more reasons, the above are some of the primary reasons as to why one would need to perform an architectural review. In spite of as many reviews and testing, issues slips through and challenges the IT architects at some point down the line. Resolution of such issues may call for certain specific reviews and the method and approach would be different based on the type of the problem. For instance, if there be a data breach, a security review of the architecture is what is needed to not only identify the root cause for the current problem, but also to identify potential vulnerabilities and come up with solutions to plug those gaps.

These specific reviews can be typically associated with the broad software quality attributes, which are also termed as non-functional-requirements. The best way to approach these specific reviews is to start with an architectural review. A review checklist would be a good tool to use for the purpose, but the checklist should be exhaustive enough to cover necessary areas, so that the reviewer can get the right and required inputs and would be in a good position to form an opinion about the possible deficiencies and can relate it with the problem being attempted to be resolved.

Keep a watch on my blog for more on specific architecture reviews.

Software Testability Review

Introduction

Close to 40% of the overall software development efforts is spent in testing activities which include test design, test preparation, test execution, and test result analysis. Improvement of test criteria and coverage, test automation, use of tools test analysis and test case re-use, etc are the popular methods used to bring this cost down.

Testability of a software application also has an influence on the testing efforts, making it harder or easier to test and analyse the test results. Testability is an important factor to achieve an efficient and effective test process. Organizations attempt to achieve higher testability by adopting an systematic approach which cuts across the SDLC, which is called Testability Engineering. Software Architects or specialized test architects are called in at different stages of product design and development to ensure that the testability is designed and built as part of the system. Being an important characteristic as it plays a vital role in influencing the cost of testing. Testability review technique is one of the primary technique which when adopted at the right stages will ensure high testability. This blog focuses primarily on the Testability Reviews.

Testability Reviews

Reviews in general is a popular and widely used technique by Software project teams to isolate defects in various stages of design and development. While testers do participate in most such reviews, whether they are called upon to bring out potential testing problems during these reviews needs validation. Let us see some of the testability heuristics as derived from testing practices and related literatures.

Architecture Level

Concentrate control structure related to a particular functionality in one cluster (or class). At times, Software Arcitects tend to emphasize this aspect, which could lead to dependent functions or methods being part of different clusters and in case of distributed development, this could even be with a different development teams. This will call for unnecessary development of test stubs or test drivers while testing.

Give higher priority to the modularity of a system than to the reuse of components. This one is debatable, but the reason why it is an important testability heuristics is that a higher stress on reusability may mean the non-availability of some such dependent methods or components, leading to delays in testing. One has to carefully evaluate the pros and cons and take a call on this heuristic.

Implement a standard test interface within each domain class. Implement a standard test interface within base classes and technical classes if they are selected for direct class testing. A standard test interface in all classes causes minimal additional implementation effort and has a high potential payback.

Introduce observation points at semantically meaningful points. Observation points are helpful in being able to collect or watch states and values of various objects or variables as the code execution pass through different stage. Observation points will be very helpful to debug or troubleshoot some of the hard to simulate defects.

Map your test strategy and your design approach with respect to inheritance hierarchies. Polymorphic method calls between classes within an inheritance hierarchy may require re-testing of the superclass when a subclass changes and vice versa. There are at least three different ways you can deal with this problem:

1. Avoid inheritance, use delegation instead: re-testing is not necessary.
2. Use inheritance, perform an analysis of the dependencies between super- and subclasses, and use a selective regression testing strategy.
3. Use inheritance, don’t analyse dependencies and rerun all test cases (test automation strategy).

Design Level

Make control structures explicit. Sometimes, control structure can be hidden in data, which gives room for chances of omitting important test cases there by leading to reduced test coverage. Being explicit will help testers to overcome this problem.

Avoid cyclic dependencies, especially between methods. Cyclic dependencies make the determination of test order complicated and also necessitate stubs and drivers and this could significantly increase the testing efforts.

Avoid unmotivated polymorphic method parameters, especially if strict subtyping does not apply. This could have a multiplier effect on the number of test cases or the test data as the all combinations of parameter classes need to be tested. An alternative here could be to use strict sub typing or minimize the number of parameter classes and restrict the type by appropriate casts wherever possible.

Avoid implicit inputs and side-effects. It is easy to miss implicit input and side-effects of methods, especially if they are not well documented.

Avoid unmotivated state behavior of objects.  This could lead to increase in number of test cases as it would be necessary to test each method for each object state.

Implement a state testing function for each test relevant class and restrict its invocation by test drivers if necessary. The state of an object is an important part of the test result after each test case execution but normally not accessible from the outside. Encapsulation makes testing more difficult. Breaking the encapsulation introduces unwanted dependencies between test drivers and the class to test.

Compensate test relevant information loss by built-in-tests. Built-in-test facilities like assertions provide a way to review and control intermediate computation results, thereby reducing the impact of information loss on the testing efficiency.

There has to be at least one input element (or combination of input values) for each output element. Unachievable output values (called output inconsistencies) may be indicators for unreachable statements and paths. This when avoided makes test result analysis a lot easier.

Provide means to trigger all exceptions. Some exceptions do never occur according to theory but needs consideration as it could improve the reliability of the system. It is a challenge for testers too to being able to trigger these exceptions so that the test cases cover all exceptions. A testable exception handling requires a design strategy and perhaps simulation of failure modes.

Code Level

Don’t squeeze the code. Code readability is an important factor which will be of great help when the testers are to design test cases and achieve maximum code coverage especially in case of unit testing.
Avoid variable reuse. Variable reuse leads to implicit information loss, i.e. loss of intermediate computation results and can be avoided by using more variables.

Minimize the number of unachievable paths. Unachievable paths will impact the code coverage as it would be difficult for the testers to design test cases that traverse through such paths. In order to reduce the number of unachievable paths, avoid correlated decisions.

Avoid recursive implementations of algorithms if there is no checking of invariants. It is difficult to test recursive algorithms because we cannot easily create a stub for the component or method we want to test. A solution to this problem is to split it into pairs that call each other or to use built-in assertions.

Summary

Dealing with testability issues during reviews of software artifacts should be the first choice to achieve an effective and efficient test process. Testability is not a characteristic of code alone but applies to the architecture and design as well.  The heuristics presented above are not exhaustive as more heuristics may need consideration depending on the test & design strategies, the nature of the system, the tools and technologies used for development and testing.

Developers’ take away from a support project

Developers usually tend to prefer development projects over production support projects. Developers always want new challenges in terms of technology and would like to be using the latest tech tools and platforms. As most development projects offer them this advantage, they usually prefer to get away from production support projects.  But in reality, the production support projects do offer them certain key benefits, which are very much required as they move up in their career path. Let us examine some of these here.

The real life business scenarios

A software project begins with perceived business requirements as drafted by the Business Analysts and approved by customers. In most cases, the requirements are far from complete and that leads the developers to live with ambiguity giving room for more defects in the product that they develop. How much ever the software is tested, when it hits the production use, the real life business scenarios will for sure throw the software out of gear and makes it fail. Thus, those involved in the support projects get the opportunity to deal with production business scenarios which will sharpen their business / domain knowledge. Given that the world has started embracing the cloud and SaaS applications, there will be less of development and more of customization and managing the configurations. That means that the need for domain skills with the developers will rank very high amongst the SaaS providers and consumers.

Better product / domain knowledge

In product development, it is quite possible that a developer or a team of developers would be working on just a small part of a product. That means, the developers associated with development projects have very little opportunity to have complete understanding of the product. Whereas the developers involved in the production support would get opportunities to work with all parts of the product and some times across other products too. They get better visibility on the operating processes / practices associated with a use case, there by getting a better product / domain knowledge.

Solution design skill

Developers tend to believe that support projects do not have much opportunity in the solution design space, which is a myth. A production defect is far more difficult to deal with than a defect identified during the development life cycle. Resolution of a production defect involves at a high level the following steps:

• Quickly come up with a data fix to maintain the data integrity if impacted by the defect.
• Perform a root-cause analysis and come up with the real life scenarios that could lead to this defect being encountered.
• Come up with an interim work around if any available to prevent it from recurring in the shorter term.
• Identify a best solution to prevent it from recurring – This is rather challenging as the solution has to be designed within the existing product architecture, with lesser efforts and least impact to the already working software.

Each of the steps when done well in combination with the real life scenarios add tremendous value to the abilities of the developers and that will lead them towards software or solution architects. Solutions in support project see production quicker than the development projects and as such high appreciation from business teams. 

Code Re-factoring

Learning from one’s own mistake is a good way of learning. But, learning from other’s mistake is a smart way of learning. Every time a developer attempts to resolve a production defect, he might be looking into the code written by someone else and may come across many different ways of achieving a result. Taking it positively, a support developer may enjoy reading through the code written by others and pick up some better algorithms and at the same time, how not to write codes. This will for sure better their coding abilities.

The developers in the supporting a production instance of a software product will realize how important the readability of the code is and hopefully they will be making it their habits to write readable code with appropriate comments and indents.

Trouble shooting expertise

Usually software products are moved to production environment after atleast three levels of testing. A defect in production means that it has slipped through all the testing phases during development. So the scenario under which this comes to surface is not something that has been visualized during the development phases. Some of such defects would be very difficult to reproduce without which resolving it would be a nightmare. Those involved in support projects would quite often exposed to such scenarios and they will over a period gain good trouble shooting expertise. Read one of my other blog on Debugging performance problems.

Collaboration with other teams

During development phase, a software developer would be looking up to his lead for any clarifications on the work that is assigned to him and would not get exposed to other teams. Whereas, those involved in production support get to work with various other teams like the infrastructure, IT security, subject matter experts, quality assurance, business analysts, end users, third party vendors when any of their components are used, etc. This collaboration and interaction brings room for acquiring some additional skills both in technical space and also on the soft skill space.

Conclusion

Being in production, support projects facilitates the enterprise to perform its operations and earn profits on an ongoing basis. They play a vital part in the business continuity of the enterprise. As long as a production software is well supported and maintained, the IT heads would not think of replacing it unless a major technology overhaul is expected.

Of course, there are certain downsides of being support projects too. For instance, one may have to be on call to support any emergency and some times, a hard to crack defect could result in tremendous pressure and stress. 

Debugging a performance problem

As with any typical Application development, performance is mostly conveniently ignored in all the phases of the development life cycle. In spite of it being a key non functional requirement it mostly remains undocumented. It is more so, as the development, test and UAT environments may not really represent the real world production usage of the application as some of the performance problems could not be spotted earlier. Even if the application is put to load test, there are certain in the production environment, like data growth, user load, etc, which may lead to performance degradation over a period of time.

While most performance problems could easily be spotted and resolved, some could be a challenge and may require sleepless nights to resolve. A structured approach may help addressing such issues within reasonably quicker time frame. Here is a step by step approach which should work in most cases.

1.       Understand the production environment

It is important to understand the production environment thoroughly so as to identify the various hardware & networking resources and the middleware components involved in the application delivery. In a typical n-tiered application, it is possible that there could be multiple appliances and servers through which a requested passes through and get processed before responding back to the user with response. Also understand which of these components are capable of collecting logs / metrics or capable of being monitored in real time.

2.       Understand the specific feedback from the end users

Gather details like who noticed the performance degradation, at what time frame, whether it is repeating at pattern or just pulling the system down. Also understand if the entire application is slowing down or some specific application components are not performing. Also try to experience the problem first hand, sitting alongside an end user or if possible use an appropriate user credentials to experience the performance issue. The ‘who’ also matters as in certain circumstances, the application slow down may be for a user associated with some specific role as the amount of data to be processed and transmitted may differ based on the user role.

3.       Review available logs and metrics

Gather available logs and metrics data collected by various hardware and software components and look for information that could be relevant to the specific application, or more specifically the set of requests that could demonstrate the performance issue. As Logging itself could be performance overkill, it would be ideal to switch off the logs or to set it to collect only minimal logs. If that be the case, configure or effect necessary code change to achieve appropriate level of logging and then try to collect the required details by re-deploying the application on to a production equivalent environment.

4.       Isolate the problem area

This step is very important and could be very challenging too. Take the help of developers and performance and load testing tools, to simulate the problem and in the meanwhile monitor for key measurement data as the request and response pass through various hardware and software components.

By analyzing the data gathered from the application end user or out of the first hand experience, and with the available logs and metrics try to isolate the issue to a specific hardware or software component. This is best done by doing the following step by step:

a.       Trace the request from the UI to the final destination, which typically may be the Database.

b.      If the request could reach the final destination, then measure the time taken for the request to cross various physical and logical layers and look for any information that could cause the slow down. If a hardware resource is over utilized, it could so happen that the requests would be queued up or rejected after a time out. Look for such information in the logs.

c.       Then review the response cycle and try to spot the delays in the return path.

d.      Try the elimination technique whereby, the involved component one after the other from the bottom is cleared of performance bottleneck.

Experience and expertise on the application and the infrastructure architecture could come in handy to spot the problem area quickly. It is possible that there could be multiple problems whether contributing to the problem on hand or not. This situation may lead to shift in focus on different areas resulting in longer time to resolve the problem. It is important to always stay in focus and proceeding in the right direction.

5.       Simulate the problem in Test /UAT environment

Make sure that the findings are correct by simulating the problem multiple times. This will reveal much more data and help characterize the problem better.

6.       Perform reviews

If the problem area has already been isolated in any of the steps above, then narrow the scope of the review to the components involved in the isolated problem area. If not, then the scope of review is little wider and look for problem areas in every component involved in the request response cycle. Code reviews to debug performance issues require unique skills. For instance, looping blocks, disk usage, processor intensive operations could be the candidates for a detailed review. Similarly, in case of distributed application, look for too many back and forth calls to different physical tiers could easily contribute to performance problem. Good knowledge on the various third party components and Operating System APIs consumed in the application may sometimes be helpful.

When the problem is isolated to a server and the application components seem to have no issues, then it might be possible that any other services or components running on the server might cause load on the server resources there by impacting the application being reviewed. If the problem is isolated to Database server, then look for dead locks, appropriate indexes etc. Sometimes, lack of archival / data retention policies could result in the database tables growing in a much faster pace leading to performance degradation.

7.       Identify the root cause

By now one should have identified the specific application procedure or function that could be the cause of the problem on hand. Have it validated by doing more simulations and tests in environments equivalent to production.

8.       Come up with solution

It is just not over yet, as root cause identification should be followed by a solution. Sometimes, the solution to the problem may require change in the architecture and might have a larger impact on the entire application. An ideal solution should prevent the problem from recurring and at the same time it should not introduce newer problems and should require minimal efforts. Alternatively if the ideal solution is not a possibility with various constraints, a break-fix solution should be offered so that the business continues and also plan for having the ideal solution implemented in the longer term.

Hope this one is useful read for those of you in production support. Feel free to share your thoughts on this subject in the form of comments.

Code Review – How to get it right?

Code quality is one of the often talked about issue in a project status meeting. The magnitude of this issue will be bigger in case of maintenance products, where end users are encountering defects. While all the members of the project team very well know that code review if done well during the build phase could reduce the code quality issues by a significant percentage. The design and build process manual clearly calls it out that code review is an exit criteria for the code to move on to QA for testing. But still this issue surfaces every now and then.  

The question that comes up is whether code review was done at all or done just for the sake of process compliance. The possible reasons for reluctance among the developers to do a peer review are: Lack of reviewing skills, using the review finding against the developer for reviewer's personal advantage, inferiority complex of the developer. The other most common reason cited attributed by developers is lack of time. 

As all of us are very clear on the benefits that the code review bring on the table, let us not try to list out and discuss about the benefits. Let us attempt to list down the required skills of a code reviewer.

Subject matter (Domain) expertise:  Though most developers are not expected to be domain experts, this skill will certainly be required if the code review is expected to prevent functional defects getting slipped into the next phase. The very fact that the developers need not have domain expertise could possibly mean that the developer might have not understood the requirement as it was intended to be resulting in injection of defects. A mis-interpretation of the requirements could result in a functional defect and there is a chance to spot it if the reviewer posses the domain expertise. Some of the production defects could be unique and may not be reproducible and in such case, code review is the recourse to trouble shoot. This if done well during the build phase, could have prevented such defects surfacing in production at a later stage.

Technical Skills: The reviewer should be an expert in the technology and the programming language used. In software programming, code could be written in innumerable ways for a given requirement. However, given the standards and practices the team is expected to follow, the various quality attributes identified for the project and the goals set for the specific review, the reviewer should possess appropriate level of knowledge to spot problem areas. It is important for the reviewer to know the internal subsystems and the inter dependencies on various local and external computing resources.

Positive Attitude: This is a very important skill for the reviewer and the reviewer should never use the review findings against the developer as an individual. The issues should be considered as a team issue. The Reviewer should acknowledge the capabilities of the developer and the developer may have valid reasons for having written code in a particular way. At the end of the review, it would be a good idea to discuss the summary of the findings with the entire team, as some findings could be good learning for other team members. The organizational standards and practices should also be revisited and revised if necessary based on the nature of the findings. This may also result in identification of certain training needs for the team.

Team Skills: Both developer and the reviewer should have the common objective of producing quality code out of the build phase and they must work as a team to achieve best results. If not, the code review may happen just for name sake or may lead to personality issues which in turn would affect the project deliverable.

Attention to Details: This is an important skill a reviewer should possess to carry out effective reviews. Typically it is human to miss certain blocks of code as parts of it may appear to be correct. Unit testing is not a substitute for code review. The reviewer should with respect to each line of code, ask questions like, what if this statement fails to execute, if there is any other best way to achieve the same action, whether this could lead to potential performance issue, whether this statement may require more system resources, like Memory, CPU time, etc.

Knowledge on Tools: In addition to the above, code reviews with certain specific goals will require knowledge and expertise on appropriate debugging and diagnostic tools.

One of the key metrics in the Software Engineering space is defect injection ratio. This helps to identify the phase that has injected most number of defects. Many times, the stake holders think that it is the developers who inject defects into the delivered software. The reality however may not be completely true as much number of defects get injected in the requirements and design phase also. However, code review when rightly used, helps the development team not only to keep the defect injected in the build phase under control, but also not let the requirements / design phase defects slip into the next phase.