Building Resilient Systems - Strategies, Principles & Practices

Resilient systems are designed with the assumption that failures are inevitable. The objective is not to eliminate every failure, but to ensure that critical services can anticipate disruption, absorb impact, continue operating in a degraded but controlled mode, recover within business-defined limits, and improve after incidents. Resilience therefore combines architecture, operations, security, organizational readiness, and continuous learning.

A resilient-system strategy should align technical decisions with business outcomes such as availability targets, recovery-time objectives, recovery-point objectives, customer impact thresholds, regulatory obligations, and acceptable cost. Common strategies include redundancy, fault isolation, graceful degradation, automation, observability, incident response, disaster recovery, chaos engineering, secure-by-design practices, and disciplined post-incident learning.

1. What Resilience Means

System resilience is the capability of a system, service, or organization to continue delivering acceptable outcomes despite faults, attacks, overloads, dependency failures, operational mistakes, configuration drift, and environmental disruption. A resilient system is not merely highly available; it is observable, recoverable, adaptable, and capable of learning from failure.

Instead of trying to build a system that never breaks, resilience engineering accepts that failures are inevitable (e.g., hardware crashes, cyberattacks, spikes in traffic, or natural disasters) and focuses on ensuring the system keeps running anyway.

2. Core Principles

• Design for failure: Designing for failure means assuming everything will break and architecting the system so that no single outage can bring down the entire application. Systems, no matter how well designed, will eventually fail in some capacity. The key to maintaining service reliability and availability lies in designing systems that not only anticipate failure but are resilient enough to recover from it swiftly.
• Reduce blast radius: Isolate faults so that local failures do not cascade into full-system outages. Reducing the blast radius means limiting the damage when a component fails. It ensures that a single failure cannot crash your entire system.
• Prefer graceful degradation: Preserve essential user journeys when noncritical capabilities are unavailable. An application should maintain its core functionality and baseline user experience even when parts of the system fail, overload, or become completely unavailable. Rather than completely crashing (failing hard), the system scales back non-critical features to remain useful.
• Recover deliberately: It means choosing controlled, predictable restoration over a rushed, chaotic scramble to get services back online. When a major system fails, the instinct is to fix it as fast as possible. However, hasty recovery often triggers secondary outages or corrupts data. Deliberate recovery prioritizes correctness, safety, and stability over pure speed.
• Observe before optimizing: Build telemetry that explains symptoms, causes, dependencies, saturation, and user impact. You must measure and understand a system's actual behavior before trying to make it faster or more resilient. Without data, optimization is just guessing.
• Continuously learn: Use incidents, tests, near misses, and exercises to improve architecture and operations. Continuous learning shifts the focus from avoiding failure to actively adapting, evolving, and responding to inevitable disruptions. It is widely recognized as a foundational pillar across software architecture, organizational strategy, and socio-ecological resilience frameworks.

3. Architectural Strategies

3.1 Redundancy and Replication

 Redundancy and Replication are foundational architectural strategies used to build resilient distributed systems. While often used interchangeably, they serve different purposes: redundancy focuses on duplicating hardware, network, or software components to eliminate single points of failure (SPOF), while replication focuses on duplicating data or state across multiple nodes to ensure data durability and availability.

Redundancy removes single points of failure by running multiple instances of services, databases, queues, caches, and infrastructure components. Replication should be designed across failure domains such as availability zones, regions, racks, network segments, and cloud accounts. The design must also account for data consistency, replication lag, failover triggers, split-brain prevention, and cost.

3.2 Fault Isolation and Bulkheads

Fault isolation is an architectural strategy that limits the blast radius of a failure, ensuring that an issue in one component does not cascade and trigger a complete system outage. In distributed systems, its primary goal is to maintain overall system availability by confining malfunctions to their point of origin. Fault isolation partitions resources so one failing subsystem cannot exhaust shared capacity. Common techniques include separate thread pools, connection pools, queues, rate limits, tenant isolation, cell-based architecture, and independent deployment units. 

Named after the physical partitions in a ship's hull that keep water from flooding the entire vessel if one section punctures, the software Bulkhead Pattern splits resources into isolated pools. If a single service or downstream consumer experiences high latency or fails completely, only its dedicated resource pool is exhausted. The rest of the application continues to function normally using its own guaranteed resources. Bulkheads are especially important in distributed systems because retry storms, slow dependencies, or overloaded databases can otherwise spread failure quickly.

3.3 Timeouts, Retries, Circuit Breakers, and Backpressure

Timeouts, Retries, Circuit Breakers, and Backpressure are fundamental architectural patterns used to isolate faults, control latency, and prevent cascading system collapses. Resilient distributed systems must treat remote calls as unreliable. 

Timeouts prevent indefinite waiting. Without a timeout, a slow downstream service causes calling threads to hang indefinitely. This quickly exhausts system resource pools (like HTTP connection pools or worker threads), causing a complete system outage. Use deadline propagation (or a timeout budget). If an edge API gateway has a 5-second timeout and spends 2 seconds processing, it should pass a remaining budget of 3 seconds down to subsequent internal microservices.

Naive retries can cause a "retry storm". If a downstream service slows down due to high load, millions of clients immediately retrying will completely crush and crash that service. Increase the wait time between successive attempts (e.g., 1s → 2s → 4s → 8s) to give the downstream service time to breathe. Add randomness to the backoff delays. This breaks up synchronized traffic spikes (the "thundering herd" problem).

Circuit breakers stop repeated calls to unhealthy dependencies. Always pair a circuit breaker with a fallback mechanism. If the breaker is open, immediately return a safe default, an error message, or cached data to keep the user experience intact.

In asynchronous or event-driven architectures, an upstream service might generate events or send data far faster than the downstream consumer can process them. This causes the consumer's memory queues to bloat, eventually leading to OutOfMemory crashes. Backpressure protects services from overload by slowing or rejecting excess work before collapse, wherein the consumer signals its current capacity back upstream, forcing the producer to slow down or buffer its output.

3.4 Graceful Degradation

Graceful degradation is a foundational architectural strategy for building resilient systems that ensures an application maintains its core functionality even when specific dependencies fail or experience severe resource constraints. Instead of crashing completely or returning generic error pages, the system strategically scales back non-critical features, serving a subset of capabilities to guarantee business continuity.

Graceful degradation preserves the most important outcomes when parts of the system fail. Examples include read-only modes, cached responses, reduced personalization, delayed processing, feature flags, queue-based buffering, static fallback pages, and prioritization of critical transactions over optional features.

4. Operational Strategies

4.1 Observability and Monitoring

Monitoring tells teams when known failure conditions occur; observability helps teams understand unknown failure modes. Effective observability combines metrics, logs, traces, events, service maps, synthetic checks, real-user monitoring, and business-impact signals. Alerts should be actionable, tied to user impact, and supported by runbooks.

4.2 Incident Response and Runbooks

Incident response should define severity levels, ownership, escalation paths, communication channels, customer messaging, decision rights, and restoration priorities. Runbooks should be concise, tested, and updated after incidents. Strong incident management separates mitigation from root-cause analysis so teams can restore service first and investigate deeply afterward.

4.3 Disaster Recovery and Business Continuity

Disaster recovery planning translates business tolerance for downtime and data loss into recovery-time objectives and recovery-point objectives. Strategies range from backup-and-restore to pilot-light, warm-standby, active-passive, and active-active architectures. The right option depends on criticality, cost, operational complexity, data consistency requirements, and regulatory expectations.

5. Security and Cyber Resilience

Cyber resilience expands traditional reliability by assuming that systems may be attacked, compromised, or misused. Strategies include zero-trust access, least privilege, segmentation, immutable infrastructure, secure configuration baselines, backup protection, tamper-resistant logging, rapid credential rotation, threat detection, and rehearsed recovery from ransomware or destructive attacks.

Security controls should be integrated into the system life cycle rather than added after deployment. This includes threat modeling, secure design reviews, dependency management, vulnerability remediation, security observability, and incident-response plans that coordinate engineering, security, legal, communications, and business stakeholders.

6. Validation Through Testing and Chaos Engineering

While traditional software testing ensures that code meets business requirements under normal conditions, chaos engineering proactively injects controlled disruptions into a system to find hidden vulnerabilities before they cause costly production outages.

Resilience must be proven, not assumed. Teams should test backup restoration, failover, capacity limits, dependency outages, network partitions, deployment rollbacks, and data-recovery procedures. Chaos engineering introduces controlled experiments to validate whether systems behave as expected under failure. Experiments should begin in low-risk environments, have clear hypotheses, include safety controls, and produce actionable improvements.

Modern Resilience Testing unifies three distinct strategies to ensure comprehensive platform coverage:

• Chaos Testing: Validates fault-tolerance and auto-healing capabilities by introducing controlled infrastructure and application faults.
• Load Testing: Simulates high traffic volume to uncover performance bottlenecks and ensure auto-scaling mechanisms trigger properly.
• Disaster Recovery (DR) Testing: Evaluates full-scale backup and failover procedures when an entire cloud region or data center goes offline.

 

7. Implementation Roadmap

1. Map critical services: Identify user journeys, dependencies, data flows, owners, and business impact.
2. Define resilience objectives: Set availability, latency, RTO, RPO, error-budget, and customer-impact targets.
3. Assess failure modes: Review single points of failure, scaling bottlenecks, operational gaps, security risks, and third-party dependencies.
4. Prioritize controls: Address high-impact, high-likelihood risks first using redundancy, isolation, fallback, observability, and automation.
5. Rehearse recovery: Conduct game days, tabletop exercises, restore tests, failover drills, and incident simulations.
6. Measure and improve: Track reliability indicators and feed lessons into architecture, runbooks, staffing, and investment decisions.

8. Metrics to Track

Metric	Mandatory Trigger
Availability and error rate	Measure whether users can successfully complete key actions.
Latency percentiles	Detect degradation that averages may hide.
Mean time to detect and recover	Evaluate operational responsiveness and recovery effectiveness.
RTO and RPO achievement	Validate whether disaster recovery meets business commitments.
Change failure rate	Understand whether deployments and configuration changes introduce instability.
Backup restore success	Confirm that recovery assets are usable when needed.

9. Common Pitfalls

• Confusing high availability with complete resilience.
• Adding retries without backoff, jitter, limits, or circuit breakers.
• Replicating data without understanding consistency and recovery trade-offs.
• Creating dashboards that do not answer operational questions during incidents.
• Maintaining runbooks that are untested, outdated, or too complex for crisis use.
• Assuming backups are reliable without regular restore validation.
• Designing for technical failure while ignoring people, process, suppliers, and communication.

10. Conclusion

Building resilient systems is a strategic discipline. The strongest programs combine thoughtful architecture, operations readiness, cybersecurity resilience, automated safeguards, tested recovery plans, and a culture that learns from failure. The practical goal is to keep essential outcomes available, limit the impact of disruption, recover within agreed business limits, and continuously strengthen the system as conditions change.

How Artificial Intelligence is Reshaping the Software Development Life Cycle (SDLC)

Artificial Intelligence (AI) is no longer a futuristic concept confined to research labs. It has reshaped numerous industries, with software engineering being one of its most profoundly affected domains. It’s a powerful, tangible force transforming every stage of the Software Development Life Cycle (SDLC). From initial planning to final maintenance, AI tools are automating tedious tasks, boosting code quality, and accelerating the pace of innovation, marking a fundamental shift from traditional, sequential processes to a more dynamic, intelligent ecosystem.

In the past, software engineering depended heavily on human expertise for tasks like gathering requirements, designing systems, coding, and performing functional tests. However, this landscape has changed dramatically as AI now automates many routine operations, improves analysis, boosts collaboration, and greatly increases productivity. With AI tools, workflows become faster and more efficient, giving engineers more time to concentrate on creative innovation and tackling complex challenges. As these models advance, they can better grasp context, learn from previous projects, and adapt to evolving needs.

AI is streamlining the software development lifecycle (SDLC), making it smarter and more efficient. This article explores how AI-driven platforms shape software development, highlighting challenges and strategic benefits for businesses using Agile methods.

Impact Across the SDLC Phases

The Software Development Life Cycle (SDLC) has long been a structured framework guiding teams through planning, building, testing, and maintaining software. But with the rise of artificial intelligence—especially generative AI and machine learning—the SDLC is undergoing a profound transformation. Let’s explore how each phase of the SDLC is getting transformed into.

1. Project Planning:

AI streamlines project management by automating tasks, offering data-driven insights, and supporting predictive analytics. This shift allows project managers to focus on strategy, problem-solving, and leadership rather than administrative duties.

• Automated Task Management: AI automates time-consuming, repetitive administrative tasks like scheduling meetings, assigning tasks, tracking progress, and generating status reports.
• Predictive Analytics and Risk Management: By analyzing vast amounts of historical data and current trends, AI can predict potential issues like project delays, budget overruns, and resource shortages before they occur. This allows for proactive risk mitigation and contingency planning.
• Optimized Resource Allocation: AI algorithms can analyze team members' skills, workloads, and availability to recommend the most efficient allocation of resources, ensuring that the right people are assigned to the right tasks at the right time.
• Enhanced Decision-Making: AI provides project managers with real-time, data-driven insights by processing large datasets faster and more objectively than humans. It can also run "what-if" scenarios to simulate the impact of different decisions, helping managers choose the optimal course of action.
• Improved Communication and Collaboration: AI tools can transcribe and summarize meeting notes, identify action items, and power chatbots that provide quick answers to common project queries, ensuring all team members are aligned and informed.
• Cost Estimation and Control: AI helps in creating more accurate cost estimations and tracking spending patterns to flag potential overruns, contributing to better budget adherence.

2. Requirements Gathering

This phase traditionally relies on manual documentation and subjective interpretation. AI introduces data-driven clarity.

• Requirements Gathering: AI can transcribe meetings, summarize discussions, and automatically format conversations into structured documents like user stories and acceptance criteria. It can also analyzes raw stakeholder input, market research, and other unstructured data to identify patterns and key requirements.
• Automated Requirements Analysis: Artificial intelligence technologies are capable of evaluating requirements for clarity, completeness, consistency, and potential conflicts, while also identifying ambiguities or incomplete information. Advanced tools employing Natural Language Processing (NLP) systematically analyze user stories, technical specifications, and client feedback—including input from social media platforms—to detect ambiguities, inconsistencies, and conflicting requirements at an early stage. Additionally, AI systems can facilitate interactive dialogues to clarify uncertainties and reveal implicit business needs expressed by analysts.
• Non-Functional Requirements: AI tools help identify non-functional needs such as regulatory and security compliance based on the project's scope, industry, and stakeholders. This streamlines the process and saves time.

3. Design and Architecture

AI streamlines software design by speeding up prototyping, automating routine tasks, optimizing with predictive analytics, and strengthening security. It generates design options, translates business goals into technical requirements, and uses fitness functions to keep code aligned with architecture. This allows architects to prioritize strategic innovation and boosts development quality and efficiency.

• Optimal Architecture Suggestions: Generative AI agents can analyze project constraints and suggest optimal design patterns and architectural frameworks (like microservices vs. monolithic) based on industry best practices and past successful projects.
• Automated UI/UX Prototyping: Generative AI can transform natural language prompts or even simple hand-drawn sketches into functional wireframes and high-fidelity mockups, significantly accelerating the design iteration process.
• Automated governance and fitness functions: AI can generate code for fitness functions (which check if the implementation adheres to architectural rules) from a higher-level description, making it easier to manage architectural changes over time.
• Guidance on design patterns: AI can analyze vast datasets of real-world projects to suggest proven and efficient design patterns for complex systems, including those specific to modern, dynamic architectures.
• Focus on strategic innovation: By handling more of the routine and complex analysis, AI allows human architects to focus on aligning technology with long-term strategy and fostering innovation.

4. Development (Coding)

AI serves as an effective "pair programmer", automating repetitive tasks and improving code quality. This enables developers to concentrate on complex problem-solving and design, rather than being replaced.

• Intelligent Code Generation: Tools like GitHub Copilot and Amazon CodeWhisperer use Large Language Models (LLMs) to provide real-time, context-aware code suggestions, complete lines, or generate entire functions based on a simple comment or prompt, dramatically reducing boilerplate code.
• AI-Powered Code Review: Machine learning models are trained on vast codebases to automatically scan and flag potential bugs, security vulnerabilities (like SQL injection or XSS), and code style violations, ensuring consistent quality and security before the code is even merged.
• Documentation and Code Explanation: Using Natural Language Processing (NLP), AI can generate documentation and comments from source code, ensuring that projects remain well-documented with minimal manual effort.
• Learning and Upskilling: AI serves as an interactive learning aid and tutor for developers, helping them quickly grasp new programming languages or frameworks by explaining concepts and providing context-aware guidance.

AI is shifting developers’ roles from manual coding to strategic "code orchestration." Critical thinking, business insight, and ethical decision-making remain vital. AI can manage routine tasks, but human validation is necessary for security, quality, and goal alignment. Developers skilled in AI tools will be highly sought after.

5. Testing and Quality Assurance (QA)

AI streamlines software testing and quality assurance by automating tasks, predicting defects, and increasing accuracy. AI tools analyze data, create test cases, and perform validations, resulting in better software and user experiences.

• Automated Test Case Generation: AI can analyze requirements and code logic to automatically generate comprehensive unit, integration, and user acceptance test cases and scripts, covering a wider range of scenarios, including complex edge cases often missed by humans.
• Predictive Bug Detection: AI-powered analysis of code changes, historical defects, and application behavior can predict which parts of the code are most likely to fail, allowing QA teams to prioritize testing efforts where they matter most.
• Self-Healing Tests: Advanced tools can automatically update test scripts to adapt to UI changes, drastically reducing the maintenance overhead for automated testing.
• Smarter visual validation: AI-powered tools can perform visual checks that go beyond simple pixel-perfect comparisons, identifying meaningful UI changes that impact user experience.
• Predictive analysis: AI uses historical data to predict areas with higher risk of defects, helping to prioritize testing efforts more efficiently.
• Enhanced performance testing: AI can simulate real user behavior and stress-test software under high traffic loads to identify performance bottlenecks before they affect users.
• Continuous testing: AI integrates with CI/CD pipelines to provide continuous, automated testing throughout the development lifecycle, enabling faster and more frequent releases without sacrificing quality.
• Data-driven insights: By analyzing vast datasets from past tests, AI provides valuable, data-driven insights that lead to better decision-making and improved software quality assurance processes.

6. Deployment

Artificial intelligence is integral to modern software deployment, streamlining task automation, enhancing continuous integration and delivery (CI/CD) pipelines, and strengthening system reliability with advanced monitoring capabilities. AI-driven solutions automate processes such as testing and deployment, analyze performance metrics to anticipate and address potential issues, and detect security vulnerabilities to safeguard applications. By transitioning deployment practices from reactive to proactive, AI supports greater efficiency, stability, and security throughout the software lifecycle.

• Intelligent CI/CD: AI can analyze deployment metrics to recommend the safest deployment windows, predict potential integration issues, and even automate rollbacks upon detecting critical failures, ensuring a more reliable Continuous Integration/Continuous Deployment pipeline.
• Automated testing and code review: AI automates code quality checks, identifies vulnerabilities, and uses intelligent test automation to prioritize tests and reduce execution time.
• Streamlined processes: By automating routine tasks and using data to optimize workflows, AI helps streamline the entire delivery pipeline, reducing deployment times and improving efficiency.

7. Operations & Maintenance

AI streamlines software operations by predicting failures, automating coding and testing, and optimizing resources to boost performance and cut costs.

• Real-Time Monitoring and Observability: AI-driven tools continuously monitor application performance metrics, system logs, and user behavior to detect anomalies and predict potential performance bottlenecks or system failures before they impact users.
• Automated Documentation: AI can analyze code and system changes to automatically generate and update technical documentation, ensuring that documentation remains accurate and up-to-date with the latest software version.
• Root Cause Analysis: AI tools can sift through massive amounts of logs, metrics, and traces to find relevant information, eliminating the need for manual, repetitive searches. AI algorithms identify subtle and complex patterns across large datasets that humans would miss, linking seemingly unrelated events to a specific failure. By automating the initial analysis and suggesting remediation steps, AI significantly reduces the time-to-resolution for critical bugs.

The Future: AI as a Team Amplifier, Not a Replacement

The integration of artificial intelligence into the software development life cycle (SDLC) does not signal the obsolescence of software developers; rather, it redefines their roles. AI facilitates automation of repetitive and low-value activities—such as generating boilerplate code, creating test cases, and performing basic debugging—while simultaneously enhancing human capabilities.

This evolution enables developers and engineers to allocate their expertise toward higher-level, strategic concerns that necessitate creativity, critical thinking, sophisticated architectural design, and a thorough understanding of business objectives and user requirements. The AI-supported SDLC promotes the development of superior software solutions with increased efficiency and security, fostering an intelligent, adaptive, and automated environment.

AI serves to augment, not replace, the contributions of human engineers by managing extensive data processing and pattern recognition tasks. The synergy between AI's computational proficiency and human analytical judgment results in outcomes that are both more precise and actionable. Engineers are thus empowered to concentrate on interpreting AI-generated insights and implementing informed decisions, as opposed to conducting manual data analysis.

A Software Product Vs Project

In short, a software Project is all about to execute a Statement of Work of an internal or external customer, where what customer required is right irrespective of what is ideal or what the end user would expect. Though some projects are scoped in such a way that certain aspects of non-functional requirements are left to the choice of the project teams.

Product development isn’t about implementing what the customer wanted to. In product development, the product manager owns and comes up with the product requirements. A large product or product suite, typically comprise of many projects and will evolve over time.

Unlike a project the product will be improved continuously without an end date based on feedback from end users and the product team prioritizes what needs to be built next based on its perceived value for its target users or customers.

A project on the other hand is funded with specific goals, a business case in mind and with finite expected value and cost.

Here is an attempt to bring out the differences between a software project and product and such differences are categorised as below:

The Mindset:

Projects are many a times started off with main focus on to deliver on time, under budget, within scope and with a temporary team. All these constraints are set in stone and any deviation is viewed seriously, which may impact the course of the project depending on the methodology adopted. So, the mindset of the project team will be with primary focus on the project parameters that determine the success of delivery and may not be the success of the product that the project may form part of. This is more so as the resources keep changing and the resources with no or little knowledge on the business domain may still deliver the project, but the product may be crappy.

Products tend to have a longer lifetime than projects and mostly built with more focus on the outcome instead of the output. Product teams are given the freedom and responsibility to think of a strategy they believe will result in the best product within a boundary of product framework. This leads to less waste and more creativity being introduced into the product development process, allowing room for embracing changes continously.

Management:

The product roadmap is key for the success of the prodct and as such, the product manager shall align the product vision and strategy with that of the business. A Project Manager, on the other hand, is responsible for executing on a predefined objective.

A Project Managers function is to create a plan, that the project will follow, and then to drive the people involved in the project to follow that plan with as little change as possible. If deviations from the planned execution are beyond an accepted threshold, the Project Manager must escalate and explain the situation to the stakeholders, who in turn will either accept the deviation or may choose to fail the project.

A product manager with the focus on constantly evaluating the viability of the product, will typically follow an agile approach with shorter sprints of developments, so the product evolves incrementally, delivering values at every stage.

Motivation:

With the primary focus of the project team being on delivering on time and within budget, the team does not have enough room to be creative enough. This brings down the motivation because the teams lose a sense of purpose and the autonomy in how to operate.

On the other hand, as typically, the resources stay longer with the product teams, they get aligned to the product strategy and the vision and thus they are given the freedom to bring in their thinking and creativity into the product, process and methodology. The feedback and collaboration with stakeholders enables the right environment, where the resources reach a higher potential and operate autonomously, resulting in better problem solving, higher ownership of outcomes, and faster time to market.

Tools:

Product management software and project management software are entirely different tools — each designed for a different type of role, to help address different business needs. Product management software helps product managers organize, develop, and communicate the product strategy, while project management software helps project managers in track the execution and incidentally manage the resource allocation, risk and issue management.

Scope:

Product scope is defined as "The features and functions that characterize a product, service, or result". Whereas the project scope is defined as "The work performed to deliver a product, service, or result with the specified features and functions".

The Product Scope defines all the capabilities of a product from the User point of view. The Product is the end result of your project and characterizes by the Product Scope. Thus, the Product Scope description includes features of a product, how the product will look like using these features, and how will it work. Product Scope also describe the ways of measuring the product performance.

The Project Scope on the other hand is an agreement of the work which is needed to deliver the product, service, or result. To develop a product features, you establish a project which has a schedule, budget, and resource allocation. In other words, the work you do to construct your product is the Project Scope.

Design & Architecture:

The product owner or manger is responsible for defining the architecture and design of the product, which should take the following into consideration:

• Business Idea & Strategy
• Identifying and Creating a product feature
• Aligning with Market Trends
• Define Product Performance Indicators
• Prioritize the implementation of features and bugs

Though a project may include the product architecture and design as part of the scope, the focus of the project team will be more on the following:

• Defining the project scheduling, taking into account the deliverables at various milestones.
• Monitoring the budget
• Planning and managing resources
• Problem and issue management
• Risk management
• Managing the scope creep.

Factors Affecting Software Resiliency

The digital transformation is happening everywhere right from small private firms to government organizations. On the personal front, connected things is coming on, where by every thing that we have or use will be smart enough to connect and communicate with other things(systems). This in effect means there will be an increased reliance on IT systems to accomplish various tasks. This will call for high order of resilience on the part of such systems and the absence of which may lead to disasterous situation.

As we all know, the word resiliency means 'the ability to bounce-back after some events'. In otherwords, it is a capability of withstanding any shock or impact without any major deformation or rupture. In software terms, resilience is the persistence of the avoidance of failures when facing a change or in a deviated circumstance.

To design a resilient system, one should first understand the various factors that work against the resiliency. Here are some such factors:

Design Flaws

Design and Architecture of the systems is a major factor that works in favor or against the resiliency requirement. The architects shall while designing the system or solution should have a good understanding of what could go wrong and provide for an exception handling ability, so that all exceptions are appropriately handled, making the system not to go down and instead recover from such exception and continue to operate. The architects have many options today in terms of tools, technologies, standards, methodologies and frameworks that help buidling resiliency within. It is the ability of choosing the right combination of tools, technologies, etc for the specific systems that will decide on the resilience capability of the system. 

Software Complexity

The size and complexity of software systems is increasing, thus the ways in which a system can fail also increases. It is fair to assume that the increase in failure possibilities does not bear a linear or additive relationship to system complexity. Typically, the complexity of the software systems increases as it evolves by responding to the changing business needs. This is more so as the tools and technologies used to design and build the software are becoming outdated, making it difficult in maintaining the systems. 

This complexity attribute makes it increasingly difficult to incorporate resiliency routines that will respond effectively to failures in the individual systems and in their complex system. The cost of achieving an equivalent level of resiliency due to the complexity factor should be added to that of the individual systems

Interdependency and Interconnectivity

We are living in a connected world and systems of many of today's businesses depend on connectivity with their partner entities to do their business. This adds multiple points of failures over and above the network connectivity. The system resiliency is increasingly dependent on the resiliency of systems different other organizations over which the entity has no control. This means that a failure or outage of a business partner's system can have a ripple effect. This situation requires the systems need to be aware and capable of such failure or outage with other connected systems and the ability to recover from such events should be designed within. 

Rapid Changes

Thanks to the evolving digital economy, the business needs are changing too frequently and thus needing system changes. Every change in an existing system, for sure will add a bit of complexity, as the architecture on which the system originally designed wouldn't have considered the changes that are coming through. Many a times, considering the time to market, such changes need to be implemented quicker than expected, leaving the software designers to adopt a quick and dirty approach to deliver the change, leaving a permanent solution for a later time period. The irony is that there will never be a time when the 'permanent solution' is implemented.

Change is one of the key source of adding complexity to the Software systems. However, the evolving tools, technologies and methodologies come to the rescue, so that the Architects design systems and solutions in such a way to pave way for embracing such changes and to embed the resiliency factors in the design.

A frequently held criticism of Common Criteria testing is that, by the time the results are available, there is a good chance that the tested software has already been replaced. The danger here is that the new software may contain new vulnerabilities that may not have existed in prior versions. Thus, determining that an obsolete piece of software is sufficiently resilient is not particularly indicative of the state of the newest version and, therefore, is not very useful

Conclusion

Higher levels of resilience can be achieved by leveraging Machine Learning and Big Data tools and techniques. As the world is moving towards more and more connected things, high order of resilience is critical. With Machine Learning capability, the systems and devices can be embedded with algorithms that make them learn from past events and the data collected from various other connected networks and systems in addition to the ambient data. The systems can be designed to predict the health of various underlying components and thus its own health as well. Based on such prediction, the components may choose to use alternate approaches, like using alternate network protocols like Wireless, Bluetooth, etc, or choose to connect to a different component or system altogether.

A Checklist for Architecture & Design Review

Mostly the security requirements remain undocumented and is left to the choice or experience of the architects and developers thus leaving vulnerabilities in the application, which hackers exploit to launch an attack on the enterprise's digital assets. Security threats are on the rise and is now being considered as a Board Item as the impact of security breach is very high and could cause monetary and non monetary losses.

One of the key aspects of the IT Governance is to ensure that the investments made in software assets are optimal and there is a quantifiable return on such investments. This also means that such investment does not lead to risks that could lead to damages. Most of us are well aware that reviews play a key role in ensuring the quality of the software assets. As such, in this blog post, I have tried to come up with a checklist for reviewing the architecture and design of a software application.

While the choice of specific design best practice is interdependent on another, a careful tradeoff is necessary. For a detailed discussion on Trade off Analysis of Software Quality Attributes. Each of the checklist item listed here needs further elaboration and identification of specific practices, which will depend on the enterprise architecture and design principles of the organization.

Deployment Considerations

• The design references the security policy of the organization and is in compliance of the same.
• The application components are designed to comply with the various networking and other infrastructure related security restrictions like firewall rules, using appropriate secure protocols, etc.
• The trust level with which the application accesses various resources are known and are in line with the acceptable practices.
• The design supports the scalability requirements such as clustering, web farms, shared session management.
• The design identifies the configuration / maintenance points, and the access to the same is manageable.
• Communication with various local or remote components of the application is using secure protocols.
• The design addresses performance requirements by adhering to relevant design best practices.

Application Architecture Considerations

Input Validation

• Whether the design identifies all entry points and trust boundaries of the application.
• Appropriate validations are in place for all inputs that comes from ourside the trust boundary.
• The input validation strategy that the application adopted is modular and consistent.
• The validation approach is to constrain, reject, and then sanitize input.
• The design addresses potential canonicalization issues.
• The design addresses SQL Injection, Cross Site Scripting and other vunerabilities
• The design applies defense in depth to the input validation strategy by providing input validation across tiers.

Authentication

• The design identifies the identities or roles that are used to access resources across the trust boundaries.
• Service account or such other predefined identity requirements to, if so needed to access variuos system resources are identified and documented.
• User credentials or authentication tokens are stored in secure manner and access to the same is appropriately controlled and managed.
• Where the credentials are shared over the network, appropriate security protocol and encryption techniques are used.
• Appropriate account management policies are considered.
• In case of authentication failures, the error information displayed is minimal so that it does not reveal any clues that could make the credential guessing easier.
• The design adopts a policy of using least-privileged accounts.
• Password digests with salt are stored in the user store for verification.
• Password rules are defined so that the stronger passwords are enforced.

Authorization

• The user role design offers sufficient separation of privileges and considers authorization
• granularity.
• Multiple gatekeepers are envisaged for defense in depth.
• The application’s identity is restricted in the database to access-specific stored procedures and does not have permissions to access tables directly.
• Access to system level resources are restricted unless there is an absolute necessity.
• Code Access Security requirements are established and considered.

Configuration Management

• Stronger authentication and authorization is considered for access to administrration modules.
• Secure protocols are used for remote administration of the application.
• Configuration data is stored in a secured store and access to the same is appropriately controlled and managed
• Least-privileged process accounts and service accounts are used.

Sensitive Data

• Design recognizes sensitive data and considers appropriate checks and controls on the same.
• Database connections, passwords, keys, or other secrets are not stored in plain text.
• The design identifies the methodology to store sensitive data securely. Appropriate algorithms and
• key sizes are used for encryption. 
• Error logs, audit logs or such other application logs does not store sensitive data in plain text.
• The design identifies protection mechanisms for sensitive data that is sent over the network.

Session Management

• The contents of authentication cookies are encrypted.
• Session lifetime is limited and times out upon expiration.
• Session state is protected from unauthorized access.
• Session identifiers are not passed in query strings.

Cryptography

• Platform-level cryptography is used and it has no custom implementations.
• The design identifies the correct cryptographic algorithm and key size for the application’s data encryption requirements.
• The methodology to secure the encryption keys is identified and the same is in line with the acceptable best practices.
• The design identifies and establishes the key recycle policy for the application.

Parameter Manipulation

• All input parameters are validated including form fields, query strings, cookies, and HTTP headers.
• Sensitive data is not passed in query strings or form fields.
• HTTP header information is not relied on to make security decisions.
• View state is protected using MACs.

Exception Management

• The design outlines a standardized approach to structured exception handling across the application.
• Application exception handling minimizes the information disclosure in case of an exception.
• Application errors are logged to the error log, and the design provides for periodic review of such logs.
• Sensitive data is not logged as part of the error logs, but where necessary, the same is logged with appropriate de-identification technique

Auditing and Logging

• The design identifies the level of auditing and logging necessary for the application and identifies the key parameters to be logged and audited.
• The design considers how to flow caller identity across multiple tiers at the operating system or application level for auditing.
• The design identifies the storage, security, and analysis of the application log files

Webservice Security Standards

SOA adoption is on the rise and Webservices is predominantly used for its implementation. Webservice messages are sent across the network in an XML format defined by the W3C SOAP specification. Webservices have come a long way and has sufficiently matured to offer the required tenets especially on the security domain. In this blog let us have a quick look at the available standards with respect to the security dimensions and look at how the related security requirements are addressed.

Secure Messaging

• WS-Security - This specification was originally developed by IBM, Microsoft and Verisgn and OASIS (Organization for the Advancement of Structured Information Standards) continued the work on this standard. This standard addresses the Integrity and Confidentiality requirements of the webservice messages. The specification describes the signing, encrypting of the SOAP messages and also about attaching security tokens. Various signature formats and encryption algorithms are supported. The security tokens supported include: X.509 Certificates, Kerberos tickets, User ID/Password credentials, SAML assertions and custom tokens. Due to the increased size of the SOAP messages and the cryptographic requirements, this standard requires significantly higher compute resources and network bandwidth.
• SSL/TLS - SSL was developed by Netscape Communications Corporation in 1994 to secure transactions over the World Wide Web. Soon after, the Internet Engineering Task Force (IETF) began work to develop a standard protocol that provided the same functionality. They used SSL 3.0 as the basis for that work, which became the TLS protocol. In applications design, TLS is usually implemented on top of any of the Transport Layer protocols, encapsulating the application-specific protocols such as HTTP, FTP, SMTP, NNTP and XMPP. Historically it has been used primarily with reliable transport protocols such as the Transmission Control Protocol (TCP). This standard helps address the Strong authentication, message privacy and integrity requirements.

Resource Protection

• XACML - eXtensible Access Control Markup Language defines a declarative access control policy language implemented in XML and a processing model describing how to evaluate access requests. Version 3.0 of this standard has been published by OASIS in January 2013. The new features of the latest version of this standard include: Multiple Decision Profile, Delegation, Obligation Expressions, Advice Expressions and Policy Combination Algorithms.While there are many ways the base language can be extended, many environments will not need to do so. The standard language already supports a wide variety of data types, functions, and rules about combining the results of different policies. In addition to this, there are already standards groups working on extensions and profiles that will hook XACML into other standards like SAML and LDAP, which will increase the number of ways that XACML can be used.
• XrML - Developed by Content Guard, a subsidiary of Xerox, and supported by Microsoft, eXtensible Rights Markup Language would provide a universal method for specifying rights and issuing conditions associated with the use and protection of content in a digital rights management system. XrML licenses can be attached to WS-Security in the form of tokens. XACML and XrML both deal with authorization. They share requirements from many of the same application domains. Both share the same concepts but use different terms. Both are based on XML Schema. Microsoft's Active Directory Rights Management Services (AD RMS) uses the eXtensible rights Markup Language (XrML) in licenses, certificates, and templates to identify digital content and the rights and conditions that govern use of that content.
• RBAC, ABAC - Similar to XrML, RBAC and ABAC are established approaches to define and implement Role Based Access Control and Attribute Based Access Controls and can be attached to WS-Security as tokens. The use of RBAC or ABAC to manage user privileges (computer permissions) within a single system or application is widely accepted as a best practice.
• EPAL - The Enterprise Privacy Authorization Language (EPAL) is an interoperability language for exchanging privacy policy in a structured format between applications and can be leveraged for addressing the privacy concerns with the SOAP messages. An EPAL policy categorizes the data an enterprise holds and the rules which govern the usage of data of each category. Since EPAL is designed to capture privacy policies in many areas of responsibility, the language cannot predefine the elements of a privacy policy. Therefore, EPAL provides a mechanism for defining the elements which are used to build the policy.

Negotiation of Contracts

• ebXML - e-business XML is a modular suite of standards advanced by OASIS and UNCEFACT and approved as ISO 15000. While the ebXML standards seek to provide formal XML-enabled mechanisms that can be implemented directly, the ebXML architecture is focused on concepts and methodologies that can be more broadly applied to allow practitioners to better implement e-business solutions. ebXML provides companies with a standard method to exchange business messages, conduct trading relationships, communicate data in common terms and define and register business processes. A CPA (Collaboration Protocol Agreement) document is the intersection of two CPP documents, and describes the formal relationship between two parties.
• SWSA - The SWSA(Semantic Web Services Architecture) interoperability architecture covers the support functions to be accomplished by Semantic Web agents (service providers, requestors, and middle agents). While not all operational environments will find it necessary to support all functions to the same degree, the distributed functions to be addressed by this architecture to include: Dynamic Service Discovery, Service Engagement (Negotiating & Contracting), Service Process Enactment & Management, Semantic Web Community Support Services, Semantic Web Service Lifecycle & Resource Management Services and Cross Cutting Issues.

Trust Management

• WS-Trust - The goal of WS-Trust is to enable applications to construct trusted SOAP message exchanges. This trust is represented through the exchange and brokering of security tokens. This specification provides a protocol agnostic way to issue, renew, and validate these security tokens. The Web service security model defined in WS-Trust is based on a process in which a Web service can require that an incoming message prove a set of claims (e.g., name, key, permission, capability, etc.). If a message arrives without having the required proof of claims, the service SHOULD ignore or reject the message. A service can indicate its required claims and related information in its policy as described by WS-Policy and WS-PolicyAttachment specifications.
• XKMS - XML Key Management Specification is a protocol developed by W3C which describes the distribution and registration of public keys. Services can access an XKMS compliant server in order to receive updated key information for encryption and authentication. The XML Key Management Specification (XKMS) allows for easy management of the security infrastructure, while the Security Assertion Markup Language (SAML) makes trust portable. SAML provides a mechanism for transferring assertions about authentication of entities between various cooperating entities without forcing them to lose ownership of the information.
• SAML - Security Assertion Markup Language is a product of the OASIS Security Services Technical Committee intended for exchanging authentication and authorization data between parties, in particular, between an identity provider and a service provider. SAML allows business entities to make assertions regarding the identity, attributes, and entitlements of a subject (an entity that is often a human user) to other entities, such as a partner company or another enterprise application. SAML specifies three components: assertions, protocol, and binding. There are three assertions: authentication, attribute, and authorization. Authentication assertion validates the user's identity. Attribute assertion contains specific information about the user. And authorization assertion identifies what the user is authorized to do. Protocol defines how SAML asks for and receives assertions. Binding defines how SAML message exchanges are mapped to Simple Object Access Protocol (SOAP) exchanges.
• WS-Federation - WS-Federation extends the WS-Security, WS-Trust and WS-SecurityPolicy by describing how the claim transformation model inherent in security token exchanges can enable richer trust relationships and advanced federation of services. A fundamental goal of WS-Federation is to simplify the development of federated services through cross-realm communication and management of Federation Services by re-using the WS-Trust Security Token Service model and protocol. A variety of Federation Services (e.g. Authentication, Authorization, Attribute and Pseudonym Services) can be developed as variations of the base Security Token Service. 

Security properties

• WS-Policy, WS-SecurityPolicy - WS-Policy represents a set of specifications that describe the capabilities and constraints of the security policies on intermediaries and end points and how to associate policies with services and end points. Web Services Policy is a machine-readable language for representing these Web service capabilities and requirements as policies. Policy makes it possible for providers to represent such capabilities and requirements in a machine-readable form. A policy-aware client uses a policy to determine whether one of these policy alternatives (i.e. the conditions for an interaction) can be met in order to interact with the associated Web Service. Such clients may choose any of these policy alternatives and must choose exactly one of them for a successful Web service interaction. Clients may choose a different policy alternative for a subsequent interaction.
• WS-ReliableMessaging, WS-Reliability - WS-ReliableMessaging, was originally written by BEA Systems, Microsoft, IBM, and Tibco and later submitted to the OASIS Web Services Reliable Exchange (WS-RX) Technical Committee for adoption and approval.Prior to WS-ReliableMessaging, OASIS produced a competing standard WS-Reliability that was supported by a coalition of vendors. The protocol allows endpoints to meet the guarantee for the delivery assurances namely, Atmost Once, Atleast Once, Exactly Once and In Order. Persistence considerations related to an endpoint's ability to satisfy the delivery assurances are the responsibility of the implementation and do not affect the wire protocol.

NGINX - A High Performance HTTP Server

What you will find about NGINX from its website is that it is a free, open source light weight HTTP web server and a reverse proxy. Nginx doesn't rely on threads to handle requests. Instead it uses a much more scalable event-driven (asynchronous) architecture. This architecture uses small, but more importantly, predictable amounts of memory under load. We have found that it is holding up to what it says. Here is how we ended up using NGINX for one of our client and have found it meeting up to our expectation.


We were on a performance testing assignment of a game application, where much of the game contents were static flash files (of course they are dynamic as they had embedded action scripts for execution in the front end) and image files of which quite many are of sizes above 500KB. Need not to mention that this is the compressed size and as such enabling compression did not offered any performance gains. These files were served by Apache, which also serves dynamic php requests.


Our initial performance tests did indicated that for about 1000 concurrent hits, the Apache server memory consumption shot up considerably beyond 2 GB and also quite many requests were timing out. Examination of the server configuration did reveal that the server is configured to serve 256 child processes and also that it was configured to use the ‘prefork’ mpm module, which limits each child process to use only one thread. This technically limits the number of concurrent requests the Apache Server can serve to 256.


We also figured out that Apache needs to use ‘prefork’ module in order to safely serve PHP requests. Though later versions of PHP are found to be working with the other mpm module (‘worker’, which can use multiple threads per child process) many still have concerns around thread safety. The 256 client process limit is also a hard limit and if that needs to be increased, the Apache server needs to be rebuilt.


With this diagnosis, the choices we had with us was to have the static content served out of a server other than Apache and just leave Apache to serve the PHP requests. We just then thought of trying out NGINX and without wasting much time, we went ahead and implemented it. NGINX was then configured to listen on port 80 and has been configured to serve the contents from the root folder of Apache. Apache server was configured to listen at a non standard port and NGINX has also been configured to act as a reverse proxy for all php requests by getting them served processed by Apache.


We performed the tests again and have found tremendous improvement. The average latency at 1000 concurrent hits stress level has come down to under 5 seconds from about 80 seconds. We could also observe that NGINX was consuming just around 10MB. NGINX, by itself does not has any limitation on the number concurrent threads and it is constrained by the limits of the server itself and other daemons running on it.

Resilient Systems - Survivability of Software Systems

Resilience as we all know is an ability to withstand through tough times.There is also another term quite interchangeably used, which is Reliability. But Reliability and Resilience are different. Reliability is about a system or a process that has zero tolerance to failure or the one that should not fail. In other words, when we talk about reliable systems, the context is that failure is not expected or rather acceptable. Whereas Resilience is about the ability to recover from failures. What is important to understand about resilience is that failure is expected and is inherent in any systems or processes, which might be triggered due to changes to the platform, environment and data. While Reliability is about the system’s robustness of not failing, Resilience is its ability to sense or detect failures ahead and then prevent it from encountering such events that lead to failure and when it cannot be avoided, allow it to happen and then recover from the failure sooner.


A working definition for resilience (of a system) developed by the Resilient Systems Working Group (RSWG) is as follows:

“Resilience is the capability of a system with specific characteristics before, during and after a disruption to absorb the disruption, recover to an acceptable level of performance, and sustain that level for an acceptable period of time.“ The following words were clarified:

• The term capability is preferred over capacity since capacity has a specific meaning in the design principles.
• The term system is limited to human-made systems containing software, hardware, humans, concepts, and processes. Infrastructures are also systems.
• The term sustain allows determination of long-term performance to be stated.
• Characteristics can be static features, such as redundancy, or dynamic features, such as corrective action to be specified.
• Before, during and after – Allows the three phases of disruption to be considered.
• Before – Allows anticipation and corrective action to be considered
• During – How the system survives the impact of the disruption
• After – How the system recovers from the disruption
• Disruption is the initiating event of a reduction is performance. A disruption may be either a sudden or sustained event. A disruption may either be internal (human or software error) or external (earthquake, tsunami, hurricane, or terrorist attack).

Evan Marcus, and Hal Stern in their book Blueprints for High Availability, define a resilient system as one that can take a hit to a critical component and recover and come back for more in a known, bounded, and generally acceptable period of time.


In general Resilience is a term of concern for Information Security professionals as the final impact of disruption (from which a system needs to recover), could mostly be on Availability which is one of the three tenets of Information Security (CIA - Confidentiality, Integrity and Availability). But there is a lot for System designers and developers, especially those tasked to build mission critical systems to be concerned about Resilience and architect the systems to build in a required level of Resilience characteristics. For a system to be resilient, it should draw necessary support from dependent software and hardware components, systems and the platform. For instance a disruption for a web application can even be from network outage, security attacks at the network layer, which the software has no control over. But it is important to consider software resiliency in relation to the resiliency of the entire system, including the human and operational components.


The PDR (Protect - Detect - React) strategy is no longer as effective as it used to be due to various factors. It is time that predictive analytics and some of the disruptive technologies like big data and machine learning need a consideration in enhancing the system resiliency. Based on the logs of various inter-connected applications or components and other traffic data on the network, intelligence need to be built into the system to a combination of number of possible actions. For instance, if there is a reason to suspect an attacker attempting to gain access to the systems, a possible action could be to operate the system at a reduced access mode, i.e. parts of the systems may be shut down or parts of the networks to which the system is exposed could be blocked, etc.


OWASP’s AppSensor project is worth checking by the architects and developers. The AppSensor project defines a conceptual framework and methodology that offers prescriptive guidance to implement intrusion detection and automated response into an existing application. AppSensor defines over 50 different detection points which can be used to identify a malicious attacker.Appsensor provides guidance in the form of possible responses for each such identified malicious atacer.


The following are some of the factors that need to be appropriately addressed to enhance the resilience of the systems:

Complexity - With systems continuously evolving and many discrete systems and components increasingly integrated into today’s IT eco-system, the complexity is on the rise. This makes the resiliency routines as built in to individual systems needing a constant review and revision.

Cloud - Cloud computing is gaining higher acceptance and as organizations embrace cloud for its IT needs, the location of data, systems and components are widespread across the globe and that brings in challenge for those involved in building resilience.

Agility - To stay on top of the competition, organizations need agility in their business processes, which means rapid changes in the underlying systems and this could be a challenge as this will call for a constant check to ensure that the changes being introduced does not downgrade or compromise the resiliency level of the systems.


While there are techniques and guiding principles which when followed and applied, the resilience of the systems can be greatly improved, such design or implementation comes with a price and that is where the economics of Resiliency needs to be considered. For instance, mission critical software systems like the ones used in medical devices, need to have a high resilience characteristic, but quite many of the business systems can have a higher tolerance level and thereby being less resilient. However, it is good to document the expected resilience level at the initial stage and work on it in the early life cycle of the system development. thinking about resilience later in the life cycle may not be any good as implementation will call for higher investment.

---

References:

Crosstalk - The journal of Defense Software Engineering Vol 22 No:6

Resilient Systems Working Group

OWASP - AppSensor Project

Application Architecture Review - Security

In continuation of the Architecture review series, let us focus on security review in this blog. With information security breaches hitting the news headlines quite frequently, many enterprises are realizing the real need to manage this security risk and be resilient. As such, it is possible that as an Architect, you might have been called for to perform a security review of the existing applications. I have tried to put together the following areas of concern, which need a closer look to form an opinion whether the application architecture is secure enough.

 

 

In general the broad areas of concern for the security architects should be the following:

 

 

Authentication – Review the tools, technology and the approach used by the application to establish the identity of the application users for possible deficiencies. In this connection the following specific areas need attention.

• Look for identification of the legitimate human and system users of the application in the requirements document which in turn are validated with appropriate business scenarios. 
• If the application exposes interface to external systems, understand how access by such systems are identified and authenticated. Also understand how secure such other external systems are and if possible ask for a security assessment of such other systems.
• Identify how users are authenticated, whether two factor or three factor authentication.
• Check if Single Sign On is implemented and in such case, understand how it is implemented, what tools and technology are used. If the Identity provider is external to the system boundary, then also check how the information in transit between the identity provider and the application is secured.
• In case of external identity providers, it would also be worth checking the security practices followed by the Service Provider and whether they are being subject to regular external independent security assessment.
• If the application maintains the user information locally and authenticates against it, ensure whether identity related data is secured appropriately from unauthorized access.
• It would also be worth understanding how the database servers authenticate the application or the application user. If the application users happen to be the users of the database as well, then the mechanisms implemented to prevent such users directly accessing the database needs to be scrutinized.

 

 

Authorization - Each of the identified human or system users would be operating on the application by assuming defined roles and the authorization to access various components or information should be dependent on such roles. Get a clear view of how the roles and authorization are implemented in the application. The following specific areas are worth the attention in this regard.

• Check if there exists an information sensitivity policy or information privacy policy as relevant to the data or information being accessed or managed by the application.
• Understand how the defined roles are mapped to the various datasets in terms of the permission to the Create, Read, Update and Delete. It would also be good to examine the various roles defined by the organization whether they are in line with that of the principles of segregation of duties and look for how the users with multiple or overlapping roles are handled by the system.
• With a view to improve application performance, developers tend to create interfaces (both visual and non-visual) in such a way they are chunky as against being chatty. While this holds good in terms of application performance, the datasets being served need to be reviewed with respect to the information sensitivity and the role based permission restrictions should be applied to all internal and external APIs and interfaces.

 

 

Availability / Scalabiltiy – Systems are designed to process data in the expected and timely manner so that the information users make the most of it, and perform the business operations efficiently and effectively. The general experience is that the systems perform very well in the initial testing phase and when it is deployed in production its behaviour could be different and might slow down considerably due to various environment and load related issues. As an architect it is essential that the proper estimation is done for expected user and data growth and the application is designed to meet such needs. Examination of the following areas might reveal how the application meets this concern.

• Check out my own blog on Architecture Review – Scalability to know more on this specific area.

 

 

Auditability – The systems should be designed to log certain events, which could be potentially lead to security breach. These logs should be readable when needed by users with appropriate roles and should be monitored periodically. Event alerts also help notifying the administrators on the occurrence of certain type of events, which may require immediate attention. Examine the following areas of the application design to form an opinion on this concern.

• Review the Application architecture to understand how the event alert and logging mechanism is implemented. 
• Review for completeness of various events that are being handled and the relevant data is being logged. Examine if any sensitive data is being logged and if so, whether role based access restrictions is also implemented around the log data. 
• Check how the event log data is organized and stored and also look for existence of any policy or procedures around managing such log data.
• Understand the regulatory needs, which many times govern the data to be logged and how long such log data need to be retained.
• The log data grows too fast and many times if the storage of log data is within the same production database of the application, there is a possibility that this growth may impact the performance of the application itself impacting the Availability needs. Depending on the volume and growth rate of the data, ensure that the chosen tools and technology is adequate and appropriate.

 

 

This blog is not an exhaustive checklist and just intended to bring out the broad concerns which at a minimum should be considered in the Architecture Review. TOGAF 9.1 has in its ADM Guidelines and Techniques has listed the design considerations with respect to building security as part of the design and architecture. These security design considerations can be used for an exhaustive security review, which also covers the implementation, change management and the IT infrastructure.

Also check out my own blog titled as Building Secure Application, which is abour making security part of the SDLC.

Building Secure Applications

The awareness on security has increased manifold and thanks to the frequent news headlines on security breaches and compromises leaving organizations with heavy damage and / or loss. The board has started realizing that information security is one of the primary causes of most of the IT Risks that their organization has to mitigate or to be ready to face. But the application design and development processes has not seen significant change to ensure that the delivered application is secure and would help the organization bring down the risk exposure.

 

Most of the software engineers and architects involved in application design and development still have very less security awareness and they believe that that security is something that the IT Infrastructure team will take care of. On the other hand, the IT infrastructure team believes in the investment they would have made in robust security tools and expect that they have done their part to protect the organization and pass on any application specific security issues back to the development teams. Then, whose responsibility it is to ensure building a secure application?

 

This is where an independent, enterprise wide security consulting team needs to extend the security consulting to various project teams within the organization. But still, this will not solve the problem completely, as it is for the development teams to be security aware and design and build security applications. 

 

Here is what the security consulting team can offer to the project teams:

 

Pre-project security reviews: This helps the decision makers to be aware of what security threat landscape that they are going to be dealing with upon implementation of the project. This could even have an impact on the project costing as additional cost might have to be incurred to mitigate the security risks that this project may bring on and in most cases it is not just one time investment and it has to be recurring operating expenses as well. This helps the organization to make informed decisions considering the extent of impact on the organization’s risk exposure.

 

 

Design level security reviews: Like in case of software quality, earlier one spot the security concerns, it would cheaper and easier to factor in the mitigation plans. At this stage, the security consulting teams can help the project teams in the following ways:

• A high level evaluation of the security concerns that the design could expose and suggest possible security controls to mitigate such concerns.
• Equip the project stake holders with necessary information associated with the identified security concerns, so that they can take better risk management decisions.
• Extend guidance to the project teams on the choice of controls and solutions that might best address the security concerns.
• Perform research and exploration on any the technology or feature is quite new and innovative which could potentially open a new thread landscape
• Offer training to the design and development teams about the most common security vulnerabilities and the attack patterns, so that they design and build counter measures early on.

Implementation Security Reviews: Security consulting team offers to perform the reviews and verifications in the later phase of the application development, so that vulnerabilities if any exist may not slip through to production. Typically the services offered by the security consulting team are one or more of the following:

• Ensure that the security concerns identified in the design review is fully addressed.
• Perform security specific reviews on the code and this could be on a sampling basis, depending on the acceptable risk level of the application.
• Use automated tools that will examine the code and / or the packaged component or application.
• Review the security specific test cases as created by the software testing team and suggest additional test cases to ensure better coverage in the security testing.

Having a security consulting team will not just be enough to build secure applications. The Software Engineering Process which defines and describes the approach and methodologies used for project execution should also mandate for practising and consuming the services of the Security Consulting teams and their review reports should be identified as entry and exit criteria for various phases of the application development.

 

Security education is also required to ensure that the project team is aware why security is so important for the organization. The following measures towards security education would help the teams to design and develop secure applications:

• Include a module of security education in the employee induction program, so that every new employee is oriented towards the security needs of the organization.
• Offer in-depth security training to architects and select software developers and ensure that every project team has at least couple of such trained resources. 
• Ensure that the coding guidelines document also factors writing secure code.
• Hold periodic training sessions where in addition to security related presentations, recent security review findings and or defects found during the security testing can be taken up for discussion and in the end come up with solutions to prevent such issues creeping in.
• If the organization publishes a periodic news bulletin, ensure that security is a mandated section in it and use it to publish security related updates.

As it can be observed, security is not just one person or department’s responsibility and it should be the outcome of collaborated efforts of concerned teams with the common goal to build a secure application. The security consulting team can also be external to the organization, i.e. this function can be outsourced to security consuling firms who would be specializing in the area of application security practice.

---

Solution Architecture - Basic Principles

As I have written in my earlier blog post Solution Architect – Understanding the Role, the Solution Architecture practice area demands a wider knowledge in business and technical areas. The solution Architects need to be jack of all trades rather than being the master of a specific area. The Solution Architects should be able to bridge the gaps that the business users have on technical space and those that the technical teams have on the business areas.


Architecting a good solution is always challenging as the context and the technology keeps changing too fast over time. A solution which was perfect in a time frame may not be so after a time period. Given that each of the non-functional quality attributes might adversely affect one or more other non-functional quality attributes, the Solution Architects should be able to do the balancing act on these attributes in line with the business needs and other factors that could be foreseen in the near and longer term.


Here are some of the basic principles, which when practiced, would help a solution architect to deliver a good solution.


Business Drives Information Technology

Just in case, if one has the expertise in IT, then it is important to take off that hat and wear the business hat while architecting solutions. Business does not mind which technology or tools that would be in use, but they want their business problems solved with reasonable longevity and other non-functional requirements. Ignoring or undermining business expectations or priorities could lead to building a solution with excessive engineering or technical complexity, and could result in higher cost and delays.


Just Enough Architecture, Just in Time

Let the solutions evolve based on business priorities. It would be ideal to take one business problems at a time based on its priority and architect the solution keeping in mind the first principle and factoring the ability to change and scale in response to the business needs. This will help the businesses get the solutions faster and in increments. This will also help the Architects adopt agile methods and ensure timely responses to business changes as mostly needed.


Common Business Solutions

There is a tendency for departments to go in for solutions on their own as they have their own budgets to spend. This many times leads to a situation where multiple solutions being in use for the same problem domain within various departments. It is a must for architects to have a complete view of the problems and solutions within the enterprise and the solutions architected should be common and usable across multiple departments. If need be, the existing solution can be enhanced to meet the changing requirements of different department. This principle when practiced will facilitate easy maintenance and will ensure better data governance.


Conform to Data Governance principles

Data and information is used for decision making at various levels and it is important that the data is organized and maintained at the highest possible quality level. The need to understand the sensitivity levels of the data within and across departments and partner systems is also equally important while architecting the solution. With big data era emerging, organizations are looking at handling petabytes of structured and unstructured data to be managed. It is imperative to have the Data Governance policies and processes in place and the Solution Architecture team should ensure that the solutions are in compliance with it.


Comply with Information Security Framework

Similar to the Data Governance policies, the organization should have Information Security policies and related framework in place and the Solution Architecture team should ensure that the solutions are in conformance with it. Security must be designed into the solutions from the inception and adding it later could result in higher cost and delays. This however is dependent on the organization’s business context and its risk appetite.


Also read the following other blog posts on Architecture Review - Scalability