Software Reliability

Covariate Software Failure and Reliability Assessment Tool (C-SFRAT)

Description

The Covariate Software Failure and Reliability Assessment Tool (C-SFRAT) is an open source application that applies covariate software reliability models to help guide model selection and test activity allocation. The primary functions of the C-SFRAT include:

  1. Displaying model fit and failure intensity plots of selected hazard function and covariate combinations
  2. Prediction of future failures and failure intensity based on a specified testing activity profile
  3. Comparison of fitted models based on information theoretic and predictive goodness-of-fit measures with user-defined weighting
  4. Recommendations for test activity allocation to maximize defect discovery within a specified budget or minimize the total testing resources required to discover a specified number of defects.

C-SFRAT runs under the Python 3.x framework and can be used on computers running Windows, macOS, or Linux.

Resources

Example covariate data sets
C-SFRAT GitHub repository


Software Failure and Reliability Assessment Tool (SFRAT)

Description

The key to the success of all software is its reliability. The Software Failure and Reliability Assessment Tool (SFRAT) is an open source application to estimate and predict the reliability of a software system during test and operation. It allows users to answer the following questions about a software system during test:

  1. Is the software ready to release (has it achieved a specified reliability goal)?
  2. How much more time and test effort will be required to achieve a specified goal?
  3. What will be the consequences to the system’s operational reliability if not enough testing resources are available?

SFRAT runs under the R statistical programming framework and can be used on computers running Windows, Mac OS X, or Linux

If you benefit from the SFRAT in your work or research, please consider citing the following open access article to make others aware of this free and open source tool: V. Nagaraju, V. Shekar, J. Steakelum, M. Luperon, Y. Shi, and L. Fiondella, 2019. Practical software reliability engineering with the Software Failure and Reliability Assessment Tool (SFRAT). SoftwareX, 10, p.100357.

Resources

Example failure data sets
SFRAT Github repository
User’s Guide
Contributor’s Guide

Automated Report generation script
Script instruction manual

Contributions

  1. Dr. Shinji Inoue from Kansai University, Kansai, Osaka, Japan translated the GUI to Japanese version. Available at:
    SFRAT – Japanese Version
  2. Dr. Hiroyuki Okamura from Hiroshima University, Hiroshima, Japan contributed 11 additional models to the tool. Available at:
    SFRAT_Okamura
  3. Aerospace Corporation contributed three models. Available at:
    SFRAT_Aerospace
  4. Barry Von Tobel from MITRE developed a script to automatically transport failure data collected from JIRA tool to SFRAT compatible data format. Available at:
    JIRA export data to SFRAT data format

Software Defect Estimation Tool (SweET)

Description

The Software Defect Estimation Tool (SweET) is an open source application to track error identification and removal efforts during the software development lifecycle. SwEET is a free and open source version of the SoftWare Error Estimation Program (SWEEP) and SweET uses Weibull software reliability growth model utilizing Expectation Conditional Maximization algorithm to ensure stability and performance of the model fitting process. SweET simplifies four models of SWEEP into three modes:

  1. Mode A: Time-based model: Estimates and tracks errors during system test and integration cycles.
  2. Mode B: Phase-based and planning aid model: Predict and track defects for multiple phases and can provide defect information before running any code, whereas the planning aid model generates an error discovery profile based on the phase based historical data to help a software prohect achieve its objectives.
  3. Mode C: Defect injection model: Allows the user to understand the probable defect injection profile and resulting efficiency and effectiveness of the verification process.

SweET runs under the Python 3.x programming framework and can be used on computers running Windows, Mac OS X, or Linux.

Resources

Example data sets
SweET Github repository
User’s Guide

Publications

36. V. Nagaraju, C. Jayasinghe, and L. Fiondella, Optimal Test Activity Allocation for Covariate Software Reliability Models, Journal of Systems and Software (JSS), 168, 2020.

35. V. Nagaraju, C. Jayasinghe, and L. Fiondella, A Covariate Software Reliability Model and Optimal Test Activity Allocation, In Proc. 67th Annual Reliability and Maintainability Symposium (RAMS 2021), Orlando, FL, Jan 2021.

34. M. Nafreen and L. Fiondella, Software Reliability Models with Bathtub-shaped Fault Detection, In Proc. 67th Annual Reliability and Maintainability Symposium (RAMS 2021), Orlando, FL, Jan 2021.

33. M. Nafreen, V. Nagaraju, M. Luperon, Y. Shi, T. Wandji, and L. Fiondella, Connecting Software Reliability Growth Models to Software Defect Tracking, In Proc. International Symposium on Software Reliability Engineering, Coimbra, Portugal, Nov 2020.

32. V. Nagaraju, C. Jayasinghe, L. Fiondella (To Appear). Optimal Test Activity Allocation for Covariate Software Reliability Models. Journal of Systems and Software (JSS).

31. V. Nagaraju, L. Fiondella (2020). Online Optimal Release Time for Non-homogeneous Poisson Process Software Reliability Growth Model. In Proc. 66th Annual Reliability and Maintainability Symposium (RAMS), Palm Springs, CA. Second Place in Tom Fagan Student Paper Competition.

30. M. Nafreen, S. Bhattacharya, L. Fiondella (2020). Architecture-based Software Reliability incorporating Fault-Tolerant Machine Learning. In Proc. 66th Annual Reliability and Maintainability Symposium (RAMS), Palm Springs, CA.

29. A. Gula, C. Ellis, S. Bhattacharya, L. Fiondella (2020). Software and System Reliability Engineering for Autonomous Systems incorporating Machine Learning. In Proc. 66th Annual Reliability and Maintainability Symposium (RAMS), Palm Springs, CA. Best Student Paper Award, Society of Reliability Engineers Stan Ofsthun. 

28. Vidhyashree Nagaraju, Lance Fiondella (2019). Improved Algorithm for Non-Homogeneous Poisson Process Software Reliability Growth Models Incorporating Testing-Effort. International Journal of Performability Engineering, 15(5), pp. 1265.

27. V. Nagaraju, V. Shekar, J. Steakelum, M. Luperon, Y. Shi, L. Fiondella (2019). Practical Software Reliability Engineering with the Software Failure and Reliability Assessment Tool (SFRAT). SoftwareX, 10, pp. 1-10.

26. V. Nagaraju, L. Fiondella, T. Wandji (2019). A Heterogeneous Single Changepoint Software Reliability Growth Model Framework. Software, Testing Verification and Reliability (STVR), 29(8).

25. V. Nagaraju, L. Fiondella (2018). A Survey of Change-point Software Reliability Growth Models. In Proc. International Conference on Reliability and Quality in Design, Toronto, Canada. Best Student Paper Award.

24. V. Nagaraju, L. Fiondella (2018). A Survey of Change-point Software Reliability Growth Models. In Proc. International Conference on Reliability and Quality in Design, Toronto, Canada.

23. V. Nagaraju, L. Fiondella (2018). A Heterogeneous Single Changepoint Software Reliability Growth Model Framework. Software, Testing Verification and Reliability (under review).

22. V. Nagaraju, T. Wandji, L. Fiondella (2018). An Improved Algorithm for Non-homogeneous Poisson Process Software Reliability Growth Models incorporating Testing-Effort. International Journal of Performability Engineering, 15(5), pp. 1265-1272, 2019.

21. V. Nagaraju, L. Fiondella (2018). Software Reliability in RAMS Management, Handbook of RAMS in Railway Systems: Theory and Practice, Taylor & Francis, 2018.

20. V. Nagaraju, T. Wandji, L. Fiondella (2017). A Hybrid Approach to Identify the Maximum Likelihood Estimates of a Two Changepoint Goel-Okumoto Software Reliability Growth Model. In Proc. International Conference on Reliability and Quality in Design (ISSAT), pp. 59-63, Chicago, IL.

19. V. Nagaraju, V. Shekar, T. Wandji, L. Fiondella (2017). The Software Failure and Reliability Assessment Tool (SFRAT): A Platform to Foster Collaboration. In Proc. IEICE Technical Committee on Reliability, Wakkanai, Japan.

18. V. Nagaraju, L. Fiondella (2017). A Single Change-point Software Reliability Growth Model with Heterogeneous Fault Detection Processes. In Proc. of Annual Reliability and Maintainability Symposium (RAMS), Orlando, FL.

17. V. Nagaraju, T. Wandji, L. Fiondella (2017). A hybrid approach to identify the maximum likelihood estimates of a two changepoint Goel-Okumoto Software Reliability Growth Model. In Proc. International Conference on Reliability and Quality in Design (ISSAT 2017), Chicago, IL.

16. V. Nagaraju, T. Wandji, L. Fiondella (2017). An Expectation Conditional Maximization Algorithm for the Goel-Okumoto Software Reliability Growth Model with Two Change-points. In Proc. International Conference on Reliability and Quality in Design (ISSAT 2017), Chicago, IL.

15. V. Nagaraju, L. Fiondella, T. Wandji (2017). An Open Source Tool to Support the Quantitative Assessment of Cybersecurity. In Proc. International Conference on Cyber Warfare and Security (ICCWS 2017), Dayton, OH.

14. V. Nagaraju, L. Fiondella, T. Wandji (2017). An Open Source Tool to Support the Quantitative Assessment of Cybersecurity for Software Intensive System Acquisition. Journal of Information Warfare (JIW), 16(3), pp. 31-50.

13. V. Nagaraju, L. Fiondella, . Jayasinghe, P. Zeephongsekul, T. Wandji (2017). Performance Optimized Expectation Conditional Maximization Algorithms for Non-homogeneous Poisson Process Software Reliability Models. IEEE Transactions on Reliability (T-Rel), 66(3), pp. 722-734.

12. V. Nagaraju, L. Fiondella, P. Zeephongsekul, T. Wandji (2017). An Adaptive EM Algorithm for the Maximum Likelihood Estimation of Non-homogeneous Poisson Process Software Reliability Growth Models. International Journal of Reliability, Quality and Safety Engineering (IJRQSE), (in press).

11. V. Nagaraju, A. Krishna Murthy, L. Fiondella, P. Zeephongsekul, T. Wandji (2016). Expectation Conditional Maximization Algorithms for Failure count Non-homogeneous Poisson Process Software Reliability Models. In Proc. (ISSAT 2016), pp. 372-376, CA. Best Student Paper Award, International Conference on Reliability and Quality in Design. 

10. V. Nagaraju, V. Basavaraj, L. Fiondella (2016). Software Rejuvenation of a Fault-tolerant Server subject to Correlated Failure. In Proc. of 62nd Annual Reliability and Maintainability Symposium (RAMS 2016), pp. 1-6, Tucson, AZ.

9. V. Nagaraju, K. Katipally, R. Muri, T. Wandji, L. Fiondella (2016). An Open Source Software Reliability Tool: A Guide for Users. In Proc. 2016 International Conference on Reliability and Quality in Design (ISSAT 2016), pp. 132-137, CA.

8. V. Nagaraju, A. Krishna Murthy, L. Fiondella, P. Zeephongsekul, T. Wandji (2016). Expectation Conditional Maximization Algorithms for Failure count Non-homogeneous Poisson Process Software Reliability Models. In Proc. 2016 International Conference on Reliability and Quality in Design (ISSAT 2016), pp. 372-376, CA.

7. V. Nagaraju, L. Fiondella (2016). Maximum Likelihood Estimation of a Non-homogeneous Poisson Process Software Reliability Model with the Expectation Conditional Maximization Algorithm. In JSM Proceedings, Section on Statistics in Defense and National Security, Conference on Applied Statistics in Defense 2015, pp. 1121-1131, Alexandria, VA: American Statistical Association.

6. P. Zeephongsekul, C. Jayasinghe, L. Fiondella, V. Nagaraju (2016). Maximum Likelihood Estimation of Parameters of NHPP Software Reliability Models Using EM and ECM Algorithms. IEEE Transactions on Reliability, 65(3), pp. 1571-1583.

5. A.M. Neufelder, L. Fiondella, L. Gullo, H. Daughtrey (2015). Advantages of IEEE P1633 for Practicing Software Reliability. In Proc. of Annual Reliability and Maintainability Symposium (RAMS), pp. 900-906, Palm Harbor, FL.

4. L. Fiondella, R. Duffey (2015). Software and Human Reliability: Error Reduction and Prediction. In Proc. of the International Topical Meeting on Probabilistic Safety Assessment and Analysis, Sun Valley, ID.

3. V. Nagaraju, L. Fiondella (2015). An Adaptive EM Algorithm for NHPP Software Reliability Models. In Proc. of Annual Reliability and Maintainability Symposium (RAMS), pp. 17-23, Palm Harbor, FL.

2. L. Fiondella, P. Zeephongsekul (2014). Discrete Software Reliability Growth Model based on Maximum Entropy Principle with Higher Order Polynomial Moment Constraints. In Proc. of ISSAT International Conference on Reliability and Quality in Design, pp. 263-267, Seattle, WA.

1. J. Liu, Z. Wang, Z. Peng, J.-H. Cui, L. Fiondella (2014). Suave: Swarm Underwater Autonomous Vehicle Localization. In Proc. of IEEE INFOCOM, pp. 64-72, Toronto, Canada.

Invited Talks

  1. Software Reliability Engineering: Algorithms and Tools, American Society for Quality (ASQ) Statistics, Reliability & Risk, and Software Division Joint Webinar, November 10, 2020.
  2. Covariate Software Reliability Models and Applications, Center for Scientific Computing and Visualization Research, North Dartmouth, MA, November 4, 2020.
  3. Covariate Software Reliability Models and Applications, Oak Ridge National Laboratory, Oak Ridge, TN, June 8, 2020.
  4. Covariate Software Reliability Models and Applications, Virginia Tech Research Center, Arlington, VA, March 12, 2020.
  5. Continuous Integration, Software Reliability in the 21st Century Panel Discussion held at the 66th Annual Reliability and Maintainability Symposium (RAMS 2020), Palm Springs, CA, Jan 28, 2020.
  6. Towards a Guide for Software Defect Tracking, Modeling, and Analysis, NASA Software Assurance Research Program Review, Greenbelt, MD, February 6, 2020.
  7. Software Reliability and Security Modeling and Analysis from Software Defect Tracking Data, NASA Goddard Space Flight Center, Software Engineering Division, Code 580, November 19, 2019.
  8. Relationships between Machine Learning and Reliability Engineering, Air Force Institute of Technology, Dayton, OH, October 31, 2019.
  9. Software Reliability Engineering (two day course), BAE Systems, Nashua, NH, January 16-17, 2019.
  10. Software Reliability Engineering: Algorithms and Tools, Rutgers University, New Brunswick, NJ, December 20, 2018.
  11. Practical Software Reliability Modeling and Application, NASA Software Assurance Working Group Meeting, Mountain View, CA, July 31, 2018.
  12. Software Reliability Engineering: Algorithms and Tools, Indian Institute of Science, Bangalore, India, June 12, 2018.
  13. Software Reliability Engineering: Algorithms and Tools, Indian Institute of Technology Kharagpur, Karagpur, India, May 31, 2018.
  14. Software Reliability Engineering: Algorithms and Tools, Amity University, Noida, India, May 29, 2018.
  15. Tools to Assess Software Reliability and Security, MIT Lincoln Laboratory, Lexington, MA, April, 30, 2018.
  16. Software Reliability Engineering: Algorithms and Tools, Auburn University, Auburn, AL, March 27, 2018.
  17. Practical Software Reliability Modeling and Application, NASA Goddard Space Flight Center, Greenbelt, MD, March 7, 2018.
  18. Software Reliability Engineering: Algorithms and Tools, Old Dominion University, Norfolk, VA, February 15, 2018.
  19. Software Reliability Engineering: Algorithms and Tools, Virginia Commonwealth University, Richmond, VA, February 1, 2018.
  20. Software Reliability Engineering: Algorithms and Tools, Florida International University, Miami, FL, January 20, 2018.
  21. Free and Open Source Tools to Assess Software Reliability and Security, Boston Software Process Improvement Network (SPIN), Bedford, MA, November 21, 2017.
  22. Tools to Assess Software Reliability and Security, Unmanned System Common Control System (CCS) Integrated Product Team Meeting, Patuxent River, MD, December 5, 2017.
  23. ECM Algorithms & An Open Source Tool, Tottori University, Tottori, Japan, July 4, 2017.
  24. Software Reliability Engineering: Algorithms and Tools, Florida State University, Tallahassee, FL, February 27, 2017.
  25. Software Reliability Research – Assessments & Tools, Office of the Secretary of Defense, Developmental Test and Evaluation Functional Integrated Product Team Meeting, Washington, DC, Jan 11, 2017.
  26. Software Reliability Research – Assessments & Tools, Office of the Deputy Assistant Secretary of Defense for Developmental Test & Evaluation, Alexandria, VA, Oct 28, 2016.
  27. Software Reliability: Tools and Algorithms, IEEE Boston Reliability Society, Lexington, MA, Sep 14 2016.
  28. Software Reliability Engineering: Algorithms and Tools, Naval Postgraduate School, Monterey, CA, May 27 2016.
  29. Software Reliability: Tools and Models, Workshop on Rigorous Test and Evaluation for Defense, Aerospace, and National Security, Arlington, VA, April 12, 2016.
  30. Algorithms and Tools for Software Reliability Engineering, University of Maryland, Dec 2, 2015.
  31. An Open Source Software Reliability Tool and Model Fitting Algorithm, Wright State University, Oct 7, 2015.
  32. Software Reliability Engineering: Practice and Theory, UMass- Naval Undersea Warfare Center (NUWC) Lecture Series, Nov 20, 2014.

Presentations and Tutorials

  1. V. Nagaraju, C. Jayasinghe, and L. Fiondella, Covariate Software Reliability Models and Applications, Presented to WG17 Logistics, Reliability and Maintainability and WG24 Test and Evaluation (T&E) and Experimentation, at the 88th Military Operations Research Symposium (MORS 2020), New London, CT, June 2020.
  2. M. Nafreen, M. Luperon, V. Nagaraju, Y. Shi, and L. Fiondella, Connecting Software Reliability Growth Models to Software Defect Tracking, Presented to WG17 Logistics, Reliability and Maintainability and WG24 Test and Evaluation (T&E) and Experimentation, at the 88th Military Operations Research Symposium (MORS 2020), New London, CT, June 2020.
  3. A. Gula, C. Ellis, S. Bhattacharya, and L. Fiondella, Relationships between Machine Learning and Reliability Engineering, Presented to WG17 Logistics, Reliability and Maintainability and WG35 AI and Autonomous Systems, at the 88th Military Operations Research Symposium (MORS 2020), New London, CT, June 2020.
  4. M. Nafreen, M. Luperon, V. Nagaraju, Y. Shi, and L. Fiondella, Connecting Software Reliability Growth Models to Software, Presented at the Defense and Aerospace Test and Analysis (DATA) Workshop, Springfield, VA, May, 2020.
  5. A. Gula, C. Ellis, S. Bhattacharya, and L. Fiondella, Software and System Reliability Engineering for Autonomous Systems incorporating Machine Learning, Poster presented at the 66th Annual Reliability and Maintainability Symposium (RAMS 2020), Palm Springs, CA, Jan 2020.
  6. M. Nafreen, S. Bhattacharya, and L. Fiondella, Architecture-based Software Reliability incorporating Fault-Tolerant Machine Learning, Poster presented at the 66th Annual Reliability and Maintainability Symposium (RAMS 2020), Palm Springs, CA, Jan 2020.
  7. V. Nagaraju, Y. Shi, and L. Fiondella, Software Reliability and Security Assessment: Automation and Frameworks, Presented at the Defense and Aerospace Test and Analysis (DATA) Workshop, Springfield, VA, April, 2019.
  8. V. Nagaraju, L. Fiondella, Introduction to Mathematical Software Reliability Models, In Proc. 65th Annual Reliability and Maintainability Symposium (RAMS 2019), Orlando, FL, Jan 2019.
  9. V. Nagaraju, L. Fiondella, and T. Wandji, A Single Changepoint Software Reliability Growth Model with Heterogeneous Fault Detection Processes, Poster presented at the Society of Risk Analysis (SRA) Annual Meeting, Arlington, VA, December, 2017.
  10. V. Nagaraju, T. Wandji, and L. Fiondella, Free and Open Source Tools to Assess Software Reliability and Security, Presented at the National Defense Industrial Association (NDIA) Annual Systems Engineering Conference, Springfield, VA, October, 2017.
  11. V. Nagaraju, T. Wandji, and L. Fiondella, Free and Open Source Tools to Assess Software Reliability and Security, Presented at the National Defense Industrial Association (NDIA) Annual Systems Engineering Conference, Springfield, VA, October, 2017.
  12. V. Nagaraju, T. Wandji, and L. Fiondella, Software Reliability: Modeling and Tools, Tutorial presented at the 85th Military Operations Research Symposium (MORS 2017), West Point, NY, June 2017.
  13. V. Nagaraju, T. Wandji, and L. Fiondella, Software Reliability Modeling, Presented at the Science of Test Workshop, Springfield, VA, April, 2017.
  14. V. Nagaraju, L. Fiondella, T. Wandji, Introduction to Mathematical Software Reliability Models, In Proc. 63nd Annual Reliability and Maintainability Symposium (RAMS 2017), Orlando, FL, Jan 2017.
  15. V. Nagaraju, T. Wandji, and L. Fiondella, Software Failure and Reliability Assessment Tool (SFRAT): An Open Source Application for the Practitioner and Research, Presented at the Conference on Applied Statistics in Defense, Washington, DC, October, 2016.
  16. L. Fiondella, A. Nikora, and T. Wandji, Software Reliability and Security: Challenges and Crosscutting Themes, Fast abstract In Proc. International Symposium on Software Reliability Engineering, Ottawa, Canada, Nov, 2016.
  17. V. Nagaraju, T. Wandji, and L. Fiondella, An Open Source Tool to support Software Reliability Assessment, Presented to WG 17 Logistics, Reliability and Maintainability and WG 24 Test and Evaluation (T&E) at the 84rd Military Operations Research Symposium (MORS 2016), Quantico, VA, June 2016. Nominated for the 85th Barchi Prize.
  18. V. Nagaraju and L. Fiondella, Introduction to Mathematical Software Reliability Models, In Proc. 62nd Annual Reliability and Maintainability Symposium (RAMS 2016), Tucson, AZ, Jan 2016.
  19. V. Nagaraju and L. Fiondella, An Adaptive EM Algorithm for NHPP Software Reliability Models, UMassD Sigma Xi Research Exhibition, Apr 2015.
  20. L. Fiondella and T. Wandji, An Open Source Application Architecture for Software Reliability Models, In Proc. International Applied Reliability Symposium (ARS), Tucson, AZ, Jun 2015.

Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant Numbers (1526128 and 1749635), the Naval Air Systems Command (NAVAIR) through the Systems Engineering Research Center, under Research Task 139, and by NAVAIR under contract N00421-16-T-0373.

Work at JPL was supported under NASA Prime Contract NNN12AA01C, under Task Plan No. 81-19648.