Software Reliability

Covariate Software Failure and Reliability Assessment Tool (C-SFRAT)

Description

The Covariate Software Failure and Reliability Assessment Tool (C-SFRAT) is an open source application that applies covariate software reliability models to help guide model selection and test activity allocation. The primary functions of the C-SFRAT include:

  1. Displaying model fit and failure intensity plots of selected hazard function and covariate combinations
  2. Prediction of future failures and failure intensity based on a specified testing activity profile
  3. Comparison of fitted models based on information theoretic and predictive goodness-of-fit measures with user-defined weighting
  4. Recommendations for test activity allocation to maximize defect discovery within a specified budget or minimize the total testing resources required to discover a specified number of defects.

C-SFRAT runs under the Python 3.x framework and can be used on computers running Windows, macOS, or Linux.

Resources

Example covariate data sets
C-SFRAT GitHub repository

Software Failure and Reliability Assessment Tool (SFRAT)

Description

The key to the success of all software is its reliability. The Software Failure and Reliability Assessment Tool (SFRAT) is an open source application to estimate and predict the reliability of a software system during test and operation. It allows users to answer the following questions about a software system during test:

  1. Is the software ready to release (has it achieved a specified reliability goal)?
  2. How much more time and test effort will be required to achieve a specified goal?
  3. What will be the consequences to the system’s operational reliability if not enough testing resources are available?

SFRAT runs under the R statistical programming framework and can be used on computers running Windows, Mac OS X, or Linux

If you benefit from the SFRAT in your work or research, please consider citing the following open access article to make others aware of this free and open source tool: V. Nagaraju, V. Shekar, J. Steakelum, M. Luperon, Y. Shi, and L. Fiondella, 2019. Practical software reliability engineering with the Software Failure and Reliability Assessment Tool (SFRAT). SoftwareX, 10, p.100357.

Resources

Example failure data sets
SFRAT Github repository(R)
SFRAT Github repository(Python)

User’s Guide
Contributor’s Guide

Automated Report generation script
Script instruction manual

Contributions

  1. Dr. Shinji Inoue from Kansai University, Kansai, Osaka, Japan translated the GUI to Japanese version. Available at:
    SFRAT – Japanese Version
  2. Dr. Hiroyuki Okamura from Hiroshima University, Hiroshima, Japan contributed 11 additional models to the tool. Available at:
    SFRAT_Okamura
  3. Aerospace Corporation contributed three models. Available at:
    SFRAT_Aerospace
  4. Barry Von Tobel from MITRE developed a script to automatically transport failure data collected from JIRA tool to SFRAT compatible data format. Available at:
    JIRA export data to SFRAT data format

Software Defect Estimation Tool (SweET)

Description

The Software Defect Estimation Tool (SweET) is an open source application to track error identification and removal efforts during the software development lifecycle. SwEET is a free and open source version of the SoftWare Error Estimation Program (SWEEP) and SweET uses Weibull software reliability growth model utilizing Expectation Conditional Maximization algorithm to ensure stability and performance of the model fitting process. SweET simplifies four models of SWEEP into three modes:

  1. Mode A: Time-based model: Estimates and tracks errors during system test and integration cycles.
  2. Mode B: Phase-based and planning aid model: Predict and track defects for multiple phases and can provide defect information before running any code, whereas the planning aid model generates an error discovery profile based on the phase based historical data to help a software prohect achieve its objectives.
  3. Mode C: Defect injection model: Allows the user to understand the probable defect injection profile and resulting efficiency and effectiveness of the verification process.

SweET runs under the Python 3.x programming framework and can be used on computers running Windows, Mac OS X, or Linux.

Resources

Example data sets
SweET Github repository
User’s Guide

Publications

53. F. Salboukh, P. Silva, V. Nagaraju, and L. Fiondella, Application of Recurrent Neural Networks to Covariate Software Defect Prediction, In Proc. International Conference on Reliability and Quality in Design (ISSAT), San Francisco, CA, August 2023.

52. V. Nagaraju, S. Pritchard, P. Silva, and L. Fiondella, Trade-off Assessment between Model Adequacy and Complexity in Nonhomogeneous Poisson Process Software Reliability Growth Models incorporating a Changepoint, In Proc. International Conference on Reliability and Quality in Design (ISSAT), San Francisco, CA, August 2023.

51. S. Pritchard, T. Flavin, V. Nagaraju, and L. Fiondella, Application of Recurrent Neural Network for Software Reliability and Defect Prediction, In Proc. International Conference on Reliability and Quality in Design (ISSAT), San Francisco, CA, August 2023.

50. B. Thibault, C. D’Amato, V. Nagaraju, and L. Fiondella, A Neural Network for Covariate Software Reliability Model Selection, In Proc. International Conference on Reliability and Quality in Design (ISSAT), San Francisco, CA, August 2023.

49. E. Yawlui, P. Silva, V. Nagaraju, and L. Fiondella, Optimal Release Policy for Covariate Software Reliability Models, In Proc. 69th Annual Reliability and Maintainability Symposium (RAMS 2023), Orlando, FL, Jan 2023.

48. S. Bhadra, B. Das, P. Silva, and L. Fiondella, A Stochastic Petri net Model of Continuous Integration and Continuous Delivery, In Proc. 69th Annual Reliability and Maintainability Symposium (RAMS 2023), Orlando, FL, Jan 2023.

47. J. Sorrentino, P. Silva, B. Gaspard, G. Kul, and L. Fiondella, Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, IEEE International Symposium on Software Reliability Engineering, Charlotte, North Carolina, Nov 2022.

46. V. Nagaraju, S. Pritchard, and L. Fiondella, Adaptive Incremental Learning for Software Reliability Growth Models, In Proc. International Conference on Human-Computer Interaction, Volume 5, LNCS 13306.

45. J. Aubertine, K. Chen, V. Nagaraju, and L. Fiondella, A Covariate Software Tool to Guide Test Activity Allocation, SoftwareX, 17, 2022.

44. S. Prichard, V. Nagaraju, and L. Fiondella, Automating Staged Rollout with Reinforcement Learning, In Proc. New Ideas and Emerging Results (NIER) track, International Conference on Software Engineering, Pittsburgh, PA, May 2022.

43. K. Chen, Z. Faddi, V. Nagaraju, and L. Fiondella, Quantifying the Impact of Staged Rollout Policies on Software Process and Product Metrics, In Proc. 68th Annual Reliability and Maintainability Symposium (RAMS 2022), Tucson, AZ, Jan 2022.

42. V. Nagaraju and L. Fiondella, A Hybrid Model Fitting Approach incorporating Particle Swarm Optimization and Statistical Algorithms, In Proc. Reliability and Maintenance Engineering Summit 2021 (RMES 2021), Nantong, China, Sep 2021.

41. M. Nafreen and L. Fiondella, A Family of Software Reliability Models with Bathtub-shaped Fault Detection Rate, International Journal of Reliability, Quality and Safety Engineering (IJRQSE), 25(5), pp. 2150034, 2021.

40. V. Nagaraju, C. Jayasinghe, and L. Fiondella, A Covariate Software Reliability Model and Optimal Test Activity Allocation, In Proc. International Conference on Reliability and Quality in Design (ISSAT), Miami, FL, August 2021.

39. M. Nafreen, V. Nagaraju, M. Luperon, Y. Shi, T. Wandji, and L. Fiondella, Software Defect Discovery and Resolution Modeling, In Proc. International Conference on Reliability and Quality in Design (ISSAT), Miami, FL, August 2021.

38. J. Aubertine, V. Nagaraju, and L. Fiondella, The Covariate Software Failure and Reliability Assessment Tool (C-SFRAT), In Proc. International Conference on Reliability and Quality in Design (ISSAT), Miami, FL, August 2021.

37. J. Steakelum, J. Aubertine, K. Chen, V. Nagaraju, and L. Fiondella, Multi-objective Search to Design Multi-phase Model Fitting Algorithms, In Proc. International Conference on Reliability and Quality in Design (ISSAT), Miami, FL, August 2021.

36. J. Steakelum, J. Aubertine, K. Chen, V. Nagaraju, and L. Fiondella, Multi-phase Algorithm Design for Stable and Efficient Model Fitting, Annals of Operations Research (ANOR).

35. V. Nagaraju, C. Jayasinghe, and L. Fiondella, Optimal Test Activity Allocation for Covariate Software Reliability Models, Journal of Systems and Software (JSS), 168, 2020.

34. V. Nagaraju, C. Jayasinghe, and L. Fiondella, A Covariate Software Reliability Model and Optimal Test Activity Allocation, In Proc. 67th Annual Reliability and Maintainability Symposium (RAMS 2021), Orlando, FL, Jan 2021.

33. M. Nafreen and L. Fiondella, Software Reliability Models with Bathtub-shaped Fault Detection, In Proc. 67th Annual Reliability and Maintainability Symposium (RAMS 2021), Orlando, FL, Jan 2021.

32. M. Nafreen, V. Nagaraju, M. Luperon, Y. Shi, T. Wandji, and L. Fiondella, Connecting Software Reliability Growth Models to Software Defect Tracking, In Proc. International Symposium on Software Reliability Engineering, Coimbra, Portugal, Nov 2020.

31. V. Nagaraju, L. Fiondella (2020). Online Optimal Release Time for Non-homogeneous Poisson Process Software Reliability Growth Model. In Proc. 66th Annual Reliability and Maintainability Symposium (RAMS), Palm Springs, CA. Second Place in Tom Fagan Student Paper Competition.

30. M. Nafreen, S. Bhattacharya, L. Fiondella (2020). Architecture-based Software Reliability incorporating Fault-Tolerant Machine Learning. In Proc. 66th Annual Reliability and Maintainability Symposium (RAMS), Palm Springs, CA.

29. A. Gula, C. Ellis, S. Bhattacharya, L. Fiondella (2020). Software and System Reliability Engineering for Autonomous Systems incorporating Machine Learning. In Proc. 66th Annual Reliability and Maintainability Symposium (RAMS), Palm Springs, CA. Best Student Paper Award, Society of Reliability Engineers Stan Ofsthun. 

28. Vidhyashree Nagaraju, Lance Fiondella (2019). Improved Algorithm for Non-Homogeneous Poisson Process Software Reliability Growth Models Incorporating Testing-Effort. International Journal of Performability Engineering, 15(5), pp. 1265.

27. V. Nagaraju, V. Shekar, J. Steakelum, M. Luperon, Y. Shi, L. Fiondella (2019). Practical Software Reliability Engineering with the Software Failure and Reliability Assessment Tool (SFRAT). SoftwareX, 10, pp. 1-10.

26. V. Nagaraju, L. Fiondella, T. Wandji (2019). A Heterogeneous Single Changepoint Software Reliability Growth Model Framework. Software, Testing Verification and Reliability (STVR), 29(8).

25. V. Nagaraju, L. Fiondella (2018). A Survey of Change-point Software Reliability Growth Models. In Proc. International Conference on Reliability and Quality in Design, Toronto, Canada. Best Student Paper Award.

24. V. Nagaraju, L. Fiondella (2018). A Survey of Change-point Software Reliability Growth Models. In Proc. International Conference on Reliability and Quality in Design, Toronto, Canada.

23. V. Nagaraju, L. Fiondella (2018). A Heterogeneous Single Changepoint Software Reliability Growth Model Framework. Software, Testing Verification and Reliability (under review).

22. V. Nagaraju, T. Wandji, L. Fiondella (2018). An Improved Algorithm for Non-homogeneous Poisson Process Software Reliability Growth Models incorporating Testing-Effort. International Journal of Performability Engineering, 15(5), pp. 1265-1272, 2019.

21. V. Nagaraju, L. Fiondella (2018). Software Reliability in RAMS Management, Handbook of RAMS in Railway Systems: Theory and Practice, Taylor & Francis, 2018.

20. V. Nagaraju, T. Wandji, L. Fiondella (2017). A Hybrid Approach to Identify the Maximum Likelihood Estimates of a Two Changepoint Goel-Okumoto Software Reliability Growth Model. In Proc. International Conference on Reliability and Quality in Design (ISSAT), pp. 59-63, Chicago, IL.

19. V. Nagaraju, V. Shekar, T. Wandji, L. Fiondella (2017). The Software Failure and Reliability Assessment Tool (SFRAT): A Platform to Foster Collaboration. In Proc. IEICE Technical Committee on Reliability, Wakkanai, Japan.

18. V. Nagaraju, L. Fiondella (2017). A Single Change-point Software Reliability Growth Model with Heterogeneous Fault Detection Processes. In Proc. of Annual Reliability and Maintainability Symposium (RAMS), Orlando, FL.

17. V. Nagaraju, T. Wandji, L. Fiondella (2017). A hybrid approach to identify the maximum likelihood estimates of a two changepoint Goel-Okumoto Software Reliability Growth Model. In Proc. International Conference on Reliability and Quality in Design (ISSAT 2017), Chicago, IL.

16. V. Nagaraju, T. Wandji, L. Fiondella (2017). An Expectation Conditional Maximization Algorithm for the Goel-Okumoto Software Reliability Growth Model with Two Change-points. In Proc. International Conference on Reliability and Quality in Design (ISSAT 2017), Chicago, IL.

15. V. Nagaraju, L. Fiondella, T. Wandji (2017). An Open Source Tool to Support the Quantitative Assessment of Cybersecurity. In Proc. International Conference on Cyber Warfare and Security (ICCWS 2017), Dayton, OH.

14. V. Nagaraju, L. Fiondella, T. Wandji (2017). An Open Source Tool to Support the Quantitative Assessment of Cybersecurity for Software Intensive System Acquisition. Journal of Information Warfare (JIW), 16(3), pp. 31-50.

13. V. Nagaraju, L. Fiondella, . Jayasinghe, P. Zeephongsekul, T. Wandji (2017). Performance Optimized Expectation Conditional Maximization Algorithms for Non-homogeneous Poisson Process Software Reliability Models. IEEE Transactions on Reliability (T-Rel), 66(3), pp. 722-734.

12. V. Nagaraju, L. Fiondella, P. Zeephongsekul, T. Wandji (2017). An Adaptive EM Algorithm for the Maximum Likelihood Estimation of Non-homogeneous Poisson Process Software Reliability Growth Models. International Journal of Reliability, Quality and Safety Engineering (IJRQSE), (in press).

11. V. Nagaraju, A. Krishna Murthy, L. Fiondella, P. Zeephongsekul, T. Wandji (2016). Expectation Conditional Maximization Algorithms for Failure count Non-homogeneous Poisson Process Software Reliability Models. In Proc. (ISSAT 2016), pp. 372-376, CA. Best Student Paper Award, International Conference on Reliability and Quality in Design. 

10. V. Nagaraju, V. Basavaraj, L. Fiondella (2016). Software Rejuvenation of a Fault-tolerant Server subject to Correlated Failure. In Proc. of 62nd Annual Reliability and Maintainability Symposium (RAMS 2016), pp. 1-6, Tucson, AZ.

9. V. Nagaraju, K. Katipally, R. Muri, T. Wandji, L. Fiondella (2016). An Open Source Software Reliability Tool: A Guide for Users. In Proc. 2016 International Conference on Reliability and Quality in Design (ISSAT 2016), pp. 132-137, CA.

8. V. Nagaraju, A. Krishna Murthy, L. Fiondella, P. Zeephongsekul, T. Wandji (2016). Expectation Conditional Maximization Algorithms for Failure count Non-homogeneous Poisson Process Software Reliability Models. In Proc. 2016 International Conference on Reliability and Quality in Design (ISSAT 2016), pp. 372-376, CA.

7. V. Nagaraju, L. Fiondella (2016). Maximum Likelihood Estimation of a Non-homogeneous Poisson Process Software Reliability Model with the Expectation Conditional Maximization Algorithm. In JSM Proceedings, Section on Statistics in Defense and National Security, Conference on Applied Statistics in Defense 2015, pp. 1121-1131, Alexandria, VA: American Statistical Association.

6. P. Zeephongsekul, C. Jayasinghe, L. Fiondella, V. Nagaraju (2016). Maximum Likelihood Estimation of Parameters of NHPP Software Reliability Models Using EM and ECM Algorithms. IEEE Transactions on Reliability, 65(3), pp. 1571-1583.

5. A.M. Neufelder, L. Fiondella, L. Gullo, H. Daughtrey (2015). Advantages of IEEE P1633 for Practicing Software Reliability. In Proc. of Annual Reliability and Maintainability Symposium (RAMS), pp. 900-906, Palm Harbor, FL.

4. L. Fiondella, R. Duffey (2015). Software and Human Reliability: Error Reduction and Prediction. In Proc. of the International Topical Meeting on Probabilistic Safety Assessment and Analysis, Sun Valley, ID.

3. V. Nagaraju, L. Fiondella (2015). An Adaptive EM Algorithm for NHPP Software Reliability Models. In Proc. of Annual Reliability and Maintainability Symposium (RAMS), pp. 17-23, Palm Harbor, FL.

2. L. Fiondella, P. Zeephongsekul (2014). Discrete Software Reliability Growth Model based on Maximum Entropy Principle with Higher Order Polynomial Moment Constraints. In Proc. of ISSAT International Conference on Reliability and Quality in Design, pp. 263-267, Seattle, WA.

1. J. Liu, Z. Wang, Z. Peng, J.-H. Cui, L. Fiondella (2014). Suave: Swarm Underwater Autonomous Vehicle Localization. In Proc. of IEEE INFOCOM, pp. 64-72, Toronto, Canada.

Invited Talks

56. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, University of Messina, Messina, Italy, July 5, 2023.

55. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, Federal University of Pernambuco, Recife, Brazil, May 15, 2023.

54. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, University of Messina, Messina, Italy, May 11, 2023.

53. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, Defense and Aerospace Test and Analysis (DATA) Workshop, Alexandria, VA, April 26, 2023.

52. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, Vietnam International Academic Network in Information Technology Webinar (VIAN-IT), March 9, 2023.

51. A Metrics-based Software Tool to Guide Software Reliability and Security Test Activity Allocation, American Society for Quality (ASQ) Reliability and Risk Division Webinar, March 9, 2023.

50. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, University of Tennessee, Knoxville, Knoxville, TN, February 13, 2022.

49. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, University of Arizona, Tucson, AZ, February 6, 2022.

48. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, George Mason University, Fairfax, VA, January 19, 2023.

47. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, Indian Statistical Institute, Kolkata, India, January 13, 2023.

46. A Metrics-based Software Tool to Guide Software Reliability and Security Test Activity Allocation, American Society for Quality (ASQ) Aviation, Space & Defense Division Webinar, December 2, 2022.

45. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, Iowa State University, Ames, IA, November 29, 2022.

44. Covariate Software Reliability Models and Applications, Florida Atlantic University, Boca Raton, FL, October 20, 2022.

43. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, American Society for Quality (ASQ) Fall Technical Conference, Park City, UT, October 14, 2022.

42. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, ASML Reliability Engineering Conference, Veldhoven, Netherlands, Sep 27, 2022.

41. Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, IEEE Systems, Man, and Cybernetics Society, Australian Capital Territory Chapter, Canberra, Australia, Sep 20, 2022.

40. Software Reliability and Security Risk Assessment: Modeling and Algorithms, DST-NSF Joint Research and Development Projects Kick-off Workshop, Indian Institute of Technology (IIT), Delhi, Delhi, India, Aug 9, 2022.

39. Multi-phase Algorithm Design for Accurate and Efficient Model Fitting, Subir Chowdhury School of Quality and Reliability, Distinguished Speaker Series on Quality, Reliability, Availability, Maintainability, and Safety (QRAMS), Indian Institute of Technology Kharagpur, Karagpur, India, April 30, 2022.

38. A Metrics-based Software Tool to Guide Test Activity Allocation, American Society for Quality (ASQ) 29th Annual Collaboration on Quality in the Space and Defense Industries Forum (CQSDI), Kennedy Space Center, FL, March 16, 2022.

37. Connecting Software Reliability Growth Models to Software Defect Tracking, University of Alabama in Huntsville, Huntsville, Alabama, March 2, 2022.

36. Connecting Software Reliability Growth Models to Software Defect Tracking, IEEE Hyderabad Section, Hyderabad, India, October 23, 2021.

35. SRGM Tools and experience with SRGMs – especially DoD, Ground Based Strategic Deterrent Program, Hill Air Force Base, Utah, August 4, 2021.

34. Software Reliability Research – Assessments & Tools, Raytheon Software Reliability Community of Practice, Portsmouth, RI, June 25, 2021.

33. Covariate Software Reliability Models and Applications, IEEE Reliability Society Distinguished Lecturer Series, Boston, MA, December 8, 2020.

32. Software Reliability Engineering: Algorithms and Tools, American Society for Quality (ASQ) Statistics, Reliability & Risk, and Software Division Joint Webinar, November 10, 2020.

31. Covariate Software Reliability Models and Applications, Center for Scientific Computing and Visualization Research, North Dartmouth, MA, November 4, 2020.

30. Covariate Software Reliability Models and Applications, Oak Ridge National Laboratory, Oak Ridge, TN, June 8, 2020.

29. Covariate Software Reliability Models and Applications, Virginia Tech Research Center, Arlington, VA, March 12, 2020.

28. Continuous Integration, Software Reliability in the 21st Century Panel Discussion held at the 66th Annual Reliability and Maintainability Symposium (RAMS 2020), Palm Springs, CA, Jan 28, 2020.

27.Towards a Guide for Software Defect Tracking, Modeling, and Analysis, NASA Software Assurance Research Program Review, Greenbelt, MD, February 6, 2020.

26. Software Reliability and Security Modeling and Analysis from Software Defect Tracking Data, NASA Goddard Space Flight Center, Software Engineering Division, Code 580, November 19, 2019.

25. Relationships between Machine Learning and Reliability Engineering, Air Force Institute of Technology, Dayton, OH, October 31, 2019.

24. Software Reliability Engineering (two day course), BAE Systems, Nashua, NH, January 16-17, 2019.

23. Software Reliability Engineering: Algorithms and Tools, Rutgers University, New Brunswick, NJ, December 20, 2018.

22. Practical Software Reliability Modeling and Application, NASA Software Assurance Working Group Meeting, Mountain View, CA, July 31, 2018.

21. Software Reliability Engineering: Algorithms and Tools, Indian Institute of Science, Bangalore, India, June 12, 2018.

20. Software Reliability Engineering: Algorithms and Tools, Indian Institute of Technology Kharagpur, Karagpur, India, May 31, 2018.

19. Software Reliability Engineering: Algorithms and Tools, Amity University, Noida, India, May 29, 2018.

18. Tools to Assess Software Reliability and Security, MIT Lincoln Laboratory, Lexington, MA, April, 30, 2018.

17. Software Reliability Engineering: Algorithms and Tools, Auburn University, Auburn, AL, March 27, 2018.

16. Practical Software Reliability Modeling and Application, NASA Goddard Space Flight Center, Greenbelt, MD, March 7, 2018.

15. Software Reliability Engineering: Algorithms and Tools, Old Dominion University, Norfolk, VA, February 15, 2018.

14. Software Reliability Engineering: Algorithms and Tools, Virginia Commonwealth University, Richmond, VA, February 1, 2018.

13. Software Reliability Engineering: Algorithms and Tools, Florida International University, Miami, FL, January 20, 2018.

12. Free and Open Source Tools to Assess Software Reliability and Security, Boston Software Process Improvement Network (SPIN), Bedford, MA, November 21, 2017.

11. Tools to Assess Software Reliability and Security, Unmanned System Common Control System (CCS) Integrated Product Team Meeting, Patuxent River, MD, December 5, 2017.

10. ECM Algorithms & An Open Source Tool, Tottori University, Tottori, Japan, July 4, 2017.

9. Software Reliability Engineering: Algorithms and Tools, Florida State University, Tallahassee, FL, February 27, 2017.

8. Software Reliability Research – Assessments & Tools, Office of the Secretary of Defense, Developmental Test and Evaluation Functional Integrated Product Team Meeting, Washington, DC, Jan 11, 2017.

7. Software Reliability Research – Assessments & Tools, Office of the Deputy Assistant Secretary of Defense for Developmental Test & Evaluation, Alexandria, VA, Oct 28, 2016.

6. Software Reliability: Tools and Algorithms, IEEE Boston Reliability Society, Lexington, MA, Sep 14 2016.

5. Software Reliability Engineering: Algorithms and Tools, Naval Postgraduate School, Monterey, CA, May 27 2016.

4. Software Reliability: Tools and Models, Workshop on Rigorous Test and Evaluation for Defense, Aerospace, and National Security, Arlington, VA, April 12, 2016.

3. Algorithms and Tools for Software Reliability Engineering, University of Maryland, Dec 2, 2015.

2. An Open Source Software Reliability Tool and Model Fitting Algorithm, Wright State University, Oct 7, 2015.

1. Software Reliability Engineering: Practice and Theory, UMass- Naval Undersea Warfare Center (NUWC) Lecture Series, Nov 20, 2014.

Presentations and Tutorials

35. J. Sorrentino, P. Silva, B. Gaspard, G. Kul, and L. Fiondella, Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, National Defense Industrial Association (NDIA) Annual Systems Engineering Conference, Norfolk, VA, October 2023.

34. P. Silva, V. Nagaraju, and L. Fiondella, Introduction to Covariate Software Reliability Models, In Proc. 69 the Annual Reliability and Maintainability Symposium (RAMS 2024), Albuquerque, NM, Jan 2024.

33. S. Bhadra, B. Das, P. Silva, V. Nagaraju, and L. Fiondella, A Stochastic Petri Net Model of Continuous Integration and Continuous Delivery, Defense and Aerospace Test and Analysis (DATA) Workshop, Alexandria, VA, April 2023.

32. F. Salboukh, P. Silva, V. Nagaraju, and L. Fiondella, Application of Recurrent Neural Networks in Software Defect Discovery, Defense and Aerospace Test and Analysis (DATA) Workshop, Alexandria, VA, April 2023.

31. Ebenezer Yawlui, P. Silva, V. Nagaraju, and L. Fiondella, Optimal Release for Covariate Software Reliability Models, Defense and Aerospace Test and Analysis (DATA) Workshop, Alexandria, VA, April 2023. Outstanding poster presentation.

30. P. Silva, V. Nagaraju, and L. Fiondella, Introduction to Covariate Software Reliability Models, In Proc. 69 the Annual Reliability and Maintainability Symposium (RAMS 2023), Orlando, FL, Jan 2023.

29. J. Sorrentino, P. Silva, B. Gaspard, G. Kul, and L. Fiondella, Covariate Software Vulnerability Discovery Model, In Proc. 69 the Annual Reliability and Maintainability Symposium (RAMS 2023), Orlando, FL, Jan 2023.

28. J. Sorrentino, P. Silva, B. Gaspard, G. Kul, and L. Fiondella, Covariate Software Vulnerability Discovery Model to Support Cybersecurity Test & Evaluation, Reliability, Maintenance & Managing Risk Conference (RMMR), Hartford, CT, Aug 2022.

27. K. Chen, Z. Faddi, V. Nagaraju, and L. Fiondella, Quantifying the Impact of Staged Rollout Policies on Software Process and Product Metrics, Presented to WG24 Test and Evaluation (T&E), 90th Military Operations Research Symposium (MORS 2022), Quantico, VA, June 2022.

26. M. Nafreen, Y. Shi, and L. Fiondella, Software Defect Discovery and Resolution Modeling incorporating Severity, Presented to WG17 Logistics, Reliability, and Maintainability, 90th Military Operations Research Symposium (MORS 2022), Quantico, VA, June 2022.

25. S. Pritchard, V. Nagaraju, and L. Fiondella, Automating Staged Rollout with Reinforcement Learning, Presented to WG34 Data Science and Analytics, 90th Military Operations Research Symposium (MORS 2022), Quantico, VA, June 2022.

24. K. Chen, Z. Faddi, V. Nagaraju, and L. Fiondella, Quantifying the Impact of Staged Rollout Policies on Software Process and Product Metrics, Defense and Aerospace Test and Analysis (DATA) Workshop, Alexandria, VA, May, 2022.

24. M. Nafreen, P. Silva, L. Dharmasena, V. Nagaraju, and L. Fiondella, Bayesian Methods for Covariate Software Reliability Model Estimation, Defense and Aerospace Test and Analysis (DATA) Workshop, Alexandria, VA, May, 2022.

23. V. Nagaraju, L. Fiondella, Introduction to Mathematical Software Reliability Models, In Proc. 68th Annual Reliability and Maintainability Symposium (RAMS 2022), Tucson, AZ, Jan 2022.

22. J. Aubertine, K. Chen, V. Nagaraju, and L. Fiondella, A Metrics-based Software Tool to Guide Test Activity Allocation, Presented to WG17 Logistics, Reliability and Maintainability and WG24 Test and Evaluation (T&E) and Experimentation, at the 89th Military Operations Research Symposium (MORS 2021), Quantico, VA, June 2021.

21. Jacob Aubertine, Vidhyashree Nagaraju, and Lance Fiondella, A Covariate Software Tool to Guide Test Activity Allocation. Presented at the Defense and Aerospace Test and Analysis (DATA) Workshop, Springfield, VA, April, 2021.

20. V. Nagaraju, C. Jayasinghe, and L. Fiondella, Covariate Software Reliability Models and Applications, Presented to WG17 Logistics, Reliability and Maintainability and WG24 Test and Evaluation (T&E) and Experimentation, at the 88th Military Operations Research Symposium (MORS 2020), New London, CT, June 2020.

19. M. Nafreen, M. Luperon, V. Nagaraju, Y. Shi, and L. Fiondella, Connecting Software Reliability Growth Models to Software Defect Tracking, Presented to WG17 Logistics, Reliability and Maintainability and WG24 Test and Evaluation (T&E) and Experimentation, at the 88th Military Operations Research Symposium (MORS 2020), New London, CT, June 2020.

18. A. Gula, C. Ellis, S. Bhattacharya, and L. Fiondella, Relationships between Machine Learning and Reliability Engineering, Presented to WG17 Logistics, Reliability and Maintainability and WG35 AI and Autonomous Systems, at the 88th Military Operations Research Symposium (MORS 2020), New London, CT, June 2020.

17. M. Nafreen, M. Luperon, V. Nagaraju, Y. Shi, and L. Fiondella, Connecting Software Reliability Growth Models to Software, Presented at the Defense and Aerospace Test and Analysis (DATA) Workshop, Springfield, VA, May, 2020.

16. A. Gula, C. Ellis, S. Bhattacharya, and L. Fiondella, Software and System Reliability Engineering for Autonomous Systems incorporating Machine Learning, Poster presented at the 66th Annual Reliability and Maintainability Symposium (RAMS 2020), Palm Springs, CA, Jan 2020.

15. M. Nafreen, S. Bhattacharya, and L. Fiondella, Architecture-based Software Reliability incorporating Fault-Tolerant Machine Learning, Poster presented at the 66th Annual Reliability and Maintainability Symposium (RAMS 2020), Palm Springs, CA, Jan 2020.

14. V. Nagaraju, Y. Shi, and L. Fiondella, Software Reliability and Security Assessment: Automation and Frameworks, Presented at the Defense and Aerospace Test and Analysis (DATA) Workshop, Springfield, VA, April, 2019.

13. V. Nagaraju, L. Fiondella, Introduction to Mathematical Software Reliability Models, In Proc. 65th Annual Reliability and Maintainability Symposium (RAMS 2019), Orlando, FL, Jan 2019.

12. V. Nagaraju, L. Fiondella, and T. Wandji, A Single Changepoint Software Reliability Growth Model with Heterogeneous Fault Detection Processes, Poster presented at the Society of Risk Analysis (SRA) Annual Meeting, Arlington, VA, December, 2017.

11. V. Nagaraju, T. Wandji, and L. Fiondella, Free and Open Source Tools to Assess Software Reliability and Security, Presented at the National Defense Industrial Association (NDIA) Annual Systems Engineering Conference, Springfield, VA, October, 2017.

10. V. Nagaraju, T. Wandji, and L. Fiondella, Free and Open Source Tools to Assess Software Reliability and Security, Presented at the National Defense Industrial Association (NDIA) Annual Systems Engineering Conference, Springfield, VA, October, 2017.

9. V. Nagaraju, T. Wandji, and L. Fiondella, Software Reliability: Modeling and Tools, Tutorial presented at the 85th Military Operations Research Symposium (MORS 2017), West Point, NY, June 2017.

8. V. Nagaraju, T. Wandji, and L. Fiondella, Software Reliability Modeling, Presented at the Science of Test Workshop, Springfield, VA, April, 2017.

7. V. Nagaraju, L. Fiondella, T. Wandji, Introduction to Mathematical Software Reliability Models, In Proc. 63nd Annual Reliability and Maintainability Symposium (RAMS 2017), Orlando, FL, Jan 2017.

6. V. Nagaraju, T. Wandji, and L. Fiondella, Software Failure and Reliability Assessment Tool (SFRAT): An Open Source Application for the Practitioner and Research, Presented at the Conference on Applied Statistics in Defense, Washington, DC, October, 2016.

5. L. Fiondella, A. Nikora, and T. Wandji, Software Reliability and Security: Challenges and Crosscutting Themes, Fast abstract In Proc. International Symposium on Software Reliability Engineering, Ottawa, Canada, Nov, 2016.

4. V. Nagaraju, T. Wandji, and L. Fiondella, An Open Source Tool to support Software Reliability Assessment, Presented to WG 17 Logistics, Reliability and Maintainability and WG 24 Test and Evaluation (T&E) at the 84rd Military Operations Research Symposium (MORS 2016), Quantico, VA, June 2016. Nominated for the 85th Barchi Prize.

3. V. Nagaraju and L. Fiondella, Introduction to Mathematical Software Reliability Models, In Proc. 62nd Annual Reliability and Maintainability Symposium (RAMS 2016), Tucson, AZ, Jan 2016.

2. V. Nagaraju and L. Fiondella, An Adaptive EM Algorithm for NHPP Software Reliability Models, UMassD Sigma Xi Research Exhibition, Apr 2015.

1. L. Fiondella and T. Wandji, An Open Source Application Architecture for Software Reliability Models, In Proc. International Applied Reliability Symposium (ARS), Tucson, AZ, Jun 2015.

Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant Numbers (1526128 and 1749635), the Naval Air Systems Command (NAVAIR) through the Systems Engineering Research Center, under Research Task 139, NAVAIR under contract N00421-16-T-0373 and the National Aeronautics and Space Administration (NASA) under award numbers (80NSSC18K0154 and 80NSSC20K0276)

Work at JPL was supported under NASA Prime Contract NNN12AA01C, under Task Plan No. 81-19648.