Deep Learning AI for On-the-Spot Defect Analysis

Key Takeaways

• AI enables real-time defect detection, reducing manual efforts and speeding up issue resolution in software development.
• Natural language processing (NLP) sentiment analysis can help to categorize defects, whether it is an incident, new business requirement or feedback, and can also assign criticality to the defect.
• Image classification using Keras and TensorFlow library ensures brand consistency by detecting client logos from received support ticket emails.
• Integrating NLP and image classification provides additional confidence and a comprehensive approach to defect analysis using text and image data.
• Continuous monitoring and deployment of machine learning models are crucial for maintaining accurate real-time defect analysis.

Artificial intelligence (AI) is significantly transforming the field of software testing, offering new tools and methodologies to enhance the efficiency, accuracy and scope of testing activities. Triaging defects, determining their root causes, and prioritizing and assigning them to the appropriate technical team is a time-consuming process that can be automated for real-time action using AI.

Urgent production incidents occur that are often critical and require swift intervention. must first carefully analyze the issue, putting in a significant amount of manual effort to pinpoint the root cause, assign the correct priority and determine if it is a defect, an enhancement or a task.

For instance, the recent CrowdStrike issue led to a support team bombarded with production incident tickets from several customers. Generally, a support team is quite small, and production incidents have short service level agreement (SLA) spans, making it complicated to perform each ticket analysis.

In this article, I am introducing an AI/machine learning (ML) solution that uses deep learning models – Image Classification with TensorFlow and Keras, along with natural language processing (NLP) – to automatically analyze production incidents to significantly reduce the manual efforts and allow for a quicker initiation of issue resolution for business users.

Real-Time Incident Flow Scenario

The clients of many service-based organizations are product-based organizations that have several types of customers. In the case of any new deployment, these end users are directly affected, and if they face any issues, they raise new support tickets. These are known as real-time defects, which are high-priority incidents and require immediate triage. Some incident examples include a newly deployed API throwing a 5xx error or a login service not working, resulting in the entire business coming to a halt. For such incidents termed as P1, the SLA window is very narrow and automating such tickets will certainly help service-based organizations.

Here, I will use text extraction for reading email body content to find the client, defect category (incident or new business requirement or feedback) and criticality (high, medium or low) of the issue. This information helps developers determine what kind of issue it is, whether they need to triage it in the current sprint or delegate it to the next based on priority. For confidence in the text classification model output, I will use image classification on the client logo in the received email, which will predict the client and then match both the text extraction model and image classification model output for verifying the client. We will predict the following from sent incidents emails:

1. Who is the sender?
2. What is the criticality of this incident?
3. What is the category/priority of this incident?
4. Identify the logo is for which client?

Text Extraction and NLP Sentiment Analysis

Sentiment analysis as a part of NLP is the act of collecting sentiments or opinions expressed in text data. These advances are key to real-time problem detection; they can use user feedback (both direct and via social media), support tickets and other textual sources to surface emerging issues.

Implementation: NLP Sentiment Analysis

Collect text data from different sources such as customer reviews, social media platforms (e.g., X), support tickets or emails. Next, feed this data into the NLP sentiment analyzer.

• Data collection. Gather the text data from various sources, such as web html support tickets or internal databases. Python provides several libraries, such as requests and BeautifulSoup – scrap and collect data from the html web. This process helps in identifying which client (product-based organization) the email is sent to from a service-based organization.
• Data preprocessing. Prepare the text for further analysis such as tokenization, lemmatization or stop-word removal. These help in segmenting the text to discernible parts and removing noise.
• Text mining: sentiment analysis. Analyze the processed text using trained machine learning models designed to recognize sentiment. The text is classified as positive, negative or neutral for such models by sentiment. This process helps to identify the criticality (priority, severity) and categorization (predicting if sent email is defect, requirement, or incident) of the defect.
• Identify ML model. Depending on text mining goals, build machine learning models to perform tasks like classification, clustering or recommendation. Common libraries for text classification include scikit-learn and TensorFlow.

What is Image Classification and Why Do We Need It?

Image classification belongs to computer vision, in which an image can be typically classified into one or multiple predefined classes based on its content.

For production incident triage, an NLP model will provide us with a category and criticality of defect. The email body content will provide the client, but we can get additional confidence on our NLP model by validating the logo present in the customer’s sent email. If this logo matches the client logo that we are getting as an output from the NLP model, we can confidently say that this incident is popping up from this client. We can route it to the respective team with minimal chances of any error. To detect a client’s logo in sent email, we need to train an image classification model (see Figure 3).

Implementation – Image Classification

• Model training. In the above image, we can see this model and dataset in which data (images) is loaded from batches into a neural network such that it derives learning patterns among themselves in the form of weights. The training and testing datasets for logo defect detection would contain a set of correct logo examples. The model learns the different classes by identifying certain patterns in images.
• Image capture in real time. When we finish training our model, we will not set it up to capture images of the logo from customer sent emails.
• Image processing. Process captured images to make them the same size, orientation and color. Then standardize these images and later perform classification.
• Opinions. Finally, the model classifies these images as matching clients or null output if logos do not match any clients.
• Defect reporting. a defect report and raises an alert when it identifies null, which means we need to retrain our data on more client logos.

End-to-End Flow Diagram for Defect Analysis Using NLP and Image Classification

In Figure 4, both NLP sentiment analysis and image classification are incorporated into one unified defect analysis system. This integrated approach ensures a comprehensive insight into potential client detection by using the power of both text and image data.

Data Integration

This comprehensive view of incident analysis is a key advantage to the integration between data from NLP sentiment analysis and image classification. For instance, if users send an email with a negative review email, then email content and client logo in their email is passed to the above shown flow diagram using text and image classification process for automatic analysis of incidents.

Automated Workflow

Using cloud frameworks, we can integrate this flow providing automated defect detection and reporting. For example, if we receive a user feedback email, it can invoke an API, further invoking ML-deployed code, which will provide client-related information using the above highlighted flow diagram.

Real-Time Defect Analysis: ML Model Deployment

The flow diagram in Figure 5 depicts how an ML model can be deployed for predicting defects from real-time emails. The models need to be kept up-to-date and maintained so that they are still useful as new types of defects occur or the software is modified.

Steps for ML Model Deployment

• Prep Model and Upload (Train and Save Model). Train our machine learning model in a framework; save it as a .pkl (pickle) file. Upload the .pkl file to an s3 bucket.
• Configure AWS Lambda. Load the model, accessing the s3 bucket, and receive prediction by feeding a new run time.
• API Gateway Integration. Configure our API gateway to link to the Lambda function so that it can be triggered when requested.

Conclusion

A software testing use case is now one step closer to real-time defect analysis and includes NLP sentiment analysis and image classification for logos or brand names, with model deployment using ML. With these technologies, organizations can detect and fix defects faster and more accurately, which in return will enhance software quality, leading to satisfied users. With the ever-evolving AI cousin, we can expect to see or hear about tools like this being integrated into and used in defect analysis going forward – opening even more avenues of innovation and efficiency for software development.