SUPERB AI.

Product

A Guide to Improving Model Performance in Just 3 Hours with Superb Platform’s Model Diagnosis: Experiment on BDD 100K (mAP Improved by 10%)

Summary: We have undertaken a thorough process of training an AI model, assessing its performance, identifying model vulnerabilities, and addressing them through Superb Platform for enhanced performance. Consequently, the model’s average accuracy (mAP) has improved from 38.9 to 42.9. The steps in our process include: 1. Curate and process (=label) data 2. Train a model 3. Diagnose the model’s performance 4. Identify and address model vulnerabilities 5. Re-train the model again 6. Augment data 7. Re-train the model, diagnose its performance, and deploy Introduction In our previous blog series, we discussed the importance of continuous improvement and optimization in AI model development and the steps we should follow to achieve them. We also covered the definitions and applications of key metrics for diagnosing model performance. In this article, we will guide you through the process of pinpointing and addressing model vulnerabilities through Model Diagnosis with a step-by-step approach. In this experiment, we utilized an open-source dataset called BDD 100K. About Released by the Berkeley Artificial Intelligence Research, BDD 100K is a comprehensive dataset comprising 102 million images, complete with GPS, IMU, timestamp, and other metadata for use in diverse applications. Its diverse data, designed for all sorts of road network applications, includes pre-labeled objects such as lanes, vehicles, people, and road infrastructure. Experiment Objective: We expect this experiment to provide valuable guidance for businesses facing challenges in AI model advancement post-adoption. Our goal is to aid companies who have experienced setbacks after huge time and resource investments due to subpar AI model performance or those who have failed AI adoption as they encountered performance degradation in real-world (Wild) applications compared to test environments (PoC). By sharing our findings and solutions through the Superb Platform experiment, we intend to facilitate successful AI integration into business operations. Experiment Overview: Continuous iteration of the following procedures is essential for companies to develop, deploy, maintain, and enhance their AI models. As we have mentioned numerously, this is a crucial part of the MLOps pipeline, which many companies (including us at Superb AI) and research institutions are now stressing as highly important. The process is comprehensive and intricate, yet we've distilled it with a focus on 'model performance improvement,' the primary goal of this experiment: Let's delve into how these steps collectively enhance the performance of a model in its early stages. 1. Data curation: First, we utilize the 'Auto-curate - Split train/validation set' feature to select 30,000 images from the BDD 100K dataset and divide them into a training dataset and a validation dataset. We opted for a 4:1 ratio, allocating 24,000 images for training and 6,000 for validation. The curated dataset forms a separate data group referred to as a ‘slice**.’ 🧐 This figure stems from our previous experiment's findings, demonstrating that models trained on smaller datasets can achieve performance comparable to those trained on larger ones. Training on a smaller dataset optimizes various resource usage, including data labeling and training time. Check our previous experiment here BDD 100K dataset size : 30,000 images Train dataset: Slice name: “train experiment v1” size: 24,000 images Validation dataset Slice name: “validation experiment v1” size: 6,000 images What is Auto-Curate is a feature that automatically curates data based on user requirements, and it offers 4 options: Curate What to Label Split Train/Validation Set) Find Edge Cases Find Mislabels What is a A slice is a subset of a dataset, serving as the basic unit for all data-related activities in Superb Curate. Users can create slices of specific characteristics or purposes. (Source: 2. Model training With data curation complete, we can now proceed to model training. Superb Model currently offers 5 high-performance baseline models, enabling users to select the most suitable one based on capacity, speed, supported annotation types, and so on. In this experiment, we used the For consistency, we designated specific train and validation slices. Training can be initiated by selecting the classes for training and naming the model. Anyone can effortlessly train a model once the dataset is designated. If you are training your model on Superb Model, it's important to note that higher performance models may require extended training times, up to 12 hours depending on the data volume. 3. Model performance diagnosis After completing the model training, you can assess your trained model's performance in the same page. This page presents a variety of metrics and graphs that depict your model's performance, ranging from the most widely-used mAP to PR Curve, IoU, F1 Score, Confidence, and Confusion Matrix. 4. Model vulnerability identification and improvement (using Superb Curate’s ‘Model Diagnosis’ and ‘Auto-Curate,’ and Superb Label’s ‘Labeling’ feature) The mAP of our model, following the initial training, stood at 38.9. We will now focus on identifying vulnerabilities in our model from a data-centric perspective, rectifying poor-performing data, and enhancing the mAP score. A quick and easy method to pinpoint vulnerable data is to use Superb Curate. 1) Using Superb Curate’s ‘Model Diagnosis’ Through Superb Platform's Grid View and Analytics View and the diverse filter options provided, you can visually identify vulnerable data, edit or augment them, and thus improve model performance. The following filter options allow you to examine the data and isolate them into a ‘slice’: The Ground Truth data that is not labeled accurately (=Mislabeled) The trained model's prediction matches the Ground Truth label (=True Positive) The Ground Truth data is accurately labeled, but the model fails to generate a prediction (=False Negative) The trained model generated a prediction, but it does not exist in the Ground Truth data (=False Positive) 2) Mislabeled data from the train set (Ground Truth) (=Find Mislabel, name: mislabel test slice) In this experiment, we will send the slice containing 2-1) mislabeled train data to Superb Label to correct labeling and use the updated data for re-training. Since the model is trained with incorrect data, addressing the mislabels will lead to an increased model performance metric. Let’s proceed to Superb Label and open the slice requiring correction. Upon examining the actual data points, we found that ‘car’ was mislabeled as ‘traffic light,’ and ‘traffic light’ as ‘rider.’ Our objective is to identify and rectify such mislabeled Ground Truth data. After successfully correcting all data, we will send the slice back to Superb Curate to prepare for re-training, using the “ 5. Model re-training (Superb Model’s “Train with more data” feature) We augmented the mAP 38.9 model from Step 2 with the refined dataset from Step 4 and initiated re-training to enhance performance. (Name: [experiment]model-diagnosis-YOLOv-6v2) 6. Data augmentation for rare classes (Superb Curate’s ‘Query’ and ‘Auto-Curate - Edge Case’ features) Following Step 5, the mAP value rose to 42.3, signaling a substantial performance improvement. To further enhance this, we can revisit the model's performance metrics. Add more ‘rare class’ data (using Superb Curate’s ‘Query’ feature): Our analysis revealed a relative scarcity of training data for ‘bike,’ ‘motor,’ and ‘rider.’ By augmenting data for these 3 classes and re-training, we expect to further improve performance. We selected approximately 3,000 images from an unused pool of 8,000 ‘bike,’ ‘motor,’ or ‘rider’ images for this purpose and created a slice to use for additional training. Add edge cases (using Superb Curate’s ‘Auto-Curate - Edge Case’): As an alternative approach, we can consider training the model with additional edge case data. From the unused data and those not qualifying for 6-1), we automatically selected 3,000 edge case data for further training through Auto-Curate. 7. Additional re-training, model performance diagnosis, and deployment (Superb Model’s ‘Train with more data’, Superb Curate’s ‘Model Diagnosis’) We added the two data slices from the above steps to the training dataset using Superb Model’s ‘Train with more data’ feature and commenced additional re-training. If the mAP value rises beyond 42.3 post-retraining, it would indicate another successful performance enhancement. The mAP value has increased from 42.3 to 42.9, marking another milestone in model performance improvement! Beyond this metric, you can analyze model performance from multiple perspectives using various performance metrics. Visit the ‘Model Diagnosis’ feature for an intuitive understanding of the performance gap between the first version of your model (mAP 38.9) and the latest version (42.9). This feature provides insights using Confusion Matrix, PR Curve, and other diverse indicators and graphs. Can We Opt to Change the Type of the Trained Baseline Model? You can opt to try out other models offered in Superb Model to find the baseline model that best fits your dataset. However, it’s important to note that the data-centric approach, which involves diagnosing model performance, identifying vulnerabilities, and addressing them, remains consistent across different models. We chose the DETA (Swin-L) model from Superb Model and followed the aforementioned process. As observed, the patterns and trends in accuracy improvement are similar. Conclusion Excluding the time for data processing and model training, it took only 3 hours to analyze and identify model vulnerabilities, and devise solutions. With Superb Curate’s Model Diagnosis, you can quickly identify incorrectly classified data or the most frequent types of errors, minimizing the trial and error often associated with interpreting and combining performance metrics. This feature also facilitates rapid insights into performance improvement opportunities, helping you understand which data needs augmentation or correction. In other words, you can quickly determine whether performance issues stem from labeling errors, insufficient edge cases, or a lack of data for specific classes. Our experiment further demonstrates that an MLOps cycle embodying a “true data-centric approach” can be established through Model Diagnosis. The key to building an effective MLOps pipeline is accelerating the cycle to enable as many iterations as possible. Superb Platform is designed for users to execute the entire MLOps cycle seamlessly in one integrated environment, saving time and resources that would otherwise be spent switching between different tools and platforms including external ones. Are you experiencing problems with AI adoption and model improvement, or interested in analyzing performance changes following improvement actions? If you wish to learn how Superb Platform can further enhance model performance, please don't hesitate to contact us. We are eager to provide you with a product demo tailored to your data and industry and to explain the benefits of the data-centric approach in more detail. We assure you, Superb Platform is the best solution for those seeking to commercialize AI models.

Tyler McKean

Head of Customer Success | 5 min read