Manage duplicate records
Before loading data into Blue for the first time or reimporting, it is important to make sure the data is as accurate as possible. Tools like the Data Integrity Gateway (DIG) are beneficial for validating data before use in Blue projects. Sometimes no matter how careful we are with data integrity and validation, we can overlook certain things and end up getting datasource import errors. Below is a way to troubleshoot datasource import errors from MS Excel or CSV datasources into Blue with regards to finding duplicates in datasource errors and correcting them.
There are two methods using MS Excel to eliminate duplicates. The first method is used to mark the duplicate records and then manually delete them at a later time, the second method immediately removes duplicate records.
It is also possible to find duplicate values in Excel for very large spreadsheets where multiple users have access to the same course ID (in other words, the course ID remains the same while the UserID changes for different users). The simplest approach to finding duplicates of values that are a combination of values in two columns is to create a third column and concatenate the two column values into one. Then, use any of the two methods for finding duplicates in the newly created column.