Implement course evaluation using 3-level relationship (3-LR)
Introduction
While most course evaluations traditionally focus on courses and instructors, a growing number of institutions are expanding their evaluations to include a course-instructor-student relationship known as 3-level relationship (3-LR) to allow students to evaluate the instructors who taught their course components (e.g., lab, lecture, seminar). This method requires the institution to maintain a comprehensive record of student enrollments and instructor assignments throughout the evaluation period.
We will provide details on how to set up an evaluation using a 3-LR datasource that links the student to the instructor and the instructor to the course via their instructor role based on the course component they taught. This 3-LR datasource enables Blue to preselect instructors for each course component. This makes it easier and quicker for students to complete their evaluations, ensuring only relevant evaluations are submitted and higher response rates.
3-LR reporting variations
Instructor reports per course can differ between instructors because the breakdown of the secondary subject (instructors) is calculated based on the number of students that attended each course component.
In the example below, Dr. James Earl Scott (left) teaches the lecture component of the course African Cultures and Societies. Professor Englund Hemmet (right) teaches the seminar component of the same course. Both professors receive an individual instructor report based on student responses tied to their specific role in teaching that course component. Notice how the primary (course) and secondary (instructor) subject response ratios are different for each report. This is because response counts can differ based on the number of students per course component (lecture, lab, seminar, etc.) who submitted evaluations (FO).
Using DIG with 3-LR
Instructor – student relationship not supported in DIG
DIG is a versatile platform that supports a variety of data structures. However, due to the complexity of the 3-LR file, which pairs students with instructors, this file must be managed outside of DIG.
We recommend enlisting the help of a Database Administrator (DBA) to create the special course-instructor-student relationship file (3-LR), ensuring each student is correctly paired with their instructors for each course.
Demographics supported
DIG fully supports course demographics (e.g., changing course dates, department names, form switches, etc.), instructor demographics (e.g., tenure, professorship, etc.), and student demographics, including relational demographics (e.g., instructor’s teaching type). Since these changes have no impact on enrollment, demographic review, modification, and publishing are fully supported.
Changes to courses - Instructor assignments
Course instructor assignments can be updated in DIG. The same updates must be made to the 3-LR file outside of DIG to ensure that the instructor assignments are linked to the students.
Changes to courses - Merges and splits
It is possible to merge and split courses in DIG, however these changes must also be made in the 3-LR file to link the students to the modified course-instructor pairs.
Blue Dashboard variations when using 3-LR
The 3-level relationship (3-LR) is supported for certain aspects of Blue Dashboard: however, when using the special course-instructor-student relationship (3-LR), the results displayed in Blue Dashboard differ from those in Blue Reports. In Blue Dashboard, instructors are not separated by course components, so the data shows the results for all instructors who taught the course.
See the example below:
Econ 101 - with 3 instructors
- 100 students attached to course in 3-LR:
- 40 students - instructor A
- 60 students - instructor B
- 0 students - instructor C
- 100% response rate
| Responses to Course Questions | Responses to Instructor Questions | |||||
|---|---|---|---|---|---|---|
| A | B | C | A | B | C | |
| Instructor Report | 40 | 60 | No report | 40 | 60 | No report |
| Course Report | 100 | 100 | 100 | 100 | 100 | 100 |
| Dashboard | 100 | 100 | 100 | 40 | 60 | 0 |
Instructor report = Individual type broken down by Secondary subject
Course report = Individual type with no breakdown
Implementation
Step 1 - Create a 3-LR file
- Open a new file (CSV or XLSX format) to create your 3-LR course-instructor-student relationship file.
- Create two columns - SourceID and TargetID in columns A and B respectively.
- From your enrollment data file, copy and paste the appropriate instructor IDs into the SourceID column, and the matching student IDs into the TargetID column.
- From your enrollment data file, copy and paste the matching course IDs into the empty column C.
- In column D, add this formula =A2&"(;)"&C2 in row 2 to concatenate both the instructor ID (column A) and course ID (column C) which will be separated by a semi-colon in brackets "(;)". To complete the rest of the rows, select cell D2 and using the Shift button on your keyboard with the down arrow copy the formula for the remaining Source and Target IDs and press Enter.
- Transfer the concatenated values of column D into column A by copying Column D and pasting it into Column A using the Value paste option. The Value paste option “123” ensures that the formula is not transferred with the values.
Delete columns C and D containing the Course IDs and CONCAT formulas. 7. You will be left with two columns - SourceID (containing both instructor and course IDs separated by a semicolon), TargetID (student IDs) and the Teaching Role. 8. Save the file.
- Import the file into Blue as a 3-LR relationship datasource. Follow the instructions in Add a user datasource section to Set up a data block.
- Use the following values: Select the Users datasource as the Source.
- Select the Users datasource as the Target.
Step 2 - Create a 3-LR project
Before you can create your project, make sure that you have already:
- Created all the required Object (course) and User (instructor) datasources and Course-Student and Course-Instructor relationships.
- Created the special course-instructor-student 3-LR relationship datasource.
- Created a Definition with groups and privileges for your project.
| Datasource name | Type | Contents |
|---|---|---|
| Courses | Object | Courses being evaluated |
| Users | User | Students, instructors, coordinators, and academic leaders |
| Course-Student | Relationship | How students are linked to courses |
| Course-Instructor | Relationship | How instructors are linked to courses |
| Special Course-Instructor-Student | Relationship | Special relationship datasource |
| Course-Coordinator | Relationship | How coordinators are linked to courses |
| Definition values | Groups & privileges |
|---|---|
| Definition Type: Multiple secondary subject pairing Subject Source: Courses Secondary Source: Users Pairing Source: Course-Instructor | Students (FO) Instructors (QP, RV, SV) Coordinators (QP, SM) |
Project requirements
Two unique settings are required for a 3-LR project:
- To create the special course-instructor-student 3-level relationship for the project in Members > 3-level relationship.
- To add a Section Title type secondary subject selection question in the Questionnaire > Question list screen such as "Please select your teaching team for courses" where the instructor is not known in your enrollment data. Also check the option Filter the list of available selections based on the 3-level rater relationship which enables Blue to identify and select instructors from your 3-LR datasource.
Configure the project
- In Blue Administration, select Project > New Project
- Select the special course-instructor-student 3-level relationship definition you created as the Definition type
- Enable the Automatic Update checkbox to subject (Courses)
- Leave the Allow secondary subject optional checkbox disabled.
Limitations of making secondary subject optional
Certain limitations can occur when the secondary subject is optional in the project such as:
- Students may only see course questions in the fillout form
- Some student responses may not display in course and export reports
- Invited counts may be incorrectly reported in teacher reports
As a workaround, a dummy secondary subject (e.g. instructor) can be created for courses without secondary subjects to avoid the examples above.
- Enable Automatic Update for all groups except organizational groups (grayed out) as these will automatically be updated
- Set Private as your Project Audience and choose Yes as the Confidentiality.
- Select Validate to apply the Definition and if there are no errors, click create project to finalize. The project information window will open in the INFO - CONTENT tab.
- Select which datasource will update automatically.
- Select Yes to make your project confidential.
- Validate and Create Project the project.
- Navigate to Members > 3-level relationship.
- Select the special course-instructor-student 3-LR datasource you created by clicking on Select Datasource.
- The datasource is validated when you update the datasource initially and then each time you save your project.
- The snapshot that is produced displays your most current data, based on the last time the data was synced when clicking on Save Project.
Sample validation log
- Navigate to the Questionnaire tab and select Add question. This question is a mandatory part of this project.
- Select Section Title as the question.
- Under Display Mode, select Secondary Subject (instructor) Selection.
Failure to add this question will result in an "Error: The 3-level relationship filter is not valid.” when publishing the project.
- Enter an appropriate Section Heading.
- Select the instructor group(s) in the Secondary Subject Grouping area to display by entering the instructor roles used in the data. Enter one group title at a time and click Add after each group. For more information, see Set up groups to be evaluated based on their roles (secondary subject (instructor) selection filter).
- Enable the option Filter the list of available selections based on the 3-level relationship.
- Select Add to List.
- Setup the rest of the project as a typical end-of-term course evaluation project and publish it.
Fillout task example
Fillout for preselected instructor
Validation log warnings
The validation log provides a list of the course, instructor, and student updates that are no longer synchronized between Blue and the 3-LR datasources. By reviewing this list regularly and making the updates, you can ensure that your project continues to preselect instructors for the students who have completed course components.
| Subject (S) exists in project | Secondary Subject (SS) exists in project | Student (R) exists in project | 3-Level Relationship file | FO / Reporting Behavior |
|---|---|---|---|---|
| Yes | Yes | Yes | S-SS-R exist | SS evaluated in FO Tasks Instructor report created Course report created |
| Yes | Yes | Yes | R doesn't exist | SS NOT evaluated in FO Task R is NOT counted in Instructor report R is counted and reported in Course report |
| Yes | Yes | Yes | SS doesn't exist | SS is NOT evaluated in FO Task Instructor report is NOT created Course report is created |
| Yes | Optional (No) | Yes | Yes | SS is NOT evaluated in FO Task Instructor report is NOT created Course report is created |
| Yes | Yes | No | Yes | FO is not created |
Instructor report = Individual type broken down by Secondary subject
Course report = Individual type with no breakdown