Significance Testing

Navigation

Create a new survey or open a saved survey from the My Surveys page.  The Design tab opens.  Click the Analyze tab.  The My Reports page opens.  Select the Cross-tab Analysis report from the Other Reports section.  The report editing dialog opens.  Click the Properties tab.

Use

Use the significance testing to compare whether the proportion of respondents in one column is significantly different from the proportion in the other column within the single variable.  You can use different confidence intervals for the significance testing.

Description

In the 6.1 release, you can perform the significance testing on the “Choose One” (including the “Matrix” sides) variables using the Cross-tab Analysis report.  You can use the significance testing to compare whether the proportion of respondents in one column is significantly different from the proportion in the other column within the single variable.  To manage the significance testing options, access the Properties tab in your Cross-tab Analysis report.  See image below.

 

image002.png

The Properties tab

 

Note:  For the steps on how to create the Cross-tab Analysis report, see the “Cross-tab Analysis Report” topic.

When you perform the significance testing, the reference letters show as the column ID in the column headers below the column titles.  For the columns 1-26, lowercase English letters are shown (a, b, c, …, y, z), and, for the columns 27-52, uppercase English letters are shown (A, B, C, …, Y, Z).  If the difference between the columns is significant, the ID of one column shows under the response data of the other column.  See image and example below.

 

image004.png

The Cross-tab Analysis report with the significance testing

 

As the report contains four columns in this example, the letters “a,” “b,” “c,” and “d” show up in the column headers.  The result of significance testing is that the respondents who have selected the “Chicago” choice are significantly different from those who have selected the “New York” choice.  Thus, the “b” letter shows in the cell in the “Chicago” row and “a” column, and the “a” letter shows in the cell in the “New York” row and “b” column.

 

If the number of degrees of freedom (responses) is less than 30, no significance testing is calculated and published, and a footnote on the table states “Base too small for significance testing between columns [a-b, a-c, b-c].”  If the number of degrees of freedom is less than 100, the significance testing is calculated and published, and a footnote on the table states “Caution – small bases for columns [c-d, c-e, c-f].”  Depending on the selected confidence interval, a footnote showing the chosen risk level appears on the table.  For example, if you select the 95% confidence interval, a footnote states “Proportions: Columns Tested (5% risk) [a-c, b-c, a-c].”

 

The significance testing is not saved if you close the report.  When you open the saved report later, the previous test results are cleared.  You can perform a new test if you want.

 

You can also export the significance testing results to the Microsoft Excel file format as a part of the Cross-tab Analysis report to view the test data outside the Vovici product.

Significance Testing Calculation

Vovici uses the same significance test calculations as described in Quantum’s Special T statistics (Quantum volume 3, Section 7) for the T-test on column proportions.  This example compares the values of the proportions in two columns of a table. For each of the two columns (k=1, 2), the sample proportions are defined as:

 

P1 = Cell frequency 1 / Base 1

P2 = Cell frequency 2 / Base 2

 

Population variance is calculated as:

P = (Cell frequency 1 + Cell frequency 2) / (Base 1 + Base 2)

 

The variable T is calculated as:

T = (P1 – P2) / Sqrt (p x (1-p) x (1 / Base 1 + 1 / Base 2) / (1 – 1 / (Base 1 + Base 2)))

Steps

How to Perform a Significance Testing

1.    Create a Cross-tab Analysis report from the My Reports page.  The Name Your Report dialog opens.  See image below.

 

image006.png

The Name Your Report dialog

 

2.    Type the name of a report in the “Report Name” field.

3.    Click OK.  The dialog opens where you can define your Cross-tab Analysis report properties.

4.    On the Properties tab, select the “On” option from the “Significance Testing” drop-down list.

5.    Select the confidence interval for significance testing from the “Confidence Interval” drop-down list.

6.    Click OK.  The report is generated with the significance testing.

How to Disable a Significance Testing for a Workgroup

1.    Navigate to the Administration tab.

2.    In the left pane, click the General Setup subtab.

3.    Select the “Disable Significance Testing in Cross-tab Analysis” check box.  See image below.

 

image008.png

The General Setup subtab

 

4.    Click Save Setup.

See Also

Other Reports

Cross-tab Analysis Report


Want to provide your feedback on the Vovici Online Help System?  Click here to take a short survey now to help us make improvements.  You also have the option to provide feedback on this particular topic below.