IMP77 Confidence Test Suite for the Advanced Personal Machine ============================================================== Introduction ------------ The object of this test suite is to provide a collection of IMP77 programs that will test a compiler for the language to such a degree where one can be confident of the "correctness" of the compiler. In no way should this suite be regarded as a comprehensive and exhaustive compiler validation suite. The main use of this suite is to enable IMP77 compiler writers and testers to check that changes made to an existing compiler have not inadvertently affected previously working parts of the compiler. It should be noted that all of the programs are expected to compile successfully and execute. The output from the programs generally has a title which says the type of test the program is about to do, and is then followed by a column of passes (or FAILs) to indicate the tests which were passed. To narrow down a problem exactly, the tester may have to edit the original programs to add more detailed tests of his own. It is recommended that any changes should be edited into a new file, leaving the original intact. In all the programs, except the first two, you have a choice of where the output is to go, that is to say either no output, output to a file, or output to the screen. How to select one of these options is explained later. All the programs, except the second program, are self-testing, that is to say that all you need to do, for each program, is to check that the number of failures is 0. Because the second program tests the output routines it is not possible to make it self-testing (How can the computer see what is on the VDU ?). The first program has no output, and thus does not need a choice of output. This particular collection of programs has been developed for use within the Advanced Personal Machine environment, with the IMP2.? and IMP3.? version of the compiler being the current test host. Some of the programs in the suite use APM-specific conventions e.g. filenames. Such programs have been identified in the descriptions below. Generally the syntax of the programs follow that published in the "EUCSD Internal Report on the IMP77 Language" except for local changes e.g. in formal array parameter names. These differences are detailed in the IMP help information available on the APM. All these programs reside in the directory Bravo::I.Test. The general philosophy behind the design of this suite is to assume nothing and go on to check the IMP facilities going from simple to advanced. Although the individual programs in the suite have been designed to test independent aspects of the language, there must be a degree of interaction. To minimise this effect the suite programs must be tested in a certain order. This ordering is described later. Output ------ In all but the first two test programs you have a choice of where you want the output of the program to go, shown by the table below. The number of failures which occurred in a program are alway written to the screen, at the end. Where the output goes to depends on the parameter after the name used to run the program. Parameter | Result ------------|------------- No parameter | No output "s" | The screen "j" | "journal" anything else | file name entered e.g. typing "strings j" will send the output of the file strings to the file "journal", while "strings" will give you no output (except the no of failures, of course). Thus, ideally, the first time you run the test program you do it with no output, and then if there is any failures you run the program with output going to the screen or a file to find out the place it failed. You can then look and the program to find the statement it failed on. Running ------- The whole test suite can be compiled and then run by using the command files C.com and X.com respectively. Thus typing "@C" will compile the programs and typing "@X" will run them (with no output). Suite Programs -------------- 1. Filename: MINIMAL ---------------------- This is the simplest possible program. It checks that the IMP comment delimiters work for all cases. There are "commented-out" statements in the program which should be ignored by the compiler. Essentially the program consists of a simple %Begin...%EndofProgram block. The program should do nothing (gracefully). 2. Filename: OUTPUT --------------------- Before any other program can be executed, we must be sure that the output primitives do actually work correctly. The tester must know what output the program is going to produce before he can say that the output routines work correctly. In this program the output consists of a number of lines that you must compared for identity. The program produces explanatory messages to help in this test. The program tests the output facilities for %Integer, %String and Characters. In addition, the following permanent procedures are tested: PRINT SYMBOL, PRINT STRING, SPACE, SPACES, NL, NEW LINE, WRITE, OPEN OUTPUT, SELECT OUTPUT, CLOSE OUTPUT. In testing the OPEN, SELECT and CLOSE routines, the program opens and writes to three temporary files: $One, $Two, $Three. These files should be inspected to make sure that they were correctly written to. Note: This program uses APM-specific filenaming conventions. NB: THIS PROGRAM IS NOT SELF-TESTING 3. Filename: INPUT -------------------- This program tests the simple input facilities for %Integer and %String. As part of this test, the program opens and reads data held in two files which should not be changed, or the test program won't work. These files are :- CHAR.DAT and READ.DAT. In addition, the program also tests the permanent procedures: OPEN INPUT, SELECT INPUT, NEXT SYMBOL, SKIP SYMBOL, READ SYMBOL, CLOSE INPUT, READ Note: This program uses APM-specific filenaming conventions. Integer Arithmetic ------------------ This suite has test programs for all types of integer arithmetic, i.e. %integer, %byte %integer, %mite %integer, %short %integer, and %half %integer. The relevant test files are similar, except for type being tested, and the file names start with the letter I, B, M, S, and H respectively (the first letter of each word). Thus when integer arithmetic is mentioned it means all these different types of it. Integer Arithmetic is, of course, extremely important in the IMP language. In order to test the various aspects of this area, several programs were developed that test independent topics in such a manner as to avoid interaction which may confuse the test. Therefore, addition and subtraction have been kept away from the multiplication, division and exponentiation. Furthermore, these have been kept away from logical operations and precedence evaluation. 4. Filenames: ICOMP, BCOMP, MCOMP, SCOMP, HCOMP ------------------------------------------------- In order to check the arithmetic conditions, these programs will check all six comparison operators (LESS THAN, LESS THAN OR EQUALS TO, GREATER THAN, GREATER THAN OR EQUALS TO, EQUALS TO and NOT EQUALS TO). As usual, a rich combination of argument representations are used: Literal, Constant, Variable and Function. 5. Filenames: IADD, BADD, MADD, SADD, HADD ------------------------------------------- The purpose of these programs is to check the integer addition and subtraction (both unary and binary). In common with most of the tests in the integer package, the operations are mostly binary in character. The tests involve a simple assignment followed by a arithmetic condition to print a PASS or FAIL message. The tests use various combinations and representations of argument. 6. Filenames: IMULT, BMULT, MMULT, SMULT, HMULT ------------------------------------------------ These programs extends the tests on the integer arithmetic to multiplication, division, exponentiation, modulus and the permanent procedure REM. Again, a rich variety of argument combinations and representations is used. 7. Filename: IARITH --------------------- This final program in the integer arithmetic package tests all the remaining aspects. These include the SIZE OF function, precedence of operators and coercion between all precisions of integers i.e. %BYTEs, %MITEs, %SHORTs, %HALFs and normal precision. In addition the initialisation of variables during declaration and the initialisation of normal and multi-character constants is tested. 8. Filename: STRUCT --------------------- This program tests the more complex data structures permitted by IMP, namely, %RECORDs and %ARRAYs and pointers to these various objects. In addition, the permanent procedures: SIZE OF, ADDR, STRING, RECORD and INTEGER are also tested. As regards to records, the program tests assignment to record variables (including initialisation to zero of all fields, whole record copy assignment and specific sub-field assignment). In addition to assignment, the various fields were accessed and the validity of the data (determined by a simple write - read mechanism) established. These tests also checked the %FORMAT statement and variant records. As regards arrays, the program tests simple indexing using explicit and expression-evaluated addresses on both constant and variable arrays. The variable arrays included arrays with upto four dimensions, those with large values subscripts, negative subscripts and bounds declared via constants. The initialisation of constant arrays using repetition counts and (*) was also tested. The arrays themselves were of different base types and include %BYTEs, %INTEGER, %RECORD and %STRING. To further test the accessing of complex structures, the program includes arrays of records and records containing arrays. This level of nesting is continued to a "depth" of three. With all of these structures, pointer variables of the relevant types were also included so as to test the accessing of structured objects via this further level of indirection. %NAME variables were included inside records so that linked chains could be constructed and the accessing of data via a long linked chain tested. In some cases, very complex accessing mechanisms are used in order to check the compiler to these pathological limits. The comparison of pointer variables using == and ## is checked. The use of the NIL record is tested. The final tests in the program are devoted to Address Modification both directly and via pointer variables. These tests were conducted on %INTEGER and %RECORD arrays. 9. Filename: LOOP ------------------- This tests the IMP iterative statements: %FOR, %WHILE and %UNTIL together with the associated control transfer statements for %CYCLE ... %REPEAT blocks, namely, %EXIT and %CONTINUE (both conditional and forced). The conditions for these cycles have been designed to make sure that the loops are executed the correct number of times with the given arguments. The effect of nesting the loops is checked together with exiting or continuing. The effect of jumping into a %FOR loop or altering the %FOR variable is not tested as that constitutes an illegal operation. 10. Filename: IF ------------------ This tests the IMP conditional grouping statements: %IF and %UNLESS. To simplify the test, the actual condition itself has been made into a simple arithmetic comparison. For both types of statement, both the long (%START ... %FINISH %ELSE %IF ... %FINISH %ELSE ) and the simplified (... %IF ) have been checked. The program prints out PASS or FAIL messages. 11. Filename: JUMP -------------------- This program tests control transfer statements used for GOTOs. This checks out the -> Statement both in conditional and unconditional modes. It also uses forward branches and tests the declaration of jump points using the %LABEL statement. The IMP case-type statement, %SWITCH is also checked with the switch itself having a large number of vectors with subscripts starting from a non-zero value. Some vectors are undeclared thus testing the (*) variant. The vectors are indexed using an integer expression to add generality to the test. In addition, the vector locations are not assigned sequentially which further tests the switch. 12. Filename: EVENTS ---------------------- This program tests the IMP %Signal and event mechanism. The program starts by checking that events in the range 0 to 9 are raised correctly when an exception arises. The program prints out what event and sub-event it was expecting and if that was not raised it will print out a warning message. The program also checks the ability to raise an event explicitly including raising events with %ON blocks. The assignment of the EVENT record is checked. One difficult test is the ability to raise an event deep within a mutually recursed, nested routine and make sure that the forced return from that depth did not corrupt the stack. 13. Filename: BLOCKS ---------------------- This program tests the block structure and procedure facilities within the IMP77 language. The first test in the program checks that blocks can be nested to depth four and that variables declared within these blocks obey the scoping rules. The next test checks that arrays can have bounds that are computed at run-time so long as they are contained within blocks. The next test is a very important one: it checks the parameter passing mechanism. Both %Name and value parameters are passed to a routine, their values are then altered and upon returning to the main program block, these values are checked to make sure that no errors arose in the passing or alteration. The routine that performs the alteration is given parameters of various types including %INTEGER, %STRING, %RECORD as well as arrays of these base types. Next the %RETURN statement is checked (in both its conditional and forced modes) followed by checks on the %RESULT statement, again in both modes. The ability of functions to take many nested function calls as an argument is checked to make sure that the compiler does not get confused. The next test checks that IMP routines can recurse (both normally and mutually (thus checking the %ROUTINE %SPEC)). Then the %OWN qualifier is tested to make sure that data is preserved between invokations of the routine. The next test on procedures concerns the use of predicates in conditional statements. The final tests check out the ability to map onto local data. 14. Filename: EXTERN ---------------------- NOTE: This program uses external modules MONE, MTWO and MTHREE. These, together with the main program, should all be INSTALLed before execution so as to make their external data and procedures available. This collection of programs and external modules constitute the most severe test to the IMP compiler: linking in to external data and procedures that have been separately compiled. The use of modules tests the %End %of %File variant. The program tests the ability to access and modify external data of various types in different external modules. This includes the initialisation of external arrays, variables and pointers. One demanding test the chaining of external routines: the main program calls a routine in MONE which calls a routine in MTWO which calls a routine in MTHREE. Here two different tests are performed: firstly, the routine in MTHREE calls an external routine in the main program and makes sure that its static variables are still accessible and that they contain the correct values. The second test demands that the routine in MTHREE raises an event. This event must then be trapped in the main program. In addition, all the externals modules contain routines that test the external data that exists in other modules to make sure that nothing is inaccessible or corrupted. As a further test, the linkage mechanism that calls the first routine in MONE uses an %Alias. After testing external data, the program then checks that it can access external procedures. It uses external functions and predicates to test this. The final tests are concerned with external %Name variables and their ability to point across module boundaries and be accessed and modified from various modules (including the main program). 15. Filename: LOGIC --------------------- In some respects this program is an extension of the integer arithmetic tests. The purpose of the program is to test the bit vector facilities that use logical operators. In IMP77 the bit vectors are 32 bits long. The program tests all the logical operators: AND (&), Inclusive OR (!), Exclusive OR (!!), Negation (\), Right Shift (>>) and Left Shift (<<) In addition, the program tests the precedence existing between these operators. The effect of combining logical operations with integer arithmetic is not tested since the effect is implementation dependent. 16. Filename: FARITH ---------------------- In a similar manner to the testing of the integer arithmetic, the floating point arithmetic is also tested. However, it was felt that the splitting up of such a program was not warranted as in the integer case. This was because all floating point arithmetic is performed by software emulation. This program contains the following tests: (i) Assignment and initialisation of %REAL constants and variables. (ii) Comparison between %REAL arguments using the six comparison operators (> < = # <= >=). In later tests, the equality between %REAL values is determined by the use of predicate REAL NEAR that returns TRUE if the absolute difference in its arguments is less than or equal to a small value (taken to be 0.0001). Various combinations and representations of arguments were used (Constants, Variables, Literals, Function). (iii) Floating-point arithmetic was tested: addition, subtraction, division, multiplication, exponentiation and modulus. These were tested for a rich combination of argument representations as discussed above. (iv) The permanent procedures: INT PT, FRAC PT, INT and SIZE OF were tested for floating-point arguments. 17. Filename: STRINGS ----------------------- This program tests the IMP %String facilities. In testing, various combinations of %String representations are used: Literals, Constants, Functions and Variables. Firstly, the comparison operators are tested with these string arguments. All six operators are tested bearing in mind that the comparisons are performed on a strictly lexicographic basis using the ASCII character set. Next, the string permanent procedures are tested: CHAR NO, TO STRING, SUB STRING, LENGTH. Again, these functions are tested using the various string representations as described above. The next test involves string concatenation, including concatenations involving the NULL string. This is followed by testing of string assignment operations, both simple and Jam Transfer. (NOTE: at present the Jam Transfer operator works as a simple assignment with no overflow checking performed). These tests are combined with testing of various combinations of string expression designed for their complexity and probing of various features. The program finishes by testing the String Resolution facility. For these tests the resolution acting both as a condition and a normal assignment is used. Various representations and combinations of arguments are used to fully test the resolution performance. 18. Filename: ARRAYS ---------------------- This program tests all types of array constructs. It tests integer arrays, string arrays, record arrays,multi-dimensional arrays, own arrays, array names, constant bounds, variable bounds, conformant bounds, variable upper bound, arrays as parameters for procedures, etc. 19. Filename: MISC -------------------- This program checks some extra string operations which were not tested before, such as passing a string array as a parameter to a routine which has the length of the string as * (any length). It also checks that long concatenations and resolutions work correctly.