# c-testsuite

This project aims to be a collaborative database of C compiler test cases,
minimal test runners, and public test results. The general idea is that
the various test suites here adhere to well defined, simple interfaces and projects can contribute
tests back that match those interfaces, or use those tests by implementing runner scripts matching
the specifications.

There are many tools that may benefit such as C compilers, transpilers, interpreters and emulators, so we should seek to agree on simple easy test interfaces, and implement
those interfaces for a variety of tools.

results are published daily to https://c-testsuite.github.io/


# Test suites

The top level test-suite runners output https://testanything.org output.

## single-exec suite

entry point is ```./single-exec```

### runners/single-exec/*

The runner will be invoked as:

```
$ ./runners/single-exec/$NAME test/single-exec/case.c
```

The runner is free to output any data it wants, but must return
nonzero on failure.

The runner will be considered a failure if it takes more than 5 minutes.

### tests/single-exec/*

- Single .c file tests.
- 'main' is the entry point.
- The file $t.c.expected must match stdout+stderr of
  the test.
- The test programs exit with 0 on success.

C standard, Portability, preprocessor and libc requirements
are specified via tags that can be filtered against using
search queries, that can generate skip lists.

### Example:

```$ ./single-exec gcc-x86_64 ```


# Skipping tests

Try to skip a test if your compiler platform can NEVER pass it, the test is not appropriate.
In that case, there is only one mechanism, add a command that prints a list of tests to skip
on stdout named:

```
./runners/*/$TOOL.skip
```

# Search and query

All tests have a matching $t.tags file. This file specifies attributes
of the test that can be filtered and queried.

The query language is based off of https://github.com/oniony/TMSU tags.

The query language grammar is shown here:

https://github.com/oniony/TMSU/blob/master/misc/ebnf/query.ebnf

Support tags are currently

```
suite={single-exec, ...}
portable
	The test should be portable C.
arch-x86_64
	The test should pass on x86_64
c89
c99
c11
needs-cpp
    Test relies on the preprocessor
needs-libc
    Test relies on libc
```

Implicit tags:

c89 implies c99 and c11
c99 implies c11

example query:
```
$ ./scripts/make-search-index
$ ./scripts/search-tests "c99 suite=single-exec (portable or arch-amd64)"
```

These queries can be used to generate skip lists.

# otags files

otag files are intended to allow
a tests origin to be discovered. They contain the following fields.

```
org=$DOMAINNAME
repository=$SRCURL
version=$UNIQUE_VERSION
path=$TEST_PATH_IN_REPOSITORY
```

# Dependencies

Running tests

- posix sh
- python3
- coreutils
- tool under test

Querying tests:

Currently test search is based on

https://github.com/oniony/TMSU

We are sympathetic to those who do not wish to deal with
installing a lesser known third party tool, so will think of
ways to ease the burden in the future.

# Tips

## Naming test cases

The names are not stable for now, so if you 
refer to a test case in your issue tracker, it is best to 
name it something like ```c-testsuite/$CTESTGITCOMMIT/path/to/test```.

## Getting a summary from the command line

./single-exec $runner | ./scripts/tapsummary | head

## The full TAP test suites report can be viewed or 'curl'ed

For example:

- https://c-testsuite.github.io/gcc-x86_64_report.html
- https://c-testsuite.github.io/gcc-x86_64-single-exec_report.tap
- https://c-testsuite.github.io/gcc-x86_64-single-exec_report.tap.txt