[4.1] How to use and modify the core-tests platform

Ask questions or leave feedback about release 4.x here.
Post Reply
jvanstraten
Posts: 6
Joined: Mon Feb 20, 2017 3:44 pm

[4.1] How to use and modify the core-tests platform

Post by jvanstraten » Fri Feb 24, 2017 10:43 am

The core-tests platform is the conformance test suite for the core itself, excluding the cache. It's made such that most tests are also not dependent on the compiler or assembler, so if you want to change the instruction set, this is your only option to test your changes without modifying the compiler toolchain first.

Platform usage

Looking at the core-tests platform directory, there's four directories. Three of them contain the simulation files for three different core configurations, while the share folder contains shared files. There's also a makefile that chains to the makefiles within the configuration-specific directories. The makefile will autodetect newly created configuration folders, so you don't need to modify it if you want to add one.

If you only make a minor change to the VHDL code of the core, running "make conformance" in the root folder of the platform is fine. Note that the Modelsim and either the ISE or Vivado environment variables must be set for the test to work (the latter is needed for the unisim library as some files refer to it, although it should be trivial to remove the references, as the core by itself does not actually depend on it). You might want to add "-j" to the command line, which will cause the simulations for the different configurations to run in parallel. The command will list all the succeeded and failed tests only when the entire simulations complete, so it will take a minute or so before you get any output.

If you encounter failed tests, you will obviously want more information. To get it, enter one of the configuration directories with failing tests and run "make vsim". This will run the conformance test for that configuration again, but now with the Modelsim GUI open. Before automatically starting the simulation, the do file will add a couple signals to the wave window:
  • sim_currentTest: string containing the name of the currently running test case.
  • sim_rvexState: array of strings containing the state of the ρ-VEX. The timing of the signal corresponds to the S_LAST pipeline stage.
  • sim_complete: signal that indicates whether the current test case is complete or not.
  • sim_failure: signal that indicates whether the current test case has failed or has succeeded/was ignored. X is used to indicate failure simply so it stands out.
To be able to debug a test case, you'll also want to have the source code of the test case open. Look for the associated "Running test case file <path>..." line in the transcript to find the location of this file.

Test case file structure

A test case file contains a number of commands that are interpreted by the VHDL test runner. The test runner is located in core-tests/share/core_tb.vhd. Note, by the way, that this is a generated file; if you for some reason need to change something here, you should change the core_tb.template.vhd file instead, and then run the makefile in the config directory in the root directory of the release. Anyway, the file starts out with a command reference in comments, so I won't go into detail about the command syntax here, but the structure of a test case file is as follows.
  1. Header: name and config commands. The name command should be obvious -- it is used for the sim_currentTest signal and the conformance test output, and that's about it. The config commands are less obvious. They don't actually change the configuration in which the core is running (this cannot be done without re-elaborating the VHDL code) but instead impose requirements on the core configuration. If the actual core configuration does not match what the config commands require, the test case is skipped/aborted. This prevents it from showing up in the conformance test output. Basically, this is a safeguard to prevent false test failures when the core configuration is changed.
  2. Memory initialization. The test runner has separate instruction and data memories, each occupying the full 4 GiB address range of the core. Memory is allocated by the simulator only when pages are accessed to save host memory. Before anything else, a test case should use the "init" command to reset these memories. The instruction memory is initialized with "load*" commands in an auto-increment fashion; the "at" command can be used to set the write pointer. The "load" command allows syllables to be loaded using a simplified assembly language. The assembly is interpreted by the VHDL test runner based on the instruction set description in the config directory, so it should always match what the core expects. The assembly language does not support labels, so you'll have to manually compute branch offsets, registers don't have the $ prefix, and the ;; marker must be at the end of the line instead of after a newline, but otherwise the syntax should be the same as that used by binutils. The data memory can be initialized with the "write dmem" command if necessary.
  3. Core reset. Before starting the test, the core should be reset using the "reset" command. You can also do this during the test if you want to restart it.
  4. Timed commands. These commands check whether the core is running as it should ("wait", "read"), access the debug bus of the core ("write dbg", "read dbg"), or modify the state of the run control pins ("rctrl"). The most common way in which proper execution is tested is by monitoring the memory bus of the core using the "wait" command. The command can for instance be used to trigger failure if the core accesses a wrong address, writes an unexpected value, or doesn't access memory at all within a specified amount of cycles. Be careful with the timeouts though; a seeded pseudorandom generator generates stalls and disables the clock to the core randomly (in order to test sane stalling behavior) without slowing down the timeout counter. You should always use a significantly longer timeout than you think the core should need: the only adverse effect of this should be slightly longer simulation times upon failure.
Test case files may also include other files using the "inc" command. The semantics are the same as a C #include command; it behaves as if the contents of the included file were copypasted to the location of the "inc" command. This is useful for running computation kernels testing some set of instructions with different inputs, without having to rely on the proper functioning of branches and calls or copypasting code around.

Adding test cases

The easiest way to add a test case is to copy and modify a test case file that is somewhat similar. If you read through the section above and use the command reference at the top of the share/core_tb.vhd file you should be able to figure it out.

Once you've created the file, you'll need to register it in the *.suite files. *.suite files are simply lists of *.test and other *.suite files that the VHDL test case runner will iterate through. So all you should need to do is add the filename to the appropriate suite file. The suite files are organized with an index.suite file in each directory listing the other files in the directory, but the test case runner doesn't care about this.

Adding a core configuration

In the release, we only test the following core configurations:
  • The "default" dynamic core: 8-way with 4 lane groups and contexts, variable-length instructions with stop bits allowed every 2 syllables, and long-immediate borrowing only from the other syllable in an aligned pair (8way-4grp-4ctxt-2ba-limmh-neigh).
  • An older dynamic core without variable-length instructions but with more long-immediate borrow slots (8way-4grp-4ctxt-8ba-limmh-all), primarily to test if all the long-immediate forwarding logic works properly.
  • A small static 2-way core (2way-1grp-1ctxt-2ba-limmh-neigh).
Each of these configurations has forwarding enabled. If you want to add a configuration to the conformance test, simply copy one of the directories and modify the core_cfg.vhd file in it. The main makefile in the core-tests platform will auto-detect this, so you don't have to change it. The filename of the directory is not used by the scripts in any way; the core_cfg.vhd file is what matters.


If you have questions, feel free to ask them in this thread or to create a new thread.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest