Integrate STF with p4testgen

I’m trying to use p4testgen to generate test case for my p4-16 program.

p4testgen --target bmv2 --arch v1model --test-backend STF --out-dir tests --max-tests 5 basic_forwarding.p4

And I’m trying to use the python script runner(backend/bmv2/run-bmv2-test.py) to run the .stf test file

python3 ../backends/bmv2/run-bmv2-test.py     -tf /p4c/tests/basic_forwarding_1.stf -ll DEBUG -b  /p4c/p4c     /p4c/basic_forwarding.p4

But the test failed without any error msg.

INFO: Executing command: /p4c/p4c/build/p4c-bm2-ss -o /p4c/p4c/build/tmpi53jeth0/tmp1szqcs_m/basic_forwarding.json /p4c/basic_forwarding.p4
INFO: Running model
Calling target program-options parser
Adding interface pcap0 as port 0 (files)
ERROR: Test failed

And i found that the packet input file (pcap0_in.pcap) and packet output file (pcap0_out.pcap) are both empty.

root@9b86482398d4:/p4c/p4c/build/tmpkw6vm2s8/tmpg07io_a_# ll
total 28
drwx------ 2 root root  4096 Nov 21 14:44 ./
drwxr-xr-x 3 root root  4096 Nov 21 14:26 ../
-rw-r--r-- 1 root root 13608 Nov 21 14:25 basic_forwarding.json
prw-r--r-- 1 root root     0 Nov 21 14:25 pcap0_in.pcap|
-rw-r--r-- 1 root root     0 Nov 21 14:26 pcap0_out.pcap
-rw-r--r-- 1 root root   493 Nov 21 14:26 switch.log.txt

I’m using the main branch(commit b652e51) of p4lang/p4c to build the docker image and run the above test. And i have fixed some dependency issues of the run-bmv2-test.py and finally ran the test. But now, I’m not sure whether it’s still having problems in my environment or just some bugs in my p4 program.

The basic_forwarding.p4 look like this:

#include <core.p4>
#include <v1model.p4>

const bit<16> TYPE_IPV4 = 0x800;

typedef bit<9>  egressSpec_t;
typedef bit<48> macAddr_t;
typedef bit<32> ip4Addr_t;

header ethernet_t {
    macAddr_t dstAddr;
    macAddr_t srcAddr;
    bit<16>   etherType;
}

header ipv4_t {
    bit<4>    version;
    bit<4>    ihl;
    bit<8>    diffserv;
    bit<16>   totalLen;
    bit<16>   identification;
    bit<3>    flags;
    bit<13>   fragOffset;
    bit<8>    ttl;
    bit<8>    protocol;
    bit<16>   hdrChecksum;
    ip4Addr_t srcAddr;
    ip4Addr_t dstAddr;
}

struct metadata {
}

struct headers {
    ethernet_t   ethernet;
    ipv4_t       ipv4;
}

parser MyParser(packet_in packet,
                out headers hdr,
                inout metadata meta,
                inout standard_metadata_t standard_metadata) {

    state start {
        transition parse_ethernet;
    }

    state parse_ethernet {
        packet.extract(hdr.ethernet);
        transition select(hdr.ethernet.etherType) {
            TYPE_IPV4: parse_ipv4;
            default: accept;
        }
    }

    state parse_ipv4 {
        packet.extract(hdr.ipv4);
        transition accept;
    }

}

control MyVerifyChecksum(inout headers hdr, inout metadata meta) {
    apply {  }
}

control MyIngress(inout headers hdr,
                  inout metadata meta,
                  inout standard_metadata_t standard_metadata) {

    action drop() {
        mark_to_drop(standard_metadata);
    }

    action ipv4_forward(macAddr_t dstAddr, egressSpec_t port) {
        standard_metadata.egress_spec = port;
        hdr.ethernet.srcAddr = hdr.ethernet.dstAddr;
        hdr.ethernet.dstAddr = dstAddr;
        hdr.ipv4.ttl = hdr.ipv4.ttl - 1;
    }

    table ipv4_lpm {
        key = {
            hdr.ipv4.dstAddr: lpm;
        }
        actions = {
            ipv4_forward;
            drop;
            NoAction;
        }
        size = 1024;
        default_action = drop();
    }

    apply {
        if (hdr.ipv4.isValid()) {
            ipv4_lpm.apply();
        }
    }
}

control MyEgress(inout headers hdr,
                 inout metadata meta,
                 inout standard_metadata_t standard_metadata) {
    apply {  }
}

control MyComputeChecksum(inout headers  hdr, inout metadata meta) {
     apply {
	update_checksum(
	    hdr.ipv4.isValid(),
            { hdr.ipv4.version,
	      hdr.ipv4.ihl,
              hdr.ipv4.diffserv,
              hdr.ipv4.totalLen,
              hdr.ipv4.identification,
              hdr.ipv4.flags,
              hdr.ipv4.fragOffset,
              hdr.ipv4.ttl,
              hdr.ipv4.protocol,
              hdr.ipv4.srcAddr,
              hdr.ipv4.dstAddr },
            hdr.ipv4.hdrChecksum,
            HashAlgorithm.csum16);
    }
}

control MyDeparser(packet_out packet, in headers hdr) {
    apply {
        packet.emit(hdr.ethernet);
        packet.emit(hdr.ipv4);
    }
}

V1Switch(
MyParser(),
MyVerifyChecksum(),
MyIngress(),
MyEgress(),
MyComputeChecksum(),
MyDeparser()
) main;

In a word, is there anything i miss that are crucial to run the test generated by p4testgen?

I can not run BMv2 right now, but I can confirm that p4testgen produces the correct packets and that the pcaps are not empty. It is a little odd that you see so little logging.

The `run-bmv2-test.py` is unfortunately very brittle, it definitely needs an overhaul.

The dependency issues you should be able to fix with uv sync and then sudo -E env PATH=“$PATH” uv run …

Thanks a lot for your quick reply. I’ve figure it out:

Problems:

  1. I’m using docker to run the test. And I didn’t run docker with –privileged, so when the simple_switch started, the run-bmv2-test.py could not connect to the switch, and I guessed it might be that the simple_switch could not run correctly without –privileged.
  2. After I build the image with Dockerfile myself, I was using the IMAGE_TYPE=build, and that will remove the /build files. Although I build it manually when i attach to the container, there may be some problems, which makes it hard to run the run-bmv2-test.py.

Solution:

I just build the image with IMAGE_TYPE=test, and it will keep all the /build files, which are needed to run the run-bmv2-test.py, e.g. the config.h in /build. And I run the docker with –privileged which makes sure the simple_switch has enough privileges to start properly.

Result:

I get the expected result for my simple test cast as follows:

root@8746a5d28ab2:/p4c/build# uv run ../backends/bmv2/run-bmv2-test.py -tf /data/tests/forwarding_2.stf -ll DEBUG  /p4c /data/forwarding.p4 
INFO: Using test file /data/tests/forwarding_2.stf
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit
    msg = self.format(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 929, in format
    return fmt.format(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 668, in format
    record.message = record.getMessage()
  File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "../backends/bmv2/run-bmv2-test.py", line 403, in <module>
    test_result = run_test(test_options)
  File "../backends/bmv2/run-bmv2-test.py", line 294, in run_test
    result = process_file(options)
  File "../backends/bmv2/run-bmv2-test.py", line 243, in process_file
    testutils.log.debug("Writing temporary files into ", tmpdir)
Message: 'Writing temporary files into '
Arguments: (PosixPath('/p4c/build/tmpjqieifif/tmpbgl9nrrx'),)
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit
    msg = self.format(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 929, in format
    return fmt.format(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 668, in format
    record.message = record.getMessage()
  File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "../backends/bmv2/run-bmv2-test.py", line 403, in <module>
    test_result = run_test(test_options)
  File "../backends/bmv2/run-bmv2-test.py", line 294, in run_test
    result = process_file(options)
  File "../backends/bmv2/run-bmv2-test.py", line 243, in process_file
    testutils.log.debug("Writing temporary files into ", tmpdir)
Message: 'Writing temporary files into '
Arguments: (PosixPath('/p4c/build/tmpjqieifif/tmpbgl9nrrx'),)
INFO: Executing command: /p4c/build/p4c-bm2-ss -o /p4c/build/tmpjqieifif/tmpbgl9nrrx/forwarding.json /data/forwarding.p4
INFO: Running model
Calling target program-options parser
Adding interface pcap0 as port 0 (files)
INFO: Running simple_switch_CLI --thrift-port 9393
INFO: STF Command: ('add', 'MyIngress.ipv4_lpm', '1', [('hdr.ipv4.dstAddr', '0b00000000111111111111111111111111')], ('MyIngress.ipv4_forward', [('dstAddr', '0x000000000000'), ('port', '0x000')]), None)
INFO: Sending 'table_add MyIngress.ipv4_lpm MyIngress.ipv4_forward 0b00000000111111111111111111111111/32 => 0x000000000000 0x000'
INFO: STF Command: ('packet', '0', '00000000000000000000000008000000000000000000000000000000000000FFFFFF000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000')
Obtaining JSON from switch...
Done
Control utility for runtime P4 table manipulation
RuntimeCmd: Adding entry to lpm match table MyIngress.ipv4_lpm
match key:           LPM-00:ff:ff:ff/32
action:              MyIngress.ipv4_forward
runtime data:        00:00:00:00:00:00	00:00
Entry has been added with handle 0
RuntimeCmd: INFO: STF Command: ('expect', '0', '00000000000000000000000008000000000000000000FF0000000000000000FFFFFF000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$')

INFO: simple_switch: exit code -15
INFO: Execution completed
INFO: Comparing outputs
INFO: Checking file /p4c/build/tmpjqieifif/tmpbgl9nrrx/pcap0_out.pcap
INFO: All went well.
INFO: Removing temporary test directory.
1 Like

Great to hear. Definitely an indication that the output of this test utility should be improved.