The host ping across switches is unreachable

Getting Started with P4
1

Hello.
I build a 4-fat-tree topology by network.py, and configure the switch flow table file in txt.
The topology is below:

4_fat_tree
4_fat_tree1080×474 78.6 KB

h1 can ping h2, but h1 ping h3 is unreachable.

mininet> pingall
*** Ping: testing ping reachability
h1 -> h2 X X X X X X X X X X X X X X 
h2 -> h1 X X X X X X X X X X X X X X 
h3 -> X X h4 X X X X X X X X X X X X 
h4 -> X X h3 X X X X X X X X X X X X 
h5 -> X X X X h6 X X X X X X X X X X 
h6 -> X X X X h5 X X X X X X X X X X 
h7 -> X X X X X X h8 X X X X X X X X 
h8 -> X X X X X X h7 X X X X X X X X 
h9 -> X X X X X X X X h10 X X X X X X 
h10 -> X X X X X X X X h9 X X X X X X 
h11 -> X X X X X X X X X X h12 X X X X 
h12 -> X X X X X X X X X X h11 X X X X 
h13 -> X X X X X X X X X X X X h14 X X 
h14 -> X X X X X X X X X X X X h13 X X 
h15 -> X X X X X X X X X X X X X X h16 
h16 -> X X X X X X X X X X X X X X h15 
*** Results: 93% dropped (16/240 received)

The links are as below:

mininet> links
a1-eth1<->e1-eth1 (OK OK) 
a1-eth2<->e2-eth1 (OK OK) 
a2-eth1<->e1-eth2 (OK OK) 
a2-eth2<->e2-eth2 (OK OK) 
a3-eth1<->e3-eth1 (OK OK) 
a3-eth2<->e4-eth1 (OK OK) 
a4-eth1<->e3-eth2 (OK OK) 
a4-eth2<->e4-eth2 (OK OK) 
a5-eth1<->e5-eth1 (OK OK) 
a5-eth2<->e6-eth1 (OK OK) 
a6-eth1<->e5-eth2 (OK OK) 
a6-eth2<->e6-eth2 (OK OK) 
a7-eth1<->e7-eth1 (OK OK) 
a7-eth2<->e8-eth1 (OK OK) 
a8-eth1<->e7-eth2 (OK OK) 
a8-eth2<->e8-eth2 (OK OK) 
c1-eth1<->a1-eth3 (OK OK) 
c1-eth2<->a3-eth3 (OK OK) 
c1-eth3<->a5-eth3 (OK OK) 
c1-eth4<->a7-eth3 (OK OK) 
c2-eth1<->a1-eth4 (OK OK) 
c2-eth2<->a3-eth4 (OK OK) 
c2-eth3<->a5-eth4 (OK OK) 
c2-eth4<->a7-eth4 (OK OK) 
c3-eth1<->a2-eth3 (OK OK) 
c3-eth2<->a4-eth3 (OK OK) 
c3-eth3<->a6-eth3 (OK OK) 
c3-eth4<->a8-eth3 (OK OK) 
c4-eth1<->a2-eth4 (OK OK) 
c4-eth2<->a4-eth4 (OK OK) 
c4-eth3<->a6-eth4 (OK OK) 
c4-eth4<->a8-eth4 (OK OK) 
e1-eth3<->h1-eth0 (OK OK) 
e1-eth4<->h2-eth0 (OK OK) 
e2-eth3<->h3-eth0 (OK OK) 
e2-eth4<->h4-eth0 (OK OK) 
e3-eth3<->h5-eth0 (OK OK) 
e3-eth4<->h6-eth0 (OK OK) 
e4-eth3<->h7-eth0 (OK OK) 
e4-eth4<->h8-eth0 (OK OK) 
e5-eth3<->h9-eth0 (OK OK) 
e5-eth4<->h10-eth0 (OK OK) 
e6-eth3<->h11-eth0 (OK OK) 
e6-eth4<->h12-eth0 (OK OK) 
e7-eth3<->h13-eth0 (OK OK) 
e7-eth4<->h14-eth0 (OK OK) 
e8-eth3<->h15-eth0 (OK OK) 
e8-eth4<->h16-eth0 (OK OK)

e1-commands:

table_set_default ipv4_lpm drop
table_add ipv4_lpm ipv4_forward 10.0.1.1/32 => 00:00:00:01:00:01 3
table_add ipv4_lpm ipv4_forward 10.0.1.2/32 => 00:00:00:01:00:02 4
table_add ipv4_lpm ipv4_forward 10.0.2.0/24 => 00:02:01:00:01:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:02:00:01:00 2

e2-commands:

table_set_default ipv4_lpm drop
table_add ipv4_lpm ipv4_forward 10.0.2.1/32 => 00:00:00:02:00:01 3
table_add ipv4_lpm ipv4_forward 10.0.2.2/32 => 00:00:00:02:00:02 4
table_add ipv4_lpm ipv4_forward 10.0.1.0/24 => 00:02:01:00:02:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:02:00:02:00 2

a1-commands:

table_set_default ipv4_lpm drop
table_add ipv4_lpm ipv4_forward 10.0.1.0/24 => 00:03:01:00:01:00 1
table_add ipv4_lpm ipv4_forward 10.0.2.0/24 => 00:03:01:00:02:00 2
table_add ipv4_lpm ipv4_forward 10.1.0.0/16 => 00:01:01:00:00:01 3
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:01:02:00:00:01 4

Is there something wrong?

2

What P4 program are you loading on the switches?

I suspect that if you are only installing table entries on 3 of the switches, and not the others, it is likely that the other switches are experiencing a miss on table lookups, and if their default action is drop, then they are dropping most are all packets they receive.

3

Thanks!
All the switches use the same P4 program as follows:

/* -*- P4_16 -*- */
#include <core.p4>
#include <v1model.p4>

const bit<16> TYPE_IPV4 = 0x800;

/*************************************************************************
*********************** H E A D E R S  ***********************************
*************************************************************************/

typedef bit<9>  egressSpec_t;
typedef bit<48> macAddr_t;
typedef bit<32> ip4Addr_t;

header ethernet_t {
    macAddr_t dstAddr;
    macAddr_t srcAddr;
    bit<16>   etherType;
}

header ipv4_t {
    bit<4>    version;
    bit<4>    ihl;
    bit<6>    dscp;
    bit<2>    ecn;
    bit<16>   totalLen;
    bit<16>   identification;
    bit<3>    flags;
    bit<13>   fragOffset;
    bit<8>    ttl;
    bit<8>    protocol;
    bit<16>   hdrChecksum;
    ip4Addr_t srcAddr;
    ip4Addr_t dstAddr;
}

struct metadata {
    /* empty */
}

struct headers {
    ethernet_t   ethernet;
    ipv4_t       ipv4;
}

/*************************************************************************
*********************** P A R S E R  ***********************************
*************************************************************************/

parser MyParser(packet_in packet,
                out headers hdr,
                inout metadata meta,
                inout standard_metadata_t standard_metadata) {

    state start {

        packet.extract(hdr.ethernet);
        transition select(hdr.ethernet.etherType){

            TYPE_IPV4: ipv4;
            default: accept;

        }

    }

    state ipv4 {

        packet.extract(hdr.ipv4);
        transition accept;
    }

}


/*************************************************************************
************   C H E C K S U M    V E R I F I C A T I O N   *************
*************************************************************************/

control MyVerifyChecksum(inout headers hdr, inout metadata meta) {
    apply {  }
}


/*************************************************************************
**************  I N G R E S S   P R O C E S S I N G   *******************
*************************************************************************/

control MyIngress(inout headers hdr,
                  inout metadata meta,
                  inout standard_metadata_t standard_metadata) {

    action drop() {
        mark_to_drop(standard_metadata);
    }

    action ipv4_forward(macAddr_t dstAddr, egressSpec_t port) {

        //set the src mac address as the previous dst
        hdr.ethernet.srcAddr = hdr.ethernet.dstAddr;

       //set the destination mac address that we got from the match in the table
        hdr.ethernet.dstAddr = dstAddr;

        //set the output port that we also get from the table
        standard_metadata.egress_spec = port;

        //decrease ttl by 1
        hdr.ipv4.ttl = hdr.ipv4.ttl -1;

    }

    table ipv4_lpm {
        key = {
            hdr.ipv4.dstAddr: lpm;
        }
        actions = {
            ipv4_forward;
            drop;
            NoAction;
        }
        size = 1024;
        default_action = NoAction();
    }

    apply {

        //only if IPV4 the rule is applied. Therefore other packets will not be forwarded.
        if (hdr.ipv4.isValid()){
            ipv4_lpm.apply();

        }
    }
}

/*************************************************************************
****************  E G R E S S   P R O C E S S I N G   *******************
*************************************************************************/

control MyEgress(inout headers hdr,
                 inout metadata meta,
                 inout standard_metadata_t standard_metadata) {
    apply {  }
}

/*************************************************************************
*************   C H E C K S U M    C O M P U T A T I O N   **************
*************************************************************************/

control MyComputeChecksum(inout headers hdr, inout metadata meta) {
     apply {
	update_checksum(
	    hdr.ipv4.isValid(),
            { hdr.ipv4.version,
	      hdr.ipv4.ihl,
              hdr.ipv4.dscp,
              hdr.ipv4.ecn,
              hdr.ipv4.totalLen,
              hdr.ipv4.identification,
              hdr.ipv4.flags,
              hdr.ipv4.fragOffset,
              hdr.ipv4.ttl,
              hdr.ipv4.protocol,
              hdr.ipv4.srcAddr,
              hdr.ipv4.dstAddr },
            hdr.ipv4.hdrChecksum,
            HashAlgorithm.csum16);
    }
}


/*************************************************************************
***********************  D E P A R S E R  *******************************
*************************************************************************/

control MyDeparser(packet_out packet, in headers hdr) {
    apply {

        //parsed headers have to be added again into the packet.
        packet.emit(hdr.ethernet);
        packet.emit(hdr.ipv4);

    }
}

/*************************************************************************
***********************  S W I T C H  *******************************
*************************************************************************/

//switch architecture
V1Switch(
MyParser(),
MyVerifyChecksum(),
MyIngress(),
MyEgress(),
MyComputeChecksum(),
MyDeparser()
) main;

I have configure all the 20 switches table files, c1-c4, a1-a8, e1-e8. But for the convenience of reading, I only wrote e1, e2, and a1. Below, I will write the complete topology, switch flow table files and network.py.

network.py:

from p4utils.mininetlib.network_API import NetworkAPI

def create_fat_tree_topology(k):
    if k % 2 != 0:
        raise ValueError("The order k of Fat-Tree must be even!")
    
    net = NetworkAPI()

    # Network general options
    net.setLogLevel('info')
    net.enableCli()

    # Network definition
    # Number of Core layer switches = (k/2) ^ 2
    # Number of Agg Layer and Edge Layer Switches = k * k/2
    core_switch_count = (k // 2) ** 2
    agg_switch_count = k * (k // 2)
    edge_switch_count = agg_switch_count

    core_switches = [f'c{i+1}' for i in range(core_switch_count)]
    aggregation_switches = [f'a{i+1}' for i in range(agg_switch_count)]
    edge_switches = [f'e{i+1}' for i in range(edge_switch_count)]


    # Add Core switch
    for core_switch in core_switches:
        net.addP4Switch(core_switch, cli_input=f'switch/core/{core_switch.lower()}-commands.txt')

    # Add Agg switch
    for agg_switch in aggregation_switches:
        net.addP4Switch(agg_switch, cli_input=f'switch/agg/{agg_switch.lower()}-commands.txt')

    # Add Edge switch
    for edge_switch in edge_switches:
        net.addP4Switch(edge_switch, cli_input=f'switch/edge/{edge_switch.lower()}-commands.txt')
    
    net.setP4SourceAll('p4src/forwarding.p4')

    # Number of servers = k ^ 3/4
    host_count = (k ** 3) // 4
    hosts = {}  #
    #hosts = [f'h{i+1}' for i in range(host_count)]
    # Add Host
    # Dynamically add hosts with IP and MAC
    host_index = 1
    for pod in range(k):
        for edge in range(k // 2):
            # Global Edge 
            global_edge_index = pod * (k // 2) + edge + 1
            for host in range(k // 2):
                host_name = f'h{host_index}'
                ip = f'10.{pod}.{global_edge_index}.{host + 1}/24'
                #ip = f'10.{pod}.{global_edge_index}.{host + 1}'
                mac = f'00:00:{pod:02x}:{global_edge_index:02x}:00:{host + 1:02x}'
                net.addHost(host_name)
                # hosts 
                if global_edge_index not in hosts:
                    hosts[global_edge_index] = []
                hosts[global_edge_index].append((host_name, ip, mac))
                host_index += 1
        
    # Links
    # 1. Core - Agg Links
    for i, core_switch in enumerate(core_switches):
        for pod in range(k):  # Traverse all Pods
            # Calculate the convergence switch index connected by the core switch
            agg_index = pod * (k // 2) + (i // (k // 2))
            net.addLink(core_switch, aggregation_switches[agg_index])

            mac_Core_Agg = f'00:01:{i + 1:02x}:00:00:{agg_index + 1:02x}'
            mac_Agg_Core = f'00:02:{i + 1:02x}:00:00:{agg_index + 1:02x}'
            net.setIntfMac(core_switch, aggregation_switches[agg_index], mac_Core_Agg)
            net.setIntfMac(aggregation_switches[agg_index], core_switch, mac_Agg_Core)


    # 2. Agg - Edge Links
    for pod in range(k):
        for agg in range(k // 2):
            for edge in range(k // 2):
                agg_index = pod * (k // 2) + agg
                edge_index = pod * (k // 2) + edge
                
                net.addLink(aggregation_switches[agg_index], edge_switches[edge_index])
                
                mac_Agg_Edge = f'00:02:{agg_index + 1:02x}:00:{edge_index + 1:02x}:00'
                mac_Edge_Agg = f'00:03:{agg_index + 1:02x}:00:{edge_index + 1:02x}:00'
                net.setIntfMac(aggregation_switches[agg_index], edge_switches[edge_index], mac_Agg_Edge)
                net.setIntfMac(edge_switches[edge_index], aggregation_switches[agg_index], mac_Edge_Agg)

    # 3. Edge - Host Links
    for access_id, host_info in hosts.items():
        edge_host = 1
        for host_name, ip, mac in host_info:
            net.addLink(edge_switches[access_id - 1], host_name)  #  access_id - 1
            net.setIntfIp(host_name, edge_switches[access_id - 1], ip)
            #net.setIntfIp(host_name, edge_switches[access_id - 1], f'{ip}/32')
            net.setIntfMac(host_name, edge_switches[access_id - 1], mac)

            mac_Edge_Host = f'00:03:{access_id:02x}:00:00:{edge_host:02x}'
            net.setIntfMac(edge_switches[access_id - 1], host_name, mac_Edge_Host)
            edge_host = edge_host + 1

    # Nodes general options
    net.enablePcapDumpAll()
    net.enableLogAll()

    # Start network
    net.startNetwork()


if __name__ == '__main__':
    # Fat-Tree 
    k = 4  # Fat-Tree
    create_fat_tree_topology(k)

The topology photo:

4_fat_tree
4_fat_tree1080×474 83.6 KB

c1-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.0.0.0/16 => 00:02:01:00:00:01 1
table_add ipv4_lpm ipv4_forward 10.1.0.0/16 => 00:02:01:00:00:03 2
table_add ipv4_lpm ipv4_forward 10.2.0.0/16 => 00:02:01:00:00:05 3
table_add ipv4_lpm ipv4_forward 10.3.0.0/16 => 00:02:01:00:00:07 4

c2-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.0.0.0/16 => 00:02:02:00:00:01 1
table_add ipv4_lpm ipv4_forward 10.1.0.0/16 => 00:02:02:00:00:03 2
table_add ipv4_lpm ipv4_forward 10.2.0.0/16 => 00:02:02:00:00:05 3
table_add ipv4_lpm ipv4_forward 10.3.0.0/16 => 00:02:02:00:00:07 4

c3-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.0.0.0/16 => 00:02:03:00:00:02 1
table_add ipv4_lpm ipv4_forward 10.1.0.0/16 => 00:02:03:00:00:04 2
table_add ipv4_lpm ipv4_forward 10.2.0.0/16 => 00:02:03:00:00:06 3
table_add ipv4_lpm ipv4_forward 10.3.0.0/16 => 00:02:03:00:00:08 4

c4-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.0.0.0/16 => 00:02:04:00:00:02 1
table_add ipv4_lpm ipv4_forward 10.1.0.0/16 => 00:02:04:00:00:04 2
table_add ipv4_lpm ipv4_forward 10.2.0.0/16 => 00:02:04:00:00:06 3
table_add ipv4_lpm ipv4_forward 10.3.0.0/16 => 00:02:04:00:00:08 4

a1-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.0.1.0/24 => 00:03:01:00:01:00 1
table_add ipv4_lpm ipv4_forward 10.0.2.0/24 => 00:03:01:00:02:00 2
table_add ipv4_lpm ipv4_forward 10.1.0.0/16 => 00:01:01:00:00:01 3
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:01:02:00:00:01 4

a2-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.0.1.0/24 => 00:03:02:00:01:00 1
table_add ipv4_lpm ipv4_forward 10.0.2.0/24 => 00:03:02:00:02:00 2
table_add ipv4_lpm ipv4_forward 10.1.0.0/16 => 00:01:03:00:00:02 3
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:01:04:00:00:02 4

a3-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.1.3.0/24 => 00:03:03:00:03:00 1
table_add ipv4_lpm ipv4_forward 10.1.4.0/24 => 00:03:03:00:04:00 2
table_add ipv4_lpm ipv4_forward 10.0.0.0/16 => 00:01:01:00:00:03 3
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:01:02:00:00:03 4

a4-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.1.3.0/24 => 00:03:04:00:03:00 1
table_add ipv4_lpm ipv4_forward 10.1.4.0/24 => 00:03:04:00:04:00 2
table_add ipv4_lpm ipv4_forward 10.0.0.0/16 => 00:01:03:00:00:04 3
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:01:04:00:00:04 4

a5-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.2.5.0/24 => 00:03:05:00:05:00 1
table_add ipv4_lpm ipv4_forward 10.2.6.0/24 => 00:03:05:00:06:00 2
table_add ipv4_lpm ipv4_forward 10.3.0.0/16 => 00:01:01:00:00:05 3
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:01:02:00:00:05 4

a6-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.2.5.0/24 => 00:03:06:00:05:00 1
table_add ipv4_lpm ipv4_forward 10.2.6.0/24 => 00:03:06:00:06:00 2
table_add ipv4_lpm ipv4_forward 10.3.0.0/16 => 00:01:03:00:00:06 3
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:01:04:00:00:06 4

a7-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.3.7.0/24 => 00:03:07:00:07:00 1
table_add ipv4_lpm ipv4_forward 10.3.8.0/24 => 00:03:07:00:08:00 2
table_add ipv4_lpm ipv4_forward 10.2.0.0/16 => 00:01:01:00:00:07 3
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:01:02:00:00:07 4

a8-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.3.7.0/24 => 00:03:08:00:07:00 1
table_add ipv4_lpm ipv4_forward 10.3.8.0/24 => 00:03:08:00:08:00 2
table_add ipv4_lpm ipv4_forward 10.2.0.0/16 => 00:01:03:00:00:08 3
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:01:04:00:00:08 4

e1-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.0.1.1/32 => 00:00:00:01:00:01 3
table_add ipv4_lpm ipv4_forward 10.0.1.2/32 => 00:00:00:01:00:02 4
table_add ipv4_lpm ipv4_forward 10.0.2.0/24 => 00:02:01:00:01:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:02:00:01:00 2

e2-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.0.2.1/32 => 00:00:00:02:00:01 3
table_add ipv4_lpm ipv4_forward 10.0.2.2/32 => 00:00:00:02:00:02 4
table_add ipv4_lpm ipv4_forward 10.0.1.0/24 => 00:02:01:00:02:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:02:00:02:00 2

e3-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.1.3.1/32 => 00:00:01:03:00:01 3
table_add ipv4_lpm ipv4_forward 10.1.3.2/32 => 00:00:01:03:00:02 4
table_add ipv4_lpm ipv4_forward 10.1.4.0/24 => 00:02:03:00:03:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:04:00:03:00 2

e4-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.1.4.1/32 => 00:00:01:04:00:01 3
table_add ipv4_lpm ipv4_forward 10.1.4.2/32 => 00:00:01:04:00:02 4
table_add ipv4_lpm ipv4_forward 10.1.3.0/24 => 00:02:03:00:04:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:04:00:04:00 2

e5-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.2.5.1/32 => 00:00:02:05:00:01 3
table_add ipv4_lpm ipv4_forward 10.2.5.2/32 => 00:00:02:05:00:02 4
table_add ipv4_lpm ipv4_forward 10.2.6.0/24 => 00:02:05:00:05:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:06:00:05:00 2

e6-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.2.6.1/32 => 00:00:02:06:00:01 3
table_add ipv4_lpm ipv4_forward 10.2.6.2/32 => 00:00:02:06:00:02 4
table_add ipv4_lpm ipv4_forward 10.2.5.0/24 => 00:02:05:00:06:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:06:00:06:00 2

e7-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.3.7.1/32 => 00:00:03:07:00:01 3
table_add ipv4_lpm ipv4_forward 10.3.7.2/32 => 00:00:03:07:00:02 4
table_add ipv4_lpm ipv4_forward 10.3.8.0/24 => 00:02:07:00:07:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:08:00:07:00 2

e8-switch:

table_set_default ipv4_lpm drop

table_add ipv4_lpm ipv4_forward 10.3.8.1/32 => 00:00:03:08:00:01 3
table_add ipv4_lpm ipv4_forward 10.3.8.2/32 => 00:00:03:08:00:02 4
table_add ipv4_lpm ipv4_forward 10.3.7.0/24 => 00:02:07:00:08:00 1
table_add ipv4_lpm ipv4_forward 10.0.0.0/8 => 00:02:08:00:08:00 2
4

I would recommend debugging this step by step.

Pick one ping between hosts that is failing, e.g. h1 ping h3 (or whatever ping is currently not working). Start a fresh run of the entire network where you haven’t sent any packets yet, so that the switch log files are initially empty. Send one packet (or stop it at only a few, all the same, and stop it quickly). Look at the directory containing the switch log files, and see which ones are all the largest, meaning that they have printed log messages about processing at least one packet. Most of the log files should be very short, meaning they have not processed any packets.

Pick the first switch that you think it should have reached, and examine its log file. It is a bit tedious the first time, but you’ll get faster at looking for the most important things after going through one of them in detail. See where that switch sends its packet next, or whether it drops the packet.

If it sends the packet to another switch, go to that switch’s log file and see what it does with the packet, continuing until you get to a switch that either (a) drops the packet, or (b) does something you did not want it to do with the packet. Figure out why it is doing that unexpected thing from the log file, and think about whether it is because it has incorrect table entries, or wrong P4 code, or both. Fix the P4 code or table entries for that switch so you believe it is correct, and do another run from empty log files again.

After getting one failing ping to work, if you have ideas for what might be wrong with other switch’s table entries, try fixing all of the table entries. If you do not have an idea for something more general that might be wrong with the table entries yet, pick another pair of hosts where the ping is failing, and repeat the debug process above.

Note: This is general advice, for any network of devices, where you have detailed logging information on how they are behaving.