# Error during P4 program compilation: "Too many heap sections: Increase MAXHINCR or MAX\_HEAP\_SECTS" **URL:** \ **Category:** P4 Dev\ **Created:** [December 12, 2024, 1:11pm UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231 "2024-12-12T13:11:15Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![sun-titto](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sun-titto](https://forum.p4.org/u/sun-titto)\ **Post date:** [December 12, 2024, 1:11pm UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/1 "2024-12-12T13:11:15Z")
I am writing a P4 program on an Ubuntu virtual machine in VMware, with 8.5GB of allocated memory and an 8-core CPU. Since my program is quite large, I encountered the error “Too many heap sections: Increase MAXHINCR or MAX\_HEAP\_SECTS” during compilation. Apart from increasing the virtual machine’s memory, are there any other ways to resolve this issue? 😭 😰 ![Snipaste_2024-12-12_16-26-13](https://us1.discourse-cdn.com/flex019/uploads/p4/original/1X/08cd657bfb20cc07f3bd21232528aaf035a353e0.png) ---
**Author:** ![andyfingerhut](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.p4.org/andyfingerhut/32/17_2.png) [@andyfingerhut](https://forum.p4.org/u/andyfingerhut)\ **Post date:** [December 12, 2024, 1:27pm UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/2 "2024-12-12T13:27:47Z")
> [@sun-titto](#): > > Too many heap sections: Increase MAXHINCR or MAX\_HEAP\_SECTS Have you run a program like `top` in a separate terminal while trying to compile your P4 program, and verified that some P4 compiler process like `p4c-bm2-ss` is using multiple GBytes of RESident memory before this error occurs? Or is it some other process using a lot of memory? If it is the P4 compiler, you say your P4 program is “quite large”. Could you usefully, or at least as an experiment, create a copy of the program and delete or comment out a large chunk of it to see if it compiles with the amount of memory your system has now? What is the output of `p4c --version` on your system? I ask because during the past 6-12 months several P4 compiler developers have been working on changes to speed up the compiler and enable it to use less memory than older versions. Trying out the most recent version, e.g something built from source code from today or any time during the last month, _might_ help reduce the memory required. All of that said, if you are using the latest version of the P4 compiler, and you need everything in your large program, I have seen some evidence that the error message you are seeing might be due to the GC library used by the P4 compiler needing more than 8GBytes of RAM. There appear to be environment variables that can be used to increase this limit, like `export GC_MAXIMUM_HEAP_SIZE=16G` that I found mentioned in an Internet search on this page: [Too many heap sections: Increase MAXHINCR or MAX\_HEAP\_SECTS when allocating many sets · Issue #5610 · crystal-lang/crystal · GitHub](https://github.com/crystal-lang/crystal/issues/5610) But if. you need that option to increase the RAM that the P4 compiler’s GC can use, then you would also need to run it on a system with more RAM available. ---
**Author:** ![sun-titto](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sun-titto](https://forum.p4.org/u/sun-titto)\ **Post date:** [December 13, 2024, 10:19am UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/3 "2024-12-13T10:19:50Z")
Thank you for your reply. Below are my responses to your questions: 1. I ran the `top` command and found that `p4c-bm2-ss` is consuming a large amount of memory. There are no other programs competing for resources with `p4c` on the system. 2. After commenting out some parts of the code, my P4 program can be compiled and run successfully. 3. The version of the `p4c` program I am using is indeed quite outdated, as I have not updated it since downloading it last September. Thanks again for your response. I will try using the latest version of the `p4c` program. If the issue persists, I will consider allocating more memory. ---
**Author:** ![sun-titto](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sun-titto](https://forum.p4.org/u/sun-titto)\ **Post date:** [December 24, 2024, 8:28pm UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/4 "2024-12-24T20:28:35Z")
I have now updated `p4c` to the latest version (v1.2.5.1). However, since my program uses a large number of macro substitutions and the codebase is quite large, even the latest version of `p4c` cannot compile my program quickly. Therefore, I would like to know if there are any solutions to speed up the compilation of my program. The command I use for compilation is as follows: `p4c --target bmv2 --arch v1model main.p4` ---
**Author:** ![andyfingerhut](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.p4.org/andyfingerhut/32/17_2.png) [@andyfingerhut](https://forum.p4.org/u/andyfingerhut)\ **Post date:** [December 24, 2024, 9:24pm UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/5 "2024-12-24T21:24:39Z")
Is your P4 program something you are able and willing to publish? If so, it might help others who have tried to improve the performance of p4c to profile the code while compiling your program, and see where most of the time is being spent, and/or look at the source code and suggest alternative ways of achieving similar behavior with lower compilation time (if such a way exists). If you are willing to publish it, rather than copying and pasting it here, I would recommend putting your code on a public web site, e.g. a public repository on Github or a similar web site, and put a link to it in a comment here. ---
**Author:** ![sun-titto](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sun-titto](https://forum.p4.org/u/sun-titto)\ **Post date:** [December 25, 2024, 8:46am UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/6 "2024-12-25T08:46:43Z")
Although this is one of my research projects, I am willing to make my P4 code public. The code repository is available at the following link: > **[GitHub - hc-tec/p4asm\_p4code](https://github.com/hc-tec/p4asm_p4code)** > > Contribute to hc-tec/p4asm\_p4code development by creating an account on GitHub. This project may appear somewhat immature, so I sincerely welcome your feedback and suggestions for improvement. ---
**Author:** ![andyfingerhut](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.p4.org/andyfingerhut/32/17_2.png) [@andyfingerhut](https://forum.p4.org/u/andyfingerhut)\ **Post date:** [December 26, 2024, 3:23am UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/7 "2024-12-26T03:23:21Z")
I was able to compile your program with some extra debug output showing which passes were running at the time, and I ran `top` in a separate terminal window to see when the memory grew. ```auto p4c-bm2-ss -vv --arch v1model main.p4 ``` I could see that most of the processing time, and most of the memory use increase, occurred while executing two passes, both with `SimplifyDefUse` in their names. There are some recent issues where p4c developers are profiling the memory use of p4c and have seen large memory use for _some_ P4 programs in that pass, and your program certainly seems to be one of those: [Elevated memory usage in def-use · Issue #4872 · p4lang/p4c · GitHub](https://github.com/p4lang/p4c/issues/4872) I will add a comment on that p4c issue linking to this discussion, in case they may want to use your program as another example to consider. ---
**Author:** ![sun-titto](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sun-titto](https://forum.p4.org/u/sun-titto)\ **Post date:** [December 26, 2024, 11:24am UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/8 "2024-12-26T11:24:50Z")
Ok, thank you for your efforts and contributions. I am looking forward to the future improvements of p4c. ---
**Author:** ![sun-titto](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sun-titto](https://forum.p4.org/u/sun-titto)\ **Post date:** [January 8, 2025, 10:47am UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/9 "2025-01-08T10:47:41Z")
Do you have any solutions or suggestions to temporarily address my issue? Currently, I’m unable to make further progress on my project. I’ve already explored alternative approaches. Initially, I suspected the core problem was that repeatedly using the macro definition (`EXECUTE_ONE_INSTRUCTION`) caused the code size to grow excessively. Recently, I attempted a new strategy by introducing `control(instruction_execute_3)`, which executes three instances of `EXECUTE_ONE_INSTRUCTION`. I then called `instruction_execute_3` repeatedly in the original `control(MyIngress)` using `apply`, aiming to handle and execute more instructions. However, after examining the compiled JSON code, I noticed that the compiled code size increases with the number of `instruction_execute_3` calls. This suggests that `p4c` does not reuse `instruction_execute_3`, resulting in behavior similar to the previous repeated use of macro definitions. Unfortunately, this approach hasn’t resolved the issue. ```p4 control instruction_execute_3(inout headers hdr, inout metadata meta, inout standard_metadata_t standard_metadata) { #include "instruction/address_mode.p4" #include "instruction/operator.p4" apply { EXECUTE_ONE_INSTRUCTION; EXECUTE_ONE_INSTRUCTION; EXECUTE_ONE_INSTRUCTION; } } control MyIngress(inout headers hdr, inout metadata meta, inout standard_metadata_t standard_metadata) { apply { if (hdr.instruction.isValid()) { if (hdr.instruction.command_type == INSTRUCTION_COMMAND_TYPE.PUBLISH_PROGRAM) { save_intructions(hdr.instruction.start_pc); } else if (hdr.instruction.command_type == INSTRUCTION_COMMAND_TYPE.EXECUTE_PROGRAM) { instruction_execute_3.apply(hdr, meta, standard_metadata); instruction_execute_3.apply(hdr, meta, standard_metadata); instruction_execute_3.apply(hdr, meta, standard_metadata); instruction_execute_3.apply(hdr, meta, standard_metadata); instruction_execute_3.apply(hdr, meta, standard_metadata); instruction_execute_3.apply(hdr, meta, standard_metadata); } } } } ``` ---
**Author:** ![andyfingerhut](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.p4.org/andyfingerhut/32/17_2.png) [@andyfingerhut](https://forum.p4.org/u/andyfingerhut)\ **Post date:** [January 9, 2025, 6:32am UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/10 "2025-01-09T06:32:30Z")
I can understand why the modified program you describe did not improve the situation for you. It is because the P4 compiler expands all sub-controls at compile time, like “inline” can do in a C program. The resulting inline-expanded program is the same as your original, and so the compile time is no different. One possibility you may want to consider, although you may find it a bit onerous: If you can write a P4 program where you recirculate the packet after each execution of EXECUTE\_ONE\_INSTRUCTION, so that it is only invoked exactly one time in your ingress control, and the packet is recirculated as many times as needed in order to cause EXECUTE\_ONE\_INSTRUCTION to be executed the desired number of times, then the compile time for that control should be much smaller. If you restrict this repeated execution to the ingress control, you could also use the resubmit feature instead of recirculation. You can read more about the details of how to resubmit or recirculate packets in the v1model architecture here: [behavioral-model/docs/simple\_switch.md at main · p4lang/behavioral-model · GitHub](https://github.com/p4lang/behavioral-model/blob/main/docs/simple_switch.md) ---
**Author:** ![sun-titto](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sun-titto](https://forum.p4.org/u/sun-titto)\ **Post date:** [January 9, 2025, 7:34am UTC](https://forum.p4.org/t/error-during-p4-program-compilation-too-many-heap-sections-increase-maxhincr-or-max-heap-sects/1231/11 "2025-01-09T07:34:42Z")
Thank you very much for your response. Your suggestions have given me some inspiration. I have come up with a solution where a certain number of instructions (e.g., 10) are executed at a time, and additional instructions are processed through two or three `resubmit`. While this is a trade-off, sacrificing performance for generality, I believe that other optimization solutions can be found in the future.