[jachymb]

This was done mostly with a deterministic script, but also with the help of claude Opus5.5 for some nontrivial instances.

Summary

This removes 3074 #include lines that are not used, in 1702 files under stan/math/ and test/, plus 6 blank lines that the removals left doubled. The diff only removes lines; include-only aggregator headers (prim/fun.hpp, rev.hpp, …) are unchanged.

Tool. The candidates are the lines that clang-tidy 22's misc-include-cleaner reports as unused. It was run:

• on every Stan header, both plain and with STAN_OPENCL, STAN_THREADS and STAN_MPI;
• on each test, in the configurations CI builds it with.

In the analysis copy only, the aggregators, prim/fun/Eigen.hpp and Eigen's module headers carry IWYU export pragmas. So, for example, #include <stan/math/prim/meta.hpp> counts as used exactly when something from prim/meta/ is used. Files with other preprocessor conditions or with macro definitions were reviewed by hand.

What is kept. include-cleaner sees only references it can resolve. A flagged line therefore stays when one of the checks below finds that removing it would cut a file off from something it uses through that include. Each check compares against develop's include closures (clang -M), and a failing check keeps removed lines on develop's include chain to the lost header.

• Every header include-cleaner reports as needed by a file stays reachable from that file.
• Every declaration or macro a file references stays reachable, collected with libclang from develop. This covers the resolved declaration, its definition, and all overload candidates of a dependent call or using declaration. include-cleaner treats those candidates as ambiguous and never reports them as missing. This keeps, for example, neg_binomial_lccdf visible in neg_binomial_2_lccdf.hpp and log1m_exp in ordered_logistic_lpmf.hpp.
• Order is kept where a test helper, or a qualified call (stan::math::size(x), using math::exp;), saw overloads in develop only because its includer included them first. Those overloads still come first, so that the OpenCL size overload precedes prim/err's elementwise checks in OpenCL headers.
• Template specializations stay reachable for the types a file still reaches. These are the var, fvar and matrix_cl trait and std specializations, and the prim trait specializations for Eigen, std::vector and std::complex.
• A failing standalone compile (see Tests) keeps the minimal set of lines, found by bisection, that makes it compile. One example is the rev complex functions such as rev/fun/log.hpp: they instantiate prim templates with var and need the rev overloads those templates call.
• If include-cleaner reports that a file needs a standard header that develop provided through a Stan header, it still comes through a Stan header, not only through Eigen or standard-library internals.
• Each rev/fwd overload header keeps its include of its own prim counterpart, and so does each OpenCL overload that included one.
• ordered_logistic_lpmf.hpp keeps is_integer.hpp. That is the only path from stan/math.hpp to stan::math::is_integer, which prim/fun.hpp does not list. matrix_exp_action_handler.hpp keeps unsupported/Eigen/MatrixFunctions for its dependent .pow() call.

In total 669 of the 3743 flagged lines are kept.

Tests

No tests are added; the change removes includes only. Checked locally with clang 15 in both configurations above:

• Every header under stan/ passes the header check as Jenkins runs it: forced into an empty translation unit, with -Werror -Wunused-local-typedefs. The exceptions are the headers that also fail on develop: eigen_plugins.h, and the MPI headers under STAN_MPI's deleted-destructor warning, which pass with that warning off.
• test_ad.hpp, multiple_translation_units_includes.hpp, every test under test/unit/math/prim and test/unit/math/opencl, and the reduce_sum mix tests compile on their own (-fsyntax-only).
• The declaration check passes for every header, and for every test file whose own includes changed.
• runChecks.py, make cpplint and clang-format 10 report nothing new.

Left to CI:

• the generated distribution tests;
• the 786 test files whose own includes are unchanged but which now reach fewer headers.

Side Effects

Code outside the library that relied on Stan headers for unrelated includes may need its own:

• Without STAN_OPENCL, stan/math.hpp and the prim/rev/fwd/mix aggregators no longer bring in <deque> and <queue>. With STAN_OPENCL they still do.
• With STAN_OPENCL, prim.hpp and fwd.hpp no longer include opencl/rev/to_arena.hpp.
• stan/math/prim.hpp and stan/math/fwd.hpp no longer bring in the KINSOL headers; stan/math.hpp, rev.hpp and mix.hpp still do.

No file in stan-dev/stan's src/stan uses any of these without including them itself. Single headers now include less, so including one of them alone gives fewer transitively. This PR touches many include blocks and will conflict with other open PRs that edit includes. The conflicts are only in those lines.

Release notes

Removed unused #include directives across the library and its tests; no functional change.

Checklist

• Copyright holder: Jáchym Barvínek jachymb@gmail.com

  The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
  - Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
  - Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

• the basic tests are passing

  • unit tests pass (to run, use: ./runTests.py test/unit)
  • header checks pass, (make test-headers)
  • dependencies checks pass, (make test-math-dependencies)
  • docs build, (make doxygen)
  • code passes the built in C++ standards checks (make cpplint)

• the code is written in idiomatic C++ and changes are documented in the doxygen

• the new changes are tested

[stan-buildbot]

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan	0.04	0.04	1.04	3.74% faster
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan	0.16	0.16	1.0	0.28% faster
stat_comp_benchmarks/benchmarks/garch/garch.stan	0.86	0.86	0.99	-0.69% slower
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	0.99	-1.28% slower
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan	0.11	0.1	1.02	1.77% faster
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan	8.08	8.13	0.99	-0.57% slower
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan	3.67	3.68	1.0	-0.23% slower
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan	32.98	33.41	0.99	-1.29% slower
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan	0.4	0.39	1.02	2.23% faster
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan	6.82	6.81	1.0	0.14% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan	173.11	167.09	1.04	3.48% faster
stat_comp_benchmarks/benchmarks/arK/arK.stan	3.22	3.22	1.0	0.05% faster
stat_comp_benchmarks/benchmarks/arma/arma.stan	0.61	0.61	0.99	-1.02% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	25.72	25.7	1.0	0.1% faster
performance.compilation	399.18	397.16	1.01	0.5% faster
Mean result: 1.0050709948766403

Jenkins Console Log
Jenkins Build Stages
Commit hash: a2236819d562837e97a9bfe97dc9ebd6d8613729

Machine information

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           52 bits physical, 57 bits virtual
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              25
Model:                                   17
Model name:                              AMD EPYC 9474F 48-Core Processor
Stepping:                                1
Frequency boost:                         enabled
CPU MHz:                                 1498.750
CPU max MHz:                             4114.4229
CPU min MHz:                             1500.0000
BogoMIPS:                                7189.39
Virtualization:                          AMD-V
L1d cache:                               3 MiB
L1i cache:                               3 MiB
L2 cache:                                96 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

[stan-buildbot]

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan	0.04	0.04	1.0	-0.16% slower
stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan	0.16	0.16	1.0	-0.15% slower
stat_comp_benchmarks/benchmarks/garch/garch.stan	0.86	0.86	1.01	0.82% faster
stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	0.96	-3.94% slower
stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan	0.1	0.11	0.99	-1.06% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan	8.01	8.14	0.98	-1.61% slower
stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan	3.74	3.71	1.01	0.73% faster
stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan	32.89	32.74	1.0	0.44% faster
stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan	0.38	0.38	1.0	-0.08% slower
stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan	6.81	6.8	1.0	0.23% faster
stat_comp_benchmarks/benchmarks/sir/sir.stan	165.78	165.99	1.0	-0.13% slower
stat_comp_benchmarks/benchmarks/arK/arK.stan	3.27	3.31	0.99	-1.26% slower
stat_comp_benchmarks/benchmarks/arma/arma.stan	0.6	0.6	0.99	-0.54% slower
stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	25.71	27.23	0.94	-5.89% slower
performance.compilation	405.12	391.33	1.04	3.4% faster
Mean result: 0.9943158656497112

Jenkins Console Log
Jenkins Build Stages
Commit hash: a2236819d562837e97a9bfe97dc9ebd6d8613729

Machine information

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU:

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           52 bits physical, 57 bits virtual
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
NUMA node(s):                            2
Vendor ID:                               AuthenticAMD
CPU family:                              25
Model:                                   17
Model name:                              AMD EPYC 9474F 48-Core Processor
Stepping:                                1
Frequency boost:                         enabled
CPU MHz:                                 1495.300
CPU max MHz:                             4114.4229
CPU min MHz:                             1500.0000
BogoMIPS:                                7189.39
Virtualization:                          AMD-V
L1d cache:                               3 MiB
L1i cache:                               3 MiB
L2 cache:                                96 MiB
L3 cache:                                512 MiB
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap

G++:

g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:

clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

[WardBrian]

More out of curiosity than anything, can you share the script that did the bulk of this?

[jachymb]

@WardBrian Sure, see this remove_unused_includes.py It was meant as a one-timer, so it's obviously sloppy, but mostly gets the job done.

The core idea is using

clang-tidy --quiet --checks='-*,misc-include-cleaner' <file> -- -x c++ -std=c++17 -I. -isystem lib/eigen_5.0.1 -isystem lib/boost_1.87.0 ... [-DSTAN_OPENCL ...]

The script mostly goes through the filesystem tree and calls this with some additional conditions that are kinda ad-hoc based on where it was failing or inappropriate.

It reports included header X is not used directly and no header providing "Y" is directly included so I think something like that could be used on changed files to report a warning in CI if you want to use it as inspiration for #3474 :)

[jachymb]

I added some more details to the #3474 discussion.