store-merging: Apply --param=store-merging-max-size= in more spots [PR117439]

store-merging: Apply --param=store-merging-max-size= in more spots [PR117439]

Store merging assumes a merged region won't be too large.  The assumption is
e.g. in using inappropriate types in various spots (e.g. int for bit sizes
and bit positions in a few spots, or unsigned for the total size in bytes of
the merged region), in doing XNEWVEC for the whole total size of the merged
region and preparing everything in there and even that XALLOCAVEC in two
spots.  The last case is what was breaking the test below in the patch,
64MB XALLOCAVEC is just too large, but even with that fixed I think we just
shouldn't be merging gigabyte large merge groups.

We already have --param=store-merging-max-size= parameter, right now with
65536 bytes maximum (if needed, we could raise that limit a little bit).
That parameter is currently used when merging two adjacent stores, if the
size of the already merged bitregion together with the new store's bitregion
is above that limit, we don't merge those.
I guess initially that was sufficient, at that time a store was always
limited to MAX_BITSIZE_MODE_ANY_INT bits.
But later on we've added support for empty ctors ({} and even later
{CLOBBER}) and also added another spot where we merge further stores into
the merge group, if there is some overlap, we can merge various other stores
in one coalesce_immediate_stores iteration.
And, we weren't applying the --param=store-merging-max-size= parameter
in either of those cases.  So a single store can be gigabytes long, and
if there is some overlap, we can extend the region again to gigabytes in
size.

The following patch attempts to apply that parameter even in those cases.
So, if testing if it should merge the merged group with info (we've already
punted if those together are above the parameter) and some other stores,
the first two hunks just punt if that would make the merge group too large.
And the third hunk doesn't even add stores which are over the limit.

2024-11-06  Jakub Jelinek  <jakub@redhat.com>

	PR tree-optimization/117439
	* gimple-ssa-store-merging.cc
	(imm_store_chain_info::coalesce_immediate_stores): Punt if merging of
	any of the additional overlapping stores would result in growing the
	bitregion size over param_store_merging_max_size.
	(pass_store_merging::process_store): Terminate all aliasing chains
	for stores with bitregion larger than param_store_merging_max_size.

	* g++.dg/opt/pr117439.C: New test.