GCC Bugzilla has been upgraded from version 4.4.9 to 5.0rc3. If you see any problem, please report it to bug 64968.
Bug 41478 - Corrupted memory using PACK for derived-types with allocated components
Summary: Corrupted memory using PACK for derived-types with allocated components
Status: RESOLVED FIXED
Alias: None
Product: gcc
Classification: Unclassified
Component: fortran (show other bugs)
Version: 4.5.0
: P3 normal
Target Milestone: 4.5.0
Assignee: Paul Thomas
URL:
Keywords: wrong-code
Depends on:
Blocks: 32834 42361 42268
  Show dependency treegraph
 
Reported: 2009-09-26 20:05 UTC by Juergen Reuter
Modified: 2010-02-07 03:55 UTC (History)
3 users (show)

See Also:
Host:
Target:
Build:
Known to work:
Known to fail: 4.3.3 4.5.0
Last reconfirmed: 2009-09-27 17:05:57


Attachments
Test code triggering the runtime error. (243 bytes, application/octet-stream)
2009-09-26 20:06 UTC, Juergen Reuter
Details
Fix and testcase for the PR (2.90 KB, patch)
2010-01-11 06:14 UTC, Paul Thomas
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Juergen Reuter 2009-09-26 20:05:10 UTC
The following program causes a double free violation, when compiled and run, the error message is:
a.out(12292) malloc: *** error for object 0x2009d0: double free
*** set a breakpoint in malloc_error_break to debug
The code is:
program main

 type :: container_t
    integer, dimension(:), allocatable :: entry
 end type container_t

 type(container_t), dimension(2) :: a1, a2
 integer :: i

 do i = 1, 2
    allocate (a1(i)%entry (1), a2(i)%entry (1))
    a1(i)%entry = 0
    a2(i)%entry = i
 end do

 print *, "array1:"
 do i = 1, 2
    print *, a1(i)%entry
 end do
 print *, "array2:"
 do i = 1, 2
    print *, a2(i)%entry
 end do

 print *, "pack array2(2) into array1:"
 a1(1:1) = pack (a2(1:2), mask = [.false., .true.])

 print *, "array1:"
 do i = 1, 2
    print *, a1(i)%entry
 end do


end program main
Comment 1 Juergen Reuter 2009-09-26 20:06:37 UTC
Created attachment 18660 [details]
Test code triggering the runtime error.
Comment 2 kargl 2009-09-26 21:15:55 UTC
Works for me on FreeBSD.

REMOVE:kargl[201] cd tmp
REMOVE:kargl[202] setenv MALLOC_OPTIONS AJ
REMOVE:kargl[203] gfc4x -o z asd.f90 
REMOVE:kargl[204] ./z
 array1:
           0
           0
 array2:
           1
           2
 pack array2(2) into array1:
 array1:
           2
           0
Comment 3 Jerry DeLisle 2009-09-27 00:39:22 UTC
I can reproduce this on 4.3.3 and 4.5.0 r151904.

Comment 4 Dominique d'Humieres 2009-09-27 09:59:28 UTC
I also see it on *-apple-darwin9 gcc 4.2.4, 4.3.4, 4.4.1, and trunk.
Comment 5 Richard Biener 2009-09-27 10:36:46 UTC
Confirmed on i?86-linux.
Comment 6 Dominique d'Humieres 2009-09-27 11:45:46 UTC
> Works for me on FreeBSD.

Is anyone understanding why?

valgrind gives:

==77290== Invalid free() / delete / delete[]
==77290==    at 0x167FB: free (vg_replace_malloc.c:323)
==77290==    by 0x2EAE: MAIN__ (in ./pr41478.ex)
==77290==    by 0x2F0F: main (in ./pr41478.ex)
==77290==  Address 0x3e15c0 is 0 bytes inside a block of size 4 free'd
==77290==    at 0x167FB: free (vg_replace_malloc.c:323)
==77290==    by 0x2E53: MAIN__ (in ./pr41478.ex)
==77290==    by 0x2F0F: main (in ./pr41478.ex)
==77290== 
==77290== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
==77290== malloc/free: in use at exit: 448 bytes in 11 blocks.
==77290== malloc/free: 30 allocs, 20 frees, 3,482 bytes allocated.
==77290== For counts of detected errors, rerun with: -v
==77290== searching for pointers to 11 not-freed blocks.
==77290== checked 833,028 bytes.
==77290== 
==77290== 
==77290== 4 bytes in 1 blocks are definitely lost in loss record 1 of 4
==77290==    at 0x15516: malloc (vg_replace_malloc.c:193)
==77290==    by 0x257B: MAIN__ (in ./pr41478.ex)
==77290==    by 0x2F0F: main (in ./pr41478.ex)
==77290== 
==77290== LEAK SUMMARY:
==77290==    definitely lost: 4 bytes in 1 blocks.
==77290==    indirectly lost: 0 bytes in 0 blocks.
==77290==      possibly lost: 0 bytes in 0 blocks.
==77290==    still reachable: 444 bytes in 10 blocks.
==77290==         suppressed: 0 bytes in 0 blocks.
==77290== Reachable blocks (those to which a pointer was found) are not shown.
==77290== To see them, rerun with: --leak-check=full --show-reachable=yes
Comment 7 Richard Biener 2009-09-27 13:12:19 UTC
Different libc, different behavior on double-free
Comment 8 Tobias Burnus 2009-10-01 21:21:58 UTC
Minimal test case:

program main
type :: container_t
  integer, allocatable :: entry
end type container_t
type(container_t), dimension(1) :: a1, a2
allocate (a1(1)%entry, a2(1)%entry)
a2(1)%entry = 1
a1(1:1) = pack (a2(1:1), mask = [.true.])
end program main

I think what happens is the following: In pack one copies (memcpy) the bytes from A2 to A1 - that whay A1 is a one-to-one copy of A2. At the end automatic deallocation happens. First one frees (and nullifies) A1. Then one moves on to A2, which is an exact copy of A1; thus A2%entry points to the same memory as A1%entry - but the memory was already freed.

Thus we are obviously mishandling derived types with allocatable (or pointer) components. Adding "print *, loc()" before and after pack illustrates this:
              6307888 ! loc(a2(1)%entry - before pack
              6307856 ! loc(a1(1)%entry - before pack
              6307888 ! loc(a2(1)%entry - after pack
              6307888 ! loc(a1(1)%entry - after pack

Actually, ifort shows the same result:
               7020672
               7020640
               7020672
               7020672
(and openf95 and pathf95 crash in pack).

While both NAG 95, sunf95 and g95 seem to handle it correctly:
 6722520
 6722456
 6722520
 6722456

That those handle it correctly, can also be seen if one adds:
  a1(1)%entry = 2
  print *, a2(1)%entry, a1(1)%entry
It should print "1 2" (as with NAG f95, g95 and sunf95) but gfortran and ifort print "2 2".
Comment 9 janus 2009-10-20 19:56:21 UTC
Apart from the double free issue, there might be a more fundamental problem with PACK and arrays of derived types. For me, Tobias' test case from comment #8 segfaults already in the call to _gfortran_pack, and so does the following:

type :: container_t
  integer:: entry = -1
end type container_t
type(container_t), dimension(1) :: a1, a2
a2(1)%entry = 1
print *,a1,a2
a1 = pack (a2, mask = [.true.])
print *,a1,a2
end

This does not have any allocatables (so no auto-dealloc happens), but the output is:

          -1           1
Segmentation fault

which means the second print statement is not reached, i.e. the segfault seems to happen in _gfortran_pack.

The part of the dump which corresponds to the call to PACK looks like this:

  {
    struct array1_logical(kind=4) parm.4;
    static logical(kind=4) A.3[1] = {1};
    struct array1_container_t parm.2;
    struct array1_container_t parm.1;

    parm.1.dtype = 297;
    parm.1.dim[0].lbound = 1;
    parm.1.dim[0].ubound = 1;
    parm.1.dim[0].stride = 1;
    parm.1.data = (void *) &a1[0];
    parm.1.offset = -1;
    parm.2.dtype = 297;
    parm.2.dim[0].lbound = 1;
    parm.2.dim[0].ubound = 1;
    parm.2.dim[0].stride = 1;
    parm.2.data = (void *) &a2[0];
    parm.2.offset = -1;
    parm.4.dtype = 273;
    parm.4.dim[0].lbound = 1;
    parm.4.dim[0].ubound = 1;
    parm.4.dim[0].stride = 1;
    parm.4.data = (void *) &A.3[0];
    parm.4.offset = 0;
    _gfortran_pack (&parm.1, &parm.2, &parm.4, 0B);
  }

The above test case works when making a1 and a2 simple integers.
Comment 10 janus 2009-10-20 20:06:01 UTC
I have re-checked the F03 standard to verify that the first argument of PACK can indeed be of arbitrary type:

13.7.89 PACK (ARRAY, MASK [, VECTOR])
Description. Pack an array into an array of rank one under the control of a 
Class. Transformational function.
Arguments.
ARRAY    may be of any type. It shall not be scalar.
...

In the gfortran testsuite PACK seems to be tested with all intrinsic types (logical, integer, real, complex, character), but not with derived types.
Comment 11 janus 2009-10-20 20:15:19 UTC
Ok, I have identified the place in libgfortran where the segfault happens:

#0  *_gfortran_pack (ret=0x7fffec3ca650, array=0x7fffec3ca620, mask=0x7fffec3ca440, vector=0x0) at /home/jweil/gcc45/trunk/libgfortran/intrinsics/pack_generic.c:364

That line is:

      if (GFC_UNALIGNED_4(ret->data) || GFC_UNALIGNED_4(array->data)
	  || GFC_UNALIGNED_4(vector->data))
	break;

So, the problem is that we ask for the 'data' field in the optional VECTOR argument, which is not present here (i.e. vector=0x0)!
Comment 12 janus 2009-10-20 20:54:43 UTC
Here is a simple patch which cures the segfault in comment #9. However it does not tackle the double-free issue.


Index: libgfortran/intrinsics/pack_generic.c    
===================================================================
--- libgfortran/intrinsics/pack_generic.c       (Revision 153009)  
+++ libgfortran/intrinsics/pack_generic.c       (Arbeitskopie)     
@@ -350,7 +350,7 @@ pack (gfc_array_char *ret, const gfc_array_char *a

     case GFC_DTYPE_DERIVED_2:
       if (GFC_UNALIGNED_2(ret->data) || GFC_UNALIGNED_2(array->data)
-         || GFC_UNALIGNED_2(vector->data))
+         || (vector && GFC_UNALIGNED_2(vector->data)))
        break;
       else
        {
@@ -361,7 +361,7 @@ pack (gfc_array_char *ret, const gfc_array_char *a

     case GFC_DTYPE_DERIVED_4:
       if (GFC_UNALIGNED_4(ret->data) || GFC_UNALIGNED_4(array->data)
-         || GFC_UNALIGNED_4(vector->data))
+         || (vector && GFC_UNALIGNED_4(vector->data)))
        break;
       else
        {
@@ -372,7 +372,7 @@ pack (gfc_array_char *ret, const gfc_array_char *a

     case GFC_DTYPE_DERIVED_8:
       if (GFC_UNALIGNED_8(ret->data) || GFC_UNALIGNED_8(array->data)
-         || GFC_UNALIGNED_8(vector->data))
+         || (vector && GFC_UNALIGNED_8(vector->data)))
        break;
       else
        {
@@ -383,7 +383,7 @@ pack (gfc_array_char *ret, const gfc_array_char *a
 #ifdef HAVE_GFC_INTEGER_16
     case GFC_DTYPE_DERIVED_16:
       if (GFC_UNALIGNED_16(ret->data) || GFC_UNALIGNED_16(array->data)
-         || GFC_UNALIGNED_16(vector->data))
+         || (vector && GFC_UNALIGNED_16(vector->data)))
        break;
       else
        {
Comment 13 Jerry DeLisle 2009-10-21 03:04:26 UTC
With your patch, I am not seeing the double free. But I do get this:

             85078576
             85078520
             85078576
             85078576
           2           2
==27755== 
==27755== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 6 from 2)
==27755== malloc/free: in use at exit: 8 bytes in 2 blocks.
==27755== malloc/free: 18 allocs, 16 frees, 3,785 bytes allocated.
==27755== For counts of detected errors, rerun with: -v
==27755== searching for pointers to 2 not-freed blocks.
==27755== checked 89,864 bytes.
==27755== 
==27755== LEAK SUMMARY:
==27755==    definitely lost: 4 bytes in 1 blocks.
==27755==      possibly lost: 0 bytes in 0 blocks.
==27755==    still reachable: 4 bytes in 1 blocks.
==27755==         suppressed: 0 bytes in 0 blocks.
==27755== Rerun with --leak-check=full to see details of leaked memory.

using the original test case with print locs etc. Maybe my system is tolerant.
Comment 14 Thomas Koenig 2009-12-04 20:33:46 UTC
The problem is with the allocatable components for
intrinsics, at least.

This has the same problem:

program main

 type :: container_t
    integer, dimension(:), allocatable :: entry
 end type container_t

 type(container_t), dimension(2) :: a1, a2
 integer :: i

 do i = 1, 2
    allocate (a1(i)%entry (1), a2(i)%entry (1))
    a1(i)%entry = 1
    a2(i)%entry = 0
 end do

 a1 = eoshift(a2, 1)

end program main

What we must do is to deallocate anything on the
left hand side of the assignment, so instead of
using

  a1 = eoshift(a2,1)

we shoud loop over a1, deallocate everything, and then
do the assignment.

This looks like heavy front-end work, is anybody up for this? :-)
Comment 15 Thomas Koenig 2009-12-04 20:36:48 UTC
Very probably a dup of PR 40850.
Comment 16 Thomas Koenig 2009-12-04 21:03:43 UTC
We get this right on assignment, so it is probably "just" a matter
of copying over the logic from there.
Comment 17 janus 2009-12-06 22:57:28 UTC
Subject: Bug 41478

Author: janus
Date: Sun Dec  6 22:57:13 2009
New Revision: 155024

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=155024
Log:
libgfortran/
2009-12-06  Janus Weil  <janus@gcc.gnu.org>

	PR fortran/41478
	PR fortran/42268
	* intrinsics/pack_generic.c (pack): Add safety checks for the case that
	'vector' is NULL.


gcc/testsuite/
2009-12-06  Janus Weil  <janus@gcc.gnu.org>

	PR fortran/41478
	PR fortran/42268
	* gfortran.dg/intrinsic_pack_5.f90: New test.

Added:
    trunk/gcc/testsuite/gfortran.dg/intrinsic_pack_5.f90
Modified:
    trunk/gcc/testsuite/ChangeLog
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/intrinsics/pack_generic.c

Comment 18 janus 2009-12-13 21:51:01 UTC
r155024 contains the patch in comment #12, which fixes comment #9, but not the double free issue.
Comment 19 Paul Thomas 2010-01-11 06:14:30 UTC
Created attachment 19535 [details]
Fix and testcase for the PR

This will be submitted to the list tonight after checking that other transformational functions do not cause problems.

Cheers

Paul
Comment 20 Dominique d'Humieres 2010-01-11 12:29:55 UTC
The patch in comment #19, passes all my tests, but (otherwise you'ld be disappointed;-) compiling the reduced test in comment #8 gives a "Segmentation fault"

pr41478_1.f90:7:0: internal compiler error: Segmentation fault

Backtrace:

(gdb) run pr41478_1.f90
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /opt/gcc/gcc4.5w/libexec/gcc/x86_64-apple-darwin10/4.5.0/f951 pr41478_1.f90
 MAIN__
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
0x00000001000aeee1 in structure_alloc_comps (der_type=<value temporarily unavailable, due to optimizations>, decl=<value temporarily unavailable, due to optimizations>, dest=<value temporarily unavailable, due to optimizations>, rank=<value temporarily unavailable, due to optimizations>, purpose=3) at ../../work/gcc/fortran/trans-array.c:6014
6014		      tmp = gfc_duplicate_allocatable(dcmp, comp, ctype, c->as->rank);
(gdb) bt
#0  0x00000001000aeee1 in structure_alloc_comps (der_type=<value temporarily unavailable, due to optimizations>, decl=<value temporarily unavailable, due to optimizations>, dest=<value temporarily unavailable, due to optimizations>, rank=<value temporarily unavailable, due to optimizations>, purpose=3) at ../../work/gcc/fortran/trans-array.c:6014
#1  0x00000001000aed7c in structure_alloc_comps (der_type=<value temporarily unavailable, due to optimizations>, decl=0x141df5500, dest=<value temporarily unavailable, due to optimizations>, rank=<value temporarily unavailable, due to optimizations>, purpose=3) at ../../work/gcc/fortran/trans-array.c:5874
#2  0x00000001000c7ed6 in gfc_conv_procedure_call (se=0x7fff5fbfea70, sym=<value temporarily unavailable, due to optimizations>, arg=0x14180aef0, expr=<value temporarily unavailable, due to optimizations>, append_args=<value temporarily unavailable, due to optimizations>) at ../../work/gcc/fortran/trans-expr.c:3383
#3  0x00000001000d7abb in conv_generic_with_optional_char_arg (se=0x7fff5fbfea70, expr=0x141816b10) at ../../work/gcc/fortran/trans-intrinsic.c:3432
#4  0x00000001000d8aeb in gfc_conv_intrinsic_function (se=0x7fff5fbfea70, expr=0x141816b10) at ../../work/gcc/fortran/trans-intrinsic.c:5074
#5  0x00000001000c8e36 in gfc_conv_function_expr (se=0x7fff5fbfea70, expr=<value temporarily unavailable, due to optimizations>) at ../../work/gcc/fortran/trans-expr.c:3722
#6  0x00000001000ca508 in gfc_trans_assignment (expr1=0x1418165c0, expr2=0x141816b10, init_flag=0 '\0') at ../../work/gcc/fortran/trans-expr.c:4827
#7  0x00000001000a5c06 in gfc_trans_code (code=0x1418172a0) at ../../work/gcc/fortran/trans.c:1086
#8  0x00000001000c1d57 in gfc_generate_function_code (ns=<value temporarily unavailable, due to optimizations>) at ../../work/gcc/fortran/trans-decl.c:4383
#9  0x0000000100068c3f in gfc_parse_file () at ../../work/gcc/fortran/parse.c:4242
#10 0x00000001000a0d0c in gfc_be_parse_file (set_yydebug=<value temporarily unavailable, due to optimizations>) at ../../work/gcc/fortran/f95-lang.c:239
#11 0x00000001006cf01a in toplev_main (argc=2, argv=0x7fff5fbfee28) at ../../work/gcc/toplev.c:1053
#12 0x0000000100000d54 in start ()
Comment 21 Paul Thomas 2010-01-11 12:50:58 UTC
(In reply to comment #20)
> The patch in comment #19, passes all my tests, but (otherwise you'ld be
> disappointed;-) compiling the reduced test in comment #8 gives a "Segmentation
> fault"

Oh, that's weird!  Does the original testcase and the one supplied with the patch work?

Thanks

Paul 
Comment 22 Tobias Burnus 2010-01-11 13:06:03 UTC
(In reply to comment #20)
> The patch in comment #19, passes all my tests, but (otherwise you'ld be
> disappointed;-) compiling the reduced test in comment #8 gives a "Segmentation
> fault"
[...]
> ../../work/gcc/fortran/trans-array.c:6014
> 6014                  tmp = gfc_duplicate_allocatable(dcmp, comp, ctype,
> c->as->rank);

Because of that line: Dominique, do you have the draft patch for PR 42647 installed? If so, it might be the reason for the problem. Just to add another possibility for the failure.
Comment 23 Dominique d'Humieres 2010-01-11 13:50:34 UTC
(In reply to comment #21)
> Oh, that's weird!  Does the original testcase and the one supplied with the
> patch work?

Yes, it was under "all my tests".

(In reply to comment #22)
> Because of that line: Dominique, do you have the draft patch for PR 42647
> installed? If so, it might be the reason for the problem. Just to add another
> possibility for the failure.

Good catch! yes I do have the draft patch for PR 42647 (and others!). I was thinking of applying the patch to another tree, but I have not done yet.
Comment 24 Dominique d'Humieres 2010-01-11 14:04:16 UTC
(In reply to comment #23)
> > Because of that line: Dominique, do you have the draft patch for PR 42647
> > installed? If so, it might be the reason for the problem. Just to add another
> > possibility for the failure.

> Good catch! yes I do have the draft patch for PR 42647 (and others!). I was
> thinking of applying the patch to another tree, but I have not done yet.

I have reverted the patch for PR 42647 and I still see the segmentation fault (offset by -1 for the lines of trans-array.c in the backtrace).
Comment 25 Dominique d'Humieres 2010-01-11 17:50:55 UTC
I have applied the patch in comment #19 to a clean fortran-exp and I still see the segmentation fault when compiling the test in comment #8.
Comment 26 Paul Thomas 2010-01-11 18:23:14 UTC
(In reply to comment #25)
> I have applied the patch in comment #19 to a clean fortran-exp and I still see
> the segmentation fault when compiling the test in comment #8.
> 
Ahhhh....  that's because allocatable scalars are incompletely implemented.

I'll give it a try.

Paul
Comment 27 Paul Thomas 2010-01-14 06:17:51 UTC
Subject: Bug 41478

Author: pault
Date: Thu Jan 14 06:17:38 2010
New Revision: 155877

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=155877
Log:
2010-01-14  Paul Thomas  <pault@gcc.gnu.org>

        PR fortran/41478
        * trans-array.c (duplicate_allocatable):  Static version of
	gfc_duplicate_allocatable with provision to handle scalar
	components. New boolean argument to switch off call to malloc
	if true.
	(gfc_duplicate_allocatable): New function to call above with
	new argument false.
	(gfc_copy_allocatable_data): New function to call above with
	new argument true.
	(structure_alloc_comps): Do not apply indirect reference to
	scalar pointers. Add new section to copy allocatable components
	of arrays. Extend copying of allocatable components to include
	scalars.
	(gfc_copy_only_alloc_comp): New function to copy allocatable
	component derived types, without allocating the base structure.
	* trans-array.h : Add primitive for gfc_copy_allocatable_data.
	Add primitive for gfc_copy_only_alloc_comp.
	* trans-expr.c (gfc_conv_procedure_call): After calls to
	transformational functions with results that are derived types
	with allocatable components, copy the components in the result.
	(gfc_trans_arrayfunc_assign): Deallocate allocatable components
	of lhs derived types before allocation.
	

2010-01-14  Paul Thomas  <pault@gcc.gnu.org>

        PR fortran/41478
        * gfortran.dg/alloc_comp_scalar_1.f90: New test.
        * gfortran.dg/alloc_comp_transformational_1.f90: New test.

Added:
    trunk/gcc/testsuite/gfortran.dg/alloc_comp_scalar_1.f90
    trunk/gcc/testsuite/gfortran.dg/alloc_comp_transformational_1.f90
Modified:
    trunk/gcc/fortran/ChangeLog
    trunk/gcc/fortran/trans-array.c
    trunk/gcc/fortran/trans-array.h
    trunk/gcc/fortran/trans-expr.c
    trunk/gcc/testsuite/ChangeLog

Comment 28 Paul Thomas 2010-01-14 06:20:02 UTC
Fixed on trunk.

Thanks for the report

Paul