Re: [PATCH] Fix OpenACC shutdown and PTX image unloading (PR65904)

[Thomas Schwinge <thomas at codesourcery dot com>]

Hi!

On Fri, 1 May 2015 10:47:19 +0100, Julian Brown <julian@codesourcery.com> wrote:
> This patch fixes PR65904, a double-free error that started occurring
> after recent libgomp changes to the way offload images are registered
> with the runtime.
> 
> Offload images now map all functions/data using just two malloc'ed
> blocks, but the function gomp_free_memmap did not take that into
> account, and treated all mappings as if they had their own blocks (as
> they do if created by gomp_map_vars): so attempting to free the whole
> map at once failed when it hit mappings for an offload image.
> 
> The fix is to split offload-image freeing out of GOMP_offload_unregister
> into a separate function, and call that from gomp_free_memmap for the
> given device before freeing the rest of the memory map.
> 
> The patch also fixes a thinko that was revealed in image unloading in
> the NVPTX backend. Tested for libgomp with PTX offloading.

Confirming that both nvptx (PR65904 fixed) and (emulated) intelmic (no
changes) testing look fine.

> OK for trunk?

> ChangeLog
> 
>     libgomp/

Don't forget to add the PR marker here.

> --- a/libgomp/plugin/plugin-nvptx.c
> +++ b/libgomp/plugin/plugin-nvptx.c
> @@ -1687,23 +1695,22 @@ GOMP_OFFLOAD_load_image (int ord, void *target_data,
>  }
>  
>  void
> -GOMP_OFFLOAD_unload_image (int tid __attribute__((unused)), void *target_data)
> +GOMP_OFFLOAD_unload_image (int ord, void *target_data)
>  {
> -  void **img_header = (void **) target_data;
> -  struct targ_fn_descriptor *targ_fns
> -    = (struct targ_fn_descriptor *) img_header[0];
>    struct ptx_image_data *image, *prev = NULL, *newhd = NULL;
>  
> -  free (targ_fns);
> +  nvptx_attach_host_thread_to_device (ord);

By the way, do we need to lock ptx_devices in
libgomp/plugin/plugin-nvptx.c:nvptx_attach_host_thread_to_device and
libgomp/plugin/plugin-nvptx.c:GOMP_OFFLOAD_openacc_create_thread_data?
(ptx_dev_lock?  If yes, its definition as well as instantiated_devices
should be moved to a more appropriate place, probably?)  Also, several
accesses to instantiated_devices are not locked by ptx_dev_lock but
should be, from my cursory review.

> --- a/libgomp/target.c
> +++ b/libgomp/target.c
> @@ -797,32 +797,79 @@ GOMP_offload_register (void *host_table, enum offload_target_type target_type,
>    gomp_mutex_unlock (&register_lock);
>  }
>  
> -/* This function should be called from every offload image while unloading.
> -   It gets the descriptor of the host func and var tables HOST_TABLE, TYPE of
> -   the target, and TARGET_DATA needed by target plugin.  */
> +/* DEVICEP should be locked on entry, and remains locked on exit.  */

(I'm not a native speaker, but would use what I consider to be more
explicit/precise language: Âmust be locked instead of Âshould beÂ.  I'll
be happy to learn should they mean the same thing?)

>  
> -void
> -GOMP_offload_unregister (void *host_table, enum offload_target_type target_type,
> -			 void *target_data)
> +static void
> +gomp_deoffload_image_from_device (struct gomp_device_descr *devicep,
> +				  void *host_table, void *target_data)
>  {

> +/* This function should be called from every offload image while unloading.

s%from%for%, I think?  (And, s%should%must%, again?)

> +   It gets the descriptor of the host func and var tables HOST_TABLE, TYPE of
> +   the target, and TARGET_DATA needed by target plugin.  */
> +
> +void
> +GOMP_offload_unregister (void *host_table, enum offload_target_type target_type,
> +			 void *target_data)
> +{

> -/* Free address mapping tables.  MM must be locked on entry, and remains locked
> -   on return.  */
> +/* Free address mapping tables for an active device DEVICEP.  This includes
> +   both mapped offload functions/variables, and mapped user data regions.
> +   To be used before shutting a device down: subsequently reinitialising the
> +   device will repopulate the offload image mappings.  */
>  
>  attribute_hidden void
> -gomp_free_memmap (struct splay_tree_s *mem_map)
> +gomp_free_memmap (struct gomp_device_descr *devicep)
>  {
> +  int i;
> +  struct splay_tree_s *mem_map = &devicep->mem_map;
> +
> +  assert (devicep->is_initialized);
> +
> +  gomp_mutex_lock (&devicep->lock);

Need to lock before first access to *devicep?

> +  
> +  /* Unmap offload images that are registered to this device.  */
> +  for (i = 0; i < num_offload_images; i++)
> +    {
> +      struct offload_image_descr *image = &offload_images[i];

Need to take register_lock when accessing offload_images?

> +      if (image->type == devicep->type)
> +        {
> +	  void *host_table = image->host_table;
> +	  void *target_data = image->target_data;
> +
> +	  gomp_deoffload_image_from_device (devicep, host_table, target_data);
> +	}
> +    }
> +
> +  /* Clear other mappings.  */
>    while (mem_map->root)
>      {
>        struct target_mem_desc *tgt = mem_map->root->key.tgt;
>  
>        splay_tree_remove (mem_map, &mem_map->root->key);
> -      free (tgt->array);
> +
> +      if (tgt->list_count == 0)
> +	assert (!tgt->array);
> +      else
> +	free (tgt->array);
> +
>        free (tgt);
>      }
> +
> +  gomp_mutex_unlock (&devicep->lock);
>  }


GrÃÃe,
 Thomas

Attachment: pgpxKqM4nkSI_.pgp
Description: PGP signature