Re: [patch, fortran] Asynchronous I/O, take 3

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

Rainer Orth-2
Hi Thomas,

> another test run on i386-pc-solaris2.11 is underway.  However, may
> (all?) gfortran tests now SEGV.  One example is

the good news is: all three libgomp.fortran/async_io_?.f90 tests now
PASS for both 32 and 64-bit, as do gfortran.dg/f2003_inquire_1.f03 and
gfortran.dg/f2003_io_1.f03.

        Rainer

--
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University
Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

Thomas König
Hi Rainer,

> However, may
> (all?) gfortran tests now SEGV.  One example is
>
> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
>
> Backtrace for this error:
> Segmentation Fault
>
> Thread 2 received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 1 (LWP 1)]
> 0xfe1b1f03 in pthread_mutex_unlock () from /lib/libc.so.1
> (gdb) where
> #0  0xfe1b1f03 in pthread_mutex_unlock () from /lib/libc.so.1
> #1  0xfe5d1b7c in __gthread_mutex_unlock (__mutex=0x18)
>      at ../libgcc/gthr-default.h:778
> #2  _gfortran_st_rewind (fpp=0xfeffda9c)
>      at /vol/gcc/src/hg/trunk/solaris/libgfortran/io/file_pos.c:486
> #3  0x0805110f in MAIN__ ()
>      at /vol/gcc/src/hg/trunk/solaris/gcc/testsuite/gfortran.dg/backslash_2.f90:6
Ah, I see what was wrong.

The attached patch should fix this.

I have also attached a new test case which detects this error
even on Linux systems, plus a ChangeLog which fixes the typo :-)

Again regression-tested.

So, OK for trunk?

Regards

        Thomas

2018-07-02  Nicolas Koenig  <[hidden email]>
         Thomas Koenig <[hidden email]>

         PR fortran/25829
         * gfortran.texi: Add description of asynchronous I/O.
         * trans-decl.c (gfc_finish_var_decl): Treat asynchronous variables
         as volatile.
         * trans-io.c (gfc_build_io_library_fndecls): Rename st_wait to
         st_wait_async and change argument spec from ".X" to ".w".
         (gfc_trans_wait): Pass ID argument via reference.

2018-07-02  Nicolas Koenig  <[hidden email]>
         Thomas Koenig <[hidden email]>

         PR fortran/25829
         * gfortran.dg/f2003_inquire_1.f03: Add write statement.
         * gfortran.dg/f2003_io_1.f03: Add wait statement.

2018-01-02  Nicolas Koenig  <[hidden email]>
         Thomas Koenig <[hidden email]>

         PR fortran/25829
         * Makefile.am: Add async.c to gfor_io_src.
         Add async.h to gfor_io_headers.
         * Makefile.in: Regenerated.
         * gfortran.map: Add _gfortran_st_wait_async.
         * io/async.c: New file.
         * io/async.h: New file.
         * io/close.c: Include async.h.
         (st_close): Call async_wait for an asynchronous unit.
         * io/file_pos.c (st_backspace): Likewise.
         (st_endfile): Likewise.
         (st_rewind): Likewise.
         (st_flush): Likewise.
         * io/inquire.c: Add handling for asynchronous PENDING
         and ID arguments.
         * io/io.h (st_parameter_dt): Add async bit.
         (st_parameter_wait): Correct.
         (gfc_unit): Add au pointer.
         (st_wait_async): Add prototype.
         (transfer_array_inner): Likewise.
         (st_write_done_worker): Likewise.
         * io/open.c: Include async.h.
         (new_unit): Initialize asynchronous unit.
         * io/transfer.c (async_opt): New struct.
         (wrap_scalar_transfer): New function.
         (transfer_integer): Call wrap_scalar_transfer to do the work.
         (transfer_real): Likewise.
         (transfer_real_write): Likewise.
         (transfer_character): Likewise.
         (transfer_character_wide): Likewise.
         (transfer_complex): Likewise.
         (transfer_array_inner): New function.
         (transfer_array): Call transfer_array_inner.
         (transfer_derived): Call wrap_scalar_transfer.
         (data_transfer_init): Check for asynchronous I/O.
         Perform a wait operation on any pending asynchronous I/O
         if the data transfer is synchronous. Copy PDT and enqueue
         thread for data transfer.
         (st_read_done_worker): New function.
         (st_read_done): Enqueue transfer or call st_read_done_worker.
         (st_write_done_worker): New function.
         (st_write_done): Enqueue transfer or call st_read_done_worker.
         (st_wait): Document as no-op for compatibility reasons.
         (st_wait_async): New function.
         * io/unit.c (insert_unit): Use macros LOCK, UNLOCK and TRYLOCK;
         add NOTE where necessary.
         (get_gfc_unit): Likewise.
         (init_units): Likewise.
         (close_unit_1): Likewise. Call async_close if asynchronous.
         (close_unit): Use macros LOCK and UNLOCK.
         (finish_last_advance_record): Likewise.
         (newunit_alloc): Likewise.
         * io/unix.c (find_file): Likewise.
         (flush_all_units_1): Likewise.
         (flush_all_units): Likewise.
         * libgfortran.h (generate_error_common): Add prototype.
         * runtime/error.c: Include io.h and async.h.
         (generate_error_common): New function.

2018-07-02  Nicolas Koenig  <[hidden email]>
         Thomas Koenig <[hidden email]>

         PR fortran/25829
         * testsuite/libgomp.fortran/async_io_1.f90: New test.
         * testsuite/libgomp.fortran/async_io_2.f90: New test.
         * testsuite/libgomp.fortran/async_io_3.f90: New test.
         * testsuite/libgomp.fortran/async_io_4.f90: New test.


> Obviously __mutex above hasn't been properly initialized.
>
>> 2018-07-02  Nicolas Koenig  <[hidden email]>
>>          Thomas Koenig <[hidden email]>
>>
>>          PR fortran/25829
>>          * testsuite/libgfomp.fortran/async_io_1.f90: New test.
>>          * testsuite/libgfomp.fortran/async_io_2.f90: New test.
>>          * testsuite/libgfomp.fortran/async_io_3.f90: New test.
>
> You seem to have a special fondness for libgfomp ;-)
>
> Rainer
>


p13h.diff (69K) Download Attachment
async_io_1.f90 (1K) Download Attachment
async_io_2.f90 (387 bytes) Download Attachment
async_io_3.f90 (374 bytes) Download Attachment
async_io_4.f90 (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

Rainer Orth-2
Hi Thomas,

> Ah, I see what was wrong.
>
> The attached patch should fix this.

it almost did the trick indeed.  With the new patch, only two failures
remain:

FAIL: gfortran.dg/flush_1.f90   -O0  execution test
FAIL: gfortran.dg/flush_1.f90   -O1  execution test
FAIL: gfortran.dg/flush_1.f90   -O2  execution test
FAIL: gfortran.dg/flush_1.f90   -O3 -fomit-frame-pointer -funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.dg/flush_1.f90   -O3 -g  execution test
FAIL: gfortran.dg/flush_1.f90   -Os  execution test
FAIL: gfortran.dg/newunit_1.f90   -O0  execution test
FAIL: gfortran.dg/newunit_1.f90   -O1  execution test
FAIL: gfortran.dg/newunit_1.f90   -O2  execution test
FAIL: gfortran.dg/newunit_1.f90   -O3 -fomit-frame-pointer -funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.dg/newunit_1.f90   -O3 -g  execution test
FAIL: gfortran.dg/newunit_1.f90   -Os  execution test

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
Segmentation Fault

Thread 2 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1 (LWP 1)]
0xfe1b1773 in mutex_lock_impl () from /lib/libc.so.1
(gdb) where
#0  0xfe1b1773 in mutex_lock_impl () from /lib/libc.so.1
#1  0xfe1b1a03 in pthread_mutex_lock () from /lib/libc.so.1
#2  0xfe5d1da4 in __gthread_mutex_lock (__mutex=0x18)
    at ../libgcc/gthr-default.h:748
#3  _gfortran_st_flush (fpp=0xfeffda40)
    at /vol/gcc/src/hg/trunk/solaris/libgfortran/io/file_pos.c:514
#4  0x0805106f in flush_1 ()
    at /vol/gcc/src/hg/trunk/solaris/gcc/testsuite/gfortran.dg/flush_1.f90:11
#5  0x080512a7 in main (argc=1, argv=0xfeffdbfa)
    at /vol/gcc/src/hg/trunk/solaris/gcc/testsuite/gfortran.dg/flush_1.f90:28
#6  0x08050de6 in _start ()

I've not yet investigated why they only occur for 32-bit.  I tried the
following snippet to fix them


--- libgfortran/io/file_pos.c.save 2018-07-03 22:48:42.485018736 +0000
+++ libgfortran/io/file_pos.c 2018-07-04 07:51:43.494582145 +0000
@@ -511,7 +511,8 @@ st_flush (st_parameter_filepos *fpp)
       else
  {
   needs_unlock = true;
-  LOCK (&u->au->io_lock);
+  if (u->au)
+    LOCK (&u->au->io_lock);
  }
 
       /* Make sure format buffer is flushed.  */


which did so indeed, but now there's another failure, again 32-bit only:

FAIL: gfortran.dg/eof_4.f90   -O0  execution test
FAIL: gfortran.dg/eof_4.f90   -O1  execution test
FAIL: gfortran.dg/eof_4.f90   -O2  execution test
FAIL: gfortran.dg/eof_4.f90   -O3 -fomit-frame-pointer -funroll-loops -fpeel-loo
ps -ftracer -finline-functions  execution test
FAIL: gfortran.dg/eof_4.f90   -O3 -g  execution test
FAIL: gfortran.dg/eof_4.f90   -Os  execution test

At line 12 of file /vol/gcc/src/hg/trunk/solaris/gcc/testsuite/gfortran.dg/eof_4.f90 (unit = 99)
Fortran runtime error: Cannot open file 'test.dat': File exists

Error termination. Backtrace:

        Rainer

--
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University
Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

Thomas Koenig-6
Hi Rainer,

thanks a lot for your testing!  This is really making a lot
of difference.

This patch should hopefully fix what you have experienced.
I've left out the boilerplate. The tests that failed for you
will be added to the libgomp.fortran directory as async_io_5.f90,
async_io_6.f90 and async_io_7.f90.

Regards

        Thomas

p13i.diff (69K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

Dominique d'Humières-2
In reply to this post by Rainer Orth-2
I have done a full regression testing with the patch at
https://gcc.gnu.org/ml/fortran/2018-07/msg00004.html + Rainer’s patch
at https://gcc.gnu.org/ml/fortran/2018-07/msg00007.html

I am currently testing the patch at
https://gcc.gnu.org/ml/fortran/2018-07/msg00008.html

so far, so good!

IMO the tests should go to gfortran.dg (they pass my tests).

I think there is a spurious STOP in libgomp.fortran/async_io_1.f90 and the last STOP 1 should be STOP 7, as in:

--- libgomp/testsuite/libgomp.fortran/async_io_1.f90 2018-07-03 12:26:41.000000000 +0200
+++ gcc/testsuite/gfortran.dg/asynchronous_6.f90 2018-07-05 12:57:35.000000000 +0200
@@ -21,7 +21,6 @@ program main
   write (10,'(A)', asynchronous=yes) 'asdf'
   write (10,*, asynchronous=yes) cc
   close (10)
-  stop
   open (20, file='a.dat', asynchronous=yes)
   read (20, *, asynchronous=yes) i, j
   read (20, *, asynchronous=yes) k, l
@@ -42,6 +41,6 @@ program main
   open(20, file='c.dat', asynchronous=yes)
   read(20, *, asynchronous=yes) res
   wait (20)
-  if (any(res /= is)) stop 1
+  if (any(res /= is)) stop 7
   close (20,status="delete")
 end program

Thanks for the hard work!

Dominique

Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

Rainer Orth-2
Hi Thomas,

>> I am currently testing the patch at
>> https://gcc.gnu.org/ml/fortran/2018-07/msg00008.html
>>
>> so far, so good!
>
>> IMO the tests should go to gfortran.dg (they pass my tests).
>
> I put the asycn_io_*.f90 tests into libgomp.fortran because,
> under Linux, gfortran.dg does not link in pthreads, so the
> tests would not be executed in parallel, and some of them
> would fail.
>
> So, here is the final version. I would really like to get this
> into trunk, and out of the way, so Nicolas and I can focus on
> other things.

I've now regtested the patch on i386-pc-solaris2.11 and
sparc-sun-solaris2.11: no regressions and the new tests all PASS.

However, I still don't understand why you insist on the hack with
putting the async_io_*.f90 tests into the libgomp testsuite.  Why not
just make the pthread requirement explicit with

{ dg-require-effective-target pthread }
{ dg-additional-options "-pthread" }

and put them in gfortran.dg where they belong?

        Rainer

--
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University
Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

jerry DeLisle-3
In reply to this post by Dominique d'Humières-2
On 07/15/2018 04:19 AM, Thomas Koenig wrote:

> Hi everybody,
>
>> I am currently testing the patch at
>> https://gcc.gnu.org/ml/fortran/2018-07/msg00008.html
>>
>> so far, so good!
>
>> IMO the tests should go to gfortran.dg (they pass my tests).
>
> I put the asycn_io_*.f90 tests into libgomp.fortran because,
> under Linux, gfortran.dg does not link in pthreads, so the
> tests would not be executed in parallel, and some of them
> would fail.
>
> So, here is the final version. I would really like to get this
> into trunk, and out of the way, so Nicolas and I can focus on
> other things.
>
> So, OK?

OK, Go for it.

Jerry

PS and big kudos for all the effort. This is a major improvement!
Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

Rainer Orth-2
In reply to this post by Rainer Orth-2
Hi Thomas,

>> However, I still don't understand why you insist on the hack with
>> putting the async_io_*.f90 tests into the libgomp testsuite.  Why not
>> just make the pthread requirement explicit with
>>
>> { dg-require-effective-target pthread }
>> { dg-additional-options "-pthread" }
>>
>> and put them in gfortran.dg where they belong?
>
> Because this does not appear to work with Linux. I, like
> most gfortran developers, work on Linux, and I would like to
> catch any failure during regression-testing on my own system,
> if possible.

huh, what doesn't work?  I've just finished an x86_64-pc-linux-gnu
bootstrap with your patch included, added the above to the
async_io_?.f90 tests, linked them to gfortran.dg and ran the tests there
(both 32 and 64-bit multilibs), all PASSed and I verified that they were
linked with -lpthread.

> We have had this discussion with Jakub, and he advised
> us to put all the stuff requiring pthreads into libgomp.

Do you have a pointer to that previous discussion?

> It is debatable if this is a good thing, or if we should
> at least make one round of tests with -pthread enabled.
> However, this is something for the future, and requires knowledge
> of dejagnu that I don't currently have :-)

First of all, we need to see and understand the failure mode, if any.
Making this work with the testsuite is a secondary matter only, and I
can certainly help with that if necessary.

        Rainer

--
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University
Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

jerry DeLisle-3
On 07/15/2018 10:47 AM, Rainer Orth wrote:

> Hi Thomas,
>
>>> However, I still don't understand why you insist on the hack with
>>> putting the async_io_*.f90 tests into the libgomp testsuite.  Why not
>>> just make the pthread requirement explicit with
>>>
>>> { dg-require-effective-target pthread }
>>> { dg-additional-options "-pthread" }
>>>
>>> and put them in gfortran.dg where they belong?
>>
>> Because this does not appear to work with Linux. I, like
>> most gfortran developers, work on Linux, and I would like to
>> catch any failure during regression-testing on my own system,
>> if possible.
>
> huh, what doesn't work?  I've just finished an x86_64-pc-linux-gnu
> bootstrap with your patch included, added the above to the
> async_io_?.f90 tests, linked them to gfortran.dg and ran the tests there
> (both 32 and 64-bit multilibs), all PASSed and I verified that they were
> linked with -lpthread.
>
>> We have had this discussion with Jakub, and he advised
>> us to put all the stuff requiring pthreads into libgomp.
>
> Do you have a pointer to that previous discussion?
>
>> It is debatable if this is a good thing, or if we should
>> at least make one round of tests with -pthread enabled.
>> However, this is something for the future, and requires knowledge
>> of dejagnu that I don't currently have :-)
>
> First of all, we need to see and understand the failure mode, if any.
> Making this work with the testsuite is a secondary matter only, and I
> can certainly help with that if necessary.
>
> Rainer
>

Hmm, interesting. Which linux are you using?

I will try it here as well.

Jerry
Reply | Threaded
Open this post in threaded view
|

Re: [patch, fortran] Asynchronous I/O, take 3

Rainer Orth-2
Hi Jerry,

> On 07/15/2018 11:46 AM, Rainer Orth wrote:
>> Hi Jerry,
>>
>>> Hmm, interesting. Which linux are you using?
>>
>> Fedora 27.
>
> Works for me. Fedora 28. Do not know for other OS's

just tried Solaris 11/x86: works just as well.  I could try Mac OS X
10.7, but that build would take quite a while...

        Rainer

--
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University