Testing compiler reliability using Csmith

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Testing compiler reliability using Csmith

Radu Ometita
Hello everyone!

We are working on writing a paper about testing the reliability of C compilers by using Csmith (a random C99 program generator).

A previous testing effort, using Csmith, found 79 GCC bugs, and 25 of those have been marked by developers as P1 (https://www.flux.utah.edu/download?uid=114 <https://www.flux.utah.edu/download?uid=114>): . However, after this paper was published we are unaware of any further testing using Csmith, and we would like to ask you, if you are aware of any such efforts or further results.

Best regards,
Radu Ometita,
Functional compilers engineer @IOHK

Reply | Threaded
Open this post in threaded view
|

Re: Testing compiler reliability using Csmith

Andi Kleen-5
Radu Ometita <[hidden email]> writes:

> Hello everyone!
>
> We are working on writing a paper about testing the reliability of C compilers by using Csmith (a random C99 program generator).
>
> A previous testing effort, using Csmith, found 79 GCC bugs, and 25 of
> those have been marked by developers as P1
> (https://www.flux.utah.edu/download?uid=114
> <https://www.flux.utah.edu/download?uid=114>): . However, after this
> paper was published we are unaware of any further testing using
> Csmith, and we would like to ask you, if you are aware of any such
> efforts or further results.

Sameeran has been doing some additional testing with modified csmith.

There's currently no systematic effort to run csmith regularly
to find regressions.

-Andi
Reply | Threaded
Open this post in threaded view
|

Re: Testing compiler reliability using Csmith

Jeff Law
On 12/6/18 5:02 PM, Andi Kleen wrote:

> Radu Ometita <[hidden email]> writes:
>
>> Hello everyone!
>>
>> We are working on writing a paper about testing the reliability of C compilers by using Csmith (a random C99 program generator).
>>
>> A previous testing effort, using Csmith, found 79 GCC bugs, and 25 of
>> those have been marked by developers as P1
>> (https://www.flux.utah.edu/download?uid=114
>> <https://www.flux.utah.edu/download?uid=114>): . However, after this
>> paper was published we are unaware of any further testing using
>> Csmith, and we would like to ask you, if you are aware of any such
>> efforts or further results.
>
> Sameeran has been doing some additional testing with modified csmith.
>
> There's currently no systematic effort to run csmith regularly
> to find regressions.
That's my understanding as well.

I also got the impression that they'd already hit a significant
inflection point with csmith as the bugs it was finding were ultimately
duplicates of issues it'd already found.  csmith was going to need some
significant development to find new ways to stress compilers.

I have a lot of respect for John's work. It's too bad I don't see him
more often.  He's only 15 minutes up the road working with my old group
at the U.

jeff
Reply | Threaded
Open this post in threaded view
|

Re: Testing compiler reliability using Csmith

Kyrill  Tkachov-2
In reply to this post by Radu Ometita
Hi Radu,

On 06/12/18 22:10, Radu Ometita wrote:
> Hello everyone!
>
> We are working on writing a paper about testing the reliability of C compilers by using Csmith (a random C99 program generator).
>
> A previous testing effort, using Csmith, found 79 GCC bugs, and 25 of those have been marked by developers as P1 (https://www.flux.utah.edu/download?uid=114 <https://www.flux.utah.edu/download?uid=114>): . However, after this paper was published we are unaware of any further testing using Csmith, and we would like to ask you, if you are aware of any such efforts or further results.
>

We've had a large amount of really good bug reports come out of the research at:
https://people.inf.ethz.ch/suz/emi/index.html

If I understand the research correctly, the seed programs are either csmith-generated or taken from the GCC testsuite
and the mutations applied to them expose compiler bugs.

Thanks,
Kyrill

> Best regards,
> Radu Ometita,
> Functional compilers engineer @IOHK
>

Reply | Threaded
Open this post in threaded view
|

Re: Testing compiler reliability using Csmith

sameeran joshi
In reply to this post by Jeff Law
Hi, we had reported 1 issue

(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87118)

Using the extended csmith for gcc c language extensions.
We are currently working on adding more extensions to it and fuzzing in
background.

Thanks,
Sameeran joshi

On Fri 7 Dec, 2018, 9:07 PM Jeff Law, <[hidden email]> wrote:

> On 12/6/18 5:02 PM, Andi Kleen wrote:
> > Radu Ometita <[hidden email]> writes:
> >
> >> Hello everyone!
> >>
> >> We are working on writing a paper about testing the reliability of C
> compilers by using Csmith (a random C99 program generator).
> >>
> >> A previous testing effort, using Csmith, found 79 GCC bugs, and 25 of
> >> those have been marked by developers as P1
> >> (https://www.flux.utah.edu/download?uid=114
> >> <https://www.flux.utah.edu/download?uid=114>): . However, after this
> >> paper was published we are unaware of any further testing using
> >> Csmith, and we would like to ask you, if you are aware of any such
> >> efforts or further results.
> >
> > Sameeran has been doing some additional testing with modified csmith.
> >
> > There's currently no systematic effort to run csmith regularly
> > to find regressions.
> That's my understanding as well.
>
> I also got the impression that they'd already hit a significant
> inflection point with csmith as the bugs it was finding were ultimately
> duplicates of issues it'd already found.  csmith was going to need some
> significant development to find new ways to stress compilers.
>
> I have a lot of respect for John's work. It's too bad I don't see him
> more often.  He's only 15 minutes up the road working with my old group
> at the U.
>
> jeff
>