I did some Spec 2006 runs, and one of the runs I was comparing normal speed with -ffast-math.  As expected a lot of SpecFP benchmarks were faster using the -ffast-math option.  However, 3 of the benchmarks were slower, including milc.

The two hotest functions seem to be slower using -ffast-math:

slowmath  fastmath  |   slowmath   fastmath  |  Function                  
========  ========  |   ========   ========  |  ========                  
17.5310%  17.4259%  |  5,489,872  5,580,840  |  mult_su3_na               
13.4533%  15.6660%  |  4,212,953  5,017,232  |  mult_adj_su3_mat_vec      

Function mult_su3_na, 17.43 - 17.53% of total in 2 files
--------------------------------------------------------

Percent   Percent   |   Samples    Samples   |  Line #  Filename  
slowmath  fastmath  |   slowmath   fastmath  |  Line #  Filename  
--------  --------  |  ---------  ---------  |  ------  ----------
 9.8993%  75.9216%  |    543,453  4,237,070  |      38  m_mat_na.c
65.7913%   2.8468%  |  3,611,855    158,877  |      33  m_mat_na.c
           6.9169%  |               386,020  |      40  m_mat_na.c
 1.1277%   6.8534%  |     61,915    382,477  |      48  m_mat_na.c
 5.6730%            |    311,441             |      49  m_mat_na.c
 3.8513%   2.1642%  |    211,433    120,784  |      46  m_mat_na.c
 3.5253%            |    193,542             |      45  m_mat_na.c
 2.8633%            |    157,188             |      41  m_mat_na.c
 2.1619%            |    118,689             |      39  m_mat_na.c
 1.4727%            |     80,847             |      43  m_mat_na.c
 1.1929%            |     65,486             |      36  m_mat_na.c
           1.1072%  |                61,790  |      44  m_mat_na.c

Function mult_adj_su3_mat_vec, 13.45 - 15.67% of total in 2 files
-----------------------------------------------------------------

Percent   Percent   |   Samples    Samples   |  Line #  Filename   
slowmath  fastmath  |   slowmath   fastmath  |  Line #  Filename   
--------  --------  |  ---------  ---------  |  ------  -----------
71.3008%  77.9373%  |  3,003,867  3,910,288  |     116  m_amatvec.c
12.1258%  13.8781%  |    510,853    696,296  |     126  m_amatvec.c
11.0201%            |    464,272             |     121  m_amatvec.c
           3.4868%  |               174,944  |     123  m_amatvec.c
 2.1345%   1.4971%  |     89,925     75,109  |     129  m_amatvec.c
 1.3833%            |     58,278             |     132  m_amatvec.c