Even if they lose four games or so he is not going to get fired. The program is not failing but it will depending on what UM wants. Do they want a coach that can get winning records but not consistently beat good teams? Do they want to win the Big Ten, or compete to win it, and go for national titles?
If they want to be more ambitious than sure, they may fire him. Not sure who they go after though to replace him. He was a very high profile get. And he is not doing that much for them in many respects. If they give him the axe than they are probably going to have to get somebody that may struggle to recruit for a bit and fall even further behind the 8-ball.
Unless they fire him and hire somebody who ends up being a great coach diamond in the rough it is not like great coaches are a dime a dozen.