Direct vs Iterative Methods

In class I mentioned some reasons for preferring iterative methods over direct methods; one example is that direct methods are basically cubic operations, i.e. they run in time O(n3) or O(mn2), whereas iterative methods rely on matvecs that take O(s*nnz(A)), where s is the number of iterations required (usually << min(n,m) ) and nnz(A) is the number of nonzeros in the matrix A.

A friend of mine (a Purdue alum!) works in a lab in France primarily on iterative methods (he’ll admit he’s biased) but he offered the following comparison:

Why iterative methods are better:

by David Imberti [with notes from kyle]

  1. They’re faster
  2. Communication, communication, communication [if parallelization is a concern]
  3. Stability
  4. Definable accuracy [remember from class the bound on convergence that uses the residual vector’s norm? - Kyle ]
  5. It’s what most biological processes do
  6. this ; ask someone who works at an oil company (from what one Purdue math alum told me) / Intel (from what Sameh Ahmed tells me)/ Boeing / etc. what they use (hint: it’s GMRES)

Of course, to be fair, why direct methods can be better:

  1. Decompositions tend to work on any input: Robustness (you don’t have to worry about the matrix you’re working on fitting into the convergence domains that iterative methods require)
  2. If you want to solve for many right-hand sides, a matrix decomposition can handle this more efficiently
  3. direct methods can be better on extremely unstable problems (a.k.a., when things get really bad, you do QR because it’s the most stable algorithm out there at the moment).
  4. Direct methods can better take advantage of certain exploitable structures (although, I would argue the structures that can be exploited by direct methods are shrinking, but this is more or less true for things like Toeplitz solvers)

Note: Dr. Imberti got his PhD from the math department here at Purdue, and worked with Profs. Jianlin Xia (of MA) and Ahmed Sameh (of CS).