Images in this post were generated using Mathematica 8.0.0.0
"In a compartment model, what doe a determinant of 0 mean?"
On page 612, the book explains that there are three possibilities when the determinant is 0. Matter gets stuck in compartment 1, matter gets stuck in compartment 2, or the matter in the system is a constant.
How do you compute the Hessian matrix of a function $f$?
The formula for the Hessian is on page 551 and the components of the matrix are second derivatives of $f$. If we write $\mathrm{Hess}f(x,y)=\begin{bmatrix} a & b \\ c & d \end{bmatrix} $, then compute $a$ by taking two derivatives in $x$, compute $d$ by taking two derivatives in $y$, and compute $b$ and $c$ by taking one derivative in $x$ and then taking a second derivative in $y$.
"One of the exercises generated a value of 2 for every boundary. Does that mean it is a max and min, or there is none?"
You have to compare the value 2 to the value obtained at the critical point on the interior. If the value of the critical point is greater than 2, then the minimum is attained all along the boundary and the maximum was attained at the critical point. If the value of the critical point is less than 2, then the opposite. In the situation that there are multiple critical points, then you must compare all the values attained.
