Based on this paper alone, we can establish the following:
Vitamin D is extremely safe up to doses of 4,000 IU per day
Vitamin D is probably safe up to doses of 10,000 IU, provided you are monitoring blood tests and keeping the blood levels within the optimal range of 50-80 ng/ml
400 IU per day is likely a sufficient dose for infants
600 IU is way too low for pregnant women and adults, even though the IOM suggests that it is sufficient
Blood testing is recommended to assure that you are within the optimal range.
The IOM provides no definitive evidence that higher doses of vitamin D are toxic; they are assuming toxicity based on a lack of evidence that doses higher than 4,000 IU per day are safe.