At Beloit Memorial High School Caldwell played on the school's 9-0 football team in 1972. He was a guard on the school's State Championship basketball team a few months later in 1973.
Those who knew him then say he was a soft-spoken, low-key guy.
"Very much so. I think that's one of the greatest qualities he had as a leader," said Steve Ferger who went to school with Caldwell. "He was a very quiet leader, but someone that everyone looked up to."
"Jim comes from a Christian background," says another former school-mate Mardie Farr. "Jim is focused on education and Christianity."
Caldwell was also a sprinter on the track team, and he was a member of the National Honor Society.
Caldwell's coaching career took him to several universities including his alma mater Iowa. Then he broke into the NFL and in 2009 he was named the head coach of the Colts. Three years ago he had the Colts in the Super Bowl. They lost to the Saints, but Caldwell lost with dignity.
After the Colts went 2-14 last season Caldwell was fired. Ravens coach John Harbaugh wasted no time hiring Caldwell to be the Ravens quarterbacks coach.
"There was no question that Jim was going to be a perfect fit for us," said Harbaugh this week during a Super Bowl press conference.
In December Caldwell was promotred from quarterbacks coach to offensive coordinator. With his play calling the Ravens offense took off. Caldwell made changes like using a moving pocket more, and allowing Joe Flacco to throw more deep balls.
"Coach Caldwell took over as offensive coordinator toward the end of the year and you could just see the comfort level of Joe and coach Caldwell. It's been tremendous, and Joe's game has really elevated since that change took place," said Ravens safety Sean Considine in a recent interview with WTVO.