The caulk might have been a factory installed sealant or gasket material for the hull. This would have helped seal everything. Whether or not this is the case, you can use something called 3M 5200. It comes in two sizes. I got the bigger caulk sized one at Home Depot for $10. It takes a while to cure, so keep it dry for several days.
Also use it on your new transom wood and holes you make with the new bolts and stuff. It will seal it completely. Make sure not to use pressure treated wood, use a quality water sealer as well.