I may be a little nervous with my son when he was younger, but I would have been tempted to take him as well. My wife doesnt care for motorcycling, so thats where the brakes come on
. No matter what, you are creating an experience with his father that he will remember for the rest of his life! When he is an old man, and your long gone, he will think about this trip and how much he misses you. I hope you have a ton of fun!