That depends - some of them are run by families sort of like a Bed&Breakfast, and they usually have signs by the road side that tell you if they have available capacity. Usually, the sign reads "Hytte" if capacity is available, and if not, the word "Hytte" is crossed out (there are variations on this them). In these cases, no advance reservation is usually required, but "common decency" should be observed as to how late one would want to knock on the door and ask about renting the cabin. In other locations, hyttes are part of campgrounds that usually have a front office with hours of operation. These are generally closer to towns/cities, and may require advance booking. Also note that in Norway, wild camping is allowed, but there are certain rules that apply, eg. no open fires, make sure you do not disturb anyone when you set up your camp (don't camp in somebody's back yard or wheat field), and most importantly: leave no trace behind, i.e. make sure that you take everything with you that you used when camping.