You'll definitely enjoy it much more if you can get there earlier, preferably Wednesday to soak up the atmosphere. Then you can sample night qualifying, the driver's parade, the open pits on Friday, or any number of other things.
Watch your speed on the way down, they have radar traps all...