I had put nylgut 5b classic banjo red series strings on my Supertone Ragtime King banjo. While I was tuning, I noticed that the 5th string screw nut was cutting into the string. I eventually broke the string while tuning. Im changing to la bela nylon strings. Should I change the nut?
No different than the normal nut (or bridge), the notch should be fitted properly to the string you are using. Nylguts are well known to be sensitive to cuts and abrasion...the nut should have been filed or dressed to fit and to remove any sharp edges. The same really holds true for any of the plastic strings (such as La Bella or any other nylon/nylgut...or gut for that matter).
If it were my banjo, I'd get rid of the screw and replace the original (in its original position). Bone 5th-string nut blanks are available from Stewart MacDonald (www.stewmac.com) and take just a bit of file work to finish and fit. My guess would be that the original was ebony (or ebonized hardwood) as was typical of the day.
If you keep the screw, just make sure the string fits well in the slot and there are no sharp edges digging in. It will serve you for a long time.