Get out there and get riding.
I used to be nervous when I first started riding track, and that nervousnes about worrying about the other bikes made me the one that was more of a danger (switching lines so they could go around, only they don't expect you to, etc.)
Now, I'll go out and ride with the intermediate group that I can barely keep up to just for the experience.
Every so often I'll crash, usually it's from me pushing my limits, but that's just part of it and I get up and continue riding almost all of the time. There is a saying in motocross "If you aren't crashing, you're not going fast enough". In fact, I'm over due for a crash. I've managed to save all my little slip ups the last few weekends.
Yes, there are the hot headed guys out there, but they get found out very fast.
We had a guy a few weeks ago that was all over the place, not holding lines, crossing lines over jumps to "block" people, or he was trying to scrub, we're not sure. Nobody, even the fast guys could pass this jerk because he was extremely unpredictable, nobody could even get in close to him because it seemed like he was always ready to wipe out. I spent a few laps in practice on his rear end but it was too risky to even attempt a pass, I got around him after he looped himself out over a jump.
I thought maybe I was the only one that noticed this guy, but last week (about 3 weeks after) on the start gate, at least 6 of us were all talking about the guy and wondering why he hasn't been seen since that weekend. Everyone had the same complaint about the guy so I think the series organizers sorted the guy out.
Don't worry about the faster guys coming up behind you. Hold your lines and we will make the pass, it's good practice for us that way. Only here and there will I play dirty with a guy on practice days. Either he is a friend I know that we can trust each other and we find it fun. Sometimes there will be a newb playing a bit too dirty or dumb that I'll set straight on the track. Had an incident last week that made me go find the guy in the pits and explain some basic practice day track behaviour to him, but that's the first time I've ever had to do it.
I know I'm talking from the dirt world, but the street world would be very similar. We run seperate classes so nobody gets landed on or stuffed off the track (we don't get nice grass and gravel traps to slide in to, we get rocks and trees) and there is a watch out for those in over their head.
Make some great friends through riding, and it's always fun to have someone on the track to play with. Everyone is friendly and helpfull so don't be shy!