Yeah, that's just fucking stupid. I love Idris Elba and think he would make a fantastic Bond, probably better than many, but that doesn't mean not wanting him in there is racist. I don't care much about Bond either way, as I've never been a fan and have seen very few of the movies, but the character is distinct and iconic, and I absolutely hate when they pull shit like this. Not because of racism, but because that's not who the character is. Is it "important"? No, it isn't. It's a fictional character and ultimately who really gives a shit, but at the same time, when a character is supposed to be something, I don't think it's cool to swap out race or gender or anything else. If the character isn't iconic, that's another thing. Or if race ultimately has no bearing on the story or character, and the character can suitably be played by anyone with the talent to do it. But in this case, while the latter may be true, the former is still a problem.
I get irritated enough just when they cast different people for the same roles in movies. When Batman all of a sudden has a different love interest in Dark Knight from the one he had in Batman Begins, that shit is irritating. It's jarring and unpleasantly inconsistent. Or when someone suddenly has a new girlfriend or daughter or husband in a TV series, even though it's supposed to be the same person.
That said, I think in this case it's somewhat more justifiable. Bond has never been taken all that seriously, and it's become a thing to have different people play him over the years. Idris Elba is a cool, suave dude, and one with some great acting chops that would probably make a Bond role for him a ton of fun. I can see the appeal given that the character himself, despite being ultra-iconic, isn't really that important as a person. His identity is more in what he does, not who he "is". Given that, in this case I might make an exception, but again, I've never been a fan so I'm probably not qualified to say that. I think the whole Bond thing is pretty stupid, to be honest.
Either way, though, I would absolutely not consider this sentiment racism. To want to protect a beloved character's identity is only natural, and wanting to change it is pretty unnatural. Wanting Bond to be a white Scot like he's always been is not a racist sentiment. That's who Bond is and always has been. That's going to be important to many people not so much as a virtue of their desire to see a particular race, but to maintain the integrity of a pre-existing character they love.