Here's a helpful entry on "grot shop" and related usages:
noun. Trash, garbage, or filth. British slang. Famously, in the 1970s BBC comedy "The Fall and Rise of Reginald Perrin," Reggie opened a doomed (he hoped) shop that sold old bits of trash and scrap recycled into completely useless items. He called it "The Grot Shop." (To Reg's horror, the shop was wildly successful, leading to a string of Grot Shops and, eventually, Grot Industries.)
Something that is "grotty" has a look of filth about it. (This last is well familiar to me from novels and telly.)
According to the interwebs, a "grot mag" is simply a porn mag. The kind featuring lurid shots of tatas, bums and gaping fannies.