It was a reference to caning etc. at English public schools...

I wonder if perhaps the use of le vice anglais/the English vice to refer to homosexuality in VG is an indirect suggestion of the homerotic potentiality of the single-sex public school system?

I've heard the term as a reference to whipping/caning as well; also (rather unkindly!) as a reference to hypocrisy, both in the movie Wilde* and in conversation.

*Once I get home, I'd be glad to reference the script on that; I'm pretty sure it's in the scene where Oscar shows Queensbury's note to Ross.