Checking only the O_RDONLY flag to open(2)

I'm checking the flags sent to the open(2) call against permissions I've set up in some meta files. The perms here are related to the octal values typically sent to calls like chmod. I want the if block to be entered when perms is not matched by the relevant flag.

if((perms == 4 && !(flags & O_RDONLY)) ||
   (perms == 2 && !(flags & O_WRONLY)) ||
   (perms == 6 && !(flags & O_RDWR))) 

I expected this to work, and it does just fine in the the O_WRONLY and O_RDWR. However, the actual value of O_RDONLY is 0, so the & operator will return false for every value. Unfortunately, removing the negation will lead to the undesired behavior of every perms value of 4 skipping the if block. How can I achieve my goal here?