Can't anyone who knows the answer to a puzzle access my phone?
Yes. Semper is not a security feature, it is a learning tool. If you need to secure your device against unauthorized access, please use a security feature in addition to Semper. In fact, since you can always skip a puzzle or try all the possible answers, Semper does not even provide any security, if you are the only one who know the answer to a puzzle.