Boris Johnson has been spending a lot of time out of the limelight, and Brits are not happy.