How to convert an image to exactly Red, Green, and Blue

I'm curious.. I started with this image..

and was asked to clean it up. The Yellow was replaced with White. And the image should include only Red (#FF0000), Green (#00FF00), Blue (#0000FF), White (#FFFFFF), and Black (#000000).

This was the best I was able to do in a reasonably short amount of time using Photoshop's Color Selection tool and finally outputting it through the Save as Web window and explicitly removing all other colors from the GIF color table.

Is there an easier, faster way of doing this and perhaps ending with a better result without spending forever with the marque tool essentially outlining the whole thing?