First and foremost you need good fencing. Without good fencing, your goat raising venture will be short-lived. Trust me, we learned that lesson the hard way. If you haven't read the page where we discuss fencing, take a minute and do that.
Your goats will require some kind of shelter from the weather. Your locality and the weather will dictate exactly what you need to do. As a minimum, your goats need to be able to get out of the rain and cold winds. The opening to your shelter should face away from the prevailing wind. Do not build the shelter in a shallow where rain water will pool, if necessary, build a pad so that water will drain away from shelter.
You do not want to feed your goats on the ground. That means you will need a trough of some kind. Some folks will put a sheet of old roofing tin on the ground, that will work for a while, but it will soon be damaged to the point of being useless and it will be covered with goat poo. Not good. There are many ways to build simple and inexpensive feed troughs and they are commercially available. Your budget and imagination will dictate what is best for you. If you feed hay to your goats, it too should be up off the ground. Remember, goats love to climb up on top of everything, that includes your hay bale. Once they get up on top of the hay, they seem compelled to empty their bladder and bowels. 'Nuff said 'bout that.
Clean water is an absolute essential. The number of and size of your water troughs will be determined by the number of goats you intend to keep. We have several old bath tubs and a bunch of commercial troughs. They do require frequent cleaning. During warm weather, we clean the troughs with bleach to reduce algae build-up. If you have running water in your goat pasture, good for you. If not, don't be too concerned. We do not have running water where we have our goats. Our solution was to place a 275 gallon tote on a trailer. We fill the tote at our well and tow it to the goat pasture. Here again, your creative imagination comes into play.