I just published a comment from a reader that suggested that a congested area is any yellow colored portion of a VFR sectional chart. Neither the FAR nor the AIM define congested area and the key on the sectional does not state that yellow areas define congested. There have been recent enforcement actions described on the AOPA website where the FAA has punished pilots for flying too low over congested areas that would not have been identified in yellow on the chart.
I live in a neighborhood that has about 200 homes. There are about 3 to 4 homes per acre plus lots of streets. Adjacent to our neighborhood are several condominium complexes. There are thousands of cars and thousands of people. Any reasonable person flying over the area would think of this as a congested area. The Jacksonville sectional does not identify my area in yellow.
The key point is that the FAA does not clearly define "congested area" and the VFR charts cannot be used to reliably identify congested areas. Furthermore, the FAA's enforcement actions have lacked the consistency necessary to draw any conclusions about their definition of "congested". Consequently, common sense is the order of the day for defining congested areas.
The AOPA's member website has several discussions about minimum safe altitudes, FAR 91.13 and 91.119. "The FAA does not define congested area in the FARs or in the Aeronautical Information Manual. And, FAA interpretations and decisions issued by the National Transportation Safety Board in low-flight enforcement cases are not consistent for purposes of drafting a precise definition. Such a determination is usually decided on a case-by-case basis, and in the cases that we've seen, "congested" has been interpreted rather broadly. For example, a highway with moderate traffic was found to be "congested," as was a seaside area where 200 to 300 persons were sitting on the beach or bathing in the water. "
The bottom line is good judgement should be used in determining minimum safe altitudes and the only sure way to avoid an enforcement action is to err on the side of caution. Whether you think it is a congested area or not, that girl floating in her pool below you probably doesn't want you buzzing her only 500 feet above.