Libraries. There's a lot of core math and data structures for GIS. GEOS implements the basics of OpenGIS geometry: data types, simple calculations, etc. GDAL is the library + utilities for working with raster georeferenced data (maps). GDAL also includes OGR for working with vector data (shapefiles). Finally Proj is the library for translating data from one spatial reference system (SRID) to another. GEOS + GDAL + Proj give you most of the core library functions you need for working with geographic data. They're C libraries and fast but have bindings for more humane languages as well.

Data formats. GeoTIFF is a common format for raster data. There's a zillion others GDAL supports, but GeoTIFF is the native format. Esri Shapefile is a common format for vector data, but you also see a lot of GPX, KML, GeoJSON, etc.

Database. PostGIS is the most common OSgeo database. It adds a bunch of datatypes and functions to Postgres. There are also geo extensions for other databases like MySQL and Oracle. Often a relational database is unnecessary: files + GDAL go a long way.

Web development. There are lots of options for using geographic data in a web application. GeoDjango is a great choice for building a webapp. MapServer and MapNik are two options for rendering custom maps on a web server. I really like Polymaps for doing map display and compositing in Javascript.

Desktop applications. Not everyone wants to write code. QGis is a good desktop application for manipulating geographic data. There's also GRASS, OpenEV, and uDig. These programs are focused on displaying and editing geographic datasets, for making and updating custom maps by hand.

Datasets. OpenStreetMap is a huge resource for free data for road maps. geodata.gov is a portal for US government datasets where with enough digging you can find interesting data. There's a lot of other data scattered about, I haven't found a good catalog.

Community. OSGeo and OpenStreetMap are two centers of the open source geo hacking world. slashgeo is a good news site, as is the GIS forum. GIS StackExchange is looking promising.

I?m still new to all this so if I misstated or overlooked something please email me.