I was able to do this by adding an Other Calendar to my Google Calendar (on the computer, not the phone). In the section on the left where it says "Other Calendars", click Add, then Browse Interesting Calendars. In the "More" section, there is a "Contact's Birthdays". Subscribe to that.
It does not seem to add your contact's
anniversaries, however.