My experience with Cantonese
I had a similar experience with Cantonese. My father and mother both speak fluent English; the extended family with whom I have significant contact all speak English with varying degrees of fluency. But my mother's side of the family usually communicate in Cantonese - with some Mandarin and English thrown in. I don't think anyone ever directed any significant amount of Cantonese speech to me, so I wound up just learning English at home. Mandarin had to wait for school, and as for Cantonese - I now can understand it (with some help from context), but not speak it (apart from a few stock phrases).
I used to think to myself, "Why didn't I learn Cantonese when I had the chance? After all, I had all this linguistic input - maybe I just didn't put enough effort into it." And I guess I didn't, but it wasn't a conscious decision on my part. Some acquisition occurred, but not enough to make me fluent. If anything, I got only the parsing part of the grammar and not the generating part - now try devising a model of language processing that can take care of that! I don't think this is an uncommon situation. I think there are quite a lot of people who can understand certain languages but not speak them.
Anyway, I'm now trying to better my Cantonese. I've decided to try "leveraging" my knowledge of Mandarin Chinese to help me learn Cantonese. Even though they're far apart enough to be separate languages, their grammatical structure seems similar enough for me to do direct transliteration, as least for the time being. As for phonology, there are tables available for converting Cantonese syllables to Mandarin, vice versa, and even one for converting tones. The conversion isn't one-to-one by any measure, but there are patterns and principles to be found. For example, the final (i.e. rhyme) -im in Cantonese almost always converts to -ian in Mandarin.
The trouble with this approach, I've found, is that Cantonese is more conservative than Mandarin. In itself, this is not such a bad thing; however, Mandarin has undergone a lot of mergers, resulting in a much smaller syllabary. This means that a Mandarin syllable may correspond to many more Cantonese syllables than vice versa. So it's a lot easier to convert from Cantonese to Mandarin (fewer possibilities), than Mandarin to Cantonese (many more possibilities) - which is the direction I want!
To give the statistics: for any one Cantonese syllable, the mean number of corresponding Mandarin syllables is 2.47; for any one Mandarin syllable, the mean number of corresponding Cantonese syllables is 3.58 - a significant difference. For 60% of the Cantonese syllables, you have a 50-50 chance or better of guessing the correct Mandarin syllable; but for Mandarin syllables, it's only 46%. Only 2% of the Cantonese syllables have 7 or more possible correspondences, but an incredible 13% of the Mandarin syllables have 7 or more possible Cantonese correspondences. You begin to see the problem. Add to the fact that the original 8 tones of Middle Chinese merged into four (plus one neutral) in Mandarin, while Cantonese preserves 7 tones - gack!
So I don't know if this approach is really going to work, but it's fun going through the table I compiled from the tables linked to above and seeing all the regularities.