Do you take off the yellow valve and white membrane piece flap you clean? I didn't realize that part needed to come off for cleaning, and to my horror, I discovered clumps of mold growing inside the yellow part. I ended up dumping a lot of stored breast milk (so hard to do!), and I cleaned the pieces by soaking them in vinegar, scrubbing them, and then soaking them again. You could probably use bleach, but I'm a fan of vinegar as a cleaning agent.
After I started taking off the yellow valves, I haven't had a problem with mold, and I've been able to get inside the connector to scrub out the dried milk.