HTTPS encrypted traffic flows leak information on underlying contents through various statistical properties such as packet lengths and timing, enabling traffic fingerprinting attacks. Recent traffic fingerprinting attacks leveraged Convolutional Neural Networks (CNNs) to record very high accuracies undermining state-of-the-art defenses. In this paper, we analyze such CNNs to understand their inner workings which helps in building efficient traffic classifiers and effective defenses. First, we experiment on three datasets and show that website fingerprinting CNNs focus majorly on transitions between uploads and downloads in trace fronts while video fingerprinting CNNs focus more on finer shapes of periodic bursts. Next, we show that traffic fingerprinting CNNs exhibit transfer learning capabilities allowing identification of new websites with fewer data. We also demonstrate how traffic fingerprinting CNNs outperform Recurrent Neural Networks (RNNs) due to their resilience to random shifts in data, which is common in network traces. We further generalize these observations on other publicly available network traffic datasets. Leveraging our observations, we propose two new defenses against traffic fingerprinting. Our first defense FRONT-U, defends website visits by obfuscating transitions between uploads and downloads in trace fronts and provides similar privacy as the state-of-the-art defense FRONT, with half the data overhead. Our second defense STOMA, defends streaming traffic by obfuscating the finer sub-bursts within major bursts of a trace using only the nextfew seconds as opposed to using the entire trace as in the state-of-the-art.