These comments relate to the “After Dark” show that followed immediately. It does not have its own post here at asymco.com so I put this comment here.

I used to know about survey sampling a long time ago. I’m sure that other readers would be more helpful. 

It’s absolutely correct that the process of drawing the sample is hugely important. And a large sample size, by itself, is insufficient. The classic case study is the presidential poll of 1948, now known by the headline, “Dewey Defeats Truman.”  (http://www.chicagotribune.com/news/politics/chi-chicagodays-deweydefeats-story,0,6484067.story)

Afterwards, the pollsters did a post-mortem. The poll was a questionaire that was distributed to subscribers of a monthly magazine. I think sample size was 50,000 but it turned out that its readership was dominated by rich old republicans; the results of that poll simply reflected who was polled. Blue-collar families didn’t read that magazine. That’s the lesson: The poll lets you “learn” about the characteristics of the sample you draw. 

After that, pollsters developed more sophisticated sampling techinques. I think the first was “stratified random sampling” (http://en.wikipedia.org/wiki/Stratified_sampling). You divide up the population into subgroups in proportion to their numbers and then work hard to sample each subgroup (This avoids the problem of oversampling rich Republicans). 

Nowadays, a very fine-tuned sample of about 1,500 is enough, but the secret of exactly how the sample is drawn is well guarded by the commercial pollsters. 

Doing this really well is a big deal and probably not worth the time and effort until Dan takes a rough cut at it. Early on, the particulars of the sample design have to be bootstrapped since you will get a progressively better idea which strata to use. You should also be guided by the [few] research questions that you have. 

One thing that distinguishes the podcast world from traditional media is that podcasting permits and sometimes encourages direct feedback as a part of its ongoing work. Dan doesn’t encourage it as much as Horace does here, but I suppose a first step is to use the info to be gathered already to inform about which strata and data collection method to use. (Personally, I subscribe using Downcast, so I would not acquire a poll/podcast bundle if it is a separate app. It’s no good to exclude subscribers unless the purpose of that poll is only to learn about non-subscribers.) Also, 5by5 continually acquires data from sponsors in the form of all those promo codes. 

I suppose an initial step is to just think through the questions you want answers to. For ex, I wonder whether there is any reason to offer video podcasts again. Dan knows that the download numbers were small. I think its because an audio podcast is “hired” to be used in a multitasking environment (e.g., driving, gardening, exercising). 

A video podcast would have to be “hired” for a different job.  I don’t even know of any such job that would give Dan the revenue stream that he already enjoys from audio. Perhaps video is a pay podcast, while audio remains ad-supported. I donno, but a survey would be a good way to explore some possibilities.