Ten Reasons You Can’t Trust Everything You Read About the Author Earnings Report

Expert publishing blog opinions are solely those of the blogger and not necessarily endorsed by DBW.

Last week I contested some of the key conclusions Hugh Howey reached through the data in his Author Earnings report. i09.com covered that discussion, and Howey took to that site’s comments section to register a rebuttal.

In doing so he put forth a number of mistaken claims about his Author Earnings report as well as the 2014 Digital Book World and Writer’s Digest Author Survey, which I coauthored.

Here are ten of them.

Fallacy #1:
“DBW released a survey that ignored the 99% of people who query along the traditional route and never get published at all. That is, they looked at the 1% who get published, and compared this to the 100% of self-published authors, all of whom get published.”

The Digital Book World and Writer’s Digest Author Survey was open to a range of authors and did not ignore anyone, deliberately or otherwise (more on this below).

In terms of our analysis, I have never presented on this website a comparison strictly of income of self-published vs. traditionally published authors based on the findings of that survey.

Rather, my examination of author income used entirely different categories. The charts I presented compare four types of authors: aspiring (not yet published), traditionally published only, indie only and hybrid (both traditionally published and indie published).

Howey has repeatedly objected to comparisons of self-publishing vs. traditional publishing that were simply never done on this site using the Author Survey. He claims we unfairly compared the income of two types of authors—indie vs. traditional. Yet we compared four types of authors, including the unpublished ones he says we ignored.

Howey mistakenly faulted the Author Survey for not surveying authors waiting in the slush pile when in fact we had. He contended that, as a result of ignoring these aspiring authors (which we did not), we misrepresented the success of indie publishing, making it seem less than it actually is. Howey’s contention has been that our analysis took the successful authors from traditional publishing and compared them side by side with the unwashed self-published masses. Howey’s suggested solution was to include slush-pile authors in the counts of traditionally published authors even though they are not published so that we would have a “fair” comparison with income from indie publishing.

What he suggests is bad research practice. Howey’s preferred method ignores the activities of hybrid authors and where they fall in a traditional vs. indie tally. Moreover, as I explain in my post on authors’ publishing decisions, his approach would require that we guess at what people will do (e.g. self-publish, traditionally publish, continue to wait and try to make it through the slush pile or give up altogether) and that we estimate that they are making no money whether or not they eventually will—an approach tantamount to making up data.

The decision in the Author Survey to separate out the aspiring authors and report on what respondents told us they did and earned has been criticized as an example of the study’s “flawed” or “biased” analysis. Yet the analysis is based on sound research methodology and an accurate reflection of the facts the authors themselves presented.

Fallacy #2:
“Focusing on the top 7K or 50K bestsellers is the best way to avoid this oranges/apples comparison. Here, we’re looking at the top 1.5% of both routes. It’s a fair comparison. Their survey wasn’t.”

A fair comparison of what—the top indie earners to the top traditionally published earners? This was never a question our research was trying to answer.

Moreover, Howey’s data deliberately ignores the 98.5% of authors published on Amazon who haven’t made it into the elite and fully neglects the unpublished in the slush pile or with books in their file drawers—those he supposedly champions by disparaging our methods.

Fallacy #3:
“Keep in mind that this rebuttal was written by someone who defends a survey that polled the readers of Writer’s Digest, and 40% of their respondents HAD NOT YET WRITTEN A NOVEL. 30% of those who had only had A SINGLE NOVEL under their belt. They have not been open and honest about their methodologies or their sampling bias.”

If you examine any of the posts I have written for DBW, you will see that we are very “open and honest” that the survey we conducted was non-scientific and that it was drawn from a voluntary sample. The survey was distributed more broadly beyond the readers of Writer’s Digest. Invitations went to Writer’s Digest mailing list and were also sent out to the Romance Writers of America and the Science Fiction and Fantasy Writers of America. It was also announced on Twitter.

The goal of the survey was to understand what authors—all kinds of authors and not merely the elite—want. The purpose was to provide useful information to publishers, self-publishing service providers and authors alike. The data we gathered and our analysis of it forthrightly bears out that purpose.

Fallacy #4:
“They sell the results for $300 a pop (pressure from indie authors has resulted in a recent price drop). They use the result to lure people to a conference in NYC.”

The results I have been posting on this blog have all been entirely free to the public. Phil Sexton, the publisher of Writer’s Digest, and I also did a free webinar on the findings in the Author Survey, and we also appeared on the Self-Publishing Roundtable to discuss the survey and its findings with the broader community. My comments on the report and on the survey have been posted on Slideshare and on my own website.

The report is priced highly and is targeted to publishers. It also includes only a fraction of the information we collected in the survey, much of which will be disseminated on this blog and in subsequent reports aimed at authors and others (as a great deal of it already has, for free).

Fallacy #5:
“It’s a racket, which is why they are coming after me.”

I regret if Howey felt victimized by the re-analysis of his data I undertook using widely accepted methods, following the invitation implied by his decision to make his numbers available. Far from “coming after” him, the post took his data collection exercise seriously and attempted to show its contribution to the limited knowledge we have at hand. (Mike Shatzkin has written thoughtfully on the issues confronting researchers seeking to compare traditional and indie publishing.) While I endorse neither Howey’s data nor his interpretation, my examination offers conclusions that may be safely drawn from the numbers provided should others chose to do so.

Fallacy #6:
“I, on the other hand, state every limitation and bias in my survey.”

Perhaps Howey believes he has covered all possible considerations, but that is simply not the case. Mike Shatzkin and I both point to non-overlapping concerns about the limits of the data and potential bias in the sample (which from a research standpoint is not technically a “survey”). Others have thoughtfully drawn attention to further issues with the data and the unsupported conclusions Howey draws from the charts and figures he presents in his report.

Fallacy #7:
“I am up-front in saying that publishing is not a gold rush, that success comes to very few and requires a lot of luck. I make the full data set available for free, so people can reach their own conclusions. I state in the survey that I may be wrong about all of this, that we need better data, that I hope we will move forward and discover great truths together, as a community.”

Such a desire could only be furthered by embracing a scientifically grounded critique and analysis of his data and using it as a launching point for additional study.

Fallacy #8:
“I do all of this at great expense to myself in time and money. Because I want authors to have more data at hand when making decisions. Better data than a 1% to 100% comparison among people who haven’t written a book or may have written one. Bad data that will set you back three benjamins.”

Howey’s data are a comparison of the top 1.5% to 4% of genre-fiction authors published on Amazon. This may be good data if that is the population you as a writer or publisher or service provider care about. Otherwise, it tells you very little about the experiences or earnings of the average published author and nothing at all about the average aspiring author.

As stated above, the Digital Book World and Writer’s Digest Author Survey does not compare the 1% (successfully traditionally published authors) to 100% (all self-published authors), nor was it designed to do so, and much of the analysis of the Author Survey has been reported on this website for free.

The data are not perfect, but they certainly aren’t “bad.” The careful and professional design of the questionnaire ensures that the information we collected meets a very high standard of data integrity, while the volume of responses make the findings worthwhile from a qualitative perspective. In other words, it is hard to dismiss as useless or “bad” 9,210 extensive and carefully crafted interviews with authors, whether or not they are the same authors of greatest interest to another researcher.

Fallacy #9:
“It’s not hard to see where this is coming from, and it’s not from a desire to help people.”

I am not an employee of Digital Book World or Writer’s Digest, contract or otherwise. Like Howey, I have engaged in designing and analyzing the survey without compensation because I am passionate about studying authors’ careers (more information about my research is available on my website). Working with Digital Book World’s editorial director Jeremy Greenfield and Writer’s Digest publisher Phil Sexton on the Author Survey provided an unprecedented opportunity to learn about the experiences of a wide range of authors and provide information to them not otherwise available—something I care about greatly as a fiction writer myself.

In any case, questioning my personal motives and integrity or those of Digital Book World or Writer’s Digest does nothing to ameliorate the data and analysis issues around Howey’s Author Earnings data and report.

Fallacy #10:
“It’s from a desire to protect a profit-making scheme of selling to publishers the news that their ship is unsinkable, that the noise they heard in the middle of the night was a brush with an iceberg, but everything is okay.”

The results from the Author Survey have not, in fact, delivered news to publishers that “their ship is unsinkable.” Quite the contrary. Taken together, the blog posts I’ve written so far as well as the report aimed at publishers show that traditional publishing alone is not the most promising route for authors and that authors’ experiences with traditional publishing call into question publishers’ usual value propositions.

Talks I delivered at Digital Book World 2014 and in the earlier mentioned Digital Book World webinar focused precisely on the challenges and threats self-publishing poses for publishers. On both occasions I proposed that publishers rethink the services and benefits they present to authors, or risk losing those authors to the improving terms indie publishing offers.

Those recommendations were based not only on sound data, including the Author Survey, but on an understanding of the insights that could and could not be drawn from them.

16 thoughts on “Ten Reasons You Can’t Trust Everything You Read About the Author Earnings Report

  1. Pingback: Faber Factory Ten Reasons You Can’t Trust Everything You Read About the Author Earnings Report - Faber Factory

  2. Marc Cabot

    Can you elaborate more on how a “non-scientific survey” generated “sound data?”

    In general, it really seems that basing logical conclusions on data which is demonstrably shaky is going around like a cold at Christmastime, from my perspective. However, to be fair, Howey’s *methods* seem pretty sound, they just might not support his conclusions as far as he wants to build them out in some cases. Contrariwise, I have seen some pretty incredible statistical analysis in some of the attempts to show that Traditional Publishing “stands as she stood, rock-bottomed and copper sheathed,” including acting as if Amazon does not exist and treating what are obviously non-normal distributions as if they were… normal distributions. Somewhere, Gauss weeps.

  3. Linton Robinson

    Normally, I would tend to accept research from an organization over that of a single author, for several reasons, most of them obvious. But in this case the organization is WRITER DIGEST. Which means corrupt, money-grubbing, and semi-competent. DBW is a great site, WD have been scumbags feeding off writers and pimping the publishing industry (very much including the most egregious of vanity mills) since at least the first issue I read back in the mid-sixties.
    So, yeah, it’s a very slanted and incomplete snowjob, as Howey says. And the idea of selling it for $300 is just so typical of WD. Remember when they had Jane Friedman giving $100 webinars on self-publishing when she HAD NEVER PUBLISHED A BOOK and had presided over WD’s hatchet campaign against SP for so many years?
    Howey’s research may or may not be flawed, but he is spot on about the WD jerk-job.

    1. Micah Ackerman

      Wait a minute, I want to know how DBW calculated the number or writers in an agents slush pile. You are saying your data includes aspiring authors, but how did you come up with the numbers for how many manuscripts actually make it to an agents or editors slush pile? There is a huge difference between aspiring author and those of us who have treated this like a career only to be turned back time and time again by traditional publishers. Hugh’s point is that many many people send out their manuscript wait and wait for an answer that never comes or gets a rejection and quits, those are the people you haven’t counted which should be included in the traditionally published pile, or else the trad pubs and aspiring authors should be lumped together to equal the indie’s. Also what about the slop, the free unedited works that are currently counted as self-published.

      Maybe you should use only professionally edited – self-published authors in the sample.


      1. Dana Beth WeinbergDana Beth Weinberg Post author

        Please see the post on how many authors publish: http://www.digitalbookworld.com/2014/2014-author-survey-from-manuscript-to-book-how-many-authors-actually-publish/. It shows the number of unpublished authors in the Author Survey sample who have completed manuscripts to agents and editors and also describes how many are wedded to a traditional publishing route only.

        Your other comments were addressed in the post above. What I will add is that even if the method Howey proposed could be done to high research standards, whether or not to do it would depend on what question you wanted to answer. It would seem from your comment that the question of interest to you is: Which route is better in terms of author income–traditional publishing including its risk of waiting in the slush pile and not selling–or self-publishing, which makes any work available, even free “slop”? This is not a question I have tried to answer. I don’t find it useful as an author deciding how I would like to sell my next book, which would require publishing it one way or another and assumes a choice (and I have yet to meet the author who believes her own writing is slop). I also happen to disagree with calling aspiring authors “published.” Finally, I don’t find a strict dichotomy between traditional and self-publishing helpful, as there seems to be some interesting possibilities and benefits inherent in combining modes of publishing, as Howey himself might attest given his recent print contract.

  4. Intellectual and Proud of It

    Thanks, Dana, for a clear, rational, logical and well-supported refutation of Howey’s fallacies.

    Of course, those of us who passed our math and reading comprehension skills tests in fifth grade already knew the truth about the DBW/WD survey and saw though Howey’s assertions for the horse hockey they are.

    However, I’m afraid trying to refute Howey is a waste of valuable time. Instead of celebrating that writers have choices and the choice of how to publish is as individual as the writer, Howey is turning this into a religious crusade. And like all proselytizers with an agenda to grind – I think what he really wants is for the entire city of New York to issue him an apology for not recognizing his genius earlier – he is relying on emotion and appeals to faith and monetary gain.

    Logic, hard math and empirical evidence? Them’s the devil’s work. The anti-intellectual attacks aimed at you and other social scientists who thoughtfully questioned his study are especially disheartening. Unfortunately, I expected the whiff of sexism that tinged the attacks because Howey is, if anything, consistent in clinging to his beliefs (see: http://www.salon.com/2013/04/15/self_publishing_star_faces_backlash_for_misogynist_rant_partner/ for more information).

    So while I’m sure there will be more Cult of Howey followers coming here any minute to raise their banner high and call you and your associated organizations all sorts of names (cueing the Kindleboards in 3…2…1…), those of use who have the intellectual capacity to see through Howey’s seductive but scientifically suspect snake oil thank you. It’s great to have data we can actually rely on when planning our writing careers.

    1. Rob Cornell

      Seeing as the Data Guy working with Hugh has an academic background with advanced degrees in hard-science and engineering from MIT & Stanford and professional experience doing this same kind of competitive analysis of App-store charts for leading game industry companies and online casinos, I highly doubt there is really any anti-intellectualism going on here. You just might not be as intellectual as you so proudly proclaim.

  5. Pingback: Ten Reasons You Can’t Trust Everything Yo...

  6. Pingback: Publishing Opinions | Ten Reasons You Can’t Trust Everything You Read About the Author Earnings Report

  7. Pingback: Ten Reasons You Can't Trust Everything You Read...

  8. Bob Mayer

    “Say not ‘I have found the truth,’ but rather ‘I have found a truth’.” Kahlil Gibran

    No one has all the “facts”. I don’t report my data to anyone. I have over two million eBook sales.

    This whole thing is reaching ludicrous speed.

  9. Pingback: AuthorEarnings.com: Pros & Cons | CalamusCalamus

  10. Theresa M. Moore

    Similar refutations have been published elsewhere, so you are not alone in seeing the serious flaws of posting a report which does not contain actual data. Inductive logic does not reflect real world paradigms, so assuming that a small percentage of the book world which experiences success based on numbers alone is assuming that all of it does. This is not the case, and I wish people would stop using Amazon as the control group. Anyone with any degree of practicality would see that sales of ebooks do not count as sales of paper books, so lumping them together smacks of sloppy data gathering. One can have “sold” millions of ebooks without selling a single physical copy of the same title. That is what counts here, and then there are the millions of freebies, which many authors are notorious for claiming success on. With freebies, there is no guarantee that the book will be read. As I noted in my blog http://www.antellus.com/blog.html many freebie collectors do not even read what they have gathered. So once again, only real anecdotal data would make the report accurate.

  11. Pingback: Wise Bear Digital Digest: Vol. 2, Issue No. 23

  12. Intellectual and Proud of It

    You’re a little late in commenting, Rob. What happened, did the fitting for the Cult of Howey robes take longer than expected?

    I’m referring to comments like this, left on the The Cult of Howey’s primary messaging center (i.e. Howey’s blog): \I don’t understand the goals driving people who feel only they, with their degrees and experience, are qualified to quantify data. Please.\ And this is just one of dozens of similar comments left all over the internet by people whose tongues are glued to Hugh’s posterior.

    So yes, please. How dare people who are trained to work with numbers actually, y’know, work with numbers. Let’s take seriously the insults to Ms. Weinberg and other statisticians from people who probably think balancing their checkbook means making sure it doesn’t fall off the stack of unpaid bills.

    That’s what I mean by anti-intellectualism. Howeyists are attacking Ms. Weinberg and other people who find serious flaws with Howey’s report solely because the skeptics state their degrees and data analysis experience, as if that automatically discounts everything they have to say. Meanwhile, they failing to ask the provenance of their own data \analysis.\

    Howey only whispered Coder Guy’s background (because Howey obviously has no academic or real world qualifications in data analysis) so his faithful could counter the growing online outrage at how Ms. Weinberg and others are being unfairly and fallaciously maligned by him. And Coder Guy’s credentials remain as murky as San Francisco fog to people who don’t brave the Kool-Aid infested waters of Kindleboards and/or doubt Howey’s ability to say anything that isn’t spun to his agenda.

    Nice try, too bad, don’t forget to pick up your parting gifts on the way out. Howeyists should really enjoy them: I hear they include a pan for gold, some pixie dust designed to remove the logic centers of the brain, and lots and lots of blinders.

  13. Pingback: Show Me the Money


Your email address will not be published. Required fields are marked *