How to Write a Good Systems Paper

I got this paper through a friend who got it from usenet. I have not
picked up the original article, and so I don’t know if my HTML mark up
properly reflects its structure. Anyway, it’s probably good enough.

History…

From vdveen@cs.utwente.nl Mon Jan 31 14:38:54 1994
Subject: writing better (system) papers
From: Hans van der Veen < vdveen @cs.utwente.nl >

This was picked up from a UT internal newsgroup. I think that the contents
may be interesting for all of us though:

Hans.

Article: 169 of inf.spa.pegasus
Newsgroups: inf.spa.pegasus
Path: infnews.cs.utwente.nl!lars
From: lars@pegasus.esprit.ec.org (Lars K. Vognild)
Subject: How (and How Not) to Write a Good Systems Paper

source: DEC SRC’s ftp area at
gatekeeper.dec.com
/lars

[This article first appeared in ACM SIGOPS Operating Systems Review,
Vol. 17, No. 3 (July, 1983), pages 35-40. The text in this file was
scanned using OCR technology and has been proofread, but some scanning
errors may remain. This document is being made available with the
permission of the authors.]

An Evaluation of the Ninth SOSP Submissions

or

How (and How Not) to Write a Good Systems Paper

Roy Levin and David D. Redell
Ninth SOSP Program Committee Co-chairmen

Introduction

On March 21, 1983, the program committee for the 9th Symposium
on Operating System Principles, having read the eighty-three papers
submitted, selected sixteen for presentation at the symposium. This
acceptance ratio of about one in five approximates those of past
SOSPs, although the number of submissions was somewhat lower than
in recent years. Several members of the program committee found
it surprisingly easy to separate the good papers from the bad ones;
indeed, the ten committee members quickly agreed on the disposition
of over 80% of the papers. As the acceptance ratio indicates, most
of these were rejections.

After the committee had completed its selection process, several
members expressed disappointment in the overall quality of the submissions.
Many of the rejected papers exhibited similar weaknesses, weaknesses
that the committee felt should have been evident to the authors.
In the hope of raising the quality of future SOSP submissions, and
systems papers generally, the committee decided to describe the criteria
used in evaluating the papers it received. This article combines
the criteria used by all of the members of the committee, not just
the authors.

To try to avoid sounding preachy or pedagogic, we have cast this
presentation in the first and second person and adopted a light,
occasionally humorous style. Nevertheless, the intent is serious:
to point out the common problems that appear repeatedly in technical
papers in a way that will make it easier for future authors to avoid
them. As you read this article, then, suppose yourself to be a prospective
author for the 10th SOSP or for TOCS. You’ve done some work you
would like to publish, so you sit down to write a paper. What questions
should you be asking yourself as you write? These are also the questions
that we, the reviewers of your paper, will be asking to determine
its suitability for publication.

Classes of Papers

Your paper will probably fall naturally into one of three categories:

It presents a real system, either by a global survey of an
entire system or by a selective examination of specific
themes embodied in the system.
It presents a system that is unimplemented but utilizes ideas
or techniques that you feel the technical community should
know.
It addresses a topic in the theoretical areas, for example,
performance modelling or security verification.

Obviously, a single set of evaluation criteria cannot be applied
uniformly across these categories; nevertheless, many criteria apply
equally well to all three. As we describe each one below, we will
try to emphasize the classes of papers to which it applies. Often
it will be evident from context.

Criteria for Evaluation of Submissions

Original Ideas

Are the ideas in the paper new? There is no point in submitting
a paper to a conference or journal concerned with original work unless
the paper contains at least one new idea.

How do you know? You must be familiar with the state of the
art and current research in the area covered by your paper in order
to know that your work is original. Perhaps the most common failing
among the submissions in the first category (real systems) was an
absence of new ideas; the systems described were frequently isomorphic
to one of a small number of pioneering systems well-documented in
the literature.

Can you state the new idea concisely? If your paper is to advance
the state of knowledge, your readers must be able to find the new
ideas and understand them. Try writing each idea down in a paragraph
that someone generally versed in the relevant area can understand.
If you can’t, consider the possibility that you don’t really understand
the idea yourself. When you have the paragraphs, use them in the
abstract for the paper.

What exactly is the problem being solved? Your readers cannot
be expected to guess the problem you faced given only a description
of the solution. Be specific. Be sure to explain why your problem
couldn’t be solved just as well by previously published techniques.

Are the ideas significant enough to justify a paper? Frequently,
papers describing real systems contain one or two small enhancements
of established techniques. The new idea(s) can be described in a
few paragraphs; a twenty-page paper is unnecessary and often obscures
the actual innovation. Since construction of a real system is a
lot of work, the author of the paper sometimes unconsciously confuses
the total effort with the work that is actually new. (“My team worked
on this system for two years and we’re finally done. Let’s tell the
world how wonderful it is.”) If the innovation is small, a small
paper or technical note in a suitable journal is more appropriate
than an SOSP submission.

Is the work described significantly different from existing
related work? An obvious extension to a previously published algorithm,
technique, or system, does not generally warrant publication. Of
course, the label “obvious” must be applied carefully. (Remember
the story of Columbus demonstrating how to make an egg stand on end
(by gently crushing it): “it’s obvious once I’ve shown you how”.)
You must show that your work represents a significant departure from
the state of the art. If you can’t, you should ask yourself why
you are writing the paper and why anyone except your mother should
want to read it.

Is all related work referenced and have you actually read the
cited material? You will have difficulty convincing skeptical readers
of the originality of your efforts unless you specifically distinguish
it from previously published work. This requires citation. Furthermore,
you will find it harder to convince your readers of the superiority
of your approach if they have read the cited works and you haven’t.

Are comparisons with previous work clear and explicit? You cannot
simply say: “Our approach differs somewhat from that adopted in the
BagOfBits system [3].” Be specific: “Our virtual memory management
approach uses magnetic media rather than punched paper tape as in
the BagOfBits system [3], with the expected improvements in transfer
rate and janitorial costs.”

Does the work comprise a significant extension, validation,
or repudiation of earlier but unproven ideas? Implementation experiences
supporting or contradicting a previously published paper design are
extremely valuable and worthy candidates for publication. Designs
are cheap, but implementations (particularly those based on unsound
designs) are expensive.

What is the oldest paper you referenced? The newest? Have
you referenced similar work at another institution? Have you referenced
technical reports, unpublished memoranda, personal communications?
The answers to these questions help alert you to blind spots in your
knowledge or understanding. Frequently, papers with only venerable
references repeat recently published work of which the author is
unaware. Papers with only recent references often “rediscover” (through
ignorance) old ideas. Papers that cite only unpublished or unrefereed
material tend to suffer from narrowness and parochialism. Remember
that citations not only acknowledge a debt to others, but also serve
as an abbreviation mechanism to spare your readers a complete development
from first principles. If the readers need to acquire some of that
development, however, they must be able to convert your citations
into source material they can read. Personal communications and
internal memoranda fail this test. Technical reports are frequently
published in limited quantities, out-of-print, and difficult to obtain.
Consequently, such citations as source material should be avoided
wherever possible.

Reality

Does the paper describe something that has actually been implemented?
Quite a few of the SOSP submissions proceeded for fifteen pages in
the present tense before revealing, in a concluding section (if at
all), that the foregoing description was of a hypothetical system
for which implementation was just beginning or being contemplated.
This is unacceptable. Your readers have a right to know at the outset
whether the system under discussion is real or not.

If the system has been implemented, how has it been used and
what has this usage shown about the practical importance of the ideas?
Once again, a multiple man-year implementation effort does not of
itself justify publication of a paper. If the implemented system
contains new ideas, it is important to explain how they worked out
in practice. A seemingly good idea that didn’t pan out is at least
as interesting as one that did. It is important to be specific and
precise. “Our weather prediction system is up and running and no
one has complained about its occasional inaccurate forecasts” is
much less convincing than “every time we fail to forecast rain, the
users hang their wet shirts over the tape drives to dry”. In the
latter case, at least we know that people are using and depending
on the system. If the system hasn’t been implemented, do the ideas
justify publication now? This can be a difficult question for an
author to answer dispassionately, yet any reviewer of the paper will
make this judgment. It is always tempting to write a design paper
describing a new system, then follow it up in a year or two with
an “experience” paper. The successful papers of this genre nearly
always include initial experience in the closing sections of the
design paper. The subsequent experience paper then deals with the
lessons learned from longer-term use of the system, frequently in
unanticipated ways. Reviewers are very skeptical of design-only
papers unless there are new ideas of obviously high quality.

Lessons

What have you learned from the work? If you didn’t learn anything,
it is a reasonable bet that your readers won’t either, and you’ve
simply wasted their time and a few trees by publishing your paper.

What should your readers learn from the paper? Spell out the
lessons clearly. Many people repeat the mistakes of history because
they didn’t understand the history book.

How generally applicable are these lessons? Be sure to state
clearly the assumptions on which your conclusions rest. Be careful
of generalizations based on lack of knowledge or experience. A particularly
common problem in “real system” papers is generalization from a single
example, e.g., assuming that all file system directories are implemented
by storing the directory in a single file and searching it linearly.
When stating your conclusions, it helps to state the assumptions
again. The readers may not have seen them for fifteen pages and
may have forgotten them. You may have also.

Choices

What were the alternatives considered at various points and
why were the choices made the way they were? A good paper doesn’t
just describe, it explains. Telling your readers what you did doesn’t
give them any idea how carefully considered your choices were. You
want to save future researchers from following the same blind alleys.
You also want to record potentially interesting side-streets you
didn’t happen to explore. Make sure to state clearly which is which.

Did the choices turn out to be right, and if so was it for the
reasons that motivated them in the first place? If not, what lessons
have you learned from the experience? How often have you found yourself
saying “this works, but for the wrong reason”? Such a pronouncement
represents wisdom (at least a small amount) that may benefit your
readers. Many papers present a rational argument from initial assumptions
all the way to the finished result when, in fact, the result was
obtained by an entirely different path and the deductive argument
fashioned later. This kind of “revisionist history” borders on dishonesty
and prevents your readers from understanding how research really
works.

Context

What are the assumptions on which the work is based? The skeptical
readers are unlikely to accept your arguments unless their premises
are stated. Make sure you get them all; it’s easy to overlook implicit
assumptions.

Are they realistic? For “unimplemented systems” papers, this
amounts to asking whether the assumptions of the design can hope
to support a successful implementation. Many paper designs are naive
about the real characteristics of components they treat abstractly,
e.g., communication networks or humans typing on terminals. For
theoretical studies, it must be clear how the assumptions reflect
reality, e.g., failure modes in reliability modelling, classes of
security threats in security verification, arrival distributions
in queuing systems.

How sensitive is the work to perturbations of these assumptions?
If your result is delicately poised on a tall tower of fragile assumptions,
it will be less useful to your readers than one that rests on a broader
and firmer foundation.

If a formal model is presented does it give new information
and insights? Simply defining a model for its own sake is not very
useful. One deep theorem is worth a thousand definitions.

Focus

Does the introductory material contain excess baggage not needed
for your main development? “Real system” papers are particularly
guilty of irrelevant description. If your subject is distributed
file systems, the physical characteristics of the connection between
computer and communication network are probably not germane. Avoid
the temptation to describe all major characteristics of your system
at the same level of depth. Concentrate instead on the novel or
unusual ones that (presumably) will be the focus of the original
technical content of the paper.

Do you include just enough material from previously published
works to enable your readers to follow your thread of argument?
Do not assume that they have read every referenced paper within the
last week and have them at their fingertips for instant reference.
If you want your readers to get past page three, avoid introductory
sentences of the form “We adopt the definition of transactions from
Brown [4], layering it onto files as described by Green [7, 18],
with the notions of record and database introduced by Black [10]
and White [12] and later modified by Gray [6]”. On the other hand,
don’t burden your readers unnecessarily with lengthy extracts or
paraphrases from cited works.

Presentation

Are the ideas organized and presented in a clear and logical way?
Are terms defined before they are used?
Are forward references kept to a minimum? Readers get annoyed
when they repeatedly encounter statements like “Each file consists
of a sequence of items, which will be described in detail in a later
section”. Your readers have to remember the technical term “item”,
but the term has no semantics yet. It’s all right to ask them to
do this once or twice, but only when absolutely necessary. Even
if you can’t afford the digression to explain item at this point,
give readers enough information to attach some meaning to the term:
“Each file consists of a sequence of items, variable-sized, self-identifying
bit sequences whose detailed interpretation will be discussed below
under `Multimedia Files’.” Your readers may not yet understand your
concept of files completely, but at least they have some glimpse
of the direction in which you are leading them.
Have alternate organizations been considered? Theoretical papers,
particularly of a mathematical character, are generally easier to
organize than papers describing systems. The expected sequence of
definition, lemma, theorem, example, corollary works well for deductive
argument, but poorly for description. In “real system” papers, much
depends on the intent: global survey or selective treatment. Frequently,
difficulties in organization result from the author’s unwillingness
to commit to either approach. Decide whether you are surveying your
system or focusing on a specific aspect and structure the paper
accordingly.
Was an abstract written first? Does it communicate the important
ideas of the paper? Abstracts in papers describing systems are sorely
abused. The abstract is more often a prose table of contents than
a precis of the technical content of the paper. It tends to come
out something like this: “A system based on Keysworth’s conceptualization
of user interaction [4] has been designed and implemented. Some preliminary
results are presented and directions for future work considered.”
Readers skimming a journal are unlikely to keep reading after that.
Avoid the passive voice (despite tradition) and include a simple
statement of assumptions and results. “We designed and implemented
a user interface following the ideas of Keysworth and discovered
that converting the space bar to a toe pedal increases typing speed
by 15%. However, accuracy decreased dramatically when we piped rock
music instead of Muzak into the office.” Leave discussion and argument
for the paper. It helps to write the abstract before the paper (despite
tradition) and even the outline, since it focusses your attention
on the main ideas you want to convey.
Is the paper finished? Reviewers can often help you to improve
your paper, but they can’t write it for you. Moreover, they can’t
be expected to interpolate in sections marked “to be included in
the final draft”. In a mathematical paper, a reviewer regards the
statement of a theorem without proof with suspicion, and, if the
theorem is intended to culminate prior development, with intolerance.
Similarly, in a paper describing a system, a reviewer cannot tolerate
the omission of important explanation or justification. Omitting
sections with a promise to fill them in later is generally unacceptable.

Writing Style

Is the writing clear and concise?
Are words spelled and used correctly?
Are the sentences complete and grammatically correct?
Are ambiguity, slang, and cuteness avoided?

If you don’t have sufficient concern for your material to correct
errors in grammar, spelling, and usage before submitting it for publication,
why should you expect a reviewer to read the paper carefully? Some
reviewers feel that this kind of carelessness is unlikely to be confined
to the presentation, and will reject the paper at the first inkling
of technical incoherence. Remember that you are asking a favor of
your reviewers: “Please let me convince you that I have done interesting,
publishable work.” Reviewers are more favorably disposed toward
you if they receive a clean, clear, carefully corrected manuscript
than if it arrives on odd-sized paper after ten trips through a photocopier
and looking like it was composed by a grade-school dropout. Even
if you aren’t particularly concerned with precise exposition, there
is certain to be someone in your organization who is. Give your
manuscript to this conscientious soul and heed the resulting suggestions.

Summary

These thirty-odd questions can help you write a better technical
paper. Consult them often as you organize your presentation, write
your first draft, and refine your manuscript into its final form.
Some of these questions address specific problems in “systems” papers;
others apply to technical papers in general. Writing a good paper
is hard work, but you will be rewarded by a broader distribution
and greater understanding of your ideas within the community of journal
and proceedings readers.

This article was made public by

Faculty of Computer Science / SPA
University of Twente, P.O. Box 217
7500 AE Enschede, The Netherlands

lars@pegasus.esprit.ec.org
Phone: +31-53-894177
Fax : +31-53-33895

HTML mark up by Arjan Kenter

And back to Arjan’s home page…

This page was written by Arjan Kenter.