Almost the same list in frequency order

A friend, Charles Frankel, found this list in an introductory book on codebreaking. I acquired it from him. He assured me he had extensively proofed it. Its use there is in pattern matching to get hints to the contents by matching code patterns to these words. Its use in a simple substitution code should be obvious. I had a chance to read his source book.

As that was in the late 1980s the following is from fallable memory. This is from something called the Brown Corpus, or Brown's Corpus, produced by Brown University in the early 1960s. It was to establish the most common American English words. All possible sources were used for words. I did not read the formal rules. Candidly I think the English Department wanted to play with that new-fangled computer the University just bought.

From reading the list all the letters are included so clearly they did not define a word by other than what appears between whitespace. Further words like youve are on the list so they got that from you've so contractions were words.

If you look further at the list you find kennedy is on it. That indicates Kennedy was still president or recently assassinated. So this list is not appropriate for today. There are other such words on the list.

In other words whatever use you may have for this list be certain to tailor it to your needs. Do not use it blindly.

As to the needs, when indexing a document automatically eliminating these words can save a lot of disk space. That is also why to review it carefully as even in the early 60s one would not wish to eliminate the word kennedy but only the rarest of documents would want to index the letter Z.

By this method apparently used even X-Files become xfiles but if your method converts it to x files you want to eliminate x and files in not on the list. But today, or depending upon the source, file and files are much more common words. There is a standard AWK teaching script that gives you control of how to do all that.

Off and on over the years I have used it to keep only the common words and look for patterns that might distinguish between authors. That was on the observation that can write about reparing a car and debugging a program and the uncommon words would be entirely different but the common word patterns might be extremely stable.

That is essentially worthless directly but were it to work there are many documents in history where authorship is unknown or in question. If a data base of common words from the times in question could be compiled then perhaps this method could be applied. FWIW, I have had modest success sprinkled with grossly erratic results and I would like to have this list in freqency order to test it that way. I would also like to find a list of the second thousand most common words. In any event, there is no paper in the foreseeable future.

Another use I have considered is determining the authorship of source code using the reserved words as the common words. That would note preferences for different types of loops and such. But that relies upon access to large sections of source code. With GNU that is possible but large segments of source code are from multiple authors and contributors. So I do not have access to material even to test it much less verify it. Anyone with the test materials, feel free.

I presume everyone knows how to cut and paste so Without further ado, the list. [A drumroll please]

To show the problem with common words there is an extraordinary number of hits on this file from the search engines. I presume many of them are on now uncommon words such as KENNEDY related common words. That means you must intelligently edit it before using.

A
ABLE
ABOUT
ABOVE
ACCORDING
ACCOUNT
ACROSS
ACT
ACTION
ACTIVITIES
ACTIVITY
ACTUALLY
ADDED
ADDITION
ADDITIONAL
ADMINISTRATION
AFTER
AGAIN
AGAINST
AGE
AGO
AHEAD
AID
AIR
ALL
ALMOST
ALONE
ALONG
ALREADY
ALSO
ALTHOUGH
ALWAYS
AM
AMERICA
AMERICAN
AMONG
AMOUNT
AN
ANALYSIS
AND
ANOTHER
ANSWER
ANTI
ANY
ANYONE
ANYTHING
APPARENTLY
APPEAR
APPEARED
APPROACH
ARE
AREA
AREAS
ARMS
ARMY
AROUND
ART
AS
ASK
ASKED
ASSOCIATION
AT
ATTACK
ATTENTION
AUDIENCE
AVAILABLE
AVERAGE
AWAY
B
BACK
BAD
BALL
BASED
BASIC
BASIS
BE
BEAUTIFUL
BECAME
BECAUSE
BECOME
BED
BEEN
BEFORE
BEGAN
BEGINNING
BEHIND
BEING
BELIEVE
BELOW
BEST
BETTER
BETWEEN
BEYOND
BIG
BILL
BLACK
BLOOD
BLUE
BOARD
BODY
BOOK
BORN
BOTH
BOY
BOYS
BRING
BRITISH
BROUGHT
BROWN
BUILDING
BUILT
BUSINESS
BUT
BY
C
CALL
CALLED
CAME
CAN
CANNOT
CANT
CAR
CARE
CARRIED
CARS
CASE
CASES
CAUSE
CENT
CENTER
CENTRAL
CENTURY
CERTAIN
CERTAINLY
CHANCE
CHANGE
CHANGES
CHARACTER
CHARGE
CHIEF
CHILD
CHILDREN
CHOICE
CHRISTIAN
CHURCH
CITY
CLASS
CLEAR
CLEARLY
CLOSE
CLOSED
CLUB
CO
COLD
COLLEGE
COLOR
COME
COMES
COMING
COMMITTEE
COMMON
COMMUNIST
COMMUNITY
COMPANY
COMPLETE
COMPLETELY
CONCERNED
CONDITIONS
CONGRESS
CONSIDER
CONSIDERED
CONTINUED
CONTROL
CORNER
CORPS
COST
COSTS
COULD
COULDNT
COUNTRIES
COUNTRY
COUNTY
COUPLE
COURSE
COURT
COVERED
CUT
D
DAILY
DARK
DATA
DAY
DAYS
DE
DEAD
DEAL
DEATH
DECIDED
DECISION
DEEP
DEFENSE
DEGREE
DEMOCRATIC
DEPARTMENT
DESCRIBED
DESIGN
DESIGNED
DETERMINED
DEVELOPED
DEVELOPMENT
DID
DIDNT
DIFFERENCE
DIFFERENT
DIFFICULT
DIRECT
DIRECTION
DIRECTLY
DISTANCE
DISTRICT
DO
DOES
DOING
DONE
DONT
DOOR
DOUBT
DOWN
DR
DRIVE
DUE
DURING
E
EACH
EARLIER
EARLY
EARTH
EAST
EASY
ECONOMIC
EDUCATION
EFFECT
EFFECTIVE
EFFECTS
EFFORT
EFFORTS
EIGHT
EITHER
ELEMENTS
ELSE
END
ENGLAND
ENGLISH
ENOUGH
ENTIRE
EQUIPMENT
ESPECIALLY
ESTABLISHED
EUROPE
EVEN
EVENING
EVER
EVERY
EVERYTHING
EVIDENCE
EXAMPLE
EXCEPT
EXISTENCE
EXPECT
EXPECTED
EXPERIENCE
EXTENT
EYE
EYES
F
FACE
FACT
FAITH
FALL
FAMILY
FAR
FARM
FATHER
FEAR
FEDERAL
FEED
FEEL
FEELING
FEET
FELT
FEW
FIELD
FIGURE
FIGURES
FILLED
FINAL
FINALLY
FIND
FINE
FIRE
FIRM
FIRST
FISCAL
FIVE
FLOOR
FOLLOWED
FOLLOWING
FOOD
FOOT
FOR
FORCE
FORCES
FOREIGN
FORM
FORMER
FORMS
FORWARD
FOUND
FOUR
FREE
FREEDOM
FRENCH
FRIEND
FRIENDS
FROM
FRONT
FULL
FUNCTION
FURTHER
FUTURE
G
GAME
GAVE
GENERAL
GENERALLY
GEORGE
GET
GETTING
GIRL
GIRLS
GIVE
GIVEN
GIVES
GLASS
GO
GOD
GOING
GONE
GOOD
GOT
GOVERNMENT
GREAT
GREATER
GREEN
GROUND
GROUP
GROUPS
GROWING
GROWTH
GUN
H
HAD
HAIR
HALF
HALL
HAND
HANDS
HAPPENED
HARD
HAS
HAVE
HAVING
HE
HEAD
HEAR
HEARD
HEART
HEAVY
HELD
HELL
HELP
HER
HERE
HERSELF
HES
HIGH
HIGHER
HIM
HIMSELF
HIS
HISTORY
HIT
HOLD
HOME
HOPE
HORSE
HOSPITAL
HOT
HOTEL
HOUR
HOURS
HOUSE
HOW
HOWEVER
HUMAN
HUNDRED
HUSBAND
I
IDEA
IDEAS
IF
ILL
IM
IMAGE
IMMEDIATELY
IMPORTANT
IN
INCLUDE
INCLUDING
INCOME
INCREASE
INCREASED
INDEED
INDIVIDUAL
INDUSTRIAL
INDUSTRY
INFLUENCE
INFORMATION
INSIDE
INSTEAD
INTEREST
INTERNATIONAL
INTO
INVOLVED
IS
ISLAND
ISSUE
IT
ITS
ITSELF
IVE
J
JOB
JOHN
JUST
JUSTICE
KEEP
KENNEDY
KEPT
KIND
KNEW
KNOW
KNOWLEDGE
KNOWN
L
LABOR
LACK
LAND
LANGUAGE
LARGE
LARGER
LAST
LATE
LATER
LATTER
LAW
LAY
LEAD
LEADERS
LEARNED
LEAST
LEAVE
LED
LEFT
LENGTH
LESS
LET
LETTER
LETTERS
LEVEL
LIFE
LIGHT
LIKE
LIKELY
LINE
LINES
LIST
LITERATURE
LITTLE
LIVE
LIVED
LIVING
LOCAL
LONG
LONGER
LOOK
LOOKED
LOOKING
LOST
LOT
LOVE
LOW
LOWER
M
MADE
MAIN
MAJOR
MAKE
MAKES
MAKING
MAN
MANNER
MANS
MANY
MARCH
MARKET
MARRIED
MASS
MATERIAL
MATTER
MAY
MAYBE
ME
MEAN
MEANING
MEANS
MEDICAL
MEET
MEETING
MEMBER
MEMBERS
MEN
MERELY
MET
METHOD
METHODS
MIDDLE
MIGHT
MILES
MILITARY
MILLION
MIND
MINUTES
MISS
MODERN
MOMENT
MONEY
MONTH
MONTHS
MORAL
MORE
MORNING
MOST
MOTHER
MOVE
MOVED
MOVEMENT
MOVING
MR
MRS
MUCH
MUSIC
MUST
MY
MYSELF
N
NAME
NATION
NATIONAL
NATIONS
NATURAL
NATURE
NEAR
NEARLY
NECESSARY
NEED
NEEDED
NEEDS
NEGRO
NEITHER
NEVER
NEW
NEXT
NIGHT
NO
NON
NOR
NORMAL
NORTH
NOT
NOTE
NOTHING
NOW
NUCLEAR
NUMBER
NUMBERS
OBTAINED
OBVIOUSLY
OF
OFF
OFFICE
OFTEN
OH
OLD
ON
ONCE
ONE
ONES
ONLY
OPEN
OPENED
OPERATION
OPPORTUNITY
OR
ORDER
ORGANIZATION
OTHER
OTHERS
OUR
OUT
OUTSIDE
OVER
OWN
P
PAID
PAPER
PART
PARTICULAR
PARTICULARLY
PARTS
PARTY
PASSED
PAST
PATTERN
PAY
PEACE
PEOPLE
PER
PERFORMANCE
PERHAPS
PERIOD
PERSON
PERSONAL
PERSONS
PHYSICAL
PICTURE
PIECE
PLACE
PLACED
PLAN
PLANE
PLANNING
PLANS
PLANT
PLAY
POINT
POINTS
POLICE
POLICY
POLITICAL
POOL
POOR
POPULATION
POSITION
POSSIBLE
POST
POWER
PRESENT
PRESIDENT
PRESS
PRESSURE
PRICE
PRINCIPLE
PRIVATE
PROBABLY
PROBLEM
PROBLEMS
PROCESS
PRODUCTION
PRODUCTS
PROGRAM
PROGRAMS
PROGRESS
PROPERTY
PROVIDE
PROVIDED
PUBLIC
PURPOSE
PUT
QUALITY
QUESTION
QUESTIONS
QUITE
R
RACE
RADIO
RAN
RANGE
RATE
RATHER
REACHED
REACTION
READ
READING
READY
REAL
REALLY
REASON
RECEIVED
RECENT
RECENTLY
RECORD
RED
RELIGION
RELIGIOUS
REMEMBER
REPORT
REPORTED
REQUIRED
RESEARCH
RESPECT
RESPONSIBILITY
REST
RESULT
RESULTS
RETURN
RETURNED
RIGHT
RIVER
ROAD
ROOM
RUN
RUNNING
S
SAID
SALES
SAME
SAT
SAW
SAY
SAYING
SAYS
SCHOOL
SCHOOLS
SCIENCE
SEASON
SECOND
SECRETARY
SECTION
SEE
SEEM
SEEMED
SEEMS
SEEN
SELF
SENSE
SENT
SERIES
SERIOUS
SERVED
SERVICE
SERVICES
SET
SEVEN
SEVERAL
SHALL
SHE
SHORT
SHOT
SHOULD
SHOW
SHOWED
SHOWN
SIDE
SIMILAR
SIMPLE
SIMPLY
SINCE
SINGLE
SITUATION
SIX
SIZE
SLOWLY
SMALL
SO
SOCIAL
SOCIETY
SOME
SOMETHING
SOMETIMES
SOMEWHAT
SON
SOON
SORT
SOUND
SOUTH
SOUTHERN
SOVIET
SPACE
SPEAK
SPECIAL
SPECIFIC
SPIRIT
SPRING
SQUARE
ST
STAFF
STAGE
STAND
STANDARD
START
STARTED
STATE
STATEMENTS
STATES
STAY
STEP
STEPS
STILL
STOCK
STOOD
STOP
STOPPED
STORY
STRAIGHT
STREET
STRENGTH
STRONG
STUDENT
STUDENTS
STUDY
SUBJECT
SUCH
SUDDENLY
SUMMER
SUN
SUPPORT
SURE
SURFACE
SYSTEM
SYSTEMS
T
TABLE
TAKE
TAKEN
TAKING
TALK
TAX
TECHNICAL
TELL
TEMPERATURE
TEN
TERM
TERMS
TEST
TH
THAN
THAT
THATS
THE
THEIR
THEM
THEMSELVES
THEN
THEORY
THERE
THEREFORE
THERES
THESE
THEY
THING
THINGS
THINK
THINKING
THIRD
THIRTY
THIS
THOSE
THOUGHT
THREE
THROUGH
THROUGH
THROUGHOUT
THUS
TIME
TIMES
TO
TODAY
TOGETHER
TOLD
TOO
TOOK
TOP
TOTAL
TOWARD
TOWN
TRADE
TRAINING
TREATMENT
TRIAL
TRIED
TROUBLE
TRUE
TRUTH
TRY
TRYING
TURN
TURNED
TWENTY
TWO
TYPE
TYPES
U
UNDER
UNDERSTAND
UNDERSTANDING
UNION
UNITED
UNIVERSITY
UNTIL
UP
UPON
US
USE
USED
USING
USUALLY
VALUE
VALUES
VARIOUS
VERY
VIEW
VOICE
VOLUME
WAITING
WALKED
WALL
WANT
WANTED
WAR
WAS
WASHINGTON
WASNT
WATER
WAY
WAYS
WE
WEEK
WEEKS
WELL
WENT
WERE
WEST
WESTERN
WHAT
WHATEVER
WHEN
WHERE
WHETHER
WHICH
WHILE
WHITE
WHO
WHOLE
WHOM
WHOSE
WHY
WIDE
WIFE
WILL
WILLIAM
WINDOW
WISH
WITH
WITHIN
WITHOUT
WOMAN
WOMEN
WORD
WORDS
WORK
WORKED
WORKING
WORKS
WORLD
WOULD
WOULDNT
WRITING
WRITTEN
WRONG
WROTE
YEAR
YEARS
YES
YET
YORK
YOU
YOUNG
YOUR
YOURE