Geographic Positions with Two-Digit Fractions

I’m going to keep this simple:

Let’s pretend that the earth is flat (1); for our purposes it clarifies nothing to introduce spherical trig.  

Here you see part of a grid that marks locations.  The horizontal lines are latitude lines.  They measure distances north and south.  The vertical lines are longitude lines.  They measure distances east and west.  

Now let’s say that the locational precision in our system is two decimal places.  That means that no position on our flat earth can be described with numbers more precise than one one hundredth of a degree.  All lat/lon values must be in one of these forms:

0.nn

N.nn

NN.nn

NNN.nn

Our system forbids anything else.  

So let’s say that the lines in our grid representation are exactly two fractional decimal places apart – a line (lat or lon) every one one hundredth (0.01) of a degree.

I’ve put in some suggested numbers for us to work with.  The first thing that you must notice is that unless a location is sitting on one of the vertices (that’s where the lines actually cross) it is NOT REPRESENTABLE in our system.

Let’s look at an example.  Let us suppose that there is a place called, oh I don’t know, let’s make up some unusual name like ‘Prophitis Elias’.  

Now PE is a disadvantaged site because it happens to be located exactly halfway between all the vertices;  at position 34.345 N and 22.835 E.  Our system only allows us two decimal points of representation and so the position of PE is rendered as 34.35, 22.84. (2)   That shifts the position of PI onto the vertex at that position.  This is a deliberately introduced mistake (called ‘aliasing’) that tries to keep the town of PE in the system.  But it’s still a falsification which arises out of the constraints of our system and so it’s up to users to determine how serious this aliasing is.

How serious is it?  How far is PE from its true position under this kind of aliasing?

I’m going to introduce some simplifying assumptions.   I calculate that the circumference of the earth at  35.34 N is 20,291.484 miles [3] and so one one hundredth of a degree longitude at that latitude is 0.56365 miles or 2976.0851 feet.

One one hundredth of a degree in latitude is 3652.14666 feet.  Since our town of PE is sitting exactly equal distances from the vertices the error would be the hypotenuse (A) of the triangle shown here

The aliasing error for the town of PE would be 1826.0733 feet in latitude (NS) and 1488.043 feet in longitude (EW).  These numbers,  1826.1 and 1488.0, are half of the distances mentioned just above because PE is half-way from all the vertices. The actual error (A) is merely the hypotenuse of the resulting right triangle.  Solving for A by the Pythagorean theorem gives us the maximum aliasing error in this system which is 2355.592 feet or 717.984 m.   So the maximum error in this system is almost  ¾ of a kilometer.  I emphasize that this is true only for this latitude; these errors would grow smaller towards the poles and larger towards the equator.

More important: what is the average error?  How far off will we be in the usual case?

The average error has to be less than the maximum error but how much less?  Is it half?

To answer this question I created a simulation and ran 1,000,000 trials several times.  It turns out that the error is not quite normally distributed (a little skewed to the low end) and the average (the arithmetic median) size of the error converges on ~ 1272 feet (388 m.)

Here's a bar chart of 5,000,000 runs.  Each cell from L to R represents an additional 73 m. in error.

To make a long story short – In a positional system limited to two decimal places in the fraction you can expect an aliasing error of nearly 400 m.

Who would create such a limited cockamamie geographical positioning system and expect it to be used for precise work such as describing the position of, oh I don't know, say something like Bronze Age find sites?

Who indeed?

Notes

     (1)    Did you hear the one about the Flat Earth Society circular that claims ‘Thousands of members around the globe’.  Around the Globe!  Get it?, get it?  O.k. back to work you lot!  
     
     (2)   Under normal rounding rules.


     (3)  For radius and circumference of the earth calculation at 35.34 N: