Showing posts with label Minoan. Show all posts
Showing posts with label Minoan. Show all posts

Wednesday, August 8, 2018

The new Chronological Report Page on Helladic.info

The helladic.info site is pleased to announce support for chronological table creation and chronological ordering of sites in the DB.  A new database (Rev. 0.064) and new software has been delivered to implement this addition to the functionality.

The new Chrono report generates a table of occupation times based on your choices from the selection control on the Controls page. The control page allows you to select a combination of Region, Ceramic Horizon, or Type. After making your selection click on the 'Report' button. This brings you to the main report page. That page now has a 'Chrono' button. When you click on that button you will be given a chronological table on which your selection sites will be plotted with 'x's in the appropriate time span columns. Let's look at an example.




From the controls page I select 'Building' under the type label. Then press 'Report'. You'll be taken to the main report page:



You'll see the new 'Chrono Report' button. When you press that you'll be taken to the new Chrono page:



In the left-hand column are the unique place key identifiers for all the places in your selection set. They are clickable. When you click on one you'll be taken to the specific Place Key Report page for that site.

The second column is the place name. The names in this column are mouseable. When you run the mouse over them the region name will pop up in a tool-tip.




Succeeding columns, 'EH', 'EH I', 'EH II' etc., are the ceramic horizon columns. If there is evidence for your site having existed, been used, etc. during any specific ceramic horizon then there will be an 'x' in the box for that ceramic horizon. Here we can see that C1164, Troullos on Kea, existed in the LH I and the LH II.




The column headers with the ceramic horizon names are mouseable. When you do mouse over them the approximate year range will pop up in a tool tip.






It must be emphasized that these ranges are very approximate and scholars differ over what the proper year ranges should be for any specific ceramic horizon. Please keep this in mind. As I learn better ranges for specific ceramic horizons then I will probably change these tool tips.


The 'x' characters in the table boxes are also mouseable. When you do pass the mouse over them then the ceramic horizon label will pop up. This is intended to make it easier to work with the table when the column headers are out of sight.





When creating this table I faced the choice of making the table pageable (column headers never out of sight) or sortable. I chose sortable as probably the more useful alternative. When you click on the column heads the table will be re-sorted to put your criterion at the top. Because spaces in the cells sort before the character 'x' by default you'll probably have to click twice on any specific column head to get the effect that you want.

Also keep in mind that there are really two chrono pages, one in the helladic name style: 'EH', 'MH', 'LH' etc. and one in the Minoan name style 'EM', 'MM', 'LM', etc. These pages function differently. If your chosen data set has sites from Crete then the default or initial chrono page will show few or no 'x' characters in the table.



In order to see a meaningful chart you will switch to the 'Minoan Scale' page. Probably the most disorienting thing that a user can do is to make some region choice such as 'Crete' and then be faced with an empty chart and not know why. Remember to switch scales. The button to do this is at the upper right of the page and labelled 'Minoan Scale'.




Also keep in mind that some sites, particularly in the islands, have ceramic horizons in both styles. You will need to look at such sites under both scales. If you are on the Minoan Scale page there is also a button, in the same place, for switching back to Helladic.

I have not yet implemented a page for the Cycladic time scale.


There are buttons on the chrono pages that allow you to go to the elevation report page or the gazetteer report page. Those pages now also support buttons that allow you to go to the Chrono page. This should make multivariate analysis more convenient.



The tables, like any .html table, can be selected with the mouse, in whole or in part, and dropped directly onto an Excel spreadsheet. Here is an example of a table from the chrono page dropped onto Excel:


After you have dropped this table into Excel you can change the formatting or delete it completely as you choose.   But that's another topic.

~~~~~~~~~~~~~~~~~~~~~~~~

With a correspondent I've been discussing the question of whether putting exact locational information online is a good thing or a bad thing.  I'd like to hear from my readers on this question - do you have qualms about putting this kind of information online?  You can respond to this post or use my e-mail which is bobconsoli at gmail dot com.





If you like these posts then please follow me on Twitter (Squinchpix) or Google Plus (Robert Consoli).  Please do this.

You can e-mail me (and I hope you will) at  bobconsoli   at   gmail dot com

And please remember - Friends don't let friends use Facebook.

Most researchers in this field use some product like Microsoft's Excel to create a 'DB'.  This is fine; it's what is generally referred to as a 'flat file'.  That is, a single table representation of your data of interest.

The Mycenaean Atlas database, on the other hand, is a relational database with several tables that was built using MySQL.  Dumps of this DB are available but you need MySQL (or similar) and a knowledge of Structured Query Language (SQL) to run it.  Otherwise it doesn't make much sense to ask for it.  But If you'd like to have a copy of the Mycenaean Atlas database anyway then e-mail me and tell me about your project.  

And remember that useful .kml and/or .csv files can be generated directly from all the windows (including the new reports pages) of the website helladic.info   Try it out!   I'm also willing to generate custom .csv or .kml files from the atlas if you need something specific.


Saturday, June 3, 2017

The Arachne CMS seals database




I learned from the Aegeanet bulletin board that the Arachne CMS databases are online.  This includes a very large engraved seals database.

I have very little interest in engraved seals but I did go to this site anyway and looked at it.  I learned that the Heidelberg University Seals Database numbers about 13000 entries.  They do not allow access to their database except through a browser.   To do serious research work you’d have to look at every one of the 13,000 entries and you’d have to painstakingly recreate your seals DB from theirs one entry at a time.  Let’s say you can do two per hour.  That amounts to 6500 man hours.  There are only 2250 man hours in a work year.  That means that anyone who wants to use the Arachne seals DB for serious research has to spend 2.9 man years or 34.67 months in analysis and db building.   Along with all the errors that hand copying entails.

Which is an  unnecessary time expenditure because anyone who wants the Arachne seals database can have the entire thing with no more time investment than a couple of days to write a piece of software.   

How is that possible?

Well, first of all, it’s the internet and the internet is notoriously open.  It is open especially for benign purposes which this is.  But, second, the Arachne people have a sloppiness in their website which is that they show all the parameters to a page call right on their url line.  The result is that that url can be programmatically manipulated to allow you to download one page after another until you’ve downloaded every one of their pages.  The whole download process (images and all) would take, at most, a few minutes.  Once downloaded each page can be parsed for its constituent info (let me emphasize that the information being dumped is exactly the information that the web site is designed to provide in the first place and analyzing this page is exactly what your favorite web browser already does).  After download a series of sql statements can be generated which can create a customized form of their database on the local machine.  All of this can be done with a single program so that the sequence is ‘web pages in; SQL statements out’.

It would require that the would-be downloader write a program that is, in essence, a tiny customized browser that is designed just to read Arachne web pages.  That web browser can then parse the string coming from Arachne and write out sql statements that encapsulate the info of interest.  Once the sql statements are executed the database will be magically recreated on the local machine.

But let’s get back to that part about the sloppiness.  What does that look like?

A typical Arachne seals page


In this picture we see a typical Arachne seals page.   The browser I’m using, just for the record, is Chrome (Version 58.0.3029.110 (64-bit)).  See that red box towards the bottom?  That’s the information that you want to copy and put in your own database.  And the maxim is that if something is visible on your computer then it’s ON your computer and, if you can get access to it, you can do anything you want with it.  And, if you look at that little red circle at the top, you’ll see the number 495 there followed by an ampersand (&).  If you could change that number to 496 and reload the page then Arachne would serve the next image to your machine.  If you put that in a loop in a program then Arachne would serve ALL the images to your machine.  And I call this a sloppiness because those parameters don’t have to be visible.  I think that if Arachne really wanted to force you to use an approved browser then they should have used the POST method instead of GET for transmitting parameters.  That would make the passed parameters invisible.

What about the information in the red box?  How do you get hold of that?  Let’s look at a piece of the page string that is served by the Arachne servers.  This is the exact same string served to your regular browser and to see this, just use your browser to go to one of the Arachne web pages and then press the right mouse button on your mouse.  Then select 'View Source'.

...border="0" galleryimg="no"/></a><div style="font-size: 14px; padding: 3px; font-weight:bold;">Informationen</div><div class="info_wrap"><h3 class="info">Aufbewahrung: </h3><div class="info">Paläpaphos, Zypern, römisches Kammergrab.</div><div class="info">- Ortsangabe ist Fundort -</div><div class="info">Nikosia, Zypern, Cyprus Museum.</div><div class="info">- Ortsangabe ist Aufbewahrungsort -</div></div><div class="info_wrap"><h3 class="info">Herkunft: </h3><div class="info">Fundland: Zypern</div><div class="info">Fundort: Paläpaphos, CY</div><div class="info">Flur/Grabungsstätte: römisches Kammergrab</div><div class="info">Fundstelle: nein</div><div class="info">Platztypus: Grab</div><div class="info">Kontextdatierung: römisch</div><div class="info">Bereich: Zypern</div></div><div class="info_wrap"><h3 class="info">Form: </h3><div class="info">Siegelform: Lentoid</div><div class="info">Zahl der Siegelflächen: 1</div><div class="info">Wölbung der Siegelfläche: konvex</div><div class="info">Umriss der Siegelfläche: rund</div><div class="info">Dekor der Nebenseite: nein</div><div class="info">Art der Durchbohrung: diametral</div></div><div class="info_wrap"><h3 class="info">Material und Technik: </h3><div class="info">Material: Chalcedon?</div><div class="info">Bearbeitungstechnik: Rad, Rundperl, Tubus</div><div class="info">Sonstige Merkmale: nein</div></div><div class="....

Portion of the page string for an Arachne Seals web page


This string is the portion that creates the info box which I outlined in red above.  I changed some of the fields to red (I think that there are about 40 fields of interest) to show where they are.  The first ‘Fundland’ shows that this seal was found in Cyprus.  The second ‘Fundort’ gives you the location of the find city, namely Palaipaphos on Cyprus.  And so forth.  (And somewhere in here is the link to the high-resolution image of the seal.  Arachne snottily tells you that if you're just a casual user then you can only get the low-res version of the image  - or so they think.)  I emphasize that this page string is exactly the string that your Chrome or Firefox browser actually is sent by Arachne and from which your browser creates the web page that you see.  We just need a more efficient way to extract the data in a usable form and then create SQL statements out of it.  That's what's nice about this.  It is NOT a hack.  It does NOT try to change their data.  It does NOT attempt to modify their code to do something they don't want it to do.  It's merely a different way of consuming the product that they already WANT to deliver.

Much of web programming is done in a language called PHP.  PHP supports a call (leaving cURL completely out of the picture because we don’t need anything so sophisticated) called file_get_contents().  The parameter to this call is the URL whose page you are interested in acquiring.  When you execute this call the server on the Arachne machine returns the string representation of the page.  Your own favorite browser does exactly this except with much more powerful utilities.  When you get the page string back you can then do what I was suggesting which was to parse the page and write out the information in the form of SQL statements.  Writing out the SQL statements is trivial.  The whole program looks like this:

establish first URL

MAIN LOOP
web_page = file_get_contents(URL);
web_array = explode('><div class="info">', Web_page);


    FOR (every element of web_array)
            {
    IF start of element is a key word like ‘Fundort’ or ‘Fundstelle’                 then extract the argument and store it in a unique variable,                1..n
    }

Generate the SQL statement: e.g. “insert in seals values (‘Unique var 1’, ‘Unique var 2’, … ‘Unique var n’);”;


       END FOR LOOP

       update the URL to point to the next page.

END MAIN LOOP;

It’s a little more sophisticated than that.  The dates have to be extracted and placed in their own table since there can be a variable number of dates for each seal.   Basically, though, that’s all there is to it.  Now you may not want to do this but don't forget that this is always possible on the internet and for many more scholarly databases than just Arachne.  These internet tools are far more powerful than the uninitiated suppose.  What I suggest to Arachne is, if they really want to restrict access to only approved browsers, that they protect their URL line better.


And don't write back to me and say that this can't be done.  I got my version of this program running along with the associated database in about two days.  

Just for the exercise.

Blog Posts Concerning the Isthmian Wall

Since 2023 a number of posts concerning the Isthmian Wall and how we located its remaining segments, have appeared on this blog.  This post ...