.. File nm/logger/_base.js This is a comment in the documentation, which is in reStructured text. The official documentation starts here, with the title: =========== logger.js =========== :Author: Jaap Murre :Version: 1.0b :Date: April-June, 2011 *********** User Manual *********** Introduction ------------ This is a JavaScript library for logging events. Logged events are buffered and will be sent to the server at intervals set by time and number of events. The logger will buffer events in local storage thus preserving them also if the server is down, the client's computer is offline for a while, or if the page (i.e., app) is accidentally clicked away. As soon as the connection (or app) is restored, the data will be retrieved from local storage and sent to the server. Once the data has been saved at the server, they will be removed from local storage. By default, data in local storage are encrypted using advanced encryption: AES_. The data are decrypted before they are sent to the server. In order to work with the logger, you must set up server-side data processing that is capable of receiving XHR put events. Each of these save attempts (XHR put events) has attributes: .. attribute:: event_type (string) Describes the type of the event, e.g., 'page_navigation'. .. attribute:: name_space (string) This could for example identify the user or application. Usually, this is not need on the server; it is mainly important for local storage at the client. .. attribute:: data (any) The :attr:`data` attributed contains an array of event objects, each of which has attributes: .. attribute:: name (string) The name of the event. E.g., 'next_page', 'lesson_2'. .. attribute:: created (integer) Timestamp in ms since epoch (JavaScript timestamp, created with ``new Date()``). .. attribute:: data (json string) This is anything sent to the logger, JSON encoded. Requirements ------------ The logger works most reliably with dojo 1.6, which received an important upgrade of the local storage providers in ``dojox.storage``. Usage ----- We assume that the the ``nm`` directory, with the file ``logger.js`` underneath it, resides itself at the same level as the dojoroot:: dojoroot/ dojo/ dojox/ nm/ logger/_base.js docs/ Register the ``nm`` module with:: dojo.registerModulePath("nm","../../nm"); // path is relative to dojo.js file Now, the logger can easily be ``dojo.required()``. After doing this, register one or more event types, e.g., 'event', 'trial', 'keyboard', 'login_attempts', etc. Any string can be used. Registration is as follows:: dojo.require("nm.logger"); nm.logger.register("event",{url:"process_data.php",password:"pietje"}); .. WARNING:: It is advisable to ask the password directly from the user and only keep it in memory, not hardcoded in the text as is shown here. HTML and JS files may be left in the cache so that passwords can easily be retrieved. Now, you can log events with ``dojo.publish()``, like this:: dojo.publish("event",[{ name: "Next", data: currentPage }]); Notice the array brackets, ``[]``, which are mandatory with ``dojo.publish()``. .. _event_structure: Event structure +++++++++++++++ Standard events use the format:: { name: "the_name_of_your_event", data: "string, or integer or object or whatever" } If a different format is used, the logger will turn such a non-standard event into a standard event (with :attr:`name` and :attr:`data` attribute). A timestamp with attributed name :attr:`created` is always added (in ms since epoch). So, it is also possible to write:: dojo.publish("trialno",[34]); or:: dojo.publish("trial",[{no: 34, score: 98}]); or:: dojo.publish("logged in",["jaap"]); If you do not give an event name (i.e., in addition to an event *type* such as 'trial'), the logger will assign the name 'default'. It is also possible to call the logger directly, with:: nm.logger.store("trial",{name: 'exp1', data: {block: 3, score: 23}}); (no ``[]`` brackets necessary here) or store another type, e.g., a string or array:: nm.logger.store("scores",[34,29,35,43,51]); Register settings ----------------- Registration allows setting of various details using a ``details`` object. The ``details`` object must be provided, because :attr:`saveToServer` and :attr:`encrypt` are true by default and hence :attr:`url` and :attr:`password` must be specified. If you don't want saving or encryption these must be turned off explicitly. The following attributes may be set: .. attribute:: repeatSaveMs Interval in ms between save-to-server attempts, set to 30,000 ms (i.e., 30 s) by default. Save attempts occur even if the the :attr:`repeatSaveEvents` number has not been reached. .. attribute:: repeatSaveEvents Interval in number of events until next save, e.g. 20 means that the system will wait until 20 events have happened and will then save those to server, unless the :attr:`repeatSaveMs` is reached before that. Default is 10. If the save fails, it will count from 0 to :attr:`repeatSaveEvents` before another save-to-server is attempted based on counted events. .. attribute:: maxSaveAttempts Maximum number of times save-to-server will be tried (with :attr:`repeatedSaveMs`) in case of failure (e.g., due to server or connection problems). Default is 3. .. attribute:: saveToServer true (default) or false . Attempt to post data to server. If true, you *must* provide a :attr:`url` .. attribute:: url Url where to post the data. .. attribute:: encrypt true (default) or false. Encrypt local data using AES_, If true, you must provide a password. For non-critical applications, encryption may be turned off for increased performance, e.g., on mobile platforms. .. NOTE:: A `simplified version`_ of the AES implementation is used here. .. _simplified version: http://dojotoolkit.org/reference-guide/dojox/encoding/crypto/SimpleAES.html .. _AES: http://en.wikipedia.org/wiki/Advanced_Encryption_Standard .. attribute:: password Password used for encryption of local storage Namespace --------- It is possible to set the current namespace with :func:`nm.logger.setNamespace("subj134")`. This is only relevant for local storage and it is often useful to mark the ID of the subject in the namespace. If not set, it is likely that multiple users of the same application running on the same computer (sharing an account) will get mixed up in local storage. Note that all in-memory events are assumed to belong to the current user (only one namespace is active at a time). Methods ------- The following methods are available (for ``event_type`` it is best to always use a string): .. function:: logger.store(event_type,event) Send data to server, via local storage. See above for legal values of ``event``. :param string event_type: event type :param string event: see :ref:`event_structure` .. function:: logger.getData(event_type) Retrieves (and decrypts) data from local storage and in-memory data, returns these as a possibly empty array. Data from server is not retrieved. This function is useful if the logger is used without server-side storage (i.e., :attr:`saveToServer` is ``false``). :param string event_type: event type .. function:: logger.getDataByName(event_type,event_name) As :func:`logger.getData` but only returns events with :attr:`name` atribute ``event_name``. :param string event_type: event type :param string event_name: event name .. function:: logger.clear(event_type) Clear in-memory and local storage of a specific event type. :param string event_type: event type .. function:: logger.clearByName(event_type,event_name) Clear in-memory and local storage of a specific event type. Only events with :attr:`name` atribute ``event_name`` are removed. Attempts to store remaining events are to local storage. :param string event_type: event type :param string event_name: event name .. function:: logger.clearAll() Clear all in-memory and local storage of all event types. .. function:: logger.register(event_type,details) See section `Register settings`_, for an explanation of ``details``. :param string event_type: event type :param object details: properties overriding default settings .. function:: logger.setNamespace(namespace) See explanation in section Namespace_. :param string namespace: namespace identifier .. function:: logger.getNamespace() :returns: the currently set namespace (= ``"default"``, if none has been specified). .. function:: logger.saveOldEvents(event_type) If there are any existing events of type 'event_type' in local storage (i.e., the file system), it will try to save these to the server with any in-memory events of 'event_type'. This function is called in function :func:`logger.register`. :param string event_type: event type ------------------------------------------------------------------------------------- ***************************************** Appendix A: Server-side Processing in PHP ***************************************** .. code-block:: js+php < ?php // Remove space before ? in actual implementation function insert_data() { $method = $_SERVER['REQUEST_METHOD']; if ($method == 'POST' || $method == 'PUT') { $input_data = json_decode(file_get_contents('php://input')); } else { $input_data = $_GET; } $event_type = $input_data->event_type; $namespace = $input_data->namespace; $contents = $input_data->data; $con = mysql_connect("localhost","logger_user"); if (!$con) { die('Could not connect: ' . mysql_error()); } mysql_select_db("logged_events", $con); $count = 0; foreach ($contents as $event) { $sql="INSERT INTO events (event_type, namespace, event_name, data, created) VALUES ('$event_type','$namespace','$event->name','$event->data',$event->created)"; if (!mysql_query($sql,$con)) { die('MySQL error: ' . mysql_error()); } ++$count; } echo "$count records inserted"; mysql_close($con); } insert_data(); ------------------------------------------------------------------------------------- ********************************************* Appendix B: MySQL Structure of Database Table ********************************************* .. code-block:: mysql -- -- Table structure for table `events` -- CREATE TABLE IF NOT EXISTS `events` ( `index` int(11) NOT NULL AUTO_INCREMENT, `event_type` text NOT NULL, `namespace` text NOT NULL, `event_name` text NOT NULL, `data` longtext NOT NULL, `created` bigint(20) NOT NULL, PRIMARY KEY (`index`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=4 ; ------------------------------------------------------------------------------------- ********** Change Log ********** 2011-06-09 - Jaap ----------------- Updated documentation. 2011-06-08 - Jaap ----------------- Found a bug where events were stored twice in the database. Traced it to the results handler of :func:`dojox.storage.put`, which somehow did not work in the global context. Therefore clearing of in memory events did not succeed and they were thus stored twice (in memory and local storage). This happened using the ``WhatWGStorageProvider`` (though using FF3). I then noticed I was still working with dojo 1.5. After moving to 1.6, the problem disappeared and storage provider is now upgraded to one that does not have this problem: ``dojox.storage.LocalStorageProvider``. Using dojo 1.6 is thus required. Added this in the documentation. In the process of tracing this bug, also added a new function :func:`clearInMemory`. For now, only intended for private use. 2011-06-06 - Jaap ----------------- Updated the file so that the reStructured documentation can be used directly from the .js file, except for the /* comment at the beginning. 2011-05-06 - Jaap ----------------- Did several things today: - Changed fatal bug with _checkEventRegistration(), which was not a function yet - Turned logger.js into nm/logger/_base.js construction - Changed 'doc' directory name into 'docs' - Updated documentation to reflect this. Also, updated documentation in various other places. - Found bug with unicode (e.g., Chinese) characters with _encrypt/_decrypt. Apparently the algorithms expect single-byte (i.e., ASCII) characters. Solved this by wrapping the data with simple encode_utf8/decode_utf8 functions. Works for Pinyin at least. For a more complete solution, see http://www.webtoolkit.info/javascript-utf8.html. 2011-05-07 - Jaap ----------------- Added two public functions that allow getting and clearing by event name (i.e., as in the ``name`` atribute): - :func:`getDataByName` - :func:`clearByName` - Also updated documentation to reflect this. Added tests directory and wrote ``test_logger.html``. Discovered a large bug that caused server access for each additional logged event once the ``repeatSaveEvents`` maximum had been reached, because the event counter was never reset. Changed the logic. Now, the counter is set to 0 before a save to server is attempted. If the save fails, it is *not* set back to its original value. This is deliberate to prevent save-to-server attempts with each new event in case of a connection failure. .. The actual code follows below. Notice that it is indented so that it is considered one long comment by restructured text. */ dojo.provide("nm.logger._base"); //nm = {} dojo.require("dojox.storage"); // dojo.require("dojox.storage.manager"); //dojo.require("dojox.encoding.crypto.SimpleAES"); // Called in register(), if specified nm.logger = new function() { var namespace = "default", // The global namespace (optionally set with setNamespace() ) events = {}, event_details = {}, event_timers = {}; var _PREFIX = "logged_events_"; // prefix prepended before each local-storage key function fatalError(/*String*/event_type,/*String*/error_string) { var s = "FATAL ERROR ('" + event_type + "'): " + error_string; console.error(s); alert(s); } function _checkEventRegistration(event_type) { if (events[event_type] === undefined) { fatalError(event_type,"This event type must be registered before use with 'nm.logger.register()'"); } } function saveAll() { // TODO: Tries to save all data to the server fatalError("saveAll()","This function has not been implemented"); } // Tries to save the locally stored data of type 'event_type' to the server and if successful, removes it from // local storage. If there are remaining in-memory data, those are saved remotely and then // deleted as well (deletion on success only). function save(/*String*/event_type) { _checkEventRegistration(event_type); var d = event_details[event_type], count, event_data; d["saveEventCount"] = 0; if (!d['saveToServer']) { return; } var data, url = d['url']; if (!url) { fatalError(event_type,"No 'url' provided for saveToServer"); _noteFailedSave(event_type,true); // true = prevent further save attempts return; } count = d["totalSaveCount"]; ++d["totalSaveCount"]; data = getData(event_type); // does automatic decryption if specified d["saveTempStore"][count] = data; clear(event_type); // console.log("Saving to server...",event_type,data,d["saveTempStore"][count]); event_data = { 'event_type': event_type, 'namespace': namespace, 'data': data }; dojo.xhrPut( { url: event_details[event_type]['url'], putData: dojo.toJson(event_data), handleAs: "text", load: function(response) { var e = event_type, c = count, r = response; _onSuccessfulSave(e,c,r); }, error: function(error) { var e = event_type, c = count, r = response; _onFailedSave(e,c,r); } }); } function _onSuccessfulSave(/*String*/event_type,/*Integer*/count,/*String*/response) { var d = event_details[event_type]; clearTimer(event_type); event_details[event_type]["saveAttempts"] = 0; delete d["saveTempStore"][count]; // Remove temp storage // console.log("Successful save...",event_type," --- Resonse from server:\n",response); } function _onFailedSave(/*String*/event_type,/*Integer*/count,/*String?*/error) { var d = event_details[event_type]; // Rollback save: re-store elements from temporary storage to // in-memory and attempt to store in file system with storeLocal() events[event_type].concat(d["saveTempStore"][count]); delete d["saveTempStore"][count]; // Remove temp storage storeLocal(event_type); // console.log("failing save..."); _noteFailedSave(event_type); } function _noteFailedSave(/*String*/event_type,/*bool?*/prevent_further_save_attempts) { event_details[event_type]["saveAttempts"] += 1; // If the server does not respond repeatedly (or no connection), stop trying if (event_details[event_type]["saveAttempts"] >= event_details[event_type]["maxSaveAttempts"]) { clearTimer(event_type); event_details[event_type]["saveAttempts"] = 0; // console.log("No connection..."); dojo.publish("nm/logger/no_connection",["Not able to save to server"]); } if (prevent_further_save_attempts) { clearTimer(event_type); event_details[event_type]["saveAttempts"] = 0; } } function clearInMemory(/*String*/event_type) { events[event_type] = []; } // Clear in-memory and local storage of specific event function clear(/*String*/event_type) { _checkEventRegistration(event_type); clearInMemory(event_type); dojox.storage.remove(_PREFIX + event_type,namespace); } // Removes all events of ``event_type`` with ``name`` attribute // ``event_name``. Attempts to store remaining items to local storage. function clearByName(/*String*/event_type,/*String*/event_name) { var data = getData(event_type); clear(event_type); // filter out all events with ``name`` equal to ``event_name`` events[event_type] = dojo.filter(data,function(e) { var e_name = event_name; return e.name !== e_name; }); storeLocal(event_type); } // Clear all in-memory and local storage of all events function clearAll() { var a; for (a in event) { events[a] = []; } dojox.storage.clear(namespace); } function startTimer(/*String*/event_type) { event_timers[event_type] = setInterval( function() { var e_type = event_type; // console.log("Saving ",event_type," for time-out"); if (event_details[event_type]['saveToServer']) { save(event_type); } }, event_details[event_type].repeatSaveMs ); } function clearTimer(/*String*/event_type) { clearInterval(event_timers[event_type]); event_timers[event_type] = null; } function register(/*String*/event_type,/*Object?*/details) { var timer; details = details === undefined ? {} : details; details["repeatSaveMs"] = details["repeatSaveMs"] === undefined ? 5000 : details["repeatSaveMs"]; details["repeatSaveEvents"] = details["repeatSaveEvents"] === undefined ? 10 : details["repeatSaveEvents"]; details["maxSaveAttempts"] = details["maxSaveAttempts"] === undefined ? 3 : details["maxSaveAttempts"]; details["encrypt"] = details["encrypt"] === undefined ? true : details["encrypt"]; details["saveToServer"] = details["saveToServer"] === undefined ? true : details["saveToServer"]; // Accidental re-regstration will not clear existing events in memory if (!events[event_type]) { clearInMemory(event_type); } event_details[event_type] = dojo.clone(details); // console.log("details ",event_type, event_details[event_type]); event_details[event_type]["saveAttempts"] = 0; // Some extra data will be stored in the details object event_details[event_type]["saveEventCount"] = 0; event_details[event_type][""] = 0; event_details[event_type]["saveTempStore"] = {}; dojo.subscribe(event_type,function(event) { var e_type = event_type; store(e_type,event); }); if (event_details[event_type]['encrypt']) { dojo.require("dojox.encoding.crypto.SimpleAES"); } saveOldEvents(event_type); // Try to save any unsaved events from previous sessions } // Stores an event, attempts storage at the file system and schedules it // for remote storage, unless detailed otherwise during registration. // An event always has the shape {name: "aName", data: any_type_of data} // If this is not the case for ``event``, it is modified into the standard shape. function store(/*String*/event_type,/*any*/event) { _checkEventRegistration(event_type); // console.log("storing event: ",event_type,event); var e = event; if (!event_timers[event_type] && event_details[event_type]["saveToServer"]) { startTimer(event_type); //console.log("starting timer for: ",event_type); } // Turn non-standard event into standard event (with name and data attribute) if (!event.name) { e = {}; e.name = "default"; e.data = event; } e.created = +new Date(); // ms since epoch, + forces convert to number, like *1 events[event_type].push(e); storeLocal(event_type); if (event_details[event_type]['saveToServer']) { event_details[event_type]["saveEventCount"] += 1; // only count events that must be saved to server } if (event_details[event_type]["saveEventCount"] >= event_details[event_type]["repeatSaveEvents"]) { // console.log("Saving ",event_type," for saveEventCount reached"); if (event_details[event_type]['saveToServer']) { save(event_type); } } } // If there are any existing events of type 'event_type' in the file system, // it will try to save these to the server. This function is called in register() // or by an application. function saveOldEvents(/*String*/event_type) { _checkEventRegistration(event_type); var stored; stored = dojox.storage.get(_PREFIX + event_type,namespace) || []; if (stored == true && event_details[event_type]['saveToServer']) { save(event_type); } } // Convert unicode to singe-byte char string // If this fails, see http://www.webtoolkit.info/javascript-utf8.html // for a more complete solution function encode_utf8(/*String*/s) { return unescape( encodeURIComponent( s ) ); } // Convert back function decode_utf8( s ) { return decodeURIComponent( escape( s ) ); } // Convert 'data' to json and encrypts itusing SimpleAES function _encrypt(/*String*/event_type,/*any*/data) { var pw = event_details[event_type]['password']; if (!pw) { fatalError(event_type,"No 'password' provided for encryption"); return; } return dojox.encoding.crypto.SimpleAES.encrypt(encode_utf8(dojo.toJson(data)),pw); } function _decrypt(/*String*/event_type,/*Encoded JSON String*/enc_string) { var pw = event_details[event_type]['password'], data; if (!pw) { fatalError(event_type,"No 'password' provided for decryption"); return; } // console.log("_decrypting: ",enc_string); data = dojox.encoding.crypto.SimpleAES.decrypt(enc_string,pw); // console.log("_decrypted: ",decode_utf8(data)); return dojo.fromJson(decode_utf8(data)); } function getData(/*String*/event_type) { _checkEventRegistration(event_type); var stored, data; // console.log("getData events[]: ",events[event_type]); if (event_details[event_type]["encrypt"]) { // decrypt locally stored array, add in-memory array and encrypt resulting array stored = dojox.storage.get(_PREFIX + event_type,namespace) || ""; if (stored) { stored = _decrypt(event_type,stored); } else { stored = []; } // console.log("getData stored: ",stored); data = stored.concat(events[event_type]); // console.log("getData decrypting: ", data); } else { stored = dojox.storage.get(_PREFIX + event_type,namespace) || []; data = stored.concat(events[event_type]); // console.log("getData (no decrypting): ",data); } return data; } // Returns data in memory or local storage of ``event_type`` // that have ``name`` attribute ``event_name``. Return // value is a possibly empty list. function getDataByName(/*String*/event_type,/*String*/event_name) { return dojo.filter(getData(event_type),function(e) { var e_name = event_name; return e.name === e_name; }); } // Tries to store all in-memory events of type 'event_type' to local (file-based) storage function storeLocal(/*String*/event_type) { var data; // dojo.require("dojox.storage.manager"); // console.log("Storage Provider: ",dojox.storage.manager.currentProvider.declaredClass); data = getData(event_type); // does automatic decryption if specified // console.log("storeLocal: ", data); // console.log("Unencrypted data: ",data); if (event_details[event_type]["encrypt"]) { data = _encrypt(event_type,data); } dojox.storage.put(_PREFIX + event_type,data,resultsHandler,namespace); function resultsHandler(status, key, message, namespace) { var event_type = key.slice(_PREFIX.length); // Store in memory, but remove events in memory, if local (file-based) storage has succeeded if (status == dojox.storage.SUCCESS) { clearInMemory(event_type); // console.log("storeLocal events on SUCCESS: ",events[event_type]); } else { console.log("Failed save to local storage: status="+status+", key="+key+", message="+message); } }; } // Set the namespace for local storage (on the file system), typically some user ID on a computer // where several people use the same application through the browser function setNamespace(/*String*/newNamespace) { namespace = newNamespace; } function getNamespace() { return namespace; } return { store: store, getData: getData, getDataByName: getDataByName, clear: clear, clearByName: clearByName, clearAll: clearAll, register: register, setNamespace: setNamespace, getNamespace: getNamespace, saveOldEvents: saveOldEvents } }();