Showing posts with label free library. Show all posts
Showing posts with label free library. Show all posts

April 4, 2013

How to share/store Spreadsheeds as JSON data with ScriptDB!


There are many cases were we want to share data between many Apps, and one of the most popular ways it's to use Google Spreadsheets as a Central DB, and they do a great job on letting you share, download and access the Database in a very human way, but most of the time we want to efficiently query Data, but with Spreadsheets you need to know which cell has your data, filter using the visualization API or do some parsing... this makes it heavy and slow on large Data sets opposed to ScriptDB who is great for querying specific data!

Another limitation of using Spreadsheets is that they are not very good for handling shared accumulators, it's prone to over-writing when two or more users are working with the same cell at the same time, let's imagine the cell's current value its 3, and both client's add a +1, the cell's value its likely to end up as a 4 instead of 5... To prevent this ScriptDB has 2 different kinds of "locks", one public and one private, the difference is that public only allows 1 instance of the APP to run at any given time and the private lock only allows each USER to run 1 instance of the App at any given time.

Each script project get's a "sandboxed" ScriptDB for it's own use, meaning it's isolated from the rest of the users data and other ScriptDB instances from the same user. But were it really shines it's when you use a single ScriptDB for many Apps as a share point, you just need to create a tiny Apps Script webapp and share it as a library to get shared access to it's ScriptDB... not shiny enough? well, you can create many specific ScriptDBs to make everything safer, and organized by simply creating different library's and calling them as needed (creating different library's allows you to limit specific user's access) ... not enough? well, ScriptDB can be easily transferred to a Spreadsheet for a much more readable long term storage of our data. With some careful planning you can make smart data handling that's its fast, efficient and la piece de resistance... it's all free!

So, which are the limits of ScriptDB? According to the Documentation, they are actually  large enough for most cases, the quotas are per-user (so the limit is shared across all user's Apps, or in the case of the library, it goes against the owner of the library) and it depends on the kind of user:

> 50MB for consumer accounts.
> 100MB for Google Apps accounts
> 200MB for Google Apps for Business/Education/Government accounts.

Since it's all Text data wrapped inside JSON objects, this can be quite a lot of data, even for 50MB. So we are usually more limited to the query limits (default 200, maximum 50,000) and the time limit for an Apps Script instance (maximum 6 minutes processing time), in most cases it's a good idea to dump all the ScriptDB's contents inside an Apps Script sheet or sheets for more comfortable future reference and to avoid saturating the ScriptDB's query's responses.

Also keep in mind that ScriptDB it's specially useful with current and fast changing data, and Spreadsheets its better for long-term storage and for making automatic reports. In most scenarios the best approach it's to create a Sheet everyday with the Date as a name and dump the ScriptDB content on it... Also you can start a new Spreadsheet every month, all automatically and, if done properly, you will end up with a Database that your customers will love for being cheap, fast, flexible and the automatic reports are a great plus (you can even send a custom report via e-mail every time ScriptDB is dumped inside a Sheet).

Without more preamble, let's hit the code! We'll start by creating a Library that returns a ScriptDB instance that can be shared between different apps, create a new script and you only need this 3 simple lines of code!

To use this script as a library, remember you need to write down the script's project key, to get it, go to File - > Project properties.


If you plan to share access to this library with other users, remember to set the "Sharing Settings" which can be found in File - > Share...

To import your library, create (or open) a new Apps Script project and go to Resources - > Manage Libraries and write you Library's project key, inside the "Find a library" box.



Now, it's time to put some JSON data inside our library's ScriptDB! in most cases, we want to get our data from a Spreadsheet either because the Data is already on one, or simply because it is easier to write, as an example let's use the following spreadsheet:

The Apps Script code needed to do this, can be found on this part of the documentation, let's see a copy (with a few modifications):



This will make a flat JSON version of our Spreadsheet inside our shared ScriptDB! Flat JSON is very easy to work with, and remember you can put it back or make a copy in a Spreadsheet at any time.

There is a example code in the documentation to make a quick-check of your ScriptDB content, just create a new script with the following code:



Now you can see the ScriptDB's content by checking the Apps Script Log, our example spreadsheet data would look like this:

{"price":5,"name":"Razor blade","salesToday":0,"onStorage":50}
{"price":2.5,"name":"Toothbrush","salesToday":0,"onStorage":100}
{"price":3,"name":"Liquid Soap","salesToday":0,"onStorage":80}
{"price":3.7,"name":"Shampoo","salesToday":0,"onStorage":80}
{"price":2,"name":"Toothpaste","salesToday":0,"onStorage":150}


Feel free to let us know about your ideas, suggestions, requests, etc at the bottom of this article.


Happy coding!

February 20, 2013

How to easily use BigQuery's Querys & Charts with Google Apps Script! (and our FREE library)


BigQuery's ability to digest billions of rows, trillions of records stored in a 1Tb Database in seconds sounds really appealing for massive data-analysis! Using it you can, for example, digest all the sales in the last 20 years of a super market and make a chart with the evolution of the sales of the different brands of cereal inside such period... all of them or just a single one... Put it all inside some graphs and send them attached as a PDF report via Gmail....now that kind of power screams "Google" right? and hopefully might draw a smile on some of your client's face.

Google Apps Script certainly facilitates the use of Big Query! but still we need to get (and wait) for the results and iterate through them, so we (the team at MyWorkers) decided to create a library to facilitate working with BigQuery. So far it helps you with the 3 basic things you need to get started:
  • Sends your query, wait for and retrieve the results. 
  • Iterate through the results. 
  • Build a data table that can be drawn by any appropriate Google Chart. 

This functions are called and works independently, so you can make reuse them many times in your codes to save you a lot of time coding and debugging on the data handing so you can focus more on your charts and the rest of your super cool WebApp or automatic report.

This makes using Big Query VERY easy and straight forward! But before you can start using it, you need to make sure you have the following:



First you need to have (or create) a project in the Google APIS console and take note of the project number.



Also (if you haven't already) you need to enable the Big Query service in the Services tag of the Google APIS console.

To use BigQuery, you also need, well... a query. BigQuery queries are written using a variation of the standard SQL SELECT statement, you can read more details about how to make them in this Query Reference and/or using this Query Cook-book. We will be using one of the example query's (and free public tables) that Google provides:

select TOP(word, 10), COUNT(*) as word_count from publicdata:samples.shakespeare WHERE LENGTH(word) > 10;

Now you are ready to start using BigQuery via Google Apps Script! Let's check it out:



* Create a new script at script.google.com (let's call it bigQueryToCharts) and go to the Resources menu and click the Use Google APIs tag. From there, turn on the Big Query API.



* Then go to Manage libraries (also in the Resources menu) and:

  1. Go to "Find library:" and type: ME6_s9CBqVzsOfhZryNM0vzfIqMXwq4Jm .
  2. Click Select.
  3. In Version click  8 getQueryResults function added.
  4. Turn on development mode for setupBigQueryResults.

It's time to start putting some code, first lets put what we know inside some variables:

var projectNumber = '<ID>'; // The ID of your project.
var sql = 'select TOP(word, 10), COUNT(*) as word_count from publicdata:samples.shakespeare;';


IMPORTANT NOTE: I suggest using the same names for the variables, it's not necessary  but  this way the auto-complete of Apps Script will fill the names of the variables needed by the library when we call it.

Now let's use the library (setupBigQueryResults) to send our SQL to BigQuery, wait for the results and deliver them back, it will offer you options to autocomplete, choose getQueryResults and the line should look as follow:

 var queryResults = setupBigQueryResults.getQueryResults(projectNumber, sql);

To continue, we need to know 2 things: the resultCount (to iterate results) and the resultSchema to build our charts data table, both values are passed by setupBigQueryResults inside queryResults, to get them type as follows:

var resultCount = queryResults.getTotalRows();
var resultSquema = queryResults.getSchema();


Now lets hand those values back to our library, it's done this way to give more flexibility on the use of the library, (E.g. when processing multiple instances.

var resultValues = setupBigQueryResults.iterateResults(queryResults, resultCount);
var bigQueryDataTable = setupBigQueryResults.buildChartsDataTable(resultValues, resultSchema);


And that's it! We are ready to pass our data to any (appropriate) chart! Let's see our full example (with the required code to build a basic UI):

function doGet(e) {
  var projectNumber = '<YourProjectsNumber>'; 
  var sql = 'select TOP(word, 30), COUNT(*) as word_count from publicdata:samples.shakespeare WHERE LENGTH(word) > 10;';
  var queryResults = setupBigQueryResults.getQueryResults(projectNumber, sql);
  var resultCount = queryResults.getTotalRows();
  var resultSchema = queryResults.getSchema(); 
  var resultValues = setupBigQueryResults.iterateResults(queryResults, resultCount); 
  var bigQueryDataTable = setupBigQueryResults.buildChartsDataTable(resultValues, resultSchema);
  var chart = Charts.newBarChart()
    .setDataTable(bigQueryDataTable)
    .setDimensions(500, 700)
    .setYAxisTitle('Word')
    .setXAxisTitle('Count')
    .setLegendPosition(Charts.Position.BOTTOM)
    .build();
  var uiApp = UiApp.createApplication();
  uiApp.add(chart);
  return uiApp;
}


You can read all the details about what you can do with BigQuery, along with the original version of this example (without our library) in the Big Query class documentation.

Feel free to try this example, already published as a WebApp and to post your ideas, suggestions, requests, etc at the bottom of this article.

________________________________________________________________________
Important Note: The library provided in this example is setup to run as Me (César Antón Dorantes) and only gives access to the free datasets for Google BigQuery, to use it to process YOUR information (with the Billing activated on your account) you must use a library that runs as YOU. To do so, use the following library:

 "Project key" to import the library from "Resources" -> "Manage Libraries":


MsZBZDQxT88E-QS2OFguwejfIqMXwq4Jm
It will require you to authorize the script to use your BigQuery access, this is only done ONE TIME, to do it, simply run it from the following address:

https://script.google.com/macros/s/AKfycbyTD13yqIya75jntPkdubgq0f3DRsL-eTUDI8ZC9uGx5lYjHiM/exec

The FIRST time you run it, it SHOULD pop up an alert asking you to authorize the library. Once authorized, it should give this output (when run directly from the link):

"Unknown macro doGet".

________________________________________________________________________
Update: Version 3 of the library now available! It now includes Documentation and GNU License. Please Update your library to Version 3 so that your code get's the License for this Library. You can read the Library Documentation in the following link:
https://script.google.com/macros/library/d/MsZBZDQxT88E-QS2OFguwejfIqMXwq4Jm/3
________________________________________________________________________
Update: Version 4 of the library now available! It now incorporates the new method changes described on this article. Please Update your library to Version 4 so that your code works again with the new methods.
________________________________________________________________________