Tag: code

Building your Custom Connector on Google Data Studio

Prasad AjinkyaLeave a comment
Building your Custom Connector on Google Data Studio

Disclaimer – This is going to be a slightly technical post. If code scares you, then I’d suggest you skip this one. However, if data excites you, read on fellow adventurer!

When Google launched Google Data Studio, I had written an in-depth post about how one can create dashboards in Data Studio and some of the easy to use dashboard templates that Data Studio has to offer. As the product evolved, one of the most powerful features that this product had to offer was the ability to create custom data connectors to your own datasets.

What does a custom connector do?

A custom connector enables a user to access their own data source within Google Data Studio. Let’s take an example of a marketing research associate who wants to present her findings. One approach she could use would be to put all that data in Google Sheets, and then use one of the in-built connectors.

However, what would she do if her data set was large and does not fit in Google Sheet or Excel? Or if her data set included multiple surveys which are inter-related to each other?

What if this data was in a database, or available as an API? This is where the custom connector for Google Data Studio works.

I wrote my first connector a year back, and I had to do some digging around. I thought that I should pen down my notes so that more people can do this much more easily. Here are my notes for the same.

Building a custom connector

Before you jump into the implementation bit, know that this is based in JavaScript and you need to be comfortable with Google App Scripts. It’s okay if you do not know this, but JavaScript is a must.

Google has official documentation on the developer site of how to build a Community Connector, this is a pretty good resource to start. It has a step by step video and instruction guide as well.

Let’s look into what makes different parts of a connector, here is a link to a sample connector code on Github.

Community Connector sections

Each community connector is a separate Google Apps script that you deploy using Google App scripts. The connector is in itself made of the following sections –

  • Configuration section – This is to flesh out all the meta information about the community connector. Use this section to take any inputs from the user e.g API Secret and key if you don’t wish to store this in your code.
  • Authentication section – This is to authorize the app script. If your data is lying behind a secure mechanism, then use this section to authorize the script to access the data. This supports OAuth2 as well.
  • Schema section – This is used to define the structure to the data you are importing into Data Studio. Use this section to outline which fields and what data types are they. You can also add more information on what kind of aggregation do you want this field to be have (Sum, Average, Min, Max, etc).
  • Data section – This section is used to fetch the data that you are importing. Use this section for data validations or if you want to do any last minute data tweaks (e.g date conversions from string to date).

That’s all there is to it. Now, let us get into the actual flow of the script.

Connector code flow

When you are writing your connector, be sure to go through the developer reference first. In your script, you will have to include the following functions –

  1. getConfig() – this returns the configurable user options for the connector, this will be shown to the user when the user is adding the connector to their Google Data Studio accounts.
  2. getAuthType() – this is the function which is used to check if any authentication is required. If OAuth is set, then the community connector interface would check for the OAuth details
  3. getSchema() – this returns the schema of the data that is being access, this will be shown to the user when the data is being explored (where we can see the dimensions and metrics).
  4. getData() – this is the function which is used to access the data, the data format that is expected is outlined here. Normally, it is advised that the programmer write a separate function for fetching the data, a post processing function for setting up the return values, and finally call those in the correct order in this function.

Do note, that these functions will be called in the same order as they are listed. As long as you have these functions in your code, you have a functioning connector. Once you have this, you will have to deploy the code.

That’s it. Now, add this community connector to your Google Data Studio account, and make the reports you want to!

Working with markdown and gitbook

Prasad Ajinkya1 Comment
working with markdown and gitbook

For those of you who don’t know yet, I have shifted tracks to heading a tech team in a start-up. This firm focuses on helping first time home buyers with the largest hurdle in home buying, the down payment. HomeCapital is India’s first home down payment assistance program.

At HomeCapital, one of the immediate challenges that I had to face was to understand a myriad of requirements from speaking to the operations team, to the business analysts, to the developers, to some of the customers and even to some of our investors.

Since, the approach is that of a technology platform, it also means that the team had to start worrying about multiple systems all at once. Deciding to move away from one huge monolithic system to a micro-services based architecture was natural.

How does one manage loads of Micro-services?

A major challenge with a spread of micro-services was that the management overhead of systems went up. Different services were in different repos, in different languages and hosted in different methods. Yes, there was an API gateway on top to present a uniform access method for all, but the code management and documentation was a challenge.

Thankfully most popular versioning systems have solved the code management issue. One of the first steps I initiated with this was using the README.md to quickly jot down what the service is supposed to do, and how it functions. This was created more from the point of a new team member who wants to get started with the respective service. You need to be comfortable with Markdown for this. I’ll get to markdown in a minute, but this was a great starting point for me to understand what a developer really needs in the documentation.

As a person overseeing multiple services, it was essential for my team members to quickly pick up the bare essentials and use the documentation available. Having a small entry point in the repo is a perfect way to give access without creating too formal a structure. My choice of working with markdown was made.

What is Markdown?

In case if you do not know what this is, then you mostly haven’t edited a wiki. Markdown language is a super lightweight language that allows one to quickly convert the text into a rich formatted document (such as HTML, PDF, etc). To read more about this, head on to the Wiki on Markdown.

Try practicing using Markdown for some time and you will realize its almost as simple as using notepad or gedit to take down your notes. It also helps you to create a more complex structure and is super flexible for future use-cases.

Generating a usable README.md

For those of you who want to try this out, hop on to Make a README and see the basic placeholder sections needed to make a developer friendly file.

I had by this time quickly written these files and was happy that at least I had some formal documentation available in a system that was fast growing. A side note here – <rant> In most rapidly evolving systems, people often take decisions that they regret later on. This technical debt although is meant to be avoided, but often it just can’t be avoided. As long as you are willing to come back and clear the debt, it’s fine. You could re-think your approach and do it faster in a correct fashion – but then you need to be a lot more mature and I just don’t see that developer maturity yet. This side note will need to be expanded into a separate post of it’s own </rant>

What to do with a cart load of README.md files?

Quickly, I had many individual standalone files sparsely connected to each other. While this was sufficient for a developer to get started, this did not fully cover the breadth and width of the system.

This is where my past experience of working with the WordPress India community helped. The community is building an independent document made of such .md files using gitbook. Gitbook used to be a CLI based command that you could install on your machine and use to build a developer website. This using the very .md files that I now had.

At the time of writing this post, the gitbook CLI is available on npm, however, do note that the site now talks about a version 2, which is not a CLI based offer but is more of a SaaS product with a freemium offering. You could also look at some other alternatives to do this, but the ease of use of the gitbook CLI is to be applauded.

How to get started with gitbook?

  1. Head on to the npm page for gitbook-cli and install this first.
  2. Create a new folder and in the console hit gitbook init
  3. Answer the questions and create your first markdown file
  4. In the console hit gitbook serve and in your browser go to http://localhost:4000
  5. That’s it

Core concepts

Keep in mind the following things –

  • The SUMMARY.md maps to the sidebar on the left hand side. This can be styled and the content of this file pretty much decides the navigation of your gitbook
  • gitbook is extendable through the config file – book.json, not just in look and feel, but also using plugins. My must plugins are – ["collapsible-chapters","insert-logo","image-captions","tbfed-pagefooter","copy-code-button","ga","sitemap","mermaid-gb3"]
  • Create sub-folders for different modules/services
  • Have a list of all entry points in SUMMARY.md
  • Maintain a CHANGELOG.md to have a history of major changes made
  • When a particular module becomes more complex, divide that into more parts and put those parts into nested folders. Do not forget to update the links in the respective .md files
  • Make the respective indents in the SUMMARY.md file as well

Building your gitbook

You can even host this somewhere (such as an S3 bucket or a static hosting). Simply execute the following command –

gitbook build

This will create a new _book folder in your gitbook folder. Host this as the static site.

That’s all there is to it. A simple and easy way to manage an evolving set of markdown files using gitbook.

My first Ionic App

Prasad AjinkyaLeave a comment

The couple of Phonegap apps that I worked on, things were too messed up.

  1. Is it Phonegap or is it Cordova?
  2. Do I build locally or do I used Phonegap Build?
  3. Using jQuery Mobile, which event happens first – device ready or jquery ready?
  4. Why is this too slow?

There were times, when I felt that I had absolutely no control on the app. This is where my frustrations with Phonegap started growing … not to mention the insane compile complications.

Enter Ionic

This is where I first learn about the Cordova project and how there are multiple spin-offs of that. Phonegap being just one of those … I always assumed that PG and Cordova were synonyms. Guess not.

To start with Ionic, the getting started guide to Ionic is a pretty decent place to start. Keep in mind though that it’s best that you go through some basic Angular tutorials before you dive into Ionic.

Enter Angular

The thing about javascript based apps is that you need a javascript library to run the app on. You could arguably use native JS methods and create your own bespoke app.

But that’s not really a wise choice (unless you are a freaky javascript ninja). I don’t know about you, but I for certain am not. Let me at those libraries!

This is where jqm or angular really matter. My experience with jqm was pretty bad and although I am liking angular, things seem far more in my control.