Monday, January 18, 2021

Debug Google Cloud Error - Failed to enable API please make sure you have the IAM permission to enable API

 If you are not expecting this message,  and think you have the permission to enable API read on. Before using Google Cloud services, generally we will have to enable the API by searching for it and toggle enable. This is great for permission and cost management. 

Failed to enable API please make sure you have the IAM permission to enable API

If you have the right permissions but still run into this issue, these problems could be the culprit:

  • Your account does not have billing enabled yet. Set up billing. And often Google offers a generous free quota or trial. This may change though.
  • Select the right project to work on. Double check top left of the Cloud console shows the project you are expecting.
  • Your access needs to be owner role or have the ability to create resources.
  • You can check this in the IAM dashboard. 

Friday, January 8, 2021

Modern JavaScript ECMAScript ES6

 ES6, modern JavaScript in year 2020 and 2021, is short for ECMAScript version 6. It is pronounced Ehk, Ma, Script 6. It is a standard, spec drafted by an European consortium for web technology and advancement, by convention. Developers call modern javascript ES6 to distinguish from the old ways, and signal that new conventions are involved: arrow function, and semicolons are optional. One can write a book on ES6. This blog post focuses on getting started quickly, with the basics, know just enough to be effective.

undefined : refers to the variable exists but is not defined yet.

null : exists and hasn't been explicitly set.

primitive type : means there's no method attached to it the data type. Not to be confused with the objects that usually work on primitive types. A good example of primitive type is booleans true false. There's no distinguishing float vs int just number - the unified type for all numbers. Every thing is a double 64 bits.

Implication of the number type is, there are only 64 bits so large numbers don't fit. Can use exponent notation as work around. No float also means not accurate in calculation, no floating point precision. 

promise : is a first class citizen, is native to modern javascript. It is the new pattern to use, instead of callbacks (event driven, could result in callback hell, code can be unorganized,indented). A great use case is to retrieve data from APIs. Promise is an object.

let promise = myfunction(request)

Instead to get a callback, ask the function for a promise.

You might also seen fetch() being used, but that's also the slightly older way. fetch() is native to javascript and supports promise. 

Promise can have states, status PENDING, FULFILLED (successfully resolved), REJECTED (error). Can check the promise object for states.                

Instead of constantly checking the status, we can use then() and catch() in conjunction with promise to handle next steps. 

then() accepts a function as input. Will call the function when the promise preceding it is fulfilled. 

Callbacks : the old event driven way could cause nested, or long sequence code, or a callback calling another callback, hard to manage, read, not sustainable. 

var variableName : var is familiar syntax, if omitted var then a global variable is created.  If you haven't given the variable a value at declaration, it will be undefined. If the system has to make a placeholder, the initial value is null. 

Eventloop :  The javascript event loop constantly check if the stack is empty, if yes, it will check if there's anything on the queue to be put on the stack.

Date :  new Date() generate the current date time stamp. 

Return default disable automatic browser behavior like event propagation, prevent default

Commonly used JavaScript functions:

vanilla javascript functions 





charAt opposite of indexof

ES6 include template strings

Monday, December 28, 2020

Access Denied: Project User does not have permission in project If you didn’t select a project or selected the public project

This error message appears when you try to query or create new datasets when no project has been selected. You must query and select data within a project.

A similar error message.

Access Denied: Project bigquery-public-data: User does not have bigquery.datasets.create permission in project bigquery-public-data.

if you accidentally try to query while browsing BigQuery public data.

 If you try to create a database in google bigquery public data you may also run into this issue. 

Error getting document: SyntaxError: Unexpected token u in JSON at position

 This happens when calling JSON.parse() on undefined. JSON.parse(undefined). The first token is u. That's what it mean by token u.

Sunday, December 13, 2020

Data Science Machine Learning Bootcamp series on Youtube! Starting this month

 Learn machine learning 2021, make that your exciting new year resolution.

Introducing the Youtube Data Science Bootcamp video tutorial series. 

Watch the trailer here:

Subscribe to the playlist here on youtube or visit us on

Friday, December 11, 2020

Postman for API design and API testing

Postman recently became a popular library used to design and test APIs, web apps, web products, startup products and data, machine learning products. Practically modern tech is all about web API calls.

 Postman API suites in its own words 

Use Postman Collections to keep your API requests and elements organized, to document APIs, for automated testing, and for lots more.

    Learn more about collection

Comment: collections is a great way to organize your API requests, put them into folders. This is very useful for data scientists and machine learning practitioners. 

Postman API Builder helps you define, develop, test, monitor, and manage your APIs faster. 

    Learn more about Postman API builder here.

Postman's mock servers simplify API design and planning, support split-stack development, and let you see exactly how your API will run — even before it's in production.

    Learn more about Postman API mock API here.

Postman API Monitoring allows you to review your API responses, availability, and performance with each run so you can ensure that your API is always healthy.

    Learn more about Postman API monitoring here

Friday, September 25, 2020

Firebase Tutorial - Cool things you can do with Firebase 2020-2021

Introduction to Firebase

Firebase has been a part and acted like an extension of the Google Cloud platform after being acquired by Google. All resources are organized into projects, like GCP, Google Cloud (in GCP you will also want to create an organization). Each project can be connect to apps like web apps, iOS or Android. 

You control your Firebase using the Firebase console. It is a browser based admin area. 

The console manage one or many projects. 

Project dashboard shows current app usage and statistics. Quality tab shows project health and is useful for launch apps, optimizing apps for users. Can use to segment user. 

Each data resource in each project is called a collection. Like a collection of data. Read the firebase best practice article for better understanding of firebase projects. 

You can add apps to the firebase project. 

To read firebase firestore documentation first search for the docs, then select the Guides tab, select either Get Started or Beginner Guide on the left side menu. 

When you are ready add a web app to the project. Our post focuses on Firebase SDK for web apps including node code snippets and Python. 

Firebase is modular. There's no need to include a functionality if it is not used. Some functionalities include: Analytics, Authentication, Cloud Storage, Cloud Function, Remote Config (relevant for mobile apps) etc.

What is firebase : Online, in the cloud, real time database backed by Google and integrated into Google Cloud

Firebase works on all platforms

Provides iOS SDK, Android SDK, Web SDK

Getting started developing with Firebase by install the Firebase CLI command line too. 

Full list of language and libraries found on its documentation: full list of SDK

Language Libraries
Clojure taika by Cloudfuji
Dart IO Client in the official firebase-dart library
Go Firego by Steven Berlanga and Tim Gossett
Go Firebase by Cosmin Nicolaescu and Justin Tulloss
Java firebase4j by Brandon Gresham
Perl Firebase-Perl by Kiran Kumar and JT Smith
PHP firebase-php by kreait
firebase-php by Tamas Kalman
Python Pyrebase by James Childs-Maidment
python-firebase by Özgür Vatansever
python-firebase by Michael Huynh
Ruby firebase-ruby by Oscar Del Ben
BigBertha by Fernand Galiana
rest-firebase by Codementor

Even an admin SDK for administrative access. 

Firebase and google cloud

Can add Firebase to an existing GCP project

Use Firebase API to programmatically manage a firebase project. 

Firebase works Google Analytics and other offerings of the Google Cloud.
Fire Hosting is also available

Firebase in the command line | Firebase CLI | Command line tools

Use firebase CLI to launch hosting configuration. Choose JavaScript as a language unless you know Typescript, which will result in some initial error. Use firebase deploy to deploy your app. Carefully, it really is going live. Follow instruction and use npm to install, once installed, use firebase login to login. Can now access Firebase in the command line. Use firebase init to create a project and select which functionality to use. 

Firebase init create files such as the firebase.json, which can be used to configure your firebase application, write url redirects etc. 

NoSQL, object document store using Firestore
"Following Cloud Firestore's NoSQL data model, you store data in documents that contain fields mapping to values. These documents are stored in collections, which are containers for your documents that you can use to organize your data and build queries. Documents support many different data types, from simple strings and numbers, to complex, nested objects. You can also create subcollections within documents and build hierarchical data structures that scale as your database grows. The Cloud Firestore data model supports whatever data structure works best for your app."

Store hierarchical nested objects easily. 

Firestore is a document store database, data is organized into collections, each collection is a container for one or more documents. Similar to MongoDB. No schema. Yay

Retrieving Data
Quote the documentation:

There are two ways to retrieve data stored in Cloud Firestore. Either of these methods can be used with documents, collections of documents, or the results of queries:

  • Call a method to get the data.
  • Set a listener to receive data-change events.

When you set a listener, Cloud Firestore sends your listener an initial snapshot of the data, and then another snapshot each time the document changes.

In other words, when using get you likely only need and retrieve the data once. Versus receive data-change via snapshot, your application will track firestore database side changes. This is one key difference between static retrieval and real time matching.  Firestore allows to make open socket connection with the database and keep those connections open. 

As of 2019 Firebase supports Collection Group Query allows you to query across collections of documents.  Previously: Each query has to be on a specific collection, its documents or sub collections. It cannot span across multiple collections. Important : to use this feature, must enable Indexing on Collection Group Scope to allow Firestore to index collections as if they are a sub collection of a higher collection. That logic is handled by Firestore you just have to enable it in setting. 

Implement Search Functionality in Firebase Firestore
Use the query function to do SQL like filtering. 
However, Firestore does not natively support search. Instead it recommends using Algolia a search as a service platform. Algolia requires you to upload all your data to Algolia in order to make search possible. Though the term of service seems to say A cannot compete with you, you do have to expose your data to a 3rd party, it is a bit of a tie-down. The plus side is Algolia provides search UI, one click demo generation, so you can implement a prototype fast.  

firebase first need to initialize the firebase instance

using a config variable where the tokens and api keys are stored.
You can get this config variable from your firebase console in the web browser, select add firebase to web app option. 

Cost Saving

Every firebase project corresponds with a GCP project. Be sure to set quota on the GCP project, billing alert, limits. 

To Use Firebase Function First Install Firebase Tools in the Command Line

First check that node is installed before proceeding

npm install -g firebase-tools

Will also want to log into firebase in the command line. firebase login
Initialize the project with : firebase init
Deploy with : firebase deploy. Can deploy to custom domain with an easy DNS verification. Currently there doesn't seem to be native support for minifying js files. You may want to minimize before use the firebase deploy command. Source 22
Conceptually firebase deploy upload the public folder into a Google Storage and allow you to host it as a site via Firebase Hosting. domain is included.

Select firebase function to initiate a firebase function project

When initializing a firebase function you will be asked to select a few options, configurations: for example, you will want to select the language JavaScript or Typescript. Different dependencies modules and lint will be installed. Each has pros and cons. Strong typed TypeScript may be easier for debugging testing but has a learning curve. 

Firebase Authentication

New user will be added in the Auth table, you can manually add user, enable signin providers such as google. The uid is very important, an unique identifier for the user. This is very useful when want to create web apps that serve user specific contents, which is pretty much a required functionality for all social networks.

Firebase security rules

When to use Firebase Security Rule vs when to use cloud IAM

"For mobile and web client libraries, use Firebase Authentication and Cloud Firestore Security Rules to handle serverless authentication, authorization, and data validation. Learn how to secure your data for the Android, iOS, and Web client libraries with Cloud Firestore Security Rules."

"For server client libraries, use Identity and Access Management (IAM) to manage access to your database. Learn how to secure your data for the Java, Python, Node.js, and Go client libraries with IAM."

-- Firestore documentation

Firebase security rule has a monitoring tab, where it shows rule usage data, allowed, rejected etc.


Advanced Emulator

Use an emulator offline on your local machine without invoking the firebase cloud. 

Debug Google Cloud Error - Failed to enable API please make sure you have the IAM permission to enable API

 If you are not expecting this message,  and think you have the permission to enable API read on. Before using Google Cloud services, genera...