Ad

Monday, December 28, 2020

Access Denied: Project User does not have bigquery.jobs.create permission in project If you didn’t select a project or selected the public project

This error message appears when you try to query or create new datasets when no project has been selected. You must query and select data within a project.




A similar error message.

Access Denied: Project bigquery-public-data: User does not have bigquery.datasets.create permission in project bigquery-public-data.

if you accidentally try to query while browsing BigQuery public data.


 If you try to create a database in google bigquery public data you may also run into this issue. 

Error getting document: SyntaxError: Unexpected token u in JSON at position

 This happens when calling JSON.parse() on undefined. JSON.parse(undefined). The first token is u. That's what it mean by token u.

Sunday, December 13, 2020

Data Science Machine Learning Bootcamp series on Youtube! Starting this month

 Learn machine learning 2021, make that your exciting new year resolution.

Introducing the Youtube Data Science Bootcamp video tutorial series. 

Watch the trailer here:

Subscribe to the playlist here on youtube or visit us on uniqtech.medium.com

https://www.youtube.com/playlist?list=PLJhIv1LRKoqS26NBwy1opuXkhPddtjIuS

Friday, December 11, 2020

Postman for API design and API testing

Postman recently became a popular library used to design and test APIs, web apps, web products, startup products and data, machine learning products. Practically modern tech is all about web API calls.


 Postman API suites in its own words 

Use Postman Collections to keep your API requests and elements organized, to document APIs, for automated testing, and for lots more.

    Learn more about collection

Comment: collections is a great way to organize your API requests, put them into folders. This is very useful for data scientists and machine learning practitioners. 


Postman API Builder helps you define, develop, test, monitor, and manage your APIs faster. 

    Learn more about Postman API builder here.

Postman's mock servers simplify API design and planning, support split-stack development, and let you see exactly how your API will run — even before it's in production.

    Learn more about Postman API mock API here.


Postman API Monitoring allows you to review your API responses, availability, and performance with each run so you can ensure that your API is always healthy.

    Learn more about Postman API monitoring here



Friday, September 25, 2020

Firebase Tutorial - Cool things you can do with Firebase 2020-2021

Introduction to Firebase

Firebase has been a part and acted like an extension of the Google Cloud platform after being acquired by Google. All resources are organized into projects, like GCP, Google Cloud (in GCP you will also want to create an organization). Each project can be connect to apps like web apps, iOS or Android. 

You control your Firebase using the Firebase console. It is a browser based admin area. 

The console manage one or many projects. 

Project dashboard shows current app usage and statistics. Quality tab shows project health and is useful for launch apps, optimizing apps for users. Can use to segment user. 

Each data resource in each project is called a collection. Like a collection of data. Read the firebase best practice article for better understanding of firebase projects. 

You can add apps to the firebase project. 

To read firebase firestore documentation first search for the docs, then select the Guides tab, select either Get Started or Beginner Guide on the left side menu. 

When you are ready add a web app to the project. Our post focuses on Firebase SDK for web apps including node code snippets and Python. 

Firebase is modular. There's no need to include a functionality if it is not used. Some functionalities include: Analytics, Authentication, Cloud Storage, Cloud Function, Remote Config (relevant for mobile apps) etc.

What is firebase : Online, in the cloud, real time database backed by Google and integrated into Google Cloud

Firebase works on all platforms

Provides iOS SDK, Android SDK, Web SDK

Getting started developing with Firebase by install the Firebase CLI command line too. 

Full list of language and libraries found on its documentation: full list of SDK https://firebase.google.com/docs/database/rest/start

Language Libraries
Clojure taika by Cloudfuji
Dart IO Client in the official firebase-dart library
Go Firego by Steven Berlanga and Tim Gossett
Go Firebase by Cosmin Nicolaescu and Justin Tulloss
Java firebase4j by Brandon Gresham
Perl Firebase-Perl by Kiran Kumar and JT Smith
PHP firebase-php by kreait
firebase-php by Tamas Kalman
Python Pyrebase by James Childs-Maidment
python-firebase by Özgür Vatansever
python-firebase by Michael Huynh
Ruby firebase-ruby by Oscar Del Ben
BigBertha by Fernand Galiana
rest-firebase by Codementor


Even an admin SDK for administrative access. 

Firebase and google cloud

Can add Firebase to an existing GCP project

Use Firebase API to programmatically manage a firebase project. 

Firebase works Google Analytics and other offerings of the Google Cloud.
Fire Hosting is also available

Firebase in the command line | Firebase CLI | Command line tools

Use firebase CLI to launch hosting configuration. Choose JavaScript as a language unless you know Typescript, which will result in some initial error. Use firebase deploy to deploy your app. Carefully, it really is going live. Follow instruction and use npm to install, once installed, use firebase login to login. Can now access Firebase in the command line. Use firebase init to create a project and select which functionality to use. 

Firebase init create files such as the firebase.json, which can be used to configure your firebase application, write url redirects etc. 


NoSQL, object document store using Firestore
"Following Cloud Firestore's NoSQL data model, you store data in documents that contain fields mapping to values. These documents are stored in collections, which are containers for your documents that you can use to organize your data and build queries. Documents support many different data types, from simple strings and numbers, to complex, nested objects. You can also create subcollections within documents and build hierarchical data structures that scale as your database grows. The Cloud Firestore data model supports whatever data structure works best for your app."

Store hierarchical nested objects easily. 

Firestore is a document store database, data is organized into collections, each collection is a container for one or more documents. Similar to MongoDB. No schema. Yay



Retrieving Data
Quote the documentation:

There are two ways to retrieve data stored in Cloud Firestore. Either of these methods can be used with documents, collections of documents, or the results of queries:

  • Call a method to get the data.
  • Set a listener to receive data-change events.

When you set a listener, Cloud Firestore sends your listener an initial snapshot of the data, and then another snapshot each time the document changes.

In other words, when using get you likely only need and retrieve the data once. Versus receive data-change via snapshot, your application will track firestore database side changes. This is one key difference between static retrieval and real time matching.  Firestore allows to make open socket connection with the database and keep those connections open. 

As of 2019 Firebase supports Collection Group Query allows you to query across collections of documents.  Previously: Each query has to be on a specific collection, its documents or sub collections. It cannot span across multiple collections. Important : to use this feature, must enable Indexing on Collection Group Scope to allow Firestore to index collections as if they are a sub collection of a higher collection. That logic is handled by Firestore you just have to enable it in setting. 

Implement Search Functionality in Firebase Firestore
Use the query function to do SQL like filtering. 
However, Firestore does not natively support search. Instead it recommends using Algolia a search as a service platform. Algolia requires you to upload all your data to Algolia in order to make search possible. Though the term of service seems to say A cannot compete with you, you do have to expose your data to a 3rd party, it is a bit of a tie-down. The plus side is Algolia provides search UI, one click demo generation, so you can implement a prototype fast.  

firebase first need to initialize the firebase instance

using a config variable where the tokens and api keys are stored.
You can get this config variable from your firebase console in the web browser, select add firebase to web app option. 

Cost Saving

Every firebase project corresponds with a GCP project. Be sure to set quota on the GCP project, billing alert, limits. 

To Use Firebase Function First Install Firebase Tools in the Command Line

First check that node is installed before proceeding

npm install -g firebase-tools

Will also want to log into firebase in the command line. firebase login
Initialize the project with : firebase init
Deploy with : firebase deploy. Can deploy to custom domain with an easy DNS verification. Currently there doesn't seem to be native support for minifying js files. You may want to minimize before use the firebase deploy command. Source 22
Conceptually firebase deploy upload the public folder into a Google Storage and allow you to host it as a site via Firebase Hosting.  web.app domain is included.



Select firebase function to initiate a firebase function project

When initializing a firebase function you will be asked to select a few options, configurations: for example, you will want to select the language JavaScript or Typescript. Different dependencies modules and lint will be installed. Each has pros and cons. Strong typed TypeScript may be easier for debugging testing but has a learning curve. 

Firebase Authentication

New user will be added in the Auth table, you can manually add user, enable signin providers such as google. The uid is very important, an unique identifier for the user. This is very useful when want to create web apps that serve user specific contents, which is pretty much a required functionality for all social networks.

Firebase security rules



When to use Firebase Security Rule vs when to use cloud IAM

"For mobile and web client libraries, use Firebase Authentication and Cloud Firestore Security Rules to handle serverless authentication, authorization, and data validation. Learn how to secure your data for the Android, iOS, and Web client libraries with Cloud Firestore Security Rules."

"For server client libraries, use Identity and Access Management (IAM) to manage access to your database. Learn how to secure your data for the Java, Python, Node.js, and Go client libraries with IAM."

-- Firestore documentation

Firebase security rule has a monitoring tab, where it shows rule usage data, allowed, rejected etc.

Source
https://stackoverflow.com/questions/48290160/firebase-hosting-minify-served-files-mod-pagespeed
https://stackoverflow.com/questions/29300824/is-it-possible-to-do-on-the-fly-css-js-minification-combination-for-a-firebase-h

Advanced Emulator

Use an emulator offline on your local machine without invoking the firebase cloud. 

Friday, September 18, 2020

Algolia Search API Basics Tutorial

I write full time now for hi@uniqtech.co write me to say hi, request content or be notified of new tutorials like this. Unqitech writes about all the cool technologies.

  • In short Algolia is search engine SaaS software as a service
  • Tables and records in the database 
  • There is a search bar which lets you test out the Algolia service right away. And it is easy. 
  • The search results will be highlighted (yay no code)
  • They have indices important attributes
  • To display an image, include an url to image, hosted for example on google cloud
  • Note you must upload your data to Algolia for this to work
  • In the term of use it seems Algolia have a non-compete clause which theoretically prevents them from making a clone of your app
  • I have not found a way to white label it
  • The price is reasonable, based on requests. 
  • It is super easy to use, build a demo site, share it with users! But it has Algolia label everywhere. You will be doing free ad for them. 
  • Remember don't put sensitive data here. Especially not HIPAA sensitive stuff. You will have to upload your data to Algolia for the search to work. 
  • You can upload your data using a JSON file or using the dashboard. One is fast one is easy to use. Remember to use a JSONLint to check for mistakes. 
  • Will save you time if you use a lint. 
  • Remember your data is not private to any online JSON lint you use.
  • Pricing : hacker plan available, free for certain quotas. Great for testing, hackathon. 
  • Example UI is available, again not white labeled.
  • Example codes are available. Again the logo appears in the input box :(
  • Easy configuration. 
  • You can get to search in minutes! That's easy. That's a true SaaS
  • You can easily configure auto-completion. 
  • The search is typo resistant. But when the database is small, auto completion and typo handling didn't work for me. 

Wednesday, September 2, 2020

Getting started with Scikit Learn Machine Learning API - read documentat...





In this video, posted with permission, we talk about the code pattern to get started with Scikit Learn and how to read the sklearn documentation.

Thursday, July 23, 2020

AutoML Automatic Machine Learning landscape

  • Features: here are some features that may be offered
    • Automatic visualization: Automatically generate visualizations
    • Machine learning interpretability (possibly advanced feature)
      • Explain model results
        • K-LIME, Shapley, Variable Importance, Decision Tree, Partial Dependence (H2O example)
    • Model Deployment and Operations
      • Automatic deployment
      • Create REST API endpoint

H2O Driverless AI
  •  Recipes available
  •     
    

Friday, June 12, 2020

Regularization in Machine Learning, Deep Learning

Regularization can prevent overfitting and potentially make algorithm converge faster and more performant. Useful in deep learning tasks, in neural networks. Regularization acts on the loss function (cost function) by adding an extra penalty term. The penalty term depending on the method of regularization, penalizing the weight parameters so it is a function of w

Two common regularization methods:
  • Lasso 
    • Uses L1-norm
  • Ridge
    • Uses L2-norm
A trick to remember the norm is that letter L comes before letter R, so Lasso is L1 norm and Ridge is L2 norm. 

One is more likely to result in sparse solutions turning one or more coefficients zero. Which one do you think it is? 

Quiz: which formula is Lasso? Which one is ridge?

  • Regularization penalizes overly complex models
  • Large weights usually make penalty term higher, so smaller effective weights are preferred
    • Larger weights cost more
  • Regularization = regular_loss_function + extra_penalty_term(lambda, weights)
    • The extra penalty term also depends on the weights parameter and the lambda rate parameter

Thursday, June 11, 2020

WeChat Basics


  • WeChat requires approval
  • Important to decouple app with data API so we can change more after
  • Where to host WeChat app? Best hosting best API?
  • 微信开发者工具 WeChat Developer Tools
    • 三个部分组成,模拟器,编辑器和调试器
  • Developer also need a wechat account to administer and manage wechat apps and login to developer tools. It is used constantly for login, verification and testing, so it's very important. It also is like a 2 factor authentication. It is often used to verify before logging in. 
  • index.js is the main page to write your code
  • WeChat Games
    • Amazing example viral game 跳一跳
  • Performance
    • Compressed images
    • Stored locally in geo locations
  • WeChat Mini Program with AR
    • Example Armani cosmetic app allows users to try makeup
    • Other use cases for WeChat AR including gaming, real estate house previewing, open house, hotel room previewing, house shopping
    • Limited API availability, only available for some brands and developers for AR
  • WeChat voice interface
  • WeChat is a platform, not just a messaging app, it also include e-commerce, game, web browsing, search and content publishing
  • Features
  • Can create a wechat test app, previously known as sandbox
  • To test your app, click on preview on iphone to use your iphone to scan a QR code and be able to test it on your local phone
Write HelloWorld in WeChat Mini Program
When the wechat mini program launches, the onLoad function will be invoked. In index.js this code will console.log hello world. As you can see the syntax is very much similar to JavaScript and is stored in a .js script file. 

Page({

  onLoad() {

    console.log("Hello World!")

  } 

})


onLoad is a lifecycle call back function



Making WeChat Stickers - Sticker Developer
It's very important to follow community guideline and developer policies. Here's the content requirement for WeChat

Linear Algebra Review

A matrix is a grid of number

Points and vectors

||w|| length of w, also known as the magnitude of w, also the L2 norm.

U(U^T) square value of matrix is matrix U multiply by the transpose of U


Sudoku - Technical Interview

Each row can be represented using letters.Can be stored using row='ABCDEFGHI' note there are 9 letters. Iteration: for r in row : #do something

Each column can be represented using numbers. Can be store using cols ='123456789' also 9 digits. 

Use dot . as placeholder.   

Saturday, May 30, 2020

React + React Native Basics in 2020-2021

I am writing new blog posts for technologies every year because they change, they evolve. JavaScript today is nothing like the JavaScript 10 years ago. Today's topic is React and React Native. This is a narrative style cheat sheet of React Basics. It is my hope to organize those concepts in a cohesive way. It will tell you how the concepts are connected, but to find out more, to dive deeper, it is important to seek out external tutorials and resources. 

React is the declarative way to create web components, as opposed to vanilla javascript uses imperative, step by step, line by line instructional programming. ReactDOM inserts React components and render them onto a web page. Babel translates JSX to JavaScript and HTML that the browser can understand. 

Repeated elements (groups of HTML tags) are great candidates to break up and to be rewritten as components.

The most important concept in React is components. The functionality, codes are organized into components that become UI elements. Every custom react component uses the extends keyword to inherit from the React.Component class. 

Why JavaScript? It is a great time to learn modern JavaScript (ES6), you can now code both front end and back end (full stack) with JavaScript, even use it for Augmented Reality, Animation, interactivity, games, and machine learning (via Tensorflow.js). 

Post in progress, in construction. Updated daily.

Getting Started
If you want to start with a seed app, try the create-react-app
Provides basic seeding, basic scaffolding
All the javascript files are in the src folder.

Normal to see the node_module folder in all node and react apps. That's where the packages are installed. 

To get a quick start you can use CDN served React using unpackaged instead of npm install

Basic Node commands
Start the node app in command line
$ npm start

install the latest and greatest
npm install npm@latest -g

Launches the placeholder app in localhost:3000

Check versions
node -v
npm -v


Key Concepts in React:

Declarative program : opposite of imperative programming, where we specify step-by-step instruction, implementing behaviors in details. In declarative programming we tell React what we want back, like a component. HTML is a declarative, because we don't need to implement every detail just tell the browser to render a <div>.

JSX : 

JSX mix html and javascript together. Write html with javascript.

Most common way to create react element is not use createElement but using JSX
const element = <div className='container'>Hello World</div>
ReactDOM.render(element, rootElement)

React uses JavaScript to write HTML codes using JSX.
const my_html = <h1>Hello World</h1>
It is a combination of JavaScript and HTML, won't validate in vanilla JavaScript. 

JSX philosophy is perhaps for your code to resemble the web components, the declarative nature of html tags,less like pure javascript spaghetti code.


Components:
Components are great for reusable code in react. Perhaps the most important concept. 

React components, building blocks of the virtual DOM.
Every component needs a render function because it needs to know how to render the component to web page. 
React components auto attach props. When we use it it also knows which props we are referring to. It knows we want to use it

Functional component is good for a simple quick function. 

The components codes are organized into the components folder in the src subdirectory. 

To be frank, a component is a block of reusable code. 

This inside the component refers to the component 

Declarative UI

React core concept

React components rely on data passed to be dynamic

When writing babel code use the script tag <script type="text/babel"></script> instead of the regular javascript tag. 

Functional components describe how UI should look based on states and properties 

When writing JSX, we may also see {} which surrounds JavaScript expressions {my javascript expression}. it cannot interpolate complex JavaScript logic such as control flow. But can handle expressions and ternary operators. 

New component tag name needs to be capitalized. When inspect the type of object, the element will not show as a normal HTML tag such as a tag, but a special component, custom component, with custom tag name reusable using custom component. 

When initializing, component calls construct, super if any inheritance and yes generally it will, to enable the component, setState to give initial values to the state.

react.createElement() API is different from the createElement API on document
textText becomes children
just specify the type of element and pass in property value pairs as an object
can also pass in children as an array
react for creating the elements
react dom for rendering elements to the page


Babel 

Babel is the javascript compiler
tryout playground available
compile JSX to vanilla javascript


is used to convert JSX code to JavaScript, which is understood by the browser. We use the script tag with text/babel as type to tell Babel this code needs to be compiled. Babel will insert a new script tag on the page in JavaScript that the browser Js engine can understand. 

Apparently the best practice is not to use Babel stand alone. 

Libraries:
React Core
ReactDOM : insert components into DOM. Takes components insert into DOM.
Babel : compatibility, helps convert JSX to JavaScript (Browsers can understand Js code, not JSX)
React Native : Build mobile apps using just JavaScript. Write once, deploy any where. Supports iOS and Android. Dependency is React. JavaScript is bundled, transpiled from ES7 ES6 ESNext down to ES5 code. Also minified (Source: CS50 Harvard). Multiple JavaScript files compiled into a one big JavaScript bundle. Separate threads for UI, layout, JavaScript (which is Single Thread and can get locked up). Prerequisite of React Native learning is to know core React concepts

Link to React, ReactDOM, and Babel using script tags in the headers.

Basic unit of react organized around components. It inherits from React.component. Usually contains a render function. 

class CappedComponentName extends React.component{
    render(){
        return <h1>Some HTML Code</h1>
    }
}

Props : objects that are passed to elements. It looks similar to JSON, but unlike JSON which can only handle strings, props can handle other JavaScript data types. 
React uses setState to change state props.

Arrow function: 
Benefit of using arrow function, is to handle the event variable and bind this correctly. this object can get kind of funky in js. 

Best practice:

A workaround best practice to ensure compatibility is to compile JSX to JavaScript before deployment. 

What is the different between props and states. States is something the component may want to track and modify. Props is somewhat like initialization, configuration, like states but with fewer changes

Use case:
    React can convert JSON data quickly to HTML
    React is really good for repetitive HTML, generating a lot of HTML templates repetitively which vary with data
    React components make coding easy
    React helps developer handles States
    React helps HTML page display data
    
Performance : 
    Virtual DOM computing and re-rendering with delta change only makes React performant, less expensive DOM rendering, refreshing, layout change with data.
    Virtual DOM manages delta update by monitoring changes and understand what constitutes a re-render, re-render partially, instead of regenerating DOM from scratch, which is costly and expensive

In general, front end development frameworks help reduce DOM manipulation work, and improves code quality 


Post in progress, in construction. Updated daily.

Components:
React components are independent, easy to maintain, reusable. Describe how components should look based on states, properties. state can be think of as a configuration


State:
In constructor we define the state of the component, what does the component needs to keep track of
this.state = {} property value pairs. state can be think of as a configuration

State is dynamic
State contains data that can change. In contrast with props. 

State is used on a class component not a functional component. 

Initial state vs changes of state

Styling React  | Make React App Pretty
Use React UI, Material UI
React UI includes cards, breadcrumbs, tree, accordian, 

Render function
Related concept : Virtual DOM
React Render function takes 2 arguments : element, target

For the return function to span multiple lines in render use ().
render(){

    return(
        <h1>

        </h1>
    )

}

We still need to call ReactDOM.render() and give it the location and component to render in order to display the UI on the page. 

-----

If data changes, in an eligible event, can trigger changes to all the components that use the data. 

----
React modules
import React from 'react'
that's the core react module. There is also a react module for react web (react-dom) and react native (mobile). Used to be one big app. 
Even have a module for VR. 
----

Files

App.js 
main top level component, 
a wrapper for the rest of the project. Point to other component code. 

App.css 
main css file, 
Can organize css for other components, 
one css file per component. 

App.test.js 
coveres teseting
index.css --> css for the index.html

.gitignore
is good for ignoring sensitive information, large uneccessary files such as node_modules, etc.  such as data files. 

package.json
not specific to react
 common in all node projects
 contain information about the app, such as author dependencies, such as react-dom, react-scripts (used by create-react-app)
package.json the file that describes the node project
also describes dependencies

npm install command will install these dependencies

---
Best practice: 

When removing a .js or .css file be sure to also remove any related import statements.

serviceworker for push notifications.
--
React modules
react core is used for creating elements
react dom is used for rendering the elements to the DOM

----
React createElement method creates an element using React instead of using the vanilla javascript. This element can used and manipulated using react easily.

Make your React app look pretty using React Bootstrap as a starter. Animate your app using React Spring. 
----
React Fragment
Can render elements side by side which is other wise not possible
Great for rendering tables which has a lot of tags side by side
short hand <></> is the functional equivalent of fragments in JSX

Sunday, May 10, 2020

Intro to Data Visualization

REPOST from Medium with permission

Data Visualization in Machine Learning — Beyond the Basics

This is not a tutorial. These are my notes from various Machine Learning articles and tutorials. My personal cheatsheet for interviews and reviews. Any feedback and corrections are welcome. If you’d like to read more, please let me know as well. These notes are more applicable for python users. Does not include ggplot, great for R.

Prerequisites and Dependencies

This tutorial and overview is python based so we use matplotlib.pyplot. These commands can be run in command line and in Python Notebook with just a bit of modifications. Any reference to plt means the function is from the matplotlib library.
import matplotlib.pyplot as plt
# will get object does not have bar, scatter.... function_name error # if not imported

Plot a Bar Chart

Bar chart, bin chart: useful for frequency analysis, distributions and counts.
labels = ['A','B','C','D','E','F','G']
nums = [13,24,5,8,7,10,11]
xs = range(len(nums)) #[0, 1, 2, 3, 4, 5, 6]
#xs is a convention variable name for x axis
plt.bar(xs,nums)
plt.ylabel("Customize y label") 
plt.title("Customize graph label")
plt.show() #display the plot


Don’t be deceived by its simple look. Frequency analysis is very powerful in data EDA, stats and machine learning.

Plot a Histogram

Histogram will automatically divide data into bins.
import matplotlib.pyplot as plt
import pandas as pd
nums = [99, 1, 3, 5, 7,33, 23,684, 13, 3 ,0, 4]
pd.Series(nums).hist(bins=30)
# <matplotlib.axes._subplots.AxesSubplot object at 0x10d340d90>
# returns object in memory
plt.show()


Also useful for visualizing distribution and outliers.

Scatter Plot

How is scatter plot beyond the basics? Scatter plot is extremely intuitive yet powerful. Just plot the vertical coordinate and horizontal coordinate of each data point in the sample to get its scatter plot. If the relationship is non-linear, or there may be the presence of an outlier, these targets will be clearly visible in the scatter plot. In the case of many features i.e. dimensions, a scatterplot matrix can be used.
Below is a screenshot of pandas scatterplot matrix in the official documentation.


Clearly the relationship is not linear. The diagonal is the variable vs itself, so it’s showing a distribution graph instead of scatter plot. Neat, looks like the variable is normally distributed.
Scatterplot is a great first visual. Too many features? Try sampling or generating data subsets before visualizing.
Use pandas.DataFrame.describe() to summarize and describe datasets that are simply too big. This function will generate summary stats.
Scatterplots are useful for pairwise comparison of features.
Scatterplots can go beyond two dimensions. We can use marker size and color to illustrate the 3rd dimension, even 4th dimension as in the famous TED talk of economical inequality. The presenter even used timeline (animation) as the 5th dimension.

Visualizing Error

Youtube deep learning star Sraj shows a 3D visual of error function while altering y intercept aka bias and slope for linear regression. The global optima i.e. the global minimum in this case is the goal of gradient descent algorithm.
Error functions have shapes and can be visualized. Local optima which prevents your model from improving can potentially be visualized.


Gradient can be visualize as directional arrows that travel in the direction of the global minima along the shape of the 3D plot. It can also be visualized as a field of arrows in a matrix.
Each residual (y_i — y_hat) can be visualize as a vertical line connecting the data point with the fitted line in linear regression.

Data Scientists Love Box Plots

Why? It displays essential stats about distribution in a concise visual form. Aka candle stick plot. Also popular in finance.
Max, 3rd Quartile, Median, 1st Quartile, min.
This is known as the box and whisker graph too. It’s popular among statisticians. Used to visualize range. It can be drawn horizontally.
What’s between Q3 and Q1? The interquartile range, which used in analyzing outliers. Q1–1.5*IQR is too low, Q3+1.5*IQR is too high.
Box whisker plot displays outliers as a dot!
Check out Boston University’s Blood Pressure dataset box whisker plot with outliers.


Heatmap

Did you say heat map? Heat map has been in and out of favor. Web analytics still use heat map to track events and clicks on a webpage to identify key screen real estates. Why should we use heat map for machine learning?
It turns out that generating a heat map of all the feature variables — feature variables as row headers and column headers, and the variable vs itself on the diagonal— is extremely powerful way to visualize relationships between variables in high dimensional space.
For example, a correlation matrix with heat map coloring. A covariance matrix with heat map coloring. Even a massive confusion matrix with coloring.
Think less about the traditional use of heat map, but more like color is another dimension that can visually summarize the underlining data.
Correlation Matrix Heat Maps are frequently seen on Kaggle, for exploratory data analysis (EDA).


More Data Visualization Magic

Did you know that you can visualize decision trees using graphviz. It may output a very large PNG file. Remember the split of decision tree is not always stable — consistent over time. Take it with a grain of salt. The benefit of visualizing a decision tree is to understand where and how machines made decision splits. Decision tree boundaries can be visualized too, see screenshot below from Sklearn documentation.


Visualizing models, decision boundaries and prediction results may give hints whether the model is indeed a good fit or it is a poor fit for the data. For example, it is high bias to ignore the nature of our data if use a straight line to fit a circular scatter of dots.
Researchers even visualized different optimizers to see their descend to minimize loss.
Did you know you can create interactive plots using Plotly right in Jupyter Notebook? Interactive plots allow you to visualize complex data, toggle and change parameters. For example you can slide to change values of your hyperparameters and visualize how the model performance change in gridsearch and other systematic search of the space.

Wednesday, April 29, 2020

JavaScript Basic 2020

Learn what's new with JavaScript in 2020. It has changed a lot from the JavaScript you know.
  • JavaScript is interpreted as opposed to compiled
    • C is compiled
    • No need to declare variable types
    • Allows dynamic typing : given a variable there is no type associated with it until it is filled with value, it can be changed later. Some languages are not 
  • ES6 is the latest version of JavaScript full name is ECMAScript 6
    • Symbol
  • In new JavaScript languages semicolon ; is likely optional
  • JavaScript can check equality using double equal sign ==  , or triple equal sign  ===
    • == coerces the type
    • === requires to be exact, doesn't coerce the type 
  • Node command line runtime for JavaScript is built on V8 engine
  • typeof null --> returns object - one of those strange behaviors of JavaScript
  • JavaScript development is guided by ECMAScript standard. ECMA is pronounced Ehk-MA. E stands for Europe
  • You can think : the spec for JavaScript is written by ECMA
  • Each browser can have its own JavaScript engine, for example Chrome uses V8
  • Event Listener
    • Listening or subscribing events such as keydown 
    • aka the =Event handler
  • Modern JavaScript variations include Typescript, frequently used in Angular 
  • npm is the popular package manager for JavaScript
    • Has joined Github
    • In order words both Github and npm is now owned by Microsoft
  • Define a constant : const CONSTANT = 0.5
  • Enclose strings in double quotes or single quotes
  • Arrays can contain values as well as functions
    • const arr = ["value1",5, function(){console.log("Hello World")}]
      • Run the function arr[2]()
    • Can contain different types
    • Can access using indexing, starting from position zero
    • use array with for loop
for (let i = 0; i < arry.length : i++) {
    console.log(arr[i])
}

  • JavaScript allows trailing comma
  • Types
    • primitives: 
      • no methods attached?  immutable
        • boolean, string, null, number, symbol, undefined
        • Number includes both float and integer, there is no separate type
  • Sudden differences between undefined and null
  • JavaScript string
    • concatenation is implicit coercion or type casting, if we use str(variable) then that's explicit coercion
  • checking types of input using typeof, e.g. typeof undefined // --> undefined, type 5 // --> number
  • Try out JavaScript interactively using Chrome browser inspect element mode, or install node and call interactive JavaScript prompt. Use those two as a JavaScript interpreter
  • In general, undefined is returned if nothing specific is returned
  • JavaScript documentation by Mozilla https://developer.mozilla.org/en-US/docs/Web/javascript

Friday, April 3, 2020

Bootstrap Basics


  • Bootstrap is a front end framework used to quickly design, organize and beautify a modern website. It generates css fast for common front end patterns and UI elements
  • Each UI element or group of elements is called a component
  • Horizontal containers are called row s
  • Vertical containers are called col s , short for column, can be used in designing grid system
  • Bootstrap allows you to focus more on the html file rather than CSS file, write a bit less CSS
  • And no need to reinvent the wheel : writing common UIs and interactions from scratch
  • Concept : bootstrap requires using specific class names to generate desired design 
  • Pro tip: use margin to organize layout, example: can do margin left auto to push things all the way to the right  mt-3 margin top 3
  • Pro tip : use chrome inspector on Bootstrap sample and tutorial page to see what class, and configurations are used.

Grid system

Bootstrap organizes html contents into grids. Each row of the grid is called a row, each column a column. Each row has 12 columns. 

Make the website responsive

Use media query to query screen size and type. Specify the content parameter to change html content. 

Viewport is the visible area. Be sure to utilize the actual size of the phone, prevent rendering websites as desktop version on mobile device, not pretending it s desktop load. 

<meta name="viewport" content="width=Device-width, initial-scale=1.0">

Bootstrap can detect screen size and label it as lg for large, sm for small. If we use the lg and sm parameter in the class of the html element, we can specify how much space a grid column will take if the screen is small versus large (based on screen size).

<div class="row">
<div class="col-lg-3 col-sm-6"> This is a section.
</div>

<div class="col-lg-3 col-sm-6"> This is another section.
<div>
<div class="col-lg-3 col-sm-6"> This is a third section.
</div>
<div class="col-19-3 col-sm-6"> This is a fourth section.
</div>
</div>

CSS review

Pseudo class : 

selector:pseudo-class {
  property: value;
}

w3schools

example 
a:link {
  color: #FF0000;
}

but more importantly in modern css
::after
::before
are two important pseudo class

p::after { 
  content: " - add a foot note";
}


Friday, March 27, 2020

My Little Green Book of Machine Learning and Deep Learning, Artificial Intelligence

Data pre-processing

Turn Complex Data into Numbers

Turn data into features. Turn data into feature vectors. Machine Learning models can only take numeric data. All input data must be represented numerically. For example, words need to be converted to word embeddings in some Natural Language Processing tasks. 

Training vs Inference Models

There are two major tasks in machine learning 1. build and train a model 2. deploy a model for inference. Part 1 takes known data, uses it to tune parameters of the model such as weights. Part 2 takes in unknown data, real world data or test data and calls a dot predict method on the new data. 

Normalization, Scaling Data

Normalize data, need to scale data to bound it. For example in Machine Learning, an error term can be arbitrarily large because the model can be arbitrarily bad, causing the lower bound of error term for f(x) = wx+b to be essentially unbounded. Bounding the error term by scaling the features numeric value can make the result easier to compute and make the search space easier for gradient descent.

Bias Variance Tradeoff

High bias may refer to underfitting, where the model is too simple, not complex enough to make accurate predictions. It can also mean when the model is practically ignoring the data.

High variance may refer to overfitting. That's when the model overfits, hence cannot generate to future data well. 

Tuesday, March 24, 2020

Natural Language Processing (NLP) 2020

It is year 2020 and vision 20/20. It is time to do another survey article of Natural Language Processing (NLP) field. What is there to learn / know? What is new?

Getting started with NLP

Great sources for NLP

Social media : Twitter, Facebook, comments, posts, forums
Transcripts : events, conferences, speech, call transcripts, zoom transcripts
Smart voice assistants : Alexa, Siri, Google Home

Libraries for NLP

SpaCy: works for Japanese, Chinese and English up to 45 languages. Source 4
Scikit-Learn sklearn TFIDF vectorizer

Advanced NLP

Machine Translation
Transfer learning in NLP

Algorithms

Latent Dirichlet Allocation (LDA) topic modeling

Projects

Sentiment analysis
Fake news classifier
Trump tweet maker

Can combine sentiment analysis with tweet analysis. Great Natural Language Processing (NLP) projects for hackathon : retrieve keywords from tweets (entities recognition for hashtags, brand names), pipe the result into a sentiment analysis model, predict sentiment negativity to positivity zero stars to five stars. 

Monday, March 23, 2020

Growth Marketing - Technical Marketing


  • Hiring people can be a pitch for your app, startup 
  • How to use freemium monetization on content. Blur out important infographics images in newsletter, create a digital tease to motivate user to leave newsletter and head to content page 
  • Growth hack vanity address
  • Embed survey in emails. 
  • Use AMP for interactive emails. Easy to fix typos and content swamp if email is generated dynamically. 
  • When working with a Growth Manager, get to know what's her style, what's her vision. 
  • Turn super users into community managers, staffs, employees
  • Unfortunately on Youtube drama drives clicks. Dramatic Youtubers seem to grab attention. There was one joke: Uber hits a pothole, Youtuber describes it Gosh I almost died today.
  • Startups are all about growth. Funded startups are definitely about growth. Even bootstrapped startups need to think about growth, large scale growth asap. 
  • Youtube
    • Very important to have attention grabbing thumbnails

Advanced Python


  • Installation
    • import numpy as np
  • Request is deprecated
  • Python 2.x is deprecated (Python 2.7 is usually pre-installed on Macs 3-7 years from 2020). 
  • BeautifulSoup
  • Pyecharts
  • Scrapy
  • Pycharm
  • mylist = [1,2,3,4]
  • np_array= np.array(mylist)
  • Slicing
    • Entire list mylist[:]
    • Slicing zeroth to 0th, 1st element mylist[0:2]
    • The second position is exclusive
  • Function signature
    • What type of input is expected? Example CSV
    • What type of output is expected?
    • What is the functionaltiy
    • Each functionality does one task
  • Using an IDE
    • Anaconda Spyder (Scientific Python Development Environment)
    • PyCharm
    • Why use an IDE: 
    • tool kit plugin
    • pycharm can refactor update references when files move
    • get styling hints: json does not allow trailing comma
    • error highlighting
  • which python
    • this command is used to check which python version is launched
  • pip is the package management tool. It is used to install python software packages.  Can also use pip to install another package manager, specializing in data science tasks called anaconda.
  • In any language, do not overly rely on print statements for debugging.

Sunday, March 22, 2020

Technical Interview Tips - Technical Interview with Python cheat sheet


  • Time complexity
    • Big O Exponential 2**n
    • Worst case, Big O
    • Best scenario
    • Average 
  • Data structure & algorithms
  • Test case: ZeroDivisionError
  • Implement function from scratch
  • Tree algorithms
  • Use pointers while lo < hi : do xyz lo = 0th index hi = len(input)-1
  • Language: Java, Python slightly preferred
  • Exponential 

Firebase APIs Basics


  • A project is a container for resources on Google Cloud
  • Install Firebase tools first to use command line utilities
  • Defer tag in html, means don't load the resources until the page finishes loading
  • Firebase serve local port 5000

Google Cloud Basics


  • .yaml extension of configuration file
  • Export python packages and environment dependencies as requirements.txt
  • name.py python code
  • Google Cloud Function
    • By default cloud function is authenticated into other Google APIs
    • Serverless, fully managed, event triggered, considered a micro-service in the cloud
  • Google Cloud OAuth
  • Use stackdriver (build on AWS) to track performance
  • Role management using IAM
  • Google Cloud Discount
    • Google Cloud discount for students available
  • Cloud DNS is an available API
    • Add a record ANAME find ipv4 address
    • Copy external IP address
    • ANAME record is linked to external IP
    • CNAME www is connected to domain name .com
    • Now go to domain provider and do the same
    • Add the google domain one to the name server
    • Need generate an SSL a certificate to enable
  • Integrates with Firebase
    • Firebase APIs
  • Resources - Conference: Google Cloud Next Conference usually in April
  • Static versus ephemeral IP address
  • Market place: Google Cloud wordpress
  • Google Cloud partner: top partners include Accenture
  • Tutorial : Qwiklab
  • Concept : server less on the cloud, runs code in the cloud don’t need to know or manage what machine infrastructure it is operating on. No need to worry about automatic scaling, load balancing, security, patch.
  • Server side resource and library security is managed using IAM, which gives users roles. 

Friday, March 20, 2020

Learn SQL — It’s on every job listing — Part 1

SQL is not obsolete. You can now build Machine Learning models with SQL, query real time or big data with SQL. It is true if you look around you will find plenty of job postings with SQL as a desired skill, even from FANG companies (Facebook, Apple, Netflix, Google). Uniqtech writes technical tutorials for coding bootcamp graduates, free lancers, self-study, MOOC students who are in the realm of data science, software engineering, machine learning and deep learning. Read our disclaimer here. This disclaimer applies to our entire site. Please take our words with a grain of salt. They are not considered professional advice nor are they considered professional opinions. Repost from Uniqtech Medium with permission. 

Microsoft Excel is a workbook that contain work sheets just like database contains tables.
Each table can be queried separately. To query tables, jointly, we will need to use join statements and keys to look up the corresponding data.
Each table row should have a unique ID, known as the primary ID. It can also have a foreign key (FK) which associates the row, aka record, with a unique primary ID of another table.
For example each e-commerce transaction has an unique ID, which can be generated with the timestamp of when the transaction happened. Each transaction ID can have a FK such as customer ID, which uniquely identifies the customer that made the transaction. His or her full information resides in the customers table.
That is the perfect sequel to talk about the philosophy and convention behind table names. You can think of table names are natural division of the data we want to model in forms of nouns, and in noun plural form: transactions, customers, products etc. Each row in the transactions table is a transaction (singular). Each row in the customers table is a customer. Each column represents a customer attribute, such as gender, age etc.
When designing the database, an architect or Database Admin (DBA) will construct a digital blue print stating how the tables are connected with each other or they are stand alone in the database. This diagram and the relations it specify is called the database schema.

What is SQL

SQL is a database query language. It doesn’t matter what relational database you use, SQL concepts are helpful. Pandas analytics library uses similar joins, query methods. Google BigQuery allows SQL like syntax.
Newer database such as NoSQL and graph databases use different query languages. Sample code from Google Cloud Datastore nosql database
1. // List Google companies with fewer than 400 employees.
2. var companies = query.filter(‘name =’, ‘Google’).filter(‘size <’, 400);

Important SQL Keywords

SELECT

The one select statement to select them all is using the wildcard.
SELECT * FROM table_name
It is important to slow down and read the statement. It reads: select all from table_name. * means all columns.
Nested Select Statements
SELECT * FROM (SELECT "A" AS A, "B" AS B);
AS specifies the alias. When column names are not reader friendly or long, alias is your friend.
It selects the column of data.

FROM

The FROM keyword is usually followed by a tablename. FROM database.CUSTOMERS . It can also be followed by a nested query.
It specifies the table to operate on.

WHERE

Where clause narrows down the query results by specifying conditions such as where TABLE_NAME.gender == 'Female' . It works on filtering the rows of data.

Putting it all together SELECT FROM WHERE

query = """
    SELECT my_column
    FROM my_table AS m
    WHERE m.gender = ‘F’
 """

WITH

“The SQL WITH clause allows you to give a sub-query block a name (a process also called sub-query refactoring), which can be referenced in several places within the main SQL query.” — Geek for geeks

ORDER BY

Sort the query result by columns ascending or descending.
ORDER BY ASC
ORDER BY DESC

LIMIT

LIMIT 1000
LIMIT 25
Show the first xx number of rows of records in a table.
Usually at the end of the query. The last line in SQL query.
Note in big data, where managing cost and resource use is important, LIMIT does not mean the entire database is not queried.

Data Structure and Algorithms

Time Efficiency and Space Efficiency Both Matter

How to write effective tests cases

Wednesday, March 18, 2020

Google Colab Basics

In Google's own words: Colab is zero configuration, free GPU, and easy to share. I honestly have to agree with that. Google Colab is the easiest environment to get started on machine learning with scikit learn, or deep learning with Tensorflow and Pytorch. Seriously, zero installation is awesome! Back in the days, when Ruby on Rails was hot, we had installation parties all the time. Because that's what took the most time, for every one. Now Colab even has access to free TPU!

Google Colab is basically like Google Doc is for Microsoft Office as it is for Jupyter Notebook. It basically lets you create, edit and host Jupyter Notebook in the cloud.

Google Colab for Training Models

Training is essentially free and easy on Google Colab. You have access to both GPU and TPU. Though the free version can lose temporary variables, files in the home directory, because it refreshes every 12 hours or less. There are ways to save and download the files to avoid such catastrophe. 

Use Google Colab for Demo Purpose

It is easy to build an example in Colab and share it with audience, give it away instead of Github source code, instead of slides.

Coding for Good, Coding as a Service resources

 Did you know that being a developer you are at an unique position to help the society? There are many public service opportunities to contr...