Business Intelligence Data Warehousing BIDW Basics 101

BIDW may employ more stable, heavy duty and less flexible architecture, schema and data store than startups in the Silicon Valley. Such may be a sacrifice for security, stability which many fortune companies rely on.

Structured Query Language (SQL)

Despite the popularity of many new data stores and technologies such as Hadoop, Spark, Pandas etc, many companies still require Business Analysts to be fluent in sql. Never forget SQL.

Graphical User Interface (GUI)

GUI interface helps business users query and drill data without the help of the development department. The schema and database are still designed and implemented by dev.

Online Analytical Processing (OLAP)

Provides a GUI to query platform for business users to do data explorations with minimum help from dev department. 

Analysts and decision makers can quickly and efficiently do data analysis and ad hoc reporting without too much help from a data scientist or database administrator. 

The schema, reports, and drilling depth may need to be pre-planned, designed and tested before being released to business users.

This is also a large scale system, suitable for companies such as Macy's, Gap, Walmart which have millions of new sales record per hour. 

OLAP is for data exploration by large businesses.

Data Warehousing 

Data Warehousing is a serious challenge for large companies with many transactional records, product offerings across many departments. 

Many DW providers can also provide integrated data mining, business intelligence services build on top of proprietary DW hardware (including server stack) and software.

Best Practice

• Sales teams on-the-road often needs faster, better data information on mobile devices to seal a deal. Don't be surprised if they get mad when numbers are off! They bring home the dough. 

Questions that Business Analysts and Decision Makers are Interested In

• Who are the best customers? aka Who are the customers with the best Customer Life Value
• Causal relationship: 
• Results of recent experiments (More prevalent in Startup Culture)
• Hypothesis if one segmentation is actually different from another
• Is the result significant or is it random chance
• Please note that causal relationship determination requires controlled studies to control for extraneous variables. In many industries, such as biotech, statistical significance is a must, a prerequisite for next step analysis or more business investments. 
• Demo graphics of customers. Summary statistics, customer segmentation and more. 
• How to measure profitability and other Key Performance Indicators (KPI)

SQL Basics 101

SELECT, INSERT, UPDATE with SQL

The Equivalent of HelloWorld of SQL

SELECT *

FROM table_name

Select all columns and rows from a table. In real life practice, we may want to avoid using SELECT * because it may be asking and displaying a lot of unnecessary records utilizing our precious computing resource, especially for large systems, companies with large databases. 

A Basic Select Statement

SELECT ProductID, Name
FROM Product
WHERE Price > 2.00

A Fancier Select Statement

SELECT * FROM CUSTOMERS WHERE AGE > 25 AND SEX = 'F' AND REGION='CA'

The *  means all, specifically all columns and rows in this statement. All columns and all rows will be returned. 

An Advanced Select Statement with Join Statement

SELECT p.[Name] AS ProductName,
c.[Name] AS CategoryName,
FROM SalesLT.Product AS p
JOIN SalesLT.ProductCategory AS c
ON p.ProductCategoryID = c.ProductCategoryID;

An Insert Statement

INSERT INTO table_name (column1, column2, column3, ...)
VALUES (value1, value2, value3, ...);

Useful SQL interview skills

Be able to read and comprehend SQL scripts

Be able to compose advanced sql queries including aggregation, slicing and dicing.

Advanced SQL Query Select Count and Group By

It's easy to use SQL to display all the data columns and rows. But that's not practical. It's not practical for the business user to get the entire database, nor is it memory efficient. 

How to view aggregate data? Use Group By, don't forget to use Count() too, else the result is again not meaningful. 

SELECT COUNT(CUSTOMER_ID), STATE

FROM CUSTOMERS

GROUP BY STATE

ORDER BY COUNT(CUSTOMER_ID) DESC;

Group By helps aggregate and filter out data. In this case we are interested in aggregating data by State in the Customers table. What kind of state wide information are we trying to get? We are trying to count the number of customers in each state, as measured by customer_id. In addition, once data is aggregated, order the results in a descending order by count(customer_id) the largest count to the smallest. 

Compare a Select all statement which just returns all the data rows
to
Select Count() and Group By statement that aggregates data by country

SQL is great for the following queries:

• SQL Segmentation example, analyze by location, select * from sales group by location

Additional Tools

• psql is a postgres command-line tool
• postgreSQL documentation http://www.postgresql.org/docs/9.4/static/app-psql.html
• postgreSQL is popular with Ruby on Rails users

Why should you learn SQL - common SQL usage

Swift iOS Development Core Data uses sqlite as a persistent data store.

Alternatives to SQL language, sqlite, and relational databases

• ORM and ActiveRecords used in Rails
• Hadoop uses HIVE is a SQL like language
• Spark and the new way to run SQL queries on structured, distributed data
• Firebase real time database and JSON
• JSON objects
• NoSQL databases like MongoDB

SQL Security

Cross Site Scripting and SQL Injection

If allowed to enter special characters in input boxes and forms on a website, hackers may use code to run SQL queries against your database and get data illegally about your website. Many websites do not allow special characters, such as yelp. Some websites stringify the user input before processing it on the server so special characters are turned into strings so to reduce security risk. 

3D Printing Basics 101

Learn how to 3D Print with Shapeways - Getting Started with 3D Printing

Take a 3D printing class with shapeways - a famous European printing house http://www.skillshare.com/classes/design/Introduction-to-3D-Printing-An-Easy-Start-to-Your-First-3D-Design/2097968974

My personal favorite app to get started is TinkerCAD. You don't need to know 3D modeling to get started.

Getting Started with 3rd Party Printing

Here are just a few of the ways our 3D tools make it easy to 3D print with Shapeways:

1. Check to make sure your design is ready for 3D printing
2. Reinforce designs that are too thin
3. Identify loose shells in a 3D scan
4. Save on labor cost by adding a sinter shell container to multi-part designs
5. Get feedback from our engineers if we're unable to manufacture your design

Note shapeways mail from Europe. 

Useful Apps

• 123D Scan - an app that can scan real world object into 3D object
• Shapeways 3D printing model checking tool

Android Basics - Views and ViewGroups

Common views

TextView, ImageView, ButtonView

Nested View Groups

Putting views inside other views. Nesting can be costly and get expensive if there are too many layers of nesting. 

Best practice using Android Views

• The view that contains all the views is the root view. 
• Can organize material design cards into view groups, the button, icon and image inside the card are all nested views. 
• The linearlayout horizontal and vertical orientations are extremely important.
• If the elements are not horizontally or vertically spread out, relative layout may be better.
• Draw a view hierarchy diagram to organize views
• Indent children views
• Set layout_width, layout_height dynamically using match_parent or wrap_content

Android ImageView with Drawable images

The goal is for mobile developers to load images onto mobile applications when limited memory is available.

Android drawable images @drawable/my_img can be set as the source of an ImageView. Image file extension is optional. Drawable refers to the fact that the image can be drawn on the screen. Android manage all drawables in a res/drawable directory.
https://developer.android.com/guide/topics/resources/drawable-resource

Drawable supports mainly bitmap format including .jpg, .png, .gif. The unit element for these images is a pixel.

Density independent pixels (DIP) allows ImageView to scale and resize across screen sizes and pixel densities - across the wide variety of Android devices. Specifying button size using dp instead of px make sure the button is still reasonably sized and clickable on high resolution high density screens (high number of dots or pixels per inch).

Best practice to keep file size small is to include different image sizes for handling different dip's. Android does this automatically and load the corresponding dip drawable assets into the right folder: hdpi, mdpi, xhdpi, xxhdpi.

Developers also use ImageMagik to compress photos and Android Drawable Importer to convert images to drawable https://plugins.jetbrains.com/plugin/7658-android-drawable-importer

Bash - Command Line productivity for developers

Bash can improve developer productivity. It is available on Mac via terminals. Developers can use bash to write build scripts, enhance dev productivity, use curl to visit and process websites, interact with file systems, modify files, pipe outputs into files.

• ~ current directory
• pwd command to show current working directory
• cd change current directory command
• ls list files commands
• ls -l list file with long flag to display detailed info on access, directory, owner, date, file name
• ls -a list hidden file command
• . current working directory
• .. parent working directory
• cd .. to go up a directory
• Vim is a text editor in bash

More reading

- http://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything

Machine Learning SVM

SVM can use other functions to make data linearly separable. SVM can give non linear, intricate decision boundaries. SVM Decision Boundary is a straight line for linear SVM.  Apply linear SVM. If it has 0% error, your data is linearly separable.

c parameter SVM controlls trade off between smooth decision boundary and classifying training points correctly (may not generalize well, get a smooth boundary or get more points classified correctly). Effects of C especially obvious in the RVF kernel. A large c means get more training points correctly. Larger c --> more intricate boundaries

Gamma Parameter
Gamma defines how far the influence of a single training example reaches. If gamma has a low value each pointer has a far reach, if gamma has a high value each point has a closer reach.  A high gamma value will make decision boundaries pay close attention to those points that are close, but ignore those that are far. High value of gamma could mean a very wiggly decision boundary.

A point close to the frontier can really have a lot of weight and pull the frontier close to itself. Versus a low gamma, means more points will have weights of influence on the frontier, so the frontier end up being smoother.

svm kernel http://scikit-learn.org/stable/modules/svm.html#svm-kernels


Use SVM for Stock finance https://en.wikipedia.org/wiki/Support_vector_machine