Deep Learning Papers that you must know!

Word embedding: Mikolov et al., 2013b; Pennington et al., 2014 word level embedding. One embedding per language versus LASER one embedding for all languages.
Sentence embedding: Conneau et al., 2017; Peters et al., 2018; Devlin et al., 2018 sentence level embedding. One embedding per language versus LASER one embedding for all languages.

NLP
Word2vec

Paper on data augmentation. Data augmentation scaling, rotations, mirroring, and/or cropping. The Effectiveness of Data Augmentation in Image Classification using Deep Learning Jason Wang Stanford University Luis Perez Google
http://cs231n.stanford.edu/reports/2017/pdfs/300.pdf

AlexNet (wikipedia) : "AlexNet is the name of a convolutional neural network, designed by Alex Krizhevsky,[1] and published with Ilya Sutskever and Krizhevsky's PhD advisor Geoffrey Hinton,[2][3] who was originally resistant to the idea of his student.[1][4]

AlexNet competed in the ImageNet Large Scale Visual Recognition Challenge[5] on September 30, 2012. The network achieved a top-5 error of 15.3%, more than 10.8 percentage points lower than that of the runner up. The original paper's primary result was that the depth of the model was essential for its high performance, which was computationally expensive, but made feasible due to the utilization of graphics processing units (GPUs) during training.[5]" AlexNet is similar to modern CNN architecture made significant improvement in ImageNet competition performance!

Plagiarism detection: Clough, P. and Stevenson, M. Developing A Corpus of Plagiarised Short Answers, Language Resources and Evaluation: Special Issue on Plagiarism and Authorship Analysis, In Press. 

Import Sklearn libraries and functions

from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
from lightgbm import LGBMClassifier
from category_encoders import OneHotEncoder
from sklearn.model_selection import cross_val_predict
from warnings import filterwarnings
from sklearn.impute import SimpleImputer

Differential Privacy and Federated Learning 101

A list of glossary, vocabulary for differential privacy, secure AI, and federated learning on sensitive dataset. All concepts are theoretical, for discussion purpose only, are NOT intended for production nor any professional usage, and should NOT be used. This is a very new experimental concept in AI.

Before you read our article please read our disclaimer page. It is very important to note that this article and all articles, content on our website and affiliated websites are for discussion / entertainment purpose only. They should NOT be considered professional advice. Content on this site, our affiliated sites and social media are NOT intended for commercial purpose; NOT for production purpose; NOT for professional usage.

What is a scenario that differential privacy is useful? When a researcher wants to analyze a sensitive dataset, such as a dataset containing patient data, and or when a research wants to make a model that learns sensitive features, and or make a sensitive prediction: i.e. if a patient has a health issue, such as HIV.

Differential Privacy

"mathematical definition of privacy. In the simplest setting, consider an algorithm that analyzes a dataset and computes statistics about it (such as the data's mean, variance, median, mode, etc.). Such an algorithm is said to be differentially private if by looking at the output, one cannot tell whether any individual's data was included in the original dataset or not. In other words, the guarantee of a differentially private algorithm is that its behavior hardly changes when a single individual joins or leaves the dataset " - Harvard Differential Privacy Group

Query:
See the above quote
Mean variance medium mode
Advanced: machine learning and deep learning model.

Differentially private tools

Sensitive database

Anonymization: the old way, removing sensitivity personally identifiable information, has shown to sometimes fail. No guarantee.

Example where data thought to be anonymous, fails to protect privacy:

 Latanya Sweeney showed gender, date-of-birth, zipcode can identify many Americans. This is also the governor medical record example. This is known linkage attack.

Linkage Attack:
See above

"too many innocuous (even completely random) queries about a database inherently violates the privacy of its individual contributors. ... tradeoff between statistical utility and privacy." - Harvard Privacy Group

owner: the initial virtual owner is me
pointer API for send() and get() data among virtualworkers.

Pysyft https://github.com/OpenMined/PySyft

Deep Learning using pysyft

Tech Digest Summer-Fall 2019

Google Cloud

Kubernetes Clusters "What's a cluster? It's a set of master components that control the system as a whole and a set of nodes that run containers. In Kubernetes, a node represents a computing instance. In Google Cloud, nodes are virtual machines running in Compute Engine. " - Google Cloud

Service group pods together. Each pod usually contains one container.

Public IP: allows outside visitors to access Google Cloud resources.

Deep Learning 

- Input times weight add a bias activate. 

- Deep learning at home: GTX 1080 Ti
Deep Learning Neural Networks
- Inpspired by neural networks
- These are artificial neural networks
- Connected nodes, with edges
- Model sued to estimate and or approximate functions
- Need large number of inputs

Graph Convolutional Network (GCN) : 

- Adjacency matrix is the input to GCN

- Geometric deep  learning versus 2D euclidean learning

- RNN is a chained graph, replace each unit with a recurrent unit

Facebook NLP multiple languages
https://ai.facebook.com/tools/fasttext


What function turns products into sums? Choices: Sin, Cos, Log, Exp. Answer log. Log will turn products into sum. [Deep Learning math tricks]

Secured & Private AI

Private machine learning on data you are not supposed to see. 

Expression

How to resolve XYZ?

What would you do for XYZ?

How do you solve XYZ?

Best co-working place in the world

Best exhibitions at tech conferences

Technology

How to host Discourse Forum on Google Cloud? Bitnami implementation, monthly host, public ID static address with CNAME DNS configuration.

Server for Python backend : Tornado, there's also Tornado UI to aim with fast development

Host Keras machine learning deep learning models in browser: using keras js

Visualizing convolutional layers https://www.oreilly.com/ideas/visualizing-convolutional-neural-networks

Startup devtools: Marvel, VideoScribe, Youtube shorts, grapheneDB, graph adjacency matrix, 

"Discourse is an open source discussion platform with built-in moderation and governance systems that let discussion communities protect themselves from bad actors even without official moderators." - Google Cloud market place. Discourse smartly manages who can post and how often they can post. Advanced features are only available for reputable and situated community members. Bitnami provides some security measures.

FAQ for Machine Learning

Where to find open Jupyter Notebook directory?
Standard Scaler versus MinMaxScaler. Standard Scaler MinMaxScaler strength and weakness.
http://rajeshmahajan.com/standard-scaler-v-min-max-scaler-machine-learning/ MinMaxScaler takes into X.max and X.min so it is sensitive to outliers.

Tutorials

fast.ai
Pysyft tutorial, Andrew Trask in Slack
CS50
ICME
Qwiklab games

Data Science

Pandas

Pandas.Series.value_counts

Return a Series containing counts of unique values.

The resulting object will be in descending order so that the first element is the most frequently-occurring element. Excludes NA values by default

One Hot Encoding

# TODO: One-hot encode the 'features_log_minmax_transform' data using pandas.get_dummies()

features_final = pd.get_dummies(features_log_minmax_transform)

Use lambda function in Pandas

income = income_raw.apply(lambda x: 0 if x=='<=50K' else 1)

Neo4j

(Data Scientists) -[:LOVE]->(Neo4j)

nodes are in (), relationship is in [], colon : marks the label, -> is the direction

Publication pipeline

SageMaker Youtube shorts

Jobs

YC expo, startup expo jobs on Youtube live

Technical Interviews

O(n) is just m + m + m ... It is linear.

Growth Hacking

Youtube creators can now send messages to friends. 

Udacity blockchain Blockchain Developer project list syllabus curriculum

Retrieved August 2019

• Create Your Identity on Bitcoin Core
• Create Your Own Private Blockchain
• Connect Private Blockchain to Front-End Client via APIs
• Secure Digital Assets on a Private Blockchain
• Build CryptoStar Dapp on Ethereum
• Architect a Blockchain Supply Chain Solution - Part A
• Architect a Blockchain Supply Chain Solution - Part B

Pytorch Cheat Sheet for Beginners and Udacity Deep Learning Nanodegree

 A comprehensive impressive introduction to deep learning using Pytorch.  Great for beginners



Pytorch Cheat Sheet for Beginners and Udacity Deep Learning Nanodegree

Learn Tensorflow 2.0

Tensorflow High Level API Keras
"tf.Keras which is Keras integrated as part of the TensorFlow core API. Keras continues as an independent open source project." Source-2

Keras is now a part of Tensorflow, no choice, no question of choice between tf.Keras and Tensorflow.

'No need for “Keras with TensorFlow as the backend” because Keras is now part of TensorFlow. You can continue to use Theano and CNTK etc. as Keras backend if you wish.' - Source-2

Tensorflow 2.0 architecture, functions libraries Source-2

Estimator

Tensorflow Low Level API

Tensorflow in Different Languages
Python, JavaScript, Swift is supported.

Deploy to mobile Android, iOS, mobile web, Raspberry Pi,

Web


Training on CPU GPU TPU 



Is TF.keras and stand-alone Keras the same?
The short answers is no. Tensorflow team has focused more on TF.keras, not all changes are ported over to Keras, which will have its own advocate and interest group. It is safe to say TF.keras is now a different flavor of Keras. The syntax remains similar between the two. Also Keras contains other libraries and modules and that are not to be integrated into Tensorflow.  [Source 1: #AskTensorflow Youtube]

Lazy Evaluation to Eager Execution

Tensorflow 1.0  tf.layers went away in Tensorflow 2.0.




Sources:

1. https://youtu.be/wGI_VtE9CJM
2. https://medium.com/google-developer-experts/demystify-the-tensorflow-apis-57d2b0b8b6c0