Ad

Tuesday, June 26, 2018

Recurrent Neural Network RNN Basics


Recurrent Neural Network (RNN) is useful for processing sequence data like sound, words, and time series data . RNN remembers a bit of the state from before. It can predict what will come next. Time series is good for heart disease over time, hormone level, blood sugar. 

RNN weakness: sometimes gradients too close to 0 or too computationally large. It can also be bad at tracking long term memories - need to use LSTM instead, which has a forget gate, input gate, update layer, output layer


Real world usage:
Transform sequences like text, music, time series data,
Build a RNN generate new text character by character
Natural language processing, Word embedding, Word2Vec model, Semantic relationship between words, 
Combine embedding and RNN to predict sentiment of movie reviews

Hyperparameters in RNN

Hyperparameters are values that we need to set prior to applying an algorithm. Example: learning rate, mini batch size, epochs. There's no magical number. The optimal value depends on the task on hand. 

Hyperparameter concepts: starting values, intuitions

Two main types of hyperparameters optimizer hyperparameters and model hyperparameters. Optimizer hyperparameters related to the optimization and training process more than the model itself. Learning rate, mini batch size, number of training iterations are optimizer hyperparameters. Model hyperparameters are variables that relate to the structure of a model. Examples include number of hidden units, number of layers, and model specific hyperparameters.

Yoshua Bengio: learning rate is the most important hyperparameter. Good starting point = 0.01. Also frequently seen: 0.1, 0.01, 0.001, 0.00001, 0.000001

Intuition for starting small (important): If our learning rate is perfect - the multiplier is the best, then in rare ideal scenario we will land at the optimal point. Any learning rate smaller than the perfect rate, will still converge, and would not overshot the optimal point. If learning rate is too large, will never converge (if it is more than twice the optimal rate for example. If it is close enough to the ideal rate, it may still converge. ). Intuition that is must start small. Udacity Deep Learning Nanodegree Part 5 RNN Hyperparameter No.3 Learning Rate gives a great visual illustration.

If learning rate is too small, may take too long to converge, wasting valuable computing resources. 

Machine Learning and Deep Learning for Health Care

Two frequently used models are Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN). Both are useful for recognizing structures in data.

Convolutional Neural Network (CNN) is useful for image recognition in medical imaging, diagnosis based on medical imaging, tumor diagnosis. Recurrent Neural Network is the less obvious one. It is useful for processing sequence data like sound, words, and time series data - understanding what will come next. Time series is good for heart disease over time, hormone level, blood sugar

Machine Learning for Radiology
The paper Implementing Machine Learning in Radiology Practice and Research (Kohli et al.) concluded that Machine Learning will assist radiologist rather than replacing their jobs. However, Geoff Hinton, a leader in Machine Learning, Deep Learning and Artificial Intelligence, thinks there's no question radiologist will be replaced https://www.youtube.com/watch?v=2HMPRXstSvQ.

Image Processing using coding

This section is related to Convolutional Neural Networks CNN. These filters are also known as a kernel - a function that can change the original image.

Image processing - blurring filter: for each pixel, take a weighted pixel average of the pixels around it.

Image processing - edge detection, contrast: take differences of the pixels, detect horizontal lines or vertical lines by taking differences using positive and negative numbers.

There are other kernels that take variation of averages pixels.

One famous filter is called Sobel Filter


Convolutional Neural Network

Pixel illumination are the features, the input data,  in CNN models.

CNN helps figure out what kernels, filters, and features to detect all the way backwards using a process called back propagation.

Back propagation is Neural Networks' way to update weights as in Gradient Descent. Back propagation starts from the correct answer, update the weights backwards layer by layer until the final classifications become more correct.

Don't have to see the entire shape. Should be able to recognize partial shapes, obstructed views. For example, a half hidden cow is still a cow.

Should be able to recognize different styles. Van Gough's portrait in post-impressionist style is still Van Gough. It does not change the identity or the classification of the image.


Monday, June 25, 2018

Udacity nanodegree syllabus program length and projects

Computer Vision

+  3-Month Program

+  3 Reviewed Projects

+  3 Courses

 Robotics Software Engineer

+  4-Month Term

+  4 Reviewed Projects

+  6 Courses

 Intro to Self-Driving Cars

+  4-Month Program

+  7 Reviewed Projects

+  7 Courses

 Flying Car

+  3-Month Term

+  4 Reviewed Projects

+  4 Courses

Caution most new udacity nanodegrees require multiple terms. You may have to spend $2000 per program.

Thursday, June 21, 2018

Udacity Launches Blockchain Nanodegree

Wow Udacity really knows how to capitalize on the hype: just announced the blockchain nanodegree!

Udacity Blockchain Developer
Visit Uadacity blockchain developer nanodegree now.

What to watch out for? Udacity tends to be experimental. Being in the first class may mean better price tag, but potentially extremely hard-to-follow incoherent content. I definitely had the issue with Machine Learning nanodegree when it first came out.

It does seem like one of those startup prototypes: standard format, basic content, with some industry professionals supporting the course including: Coinbase, factom, madhive.

Definitely worth it if your company pays the bill. Otherwise, wait and see better results.

Sunday, June 17, 2018

XMLHttpRequest Basics API GET

response : variableName.status  // returns 200, variableName.statusText // returns "OK"

request: 
     open(method,url,async)
     send(string)
     
     var xmlhttp = new XMLHttpRequest();
     xmlhttp.open('GET', 'process.php', true);
     xmlhttp.send();

Response


-------
GET POST PUT DELETE
(Create Retrieve Update Delete CRUD)

  •      Get - retrieves info from the specified source
  • POST- sends new information to the specified source
  • Put - updates existing info of the specified source
  • Delete - removes existing info from the specified source
Status
  • 1xx server working on the request
  • 2xx okay
  • 3xx have to do something else first
  • 4xx error
  • 5xx server problem     

JavaScript Array Basics

    • JavaScript arrays are not truly multi dimensional they are more like arrays of arrays . Like matrix
    • Methods
      • Join()
      • Split()
      • Reverse() takes no element and reverse in place
      • Sort() takes no argument and sort in place
      • Concat()
      • Slice()
      • Splice() with bang. First arg is start pt and # of elements
      • Slice() 
      • Push() with bang
      • Pop() with bang
      • Unshift() operates on beginning of stack 
      • Shift() operates at beginning of stack
      • forEach()
      • Map()
      • Filter()
      • Every() if all passes return true else false 
      • Some() if some passes return true else false


Git and Github basics

$git config --global color.ui true
$git remote -v
$heroku run rake assets:precompile

git add <list o files>
git add --all
git add *.txt
git add directory/*.txt
git add directory/
git add "*.txt" (add all txt files of the entire project)

#compare diff between local and remote
git diff
# compare what's in the staged repo (after $add .) versus unstaged
git diff --staged
#unset changes in stage
git reset <filename>

## git head is the last commit on the current branch

#restoring file to the previous state
git checkout -- filename
git checkout -- fileone filetwo

# add and commit at the same time
# caveat does not add new files, still need to add it separately
git commit -a -m "comments here"

# undoing a commit
# move to the commit one before the current head (previous head)
# changes to go back to staging area
git reset --soft HEAD^
# or
git commit --amend -m "adding to the previous commit and also modify the message"
# or
#completely remove the previous commit
$git reset --hard HEAD^
# or completely remove the previous two commits
$git reset --hard HEAD^^


#sharing git
#dealing with remote repository
# git does not handle access control, it is usually is handled by GitHub or BitBucket

## check remote
$git remote -v
herokugit@heroku.com:whispering-springs-5104.git (fetch)
 herokugit@heroku.com:whispering-springs-5104.git (push)
 originhttps://github.com/theoptips/taskcloud.git (fetch)
 originhttps://github.com/theoptips/taskcloud.git (push)

 ##managing remotes
 $git remote add <name> <address>
 $git remote rm <name>

 ## heroku create will automatically add a remote called "heroku"
 git push <remote name> <branch name>
 git push heroku master

 ## branching
 $git checkout master
 $git merge <branchname>
 # git fast forward



## branching
$git checkout master
$git merge <branchname>
# git fast forward
# delete a git branch
$git branch -d <branchname>
# create a branch and checkout
$git checkout -b <branchname>

## always git pull from remote to ensure that there are no new changes
## if branch and master are at different timelines, i.e. not all new changes are committed
## in the branch that is about to be merged, then git will open an VI to view the changes
## git merge recursively and also create a node in the log about the merge

## merge commit
## if remote is ahead
## then have to pull
## which merge origin/master, the remote, with the local master
## then need to git push to add the new changes


## scenario: same file, conflicts
## git pull, syncs repo, pulls down all the changes, and try to merge
## discover a conflict
## need to edit the file, diff, edit manually to fix it,
## git commit -a
## vi editor will give a detailed message


#list all remote branches
$git branch -r
# help check branch histories versions
$git remote show <remotename>
#delete a remote branch ONLY
$git push <remotename> :<remote branch name>
$git branch -d <localbranchname>
# to delete LOCAL branch forcefully
$git branch -D <localbranchname>

Command Line basics


$less
space commands take us to the end of an document

$cat can take two files separated by a space
nano full screen program. Mainly used to edit a file

Move is very powerful for moving files around and renaming them. Move is also very powerful for moving directories.

$mv within the same directory

mv filename1.txt filename2.txt is just like rename

if instead say

mv filename.txt documents/ it is just to move doc to new directory

dot is a short cut for the current directory

cp can leave the original file in place, just like copy.

cp hello.txt hi.txt

Can't just copy a directory, but need to copy recursively,

$cp -r sourcedirectoryname destination directoryname

$rm command is very dangerous via command console

make directory $mkdir

cannot recurse, creating nested directory lil document/notes/console

mkdir -p documents/notes/console/part_1
this chaining works for nested directories


Create User Managing Users
console command:
print out the name of the user
$whoami

add a new user

$sudo adducer user_name

switch user

$su user_name

switch back to the previous user

$su previous_name

or can use exit which stops the previous su command


$exit

User permission
rwx read write execute
ugo user group other
drwxrwxrwx  d for directory, 9 permissions in ufo order
chmod change mode also changes permission
e.g.
$ chmod o+w hello.txt
$ chmod o-w hello.txt
$ chmod +x hello.txt (add execution for everyone)

Octal Notation:
01234567
10 11 12 13
one eight
one eight one one
one eight two one

r = 4 w = 2 x =1
first digit  = u
2nd digit = g
3rd digit = o

rwx = 7 --- = 0
r-x = 5, -w- = 2

e.g. chmod 751 hello.txt

File Owner Ship
$sudo chown username filename.txt
will be prompted for password
$cat filename.txt
$nano filename.txt
but with nano not able to write, only view after changing owner
CHANGING GROUP NAME
$sudo chown username:groupname filename.txt
$su username
CAN NOW MODIFY because username is the same as the owner. Before when looking at ano with a the old owner which no longer have write permission
$nano filename.txt

$sudo
a super user command, run as root super user without having to know the password etc, sudo grants permission to other users, will requires own password not the root user password,

$!! runs the previous command
$sudo !!

Process
When running nano which is a program, creates a process, which is a running instance of the program. There can be multiple instances.
task manager for processes, resources consumed like memory or CPU, process has an ID, owner,
$top
question mark key brings up help function
e.g. sort the processes
capital F allows sorting
q to quit the program

Another program , will list out all the processes, represent running processes
$ps
(can see bash, and ps) ps is itself a program that creates a process, only in the current session
$ps aux
all process for user x, mean all processes for all users. it is not an interactive program. good for knowing what's running and what is the process ID. PID

Getting specific with piping, grep takes some input search for pattern only return lines with the pattern on it. This will also show up in its own ps search.
$ps aux | grep "top"

Understand the Softmax Function in Minutes

Reposted from Uniqtech's Medium publication with permission. This is retrieved on May 14 2019. Uniqtech may have a newer version. Unde...