Building a Binary with arguments and options - flags package in golang

In this post, I will provide information about building a go program with arguments and options.

Typically, an executable can contain

• Arguments: Go expects these to be provided after all the options.
• options (specified with - & provided with an option value). Example: "-srcDir /tmp". Go calls these as "flags".
• flags: True if specified, else false. Example: "-e" to enable/disable. Go doesn't support this type of flags !!!

Here is a sample binary I built to test go flags. Sample outputs running the above binary:

• How to get help text?

 

So, we get help text free if we use flags package !! 

• Run the program without any inputs:

• Run the program without any additional arguments

                

• Run the program with an additional argument

      

• 
  

Some drawbacks I see with go flags package:

• As we saw earlier, go flags package doesn't provide a way for us to specify mandatory options. We need to either write additional code (like I provided above) or use custom packages such as go-flags
• go doesn't support true/false kind of flags

TBD:

• flagSet (building subcommands)

References:

https://golang.org/pkg/flag/
https://blog.rapid7.com/2016/08/04/build-a-simple-cli-tool-with-golang/
https://gobyexample.com/command-line-flags
https://stackoverflow.com/questions/31786215/can-command-line-flags-in-go-be-set-to-mandatory/31786480#:~:text=The%20flag%20package%20does%20not,halt%20with%20an%20error%20message

Volatile Variables in Golang

   

In the past, I have used lockless code when there is a single reader and single writer and the reader can accept stale information to some extent (for example, this sample code on my github) .

However, there is a problem in this code. If the two go routines run on different cpu cores, then the reader go routines may never get changes from writer go routine? So, the solution I typically employed in languages such as C/C++ is to use volatile type for flag. However, go doesn't seem to support volatile variables as per these discussions: google groups discussion, stackoverflow discussion!! 

So, the solution for the above situation is to use an atomic variable. To find the correct way of writing the above code, please check here.

Skipping Go Race detector for some packages

  

In my previous post, I covered the basics of Go Race detector. In most Golang, projects, we would have multiple projects. 

For some reason, if we want to skip race detector for some packages, we can use this option as the first line of the package & an empty second line:

// +build !race

I ran the sample code on my github with the above option to get this:

Babus-MacBook-Pro:Race_Tests_inGolang babuneelam$ go build -race

package racetest: build constraints exclude all Go files in /Users/babuneelam/Projects/GoProjects/Race_Tests_inGolang

Babus-MacBook-Pro:Race_Tests_inGolang babuneelam$ 

I have only a single package in this, so this error. If I were to have multiple packages in this repository, it would have build those.

Exploring Go Race Detector

   

I got to know about Golang Race Detector from a colleague & wanted to give this a try. Here is the sample code on my github  I tested (also, same code on Go playground for quicker testing).

We can use race detector in multiple ways:

1) Using go run -race with the specific go file:

Babus-MacBook-Pro:Race_Tests_inGolang babuneelam$ go run -race main.go

flag:  true

==================

WARNING: DATA RACE

Write at 0x0000012742a1 by goroutine 7:

  main.writer()

      /Users/babuneelam/Projects/GoProjects/Race_Tests_inGolang/main.go:12 +0x55

Previous read at 0x0000012742a1 by main goroutine:

  main.reader()

      /Users/babuneelam/Projects/GoProjects/Race_Tests_inGolang/main.go:19 +0x3e

  main.main()

      /Users/babuneelam/Projects/GoProjects/Race_Tests_inGolang/main.go:27 +0x5e

Goroutine 7 (running) created at:

  main.main()

      /Users/babuneelam/Projects/GoProjects/Race_Tests_inGolang/main.go:26 +0x59

==================

flag:  true

^Csignal: interrupt

Babus-MacBook-Pro:Race_Tests_inGolang babuneelam$ 

 

2) Build the go program with "-race" flag:

Babus-MacBook-Pro:Race_Tests_inGolang babuneelam$ go build -race

Babus-MacBook-Pro:Race_Tests_inGolang babuneelam$ 

Then, run the program:

Babus-MacBook-Pro:Race_Tests_inGolang babuneelam$ ./racetest 

flag:  true

==================

WARNING: DATA RACE

Write at 0x0000012742a1 by goroutine 7:

  main.writer()

      /Users/babuneelam/Projects/GoProjects/Race_Tests_inGolang/main.go:12 +0x55

Previous read at 0x0000012742a1 by main goroutine:

  main.reader()

      /Users/babuneelam/Projects/GoProjects/Race_Tests_inGolang/main.go:19 +0x3e

  main.main()

      /Users/babuneelam/Projects/GoProjects/Race_Tests_inGolang/main.go:27 +0x5e

Goroutine 7 (running) created at:

  main.main()

      /Users/babuneelam/Projects/GoProjects/Race_Tests_inGolang/main.go:26 +0x59

==================

flag:  true

flag:  true

^C

Babus-MacBook-Pro:Race_Tests_inGolang babuneelam$ 

 

3)  go test -race mypkg - TBD

4)  go install -race mypkg - TBD

As we can see the go race detector displayed us two pieces needed:

1. A race warning  
2. The two code locations that are causing the race

So, it's quite useful. To find the correct way of writing the above code, please check here.

However, this is not a static analysis tool. So, we need to run the program to detect the races. And only those races that happen during execution are detected. If we use race as part of test cases, then race detector is only as effective as your test coverage !!

In terms of performance, Golang Race Detector says this:

The cost of race detection varies by program, but for a typical program, memory usage may increase by 5-10x and execution time by 2-20x.

The race detector currently allocates an extra 8 bytes per defer and recover statement. Those extra allocations are not recovered until the goroutine exits. This means that if you have a long-running goroutine that is periodically issuing defer and recover calls, the program memory usage may grow without bound. These memory allocations will not show up in the output of runtime.ReadMemStats or runtime/pprof.

So, this is definitely not for production environments.

List of Standard Packages and Sub-repositories in Golang

 
I have been working in Golang for a while now. However, I only explored some packages like fmt, context, sync, time etc. But, I always wanted to know the entire list of standard packages available in Golang. 

Today, I found the list here: Go Standard Packages

Also, check sub-repositories here: Go sub-repositories

In future, I also would like to research a list of useful third party packages as well. 

Online IDE for Golang: Go Playground

Golang has an online IDE here: Go Playground. Here we can experiment sample programs and share code snippets with others. As per the discussion in the google groups here, the shared urls seem to be retained forever - at least for now.

The online IDE doesn't autofill methods, imports etc like IDEs like IntelliJ, Eclipse do. That would have been even nicer.

More details about this playground:

• Go Playground Implementation Details
• Source code of go playground

Also, "About" page on the online IDE page has a bit more details such as limitations etc.


TBD:

• How do we run a go program in the online IDE with input arguments?

Setting up a Go project in Vscode

  

Here are the steps I followed to setup a Go project in Vscode:

1) Install Vscode Go plugin.

2) Create a project

3) I still had this error:

4) To fix this, I downloaded the go 1.15 separately.  

5) Set GOPATH - export PATH=$PATH:/usr/local/go/bin/go 

Even after this, I am still not able to run my go program directly from UI. I could only the go program from the terminal.

One of my friends who uses Vscode helped me run go programs from vscode. He suggested these additional steps:

1) Open settings in vscode

 

2) Search for "gopath" & open "settings.json"

 

3) Set goroot & gopath in this file

 

4) After this, I had to restart vscode. After restart, vscode threw some pop ups to install a few things & I proceeded with those:

 

Tools environment: GOPATH=/Users/babuneelam/Projects/GoProjects/go_path

Installing 1 tool at /Users/babuneelam/Projects/GoProjects/go_path/bin in module mode.

gopls  

Installing github.com/mdempsky/gocode SUCCEEDED

Installing github.com/uudashr/gopkgs/v2/cmd/gopkgs SUCCEEDED

Installing github.com/ramya-rao-a/go-outline SUCCEEDED

Installing golang.org/x/tools/gopls SUCCEEDED

All tools successfully installed. You are ready to Go :).

Installing github.com/acroca/go-symbols SUCCEEDED

Installing golang.org/x/tools/cmd/guru SUCCEEDED

Installing golang.org/x/tools/cmd/gorename SUCCEEDED

Installing github.com/cweill/gotests/... SUCCEEDED

Installing github.com/fatih/gomodifytags SUCCEEDED

Installing github.com/josharian/impl SUCCEEDED

Installing github.com/davidrjenni/reftools/cmd/fillstruct SUCCEEDED

Installing github.com/haya14busa/goplay/cmd/goplay SUCCEEDED

Installing github.com/godoctor/godoctor SUCCEEDED

Installing github.com/go-delve/delve/cmd/dlv SUCCEEDED

Installing github.com/stamblerre/gocode SUCCEEDED

Installing github.com/rogpeppe/godef SUCCEEDED

Installing github.com/sqs/goreturns SUCCEEDED

Installing golang.org/x/lint/golint SUCCEEDED

All tools successfully installed. You are ready to Go :).

Tools environment: GOPATH=/Users/babuneelam/Projects/GoProjects/go_path

Installing 1 tool at /Users/babuneelam/Projects/GoProjects/go_path/bin in module mode.

goimports

Installing golang.org/x/tools/cmd/goimports SUCCEEDED

All tools successfully installed. You are ready to Go :).

5) Run & Launch. We can also set breakpoints too now.

 

By setting up vscode, I could avoid paying for Goland/IntelliJ, atleasr for now :)

Hope this helps.

How to sync local repository sources from the github repository?

At times, the local sources repository could be obsolete as the gitgub repository changes after we imported. In this post, I will provide the command required to sync the local repository from the github repository.

babu@babu-VirtualBox:~/EKiD$ git pull origin master
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 3), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (5/5), done.
From https://github.com/babuneelam/EKiD
 * branch            master     -> FETCH_HEAD
   2005d11..46764be  master     -> origin/master
Updating 2005d11..46764be
Fast-forward
 .README.swp | Bin 12288 -> 0 bytes
 README      |  51 ++++++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 42 insertions(+), 9 deletions(-)
 delete mode 100644 .README.swp
babu@babu-VirtualBox:~/EKiD$

TBD: How are conflicts between local and remote changes handled?

Hope this helps.

How to import from and export changes to an existing github repository?

In this post, I will explain how to import sources from an existing github repository to a local directory, make changes locally as necessary and then export the changes back to the github repository.

• Import from your github repository to local directory:

babu@babu-VirtualBox:~$ git clone https://github.com/babuneelam/EKiD.git
Cloning into 'EKiD'...
remote: Counting objects: 54, done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 54 (delta 6), reused 46 (delta 4), pack-reused 0
Unpacking objects: 100% (54/54), done.
Checking connectivity... done.
babu@babu-VirtualBox:~$ cd EKiD/
babu@babu-VirtualBox:~/EKiD$ ls
ConfigControls  LICENSE  README  SampleOutputs  Sources
babu@babu-VirtualBox:~/EKiD$ 

• Make change as necessary

• See the diff

babu@babu-VirtualBox:~/EKiD$ git diff
diff --git a/ConfigControls/EnvCfg b/ConfigControls/EnvCfg
index 485a346..63a1976 100644
--- a/ConfigControls/EnvCfg
+++ b/ConfigControls/EnvCfg
@@ -1,5 +1,5 @@
-GECKO_DRV_PATH=/Users/babuneelam/Downloads/geckodriver
-FONTS_FOLDER=/Library/Fonts
+GECKO_DRV_PATH=/home/babu/bin/geckodriver
+FONTS_FOLDER=/usr/share/fonts/truetype/msttcorefonts/
 #GECKO_DRV_LOG_FILE=RunTimeInfo/geckodriver.log
 #OUTPUT_DIR=RunTimeInfo/output
 #IMAGE_STORE_PATH=RunTimeInfo/google_images
babu@babu-VirtualBox:~/EKiD$ 

babu@babu-VirtualBox:~/EKiD$ git diff --name-only
ConfigControls/EnvCfg
babu@babu-VirtualBox:~/EKiD$ 

• Tell git who you are:

      git config --global user.email "you@example.com"

      git config --global user.name "Your Name"

• TBD: Add/Remove existing files from git repository
• Export ALL the local changes to the github repository:

      babu@babu-VirtualBox:~/EKiD$ git commit -a
        [master 32a66e9] A few fixes
         2 files changed, 3 insertions(+), 3 deletions(-)
      babu@babu-VirtualBox:~/EKiD$

        babu@babu-VirtualBox:~/EKiD$ git pull origin master
        From https://github.com/babuneelam/EKiD
        * branch            master     -> FETCH_HEAD
        Already up-to-date.
      babu@babu-VirtualBox:~/EKiD$ 

      babu@babu-VirtualBox:~/EKiD$ git push -u origin master
        Username for 'https://github.com': babuneelam
        Password for 'https://babuneelam@github.com':
        Counting objects: 6, done.
        Compressing objects: 100% (6/6), done.
        Writing objects: 100% (6/6), 604 bytes | 0 bytes/s, done.
        Total 6 (delta 4), reused 0 (delta 0)
        remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
        To https://github.com/babuneelam/EKiD.git
           46764be..32a66e9  master -> master
        Branch master set up to track remote branch master from origin.
      babu@babu-VirtualBox:~/EKiD$ 

• Export only a few files of the local changes to the github repository:

babu@babu-VirtualBox:~/EKiD$ git commit ConfigControls/AlgoCfg -m 'Documentation fixes'

[master 4668ebb] Documentation fixes

1 file changed, 8 insertions(+)

babu@babu-VirtualBox:~/EKiD$ git commit ConfigControls/FontCfg -m 'Documentation fixes'

[master a3437e1] Documentation fixes

1 file changed, 5 insertions(+), 2 deletions(-)

babu@babu-VirtualBox:~/EKiD$

babu@babu-VirtualBox:~/EKiD$ git pull origin master

From https://github.com/babuneelam/EKiD

* branch            master     -> FETCH_HEAD

Already up-to-date.

babu@babu-VirtualBox:~/EKiD$ git push -u origin master

Username for 'https://github.com': babuneelam

Password for 'https://babuneelam@github.com':

Counting objects: 8, done.

Compressing objects: 100% (8/8), done.

Writing objects: 100% (8/8), 1.02 KiB | 0 bytes/s, done.

Total 8 (delta 3), reused 0 (delta 0)

remote: Resolving deltas: 100% (3/3), completed with 1 local object.

To https://github.com/babuneelam/EKiD.git

  b61c9a5..a3437e1  master -> master

Branch master set up to track remote branch master from origin.

babu@babu-VirtualBox:~/EKiD$

Hope this helps.

How to add collaborators to your github repository?

Recently, I wanted to invite my friends to collaborate on a project I published on Github. In this post, I will provide steps I used to invite a collaborator for my github repository.

• Go to the github repository page. Say https://github.com/babuneelam/EKiD.

• Click "Settings" Tab 

• Click "Collaborators" on left hand side

• In the text box, search by providing name or github username or email address of the collaborator you want to invite. It automatically displays matching github users. Select correct user from among the drop down entries.

• Click "Add Collaborator"

• Now, github sends the invite to the collaborator. Once the collaborator accepts the invite, he/she can start contributing !!

Hope this helps.

How to add local project sources to a new github repository?

In this post, I will provide steps to publish source code from a local machine to github repository.

• Create a new GitHub repository. Use "+ --> New Repository" at the top right & the follow the steps there after.
• In the local machine containing your project source code, open a terminal.
• Go to the project sources directory.
• git init (initializes git repository locally)

Babus-MacBook-Pro:EKiD babuneelam$ git init
Initialized empty Git repository in /Users/babuneelam/EKiD/.git/
Babus-MacBook-Pro:EKiD babuneelam$ 

• git add . (adds all the files to local git repository)

Babus-MacBook-Pro:EKiD babuneelam$ git add .
Babus-MacBook-Pro:EKiD babuneelam$ 

• git config --global user.name "Your Name"
• git config --global user.email you@example.com
• git commit -m "First Commit" (commits all the code to local repository)

                       Babus-MacBook-Pro:EKiD babuneelam$ git commit -m "First Commit"

                                  Committer: Babu Neelam <babuneelam@Babus-MacBook-Pro.local>

                                  Your name and email address were configured automatically based

                                  on your username and hostname. Please check that they are accurate.

                                  You can suppress this message by setting them explicitly. Run the

                                   following command and follow the instructions in your editor to edit

                                   your configuration file:

                                         git config --global --edit

                                    After doing this, you may fix the identity used for this commit with:

                                         git commit --amend --reset-author
                                  41 files changed, 674 insertions(+)

                              create mode 100644 .README.swp

                              ....

                              .... ....

                     Babus-MacBook-Pro:EKiD babuneelam$ 

• Go to the Github repository page. In "Clone or download" dropbox, copy the github web url. In my case, it was https://github.com/babuneelam/EKiD.git.
• In the terminal, git remote add origin <github-web-url>

Babus-MacBook-Pro:EKiD babuneelam$ git remote add origin https://github.com/babuneelam/EKiD.git
Babus-MacBook-Pro:EKiD babuneelam$ 

• git merge origin/master --allow-unrelated-histories (pulls files from github-web-url to local repository

Babus-MacBook-Pro:EKiD babuneelam$ git merge origin/master --allow-unrelated-histories



Merge made by the 'recursive' strategy.

 .gitignore | 101 +++++++++

 LICENSE    | 674 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 2 files changed, 775 insertions(+)

 create mode 100644 .gitignore

 create mode 100644 LICENSE
Babus-MacBook-Pro:EKiD babuneelam$

• git push -u origin master (pushes the local project's changes to the github repository)

Babus-MacBook-Pro:EKiD babuneelam$ git push -u origin master



Username for 'https://github.com': babuneelam

Password for 'https://babuneelam@github.com': 

Counting objects: 43, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (41/41), done.

Writing objects: 100% (43/43), 10.96 MiB | 593.00 KiB/s, done.

Total 43 (delta 1), reused 0 (delta 0)

remote: Resolving deltas: 100% (1/1), done.

To https://github.com/babuneelam/EKiD.git

   b447218..8d823bf  master -> master

Branch master set up to track remote branch master from origin.
Babus-MacBook-Pro:EKiD babuneelam$ 

• Refresh the github repository to see the local changes on the github !!

Hope this helps.


References:
https://help.github.com/articles/adding-an-existing-project-to-github-using-the-command-line/
https://help.github.com/articles/creating-a-new-repository/
https://stackoverflow.com/questions/38255655/trying-to-pull-files-from-my-github-repository-refusing-to-merge-unrelated-his

Resources to learn Python language

I am currently learning Python language. In this post, I will go over some resources (work in progress) that helped me pick up Python.

• Websites
• https://developers.google.com/edu/python/ -- could pick up basics pretty fast from these
• https://docs.python.org/2.7/tutorial/index.html -- this seemed comprehensive, but couldn't get some concepts as clearly as I wanted.
• https://www.programiz.com/python-programming 
• Videos
• Python classes from Google Developers: Could pick up basics pretty fast from these videos.
• https://www.youtube.com/watch?v=tKTZoB2Vjuk 
• https://www.youtube.com/watch?v=EPYupizJYQI 
• https://www.youtube.com/watch?v=haycL41dAhg 
• https://www.youtube.com/watch?v=kWyoYtvJpe4 
• https://www.youtube.com/watch?v=uKZ8GBKmeDM 
• https://www.youtube.com/watch?v=Nn2KQmVF5Og 
• Python OOP classes from Corey Schafer: 
• https://www.youtube.com/watch?v=ZDa-Z5JzLYM&list=PL-osiE80TeTsqhIuOqKhwlXsIBIdSeYtc 
• Books
• Python Crash Course by Eric Matthes ( https://www.amazon.com/Python-Crash-Course-Hands-Project-Based-ebook/dp/B018UXJ9RI)
• Automate the Boring stuff with Python by Al Sweigart (https://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994/ )
• Misc
• 
  

Hope this helps.