Rohan Verma

Using Minio in Github actions to mock s3

Tue, 09 Feb 2021 19:48:00 +0530

I recently added Github Actions for simples3. In a recent PR, a contributor added support for custom endpoints. I was currently running go test on the library using a local .env files which contains real AWS keys and an actual s3 bucket. I realized that we can mock this using a minio server as well.

Coming from having used Gitlab pipelines, it seemed to be straightforward to do. I would simply have to run a service and just set the appropriate environment vars. Unfortunately, Github pipelines does not seem to support container args for services. The conventional hack on most repos, is to build a custom Dockerfile with the custom command, but it seemed unnecessary for just mocking minio for tests.

So after reading a bit, I found out that you can actually just run docker run inside the job command as a step. Essentially, it boils down to just adding it to your pipeline and setting up a few env vars. Minio accepts MINIO_ACCESS_KEY and MINIO_SECRET_KEY in the docker env, which it uses to bootstrap the admin credentials for the first time. These can be further used to create a mock bucket using aws s3 mb and provide the custom endpoint as a flag. Finally, simples3 will use its own environment vars to interact with the mock minio, and run the tests on it.

Here is the full workflow for reference:

name: Go Test

on:
 push:
 branches: [master]
 pull_request:
 branches: [master]

jobs:
 test:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v2

 - name: Set up Go
 uses: actions/setup-go@v2
 with:
 go-version: 1.15
 - name: Setup minio
 run: |
 docker run -d -p 9000:9000 --name minio \
 -e "MINIO_ACCESS_KEY=minioadmin" \
 -e "MINIO_SECRET_KEY=minioadmin" \
 -v /tmp/data:/data \
 -v /tmp/config:/root/.minio \
 minio/minio server /data

 export AWS_ACCESS_KEY_ID=minioadmin
 export AWS_SECRET_ACCESS_KEY=minioadmin
 export AWS_EC2_METADATA_DISABLED=true

 aws --endpoint-url http://127.0.0.1:9000/ s3 mb s3://testbucket

 - name: Run Go Test
 run: go test -v ./...
 env:
 AWS_S3_BUCKET: testbucket
 AWS_S3_ACCESS_KEY: minioadmin
 AWS_S3_SECRET_KEY: minioadmin
 AWS_S3_ENDPOINT: http://127.0.0.1:9000

The Ethics of Invention

Sun, 20 Dec 2020 03:23:00 +0530

The Ethics of Invention by Sheila Jasanoff is not a book on ethics, but rather a book about the complicated relationship between Technology, Law, and Policy. By the title, one might think that this is yet another “Techlash” book written to be read by Tech Luddites. Rather, the book argues for a middle ground between uncontrolled enthusiasm for technology and the timeless, often understandable, hatred for technological progress. The main focus of the book is to highlight the tendency of humans to give or delegate power to technological systems, which end up governing human behavior without even them noticing it happening under their noses.

Examples and case studies are abundant in this book. To explain the point, she takes us through how traffic signals came to dictate the laws of the road. “Inanimate lights backed by invisible experts and unseen electrical circuits have stepped in to discipline behavior that was once risky”. No one questions it, and we often just go about our day being governed by them and reshaping our world over time. Although, it is left to the reader to ponder if it is truly possible to build democratic consensus around these issues. She compares the power given to technological systems with legal constitutions. We can comprehend our delegation to the lawmakers but often the delegation to technological systems does not compute.

She highlights three major fallacies that we fall prey to while designing policies and how the over/under reliance on these impacts people.

Technological determinism: Once a new technology is invented, it “possesses an unstoppable momentum, reshaping society to its insatiable demands”. It is a belief that innovation is always beneficial or good for society and should be pushed as far as possible. She takes the example of refrigerants being beneficial in the short term but causing Ozone Depletion in the long term. Or how automobiles led to enormous progress for humanity in the short term but probably caused so many unaccounted externalities in the long term.
Myth of technocracy: That “only those with specialist knowledge and skills can manage and control technology”. There are two points she makes here. The first being a critique of the technical risk assessment. She argues that technology is value-laden from start to finish and the inventors desired end clouds our judgment and often forces externalities to be classified as “unintended”. To expand on this, she says that “Experts’ imaginations are often circumscribed by the nature of their expertise.". It is often our first instinct to rely on experts but they often “overestimate the degree of certainty behind their positions” on a matter and “blind themselves to knowledge coming from outside their closed ranks.” She goes through the example of the Challenger shuttle and the 2008 financial crisis.
Unintended consequences: “If technological mishaps, accidents, and disasters seem unintended, it is because the process of designing technologies is rarely exposed to full public view”. She asks if it is fair to use such a fuzzy word like “unintended”, can there ever be intended consequences of a technology, and if so shouldn’t policy be designed to tackle these in the first place? Another problem raised is that intention is fixed at a specific moment in time, morally, which would not be static in the long term and is bound to change and who is responsible to track these changes.

The rest of the book looks at problems of “risks, inequality, and human dignity” that need to be addressed if our society is to progress and responsibly grow alongside our technological innovations. Ending with a bunch of questions, do we exist to further technology or technology exists to further our goals and ambitions. Do we rely on technology to solve climate change or do we accept and deal with the fact that we have mismanaged it in the first place. We need to dispel our thought process from the three fallacies and start agreeing that “ethical analysis, political supervision and long impeded systematic thinking” can be applied to technological innovations.

She argues that we need to rethink tech-policy, where we first look at risk assessment to anticipate and rather bring in values into our decision making at an earlier stage. At the same time, she asks us to not rely on “Trickle down innovation” and look at collective ownership and collective good rather than treating all tech as extractive and solely for profiteering. Another caution is to keep ethical discourse public and draw a balance between private discourse so that it does not spawn further public alienation between views of experts hired by private bodies and democratic institutions.

Does NPD Governance Framework give Government a backdoor into private companies

Thu, 03 Dec 2020 04:34:00 +0530

Ministry of Electronics and Information Technology (MEITY) had released a report 1 on the “Non-Personal Data Governance Framework” in July 2020 2. The report recommends the establishment of a Non-Personal Data Regulatory Authority with an enabling role as well as enforcing role. 2 Along with this, the report defines Non Personal Data as: 3

Non-personal data: When the data is not ‘personal data’ (as defined under the PDP Bill), or the data is without any personally identifiable information (PII). It could be:

Data that never related to an identified or identifiable natural person, such as data on weather conditions, data from sensors installed on industrial machines, data from public infrastructures, etc., or

Data that was initially personal data but was later made anonymous. “Data which are aggregated and to which certain data transformation techniques are applied, to the extent that individual specific events are no longer identifiable, can be qualified as anonymous data”.

Every tech company, which collects non-personal data would be obliged to register itself as a Data Business.

Access to non-personal data may be requested for three purposes, one of those being: 3

Sovereign purposes such as national security, law enforcement, legal or regulatory purposes. These purposes could include mapping physical and cyber security vulnerabilities, mapping crime and taking preventive measures, a regulator wanting to stay abreast of developments in the sector, or national security (via telecommunications metadata, geospatial or financial data, etc.)

Putting two plus two together, it is easy to understand that this could mean that the government could get private tech companies to anonymize and give data collected of its users, that could be private in nature, and hand it to them. Another failure here is that it is clearly understood by the tech community that using metadata, it is often trivial to de-anonymize data. Now the scope and urgency of the matter is not specified. Neither is any recommendation given by the report to limit the usage of this under exceptional circumstances, or maybe atleast only when warranted by say the judiciary. Even though the report has mentioned the need of “adequate checks against abuse of power by government or other representative agencies” there is little emphasis on what these will be. This was even mentioned in the Medianama article on the concerns 6.

Here I would like to invoke the Iron Law of Data Collection, by Jon Snader 4, which states that no matter what the original rationale given for its collection new uses will be found for it and will eventually be abused.

DEA expresses an interest in providing its agents “ ‘unlimited access to patient de-identified data’ on re/filled prescriptions, daily supply, payment type, dosing information and gender, among other characteristics, until at least 2025.” 5

He gives a concrete example of how even anonymized data collected for the medical use is being appropriated by law enforcement and is going to be essentially abused.

Now the issue here is that, when unlimited and multi-genre data collected by private entities, lands into the laps of the government; how creative will our government be?

The x4ivygA51F Rabbit Hole

Wed, 25 Nov 2020 23:25:00 +0530

I recently read the blog about the 200 most common passwords of 2020. It was also featured in Jon Snader’s recent blog post. When the original post was made, I had glanced at the mysterious x4ivygA51F, which appeared at the 148th spot. But when I read the blog post again and searched for an update on it, no one had been able to figure out what that meant.

There were some guesses. Few folks on reddit and 4chan, guessed that it might be password written in another language. This guess was based on a gizmodo article, which mentions something similar. To confirm this hypothesis, I tried searching for Chinese websites if I could find some mention of this. There was little to no mention of this to be able to confirm this, at least on what is to be found through the indexed web on Google or Bing.

Although, one interesting finding here that I saw no one talk about was a post on the forum of a website called 51haoyou. The post on this forum, which talks about a breach by hackers on January 10, 2019 talks about this specific password. In a footnote on this post, it mentions the following:

Note: x4i***1F is the trial password originally displayed by discuz, and x4ivygA51F is the result of my modification of the display program so that the password is displayed completely. On 2019-10-28, I completed “The server-side program is added to determine whether the user login password is encrypted on the client side, otherwise the php program is interrupted”, “Improved the Discuz mobile version, and also realized that the user password is encrypted on the client side. “Retransmit” time.

I guess that it could mean that the Discuz forum software by default suggests this password somewhere on the registration page. Or administratively this is set to be the default password somewhere by this BB software. Using the Cangjie input method, I found those keystrokes build and translate to Question 1. Something similar does come up on the login page this does come up in terms of security question, although this could be entirely incorrect as I don’t know Chinese or Cangjie. Another guess here is that this is the breach that led to the inclusion of this password in the list itself.

Generating Go docs using gomarkdoc and pandoc

Tue, 24 Nov 2020 02:19:00 +0530

I recently had to distribute the documentation of an internal library to users. Go developers are used to their editor to providing them with auto completion, usually with the help of gopls. A lot of times it is necessary for users to be able to browse a comprehensive list of methods and data types available for reference. For open source projects, we rely on godoc.org and pkg.go.dev. But unfortunately, it is not easy to host one for internal projects.

After a bit of searching, I found princejef/gomarkdoc. This project is able to process packages and use the docs, to generate a markdown file. Using this I was able to produce a markdown of the packages I wished to share with the users.

# install as a cli-tool
go get -u github.com/princjef/gomarkdoc/cmd/gomarkdoc
# generate the docs using specific packages
gomarkdoc ./pkg/foo \
 ./pkg/bar \
 ./pkg/lorem/ipsum > docs/user-docs.md

This was good enough, but a better way would be to distribute it as a single html file. Pandoc is a swiss army knife, which can convert one markup format to another. Combining this into the pipeline, I was able to generate an html file which has the documentation, which is viewable in the browser and contains links to the git repository as well.

To make the web document look good, I found two interesting things.

The first being, sakura.css, which is a classless css stylesheet, which can be applied to any html file and make it look like a modern website.

The second useful thing, are a few flags available in pandoc which help with making the documentation html file truly standalone.

–css: This flag accepts a link to a CSS stylesheet
–self-contained: This flag produces the html file in a way, such that no external dependency is needed.

Using these flags, we can come up with the following script to generate the html file from the markdown file.

pandoc docs/user-docs.md \
 --toc \
 --metadata title="My Package - User Docs" \
 -c https://unpkg.com/sakura.css/css/sakura.css \
 --self-contained \
 -o docs/user-docs.html

This results in well rendered documentation which is standalone and looks modern as well.

Use of Facial Recognition Tech by Law Enforcement

Mon, 16 Nov 2020 03:59:00 +0530

The use of Facial Recognition by law enforcement to identify offenders is one of the cases where the regulation of the technology lags way behind, and there is an urgent need to bring about policy to reduce potential pain in our society. Apart from the serious problems that are inherent with the technology like mis-identification and minimal mapping between the training sets and the actual photos. According to a recent report in TechCrunch 1, the Home Minister of India admitted to using a data set trained on the Aadhaar Database to identify individuals involved in a recent riot. This software was originally obtained in the name of a good cause, to identify missing children, although according to the report 1 the software failed to even distinguish between boys and girls. Recent advances in AI have made certain model that perform recognition by training on huge datasets, unfortunately they are considered to be black boxes. Along with this, there is a question of how in line it is with the Privacy Laws and the inherent questions of cyber-security. To address these issues, we should open discourse on these topics with experts. Unlike the approach in the west 2 to ban the tech, we should try to come up with a middle ground. Ideally, a policy should be discussed to thoroughly document the process of how the model was trained and the dataset that was used in the process. After the model is trained and put into production, there should a mandatory requirement to disclose the uses of the model along with the false-negatives in a proper manner. Using this information, there should be regular reviews to revisit the efficacy of the model. A mechanism should be devised to be able to appeal for manual review by an expert in case a person who is identified by this system. A person should also be able to opting out of being incriminated by the use of this technology. Such mechanisms, in such a policy, would address the privacy concerns as well.

Goodbye upi.link

Mon, 24 Feb 2020 12:38:24 +0000

Dear upi.link users

Thanks for using upi.link and making it grow organically to over ~2k daily hits.

Due to a takedown request for the domain upi.link from the NPCI I have had to shutdown upi.link for now.

I had created upi.link to solve my own problem. Till date, sharing a upi request outside an App is not possible without an intermediary for free. After sharing a prototype with people in my workplace, I found out that it was useful beyond what I had imagined. I shared it with more people and finally put it up online. It has not been my best work but more or less was an experiment for me. Running upi.link with the least amount of cost and infra setup was personal challenge for me. For over a year I had run this without any extra cost to me except the domain name.

Unfortunately, I was naive when I started building this to assume that the NPCI is a government body and UPI is a platform. In reality, the NPCI is a non-profit (not non-commercial) body and UPI is its product. Even though some
of the UPI spec is open, it can hardly be called an open product. NPCI owns a trademark for UPI and also has a guideline for domains using the trademark.

At the moment, I am working on other things and I don’t have the resources to keep the upi.link domain online by either asking/paying NPCI for allowing use of this domain nor do I have any bandwidth in further developing anything on top of upi.link as of now.

I would also like to add that being involved in the fintech space in India, I have come to appreciate what UPI has provided all of us in the short term. I am also aware of the rising amounts of scams happening with UPI which the NPCI tries to prevent through partner apps like BHIM, GooglePay, PhonePe etc. Also, the fact that it has successfully reached out and onboarded so many people and organizations is a monumental feat in itself.

Some people, had suggested open sourcing this project but unfortunately; some decisions, I had taken to reduce the cost of hosting the website, hinder the scope of opensourcing the project.

Regards

Utopia for Realists

Tue, 26 Nov 2019 15:30:03 +0530

This book was a recent recommendation from Dr Kailash Nadh. A small but heavy book, that is engaging and well researched. Rutger Bregman has done a thorough job of making arguments for three major policy ideas whose time has come in the land of plenty that we live in (as compared to a human living a century ago).

Universal Basic Income
Reducing Working Hours
Open Borders

Universal Basic Income or UBI, might sound ludicrous at the first glance, but numerous studies and on-field experiments have time and again shown that the way ahead for the welfare-state might be direct money transfer (not just benefits). The basis of the argument for UBI stems from the dignity given to recipients of the no-strings attached benefits leading to a more efficient and effective utilization of the safety net.

The second policy of reducing working hours eventually, stems from the current state of “Bullshit Jobs” that produce nothing of value but just move value around. He mentions Keynes prediction that soon we will have 15 hour work weeks based on an interpolation of the work hours at the time of writing his study. He critiques the measure of economic progress used in this era, the Gross Domestic Product (GDP), a perfectly reasonable measure nonetheless created during wartime when it was not unreasonable to borrow from the future. A quote from the book on where all the time has gone when productivity has skyrocketed.

It’s quite simple really. Time is money. Economic growth can yield either more leisure or more consumption. From 1850 until 1980, we got both, but since then it is mostly consumption that has increased. Even where real incomes have stayed the same and inequality has exploded, the consumption craze has continued, but on credit.

The only way to solve this would be to legally reduce the working hours and make multiple people share the work and giving everyone more time for leisure. He even goes on to mention how this could end up solving a lot issues plaguing our society such as inequality between the sexes.

Also, he suggests taxing ventures such as banking and hedge funds that move value around and to rather give incentives such as salary hikes or grants to people to take up careers that are useful for the society like research, teaching, maintaining cities etc.

His third idea, of open borders, is a critique of what he calls “apartheid on a global scale” that has been committed by closing down borders. It is a fact that there is a huge disparity in pay and lifestyle around the world. He mentions that economists predict an approximate >2 fold increase in global prosperity if the world were to open their borders today and allow free movement of human capital.

He ends it with a final note on ideas. How ideas like Neo-Liberalism changed and mobilized the world at a time of crisis. It made me realize that the only way for realists to achieve the utopia we crave for, we must discuss these ideas in the open. It may be through talking about these in your workplace, discussing with friends and family, or publicly writing on them. Makes you wonder, maybe we are living in a revolutionary time.

Building Go Plugins inside Docker

Sun, 24 Nov 2019 15:30:03 +0530

Using Go plugins in your projects comes with a lot of caveats. As of writing, there hasn’t been much development on the feature recently. The commit history shows us that the last commit happened nearly 2 years ago. On the gopher slack, the sentiment, more or less, is that this is not a priority anymore. Along with this, there are multiple issues that come up with maintaining projects that use it:

The go version for both host and plugin should match exactly
External dependencies should match
Host and plugin GOPATH needs to exactly match while building
Plugins cannot depend on interfaces or structs of the host

To learn more you can refer this issue detailing some of the problems with go plugins

It works well if the projects bundles all the plugins in its own source tree and both the host and plugins are built together at the same time. But, that limits the scope of the project. Externally maintained plugins are impossible to build independent of the host program.

To solve this, we can use Docker to build both the host program, the bundled plugins, and the custom plugin together. Then this image can be distributed instead of distributing the host and loading the plugins separately in production.

FROM golang:1.12-alpine AS builder
RUN apk update && apk add gcc libc-dev make git
WORKDIR /myproject-plugin/
# Clone and build myproject and myproject-plugin together
# prevent version conflict for go plugins
RUN git clone https://github.com/rhnvrm/myproject.git && \
 mkdir -p myproject/bundled_plugins/myplugin
# Load our custom plugin from disk
COPY ./myplugin.go ./myproject/bundled_plugins/myplugin/myplugin.go
# CGO_ENABLED=1 is required
ENV CGO_ENABLED=1 GOOS=linux
# `make build` builds the host program and bundled plugins
# `go build` our custom plugin 
RUN cd myproject && \
 make deps && make build && \
 go build -ldflags="-s -w" -buildmode=plugin -o myplugin.prov \
 bundled_plugins/myplugin/myplugin.go

FROM alpine:latest AS deploy
RUN apk --no-cache add ca-certificates
WORKDIR /myproject/
# Copy the assets from the builder image
COPY --from=builder /myproject-plugin/myproject/myproject /myproject-plugin/myproject-plugin/myproject/bundled0.prov /myproject-plugin/myproject/bundled1.prov /myproject-plugin/myproject/bundled2.prov /myproject-plugin/myproject/myplugin.prov ./

CMD ["./myproject", "--config", "/etc/myproject/config.toml", "--prov", "bundled0.prov", "--prov", "bundled1.prov", "--prov", "bundled2.prov", "--prov", "myplugin.prov"]

This image can then be pushed and used where it needs to be deployed.

Creating archive pages grouped by year in Hugo

Fri, 15 Nov 2019 15:30:03 +0530

I was on the hunt to find out how to organize my blog and project archive pages by year in Hugo. After being unable to find any easy solutions I decided I would sit down and write the go template to render these pages myself. The idea was simple, iterate over the list divided by year into sub lists and render tables, but it turned out to be a bit tricky.

Here is the template, hope it is useful for someone who wants to have a similar page.

{{ define "title" -}}
Blog List | {{ .Site.Title }}
{{- end }}
{{ define "header" }}
{{ partial "masthead.html" . }}
{{ end }}
{{ define "main" }}
<header>
<h1>{{ .Title }}</h1>
</header>
<section>
{{ $prev := 3000}}
{{range .Site.RegularPages}}
{{if .Date}}
{{if gt $prev (.Date.Format "2006")}}
{{ if ne $prev 3000}}
</table>
{{ end }}
<h2>{{ .Date.Format "2006" }}</h2>
<table class="all-posts">
{{end}}
<tr>
<td>{{.Date.Format "02 Jan"}}</td>
<td><a class="no-underline" href="{{.Permalink}}">{{.Title}}</a></td>
</tr>
{{ $prev = .Date.Format "2006"}}
{{end}}
{{end}}
</table>
</section>
{{ end }}
{{ define "footer" }}
{{ partial "powered-by.html" . }}
{{ end }}

The above template loops through all the blogs (having a date) and renders multiple tables which are grouped by the year. We render h2 tags with the year if the previous year is greater than the current year. Followed by a <table> tag having each next blog rendered as a row in the table. The previous seen date is updated for the next iteration. In each iteration, before rendering the table tag we check if the previous date was greater than the date or3000 (an arbitrarily high year) which we set as the previous seen year before starting the loop, to decide if we want to close the table tag. In the next iteration, the previous year will be the same as the current year, and so we can continue to render rows.

Containerized development workflow using remote gopls server

Tue, 05 Nov 2019 15:30:03 +0530

If your CGO development toolchain depends on external dependencies such as system libraries, or you want to develop on an older version of go while having a different version on your host, you can use a docker container and mount the source from your host machine and build the project inside the container. This can enable us to have a consistent development environment across various developers and their host systems without having to modify system libraries.

This article is a follow up to my previous article on using containers to develop on archaic projects that require specific system libraries. Last time I discussed how to create a container having all the dependencies and dev tools installed and use them to operate on the host file system. Taking this a step forward, the biggest roadblock that arises with this approach is that our host machine
now loses the ability to run gotools on our source code. That means your editor cannot run any of the tools you depend on to write, lint, or format your go source code.

To overcome this, we can use the experimental remote lsp server feature available in gopls. gopls (pronounced: “go please”) is the official language server for the Go language. With this mode, the gopls running on the host forwards all the commands to the gopls in our container. This way the host system does not need to have any of the dependencies installed and can then use the container gopls server to modify the source in the editor of the host.

Let us look at the Dockerfile for this system.

FROM ubuntu:xenial AS builder
RUN apt-get update && \
 apt-get install -y gcc g++ libssl-dev cmake wget libpcre3 libpcre3-dev git

WORKDIR /tmp

# Install Go.
RUN wget https://dl.google.com/go/go1.12.9.linux-amd64.tar.gz && \
 tar -C /usr/local -xzf go1.12.9.linux-amd64.tar.gz

# Install Swig.
RUN wget http://prdownloads.sourceforge.net/swig/swig-4.0.1.tar.gz && \
 tar -zxvf swig-4.0.1.tar.gz && \
 cd /tmp/swig-4.0.1 && ./configure && make && make install 

ARG MY_USER_ID
ENV MY_USER_ID ${MY_USER_ID}

ARG MY_PWD
ENV MY_PWD ${MY_PWD}

RUN mkdir -p $MY_PWD

RUN useradd -r -m -u $MY_USER_ID myuser
RUN groupmod -g $MY_USER_ID myuser
USER myuser

ENV PATH="/usr/local/go/bin:${MY_PWD}/go/bin:${PATH}"
RUN GOPROXY=https://proxy.golang.org GO111MODULE=on go get golang.org/x/tools/gopls@v0.2.0
WORKDIR $MY_PWD

CMD /home/myuser/go/bin/gopls -listen=":7050"

Firstly, we install all the dependencies and then create a user which will have access to the source code. We pass the same user ID as our host user by using the MY_USER_ID as the build argument.

Then we have to replicate the host path in the container and set that as our work directory. This is a quirk of gopls, if there is a mismatch in the path, the host will communicate the wrong path to the server running in the container and will be unable to find it inside the container.

Finally, we install gopls in the container and then start it in the container with the -listen=":7050" flag.

We will now build the image.

mkdir -p go
docker build \
--build-arg MY_PWD=$PWD \
--build-arg=MY_USER_ID=`id -u $USER` \
-t myapp:latest .

Note, we also create a directory called go which will be the mock GOPATH inside the container.

Now we can expose this port to our host machine.

docker run -d --name "myapp" \
-u `id -u $USER` \
-e "GOPATH=$PWD/go" \
-p 7050:7050 \
-v $PWD:$PWD myapp:latest

Notice, that we also mount the host source in the container with the same path, the reason is explained above.

We must now add the following flag to your editor config along with any other flags we use for the language server to connect to the remote gopls server.

gopls -remote "localhost:7050"

For VSCode, you can add the following to your settings.json

{
"go.languageServerFlags": [
"-remote=localhost:7050",
"-v"
],
"go.useLanguageServer": true,
"go.gopath": "go"
}

After adding this flag, you will now be able to edit your source with all the added benefits that come with a language server like auto completion and auto formatting.

Since this is an experimental feature, it might break (a lot). Do let me know in the comments if this was helpful for you.

PaisaVasool - HackInOut 2019

Fri, 25 Oct 2019 15:30:03 +0530

Last weekend I attended a hackathon after a long time. HackInOut, which is India’s biggest community hackathon, was happening very close to where I live. I went with my office colleagues Vivek and Karan and we had discussed a lot of ideas before reaching the venue, but we were not sure what we will finally end up building.

After discussing our ideas with Siddharth Shetty, an iSpirit fellow and one of the mentors at the hackathon, we finally decided to build PaisaVasool.

What is PaisaVasool

Split subscriptions and collect money automatically and instantly over UPI seamlessly.

The problem it solves

Sharing subscriptions like Netflix friends is not seamless. Collect payments online and reconciling them manually using Apps still requires us to manually confirm payments, and APIs/SDKs by PSPs are expensive.

The hack

We piggyback on paisa values in the UPI request amount as the transaction ID, and reconcile the payment seamlessly and automatically into the subscriptions.

The app

To demonstrate the hack for this hackathon, we decided to concentrate our efforts to Splitting Recurring Payments such as those for Netflix/Amazon Prime/Hotstar/Spotify etc. Although, our hack can be extended to any kind of collect requests.

A user can create groups on our app and enter his friends contact details and instantly get reconciliations from our app when he receives payments.

Our hack allows you to consolidate your UPI requests for free!

Daily UPI requests are limited by PSP apps like Google Pay.
- You can request upto 5 times on Google Pay a day
- New users can’t be requested more than once a day
Payment API’s are expensive
- Costs (2% per transaction) are added onto the consumers
You need to check your Bank transaction details and reconcile manually without a PSP
Apps like Paisa Vasool are impossible on top of UPI as of today

Our hack allows payers to pay with any VPA.

No need to ask your user to be on a specific UPI app
No need for payee to pay from his own account (parents/friends can pay on behalf)

Instant reconcilation

We process the transacation as soon as we get the bank SMS
Users don’t need to check their bank details

Challenges we ran into

The biggest challenge was to figure out a way to solve these issues:

How to identify a transaction
How to not need the payers VPA

We figured out that:

The identifier can be embedded in the paisa value
We can read SMS to get the amount and parse the transaction ID.

Detailed technical flow

User installs our app and gives read permission for SMS to our app.
- Our app is open source and only sends SMS with UPI VPAs to our server
They create a collect request in the UI with the amount and users, and create a group for recurring weekly/monthly transactions.
- User adds his clients/friends to the group using mobile number and/or email
  - These are saved for the future
The collect request is then processed by the server
- UPI link (which can be embedded in SMS/Whatsapp/Telegram) is created using https://upi.link/ API - It is sent to all the clients/friends with appropriate amount that has been hacked to include the transaction identifier in the amount paisa value.
The app waits for SMS from the bank of the User, which includes the amount (plus paisa value identifier) inside it.
- On receiving the SMS, we change the state of the pending transaction to complete.
The user can check the state and see all his transactions from our app.

Future

The Sahamati framework in the future will allow users to share their bank transaction ledger details with apps directly in a standardized format via Aggregators.
UPI roadmap has split requests planned for the future.

Facebook Birthday List to ICS file

Fri, 11 Oct 2019 01:03:03 +0530

I have recently stopped using facebook, but I really miss the convenience of the birthday notifications. I tried to find the .ics which you could export from facebook events page to your calendar program but it was nowhere to be found and I thought I would have to script scraping it myself.

A quick search on github later, I found that someone had already beat me to it.

You can find the repository on github and follow the documentation, but I have documented the steps I followed below:

The first step is to fetch the repository from github and editing the config.

git clone git@github.com:mobeigi/fb2cal.git
cd fb2cal
cp config/config-template.ini config/config.ini
vim config/config.ini

I did not want to upload to google drive which is default so I edited the default config to save the file locally. In case you face errors, you might need to set logging to DEBUG to figure out what went wrong.

[AUTH]
fb_email = myemail@gmail.com
fb_pass = xxxxxxxxxxx
[DRIVE]
upload_to_drive = False
drive_file_id =
ics_file_name = birthdays.ics
[FILESYSTEM]
save_to_file = True
ics_file_path = ./birthdays.ics
[LOGGING]
level = INFO

Now we need to download the dependencies, I prefer to use pipenv to automate the virtual environment creation but you can install them using pip as well.

pipenv shell
pipenv install

Now we need to run the script and wait for it to complete.

cd src
python fb2cal.py

After it is complete you can view the .ics file and import it into your calendar app.

cat birthdays.ics

Afterwards, don’t forget to delete the config.ini file containing your facebook password in plaintext.

Using Docker containers for building Archaic Projects

Sun, 04 Aug 2019 07:30:00 +0000

Sometimes old projects and libraries require certain specific system library versions that are hard to reproduce, or cumbersome to replicate on our development environment. Docker can be used to simplify the build process of these projects and it is often a lot faster than starting a VM in your machine and using that to build these projects.

I recently found out about such a project at my workplace. This was a C++ binary that was statically compiled, so there was no issue with distribution but the build process had become very complex over time. It seemed to only compile on dependencies that ship with Ubuntu 16.04 Xenial Xerxes. Currently, the project was using a vagrant box, and provisioning it with dependencies. Each time a new binary had to be complied, the VM had to be started, provisioned (install development tools) using a script, setup SSH, and then copy the source into the virtual machine. After that, the VM would be used to run the bash scripts which would build multiple binaries along with the required libraries. These final binaries were then again copied to the host machine using scp.

The initial idea was to replicate the same thing in a Docker container. But can it be done in a better way?

Yes there is a way. A cool thing about Docker is that we can share the filesystem between the host and the container using mounts. Yes, mounting is also possible using VMs but the sheer speed at which you can perform in-place operations using docker run and mounts as compared to a VM is ludicrous. Plus, you don’t need to have a VM always running. What we just need to do is to build a container having all the dependencies and dev tools installed and use the dev tools of the container to operate on the host file system.

Let us dive in,

First we will make a Dockerfile having all the required tools and libraries installed. We can save this file in our projects root directory.

FROM ubuntu:xenial
RUN apt-get update 
RUN apt-get install -y gcc g++ libssl-dev cmake
RUN mkdir -p /etc/myproject
WORKDIR /etc/myproject

Notice above, that we have earmarked /etc/myproject as the directory where we will mount our project source tree.

The following sample makefile is used to build our project. Only two steps are involved with this file.

make docker-image
make build

build: libs myproject ## build all

docker-image: ## build the docker image using the Dockerfile
 docker build -t myproject:latest .
libs: ## build the libs
 docker run -it -v "${PWD}":/etc/myproject myproject:latest /bin/bash -c 'cd build && chmod +x build-libs && ./build-libs'
myproject: ## build myproject.bin
 docker run -it -v "${PWD}":/etc/myproject myproject:latest /bin/bash -c 'cd build && chmod +x build-main && ./build-main'
help:
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
.DEFAULT_GOAL := help
.PHONY: help build libs myproject

Let us look closely at each step required to finally produce the binary.

First, we will run the command make docker-image to create a local image from the docker file we hade written. This image has all the required dependencies.

Now, the next step is to run the libs and myproject steps in order using make build. In this step, we mount our hosts working directory to /etc/myproject which we had mentioned above. After that we run a command that will run a script that runs g++ in our case to build the binary. And voila, it is done.

A review of Siempo Launcher for Android

Sun, 17 Mar 2019 20:13:13 +0000

Three months of self inflicted digital pain and how it changed my habits

Last December, I decided to start an experiment and adopt a new launcher called Siempo apart from the OnePlus and Nova launcher that I’m used to since I started using Android. After three months, even though I have stopped using it, I have observed some changes and would like to share it with my readers who are also interested in digital well being. There has been a recent surge in people starting to restart blogging even though on closed platforms like Medium, but I guess it at least that is searchable outside from search engines, and hope to see more self hosted blogs following Hackernoon’s recent split. Apart from this, people like me who were active social media users who now realize the downright issues with platforms like Facebook, Instagram, YouTube and Snapchat and their algorithms, and are no longer interested in being driven by them throughout their day. In this article I talk about my experience with it and how it changed my usage habits.

After surveying all the options, the only fully featured launcher (that was usable) I found was Siempo. An other notable mention was the Minimal Launcher but it did not have a free dark mode or even proper app search, making it unusable apart from phone calls and messages. I did not want to go to the extreme with this experiment so Siempo seemed to be the best option out there for Android. A few notable features of this app based on my experience are mentioned below. But before that, I must mention what I guess mostly the ideas on which the app is based on.

Tristan Harris, a Former Design Ethicist at Google had around 2-3 years ago started a movement called Time Well Spent now called Humane Tech. Nothing better to explain this than his TED Talk on “How a handful of tech companies control billions of minds every day”

A few notable things from the website are copied below for reference

Unfortunately, what’s best for capturing our attention isn’t best for our well-being:

Snapchat turns conversations into streaks, redefining how our children measure friendship.
Instagram glorifies the picture-perfect life, eroding our self worth.
Facebook segregates us into echo chambers, fragmenting our communities.
YouTube autoplays the next video within seconds, even if it eats into our sleep.

http://humanetech.com/problem

Highlights about Siempo

Nothing on the screen when you tap that power button

There is nothing more that forms habits when you tap that power button and click that enticing app which you see that you had yourself originally placed on your home screen for convenience but now are more or less compelled to check since you have formed a habit to click it as soon as you press that physical button and the screen lights up. So this app displays nothing more than a blank screen with an optional text message which it calls an Intention although I kept it as a blank screen as I find motivational quotes a bit gimmicky and silly but might be it useful for you if you feel it helps you stay motivated to use this launcher.

Everything is Monochrome and Apps have no Logos

Colorful app icons entice you to open them. When you swipe right in Siempo you will find your most useful apps utility apps as Icons (and not logos). But wait here is the catch, these are supposed to be set by you and are based on categories. You decide the app but not the categories that can be placed here, so you can have a food app but not “Uber Eats”, “Zomato” or “Swiggy”, similarly for for your travel app, you can select only one and the icon will be a generic cab icon.

The essentials which don’t belong to a category

There will be some apps which you use which don’t belong to a category from the first page, so you again swipe right and find the frequently used apps along with a search bar incase you forgot to add the app below. These are again not based on your usage analytics or recent usage, another tactic used to form habits but by you. But here is the catch with this page, no logos here as well for the pinned apps but no icon either, only the first letter of the app and the full name below it.

And finally the biggest offenders of the attention economy. These are not even spared with fixed placement, and are randomly shuffled each time you swipe three times after you press your physical power button and unlock your screen making you literally think twice before opening this app.

Apart from this, Siempo allows you to set a custom timer of a set duration after which you use these apps such as 15 minutes, all such apps cease to function and a full page block is shown asking you to reflect for one minute if you wish to continue using this app.

Reflections on this experiment

Three months down with this, I have to admit my overall phone usage drastically declined. I stopped playing PUBG, reduced Netflix/Prime Video, nearly have stopped using Facebook (the last time I checked it was like a month ago), minimized Instagram usage to only view direct messages. Apart from this, I felt more energized throughout the day as I tended to sleep earlier although I woke up late as well, I ended up having more time on my hands as I had more energy. I ended up reading on the Kindle app or watching single small 5-10 youtube curated videos through twitter or reddit instead of a few 30 minute videos suggested by Youtube. It definitely had changed my usage patterns but I will have to caution others about a few reasons mentioned below, and would heavily suggest self moderation instead of changing their launcher.

Absolute pain to use when you actually need your phone – There were times when I needed to use my phone and could not use it properly especially when with friends and family and there were times when I actually needed these apps when the block would impact my productivity.
I started to use YouTube and Twitter – These apps which I had marked as harmful, were now my most used apps even after going through trouble to use them. After the banner comes for 15 minutes of usage, I formed a habit of doing something else for a minute and reusing these two afterwards. My mind had adapted to the pain and eased it for these two. Earlier I used to use these less on my phone, but was using them more on my laptop back then so I guess it was just a move from not using the laptop for these apps.
App choices changed – This might be relevant to those who are price wary especially in the food/travel category. Before Siempo I certainly had a 50-50 split in terms of usage of certain apps as they offer equal number of discounts. For food, I switched between Swiggy and Zomato and Ola/Uber for cabs, it was again a 50-50 split as well. But since I had set Swiggy as my food app, and Ola as my cab app, I began to prefer convenience more for both of these after I switched back as I have now formed a deep association with these and the time I need their services I can only think of these two.

I hope, dear reader, I was able to help you with informing more about the digital well being space and I have motivated you to journey on your own quest and read more about it. Do comment about your experience with Siempo and similar apps along with what your thoughts are, would love to discuss this further.

upi.link - Progress Update 2

Sun, 24 Feb 2019 12:38:24 +0000

Highlights

I wanted to finish the underlying infrastructural things so that I can start to focus on the higher level features.

Features

Implemented User Profiles

Added support for user profiles this week. Now, I can extend the user authentication table with profile data. Since, I want to store minimal data about the users on my server, I have used gravatar to load the profile photo and as of now no other information other than email is stored. Maybe, next week based on user feedback I might save the preferred UPI VPA so that the user does not have to type it every time.

Track generated links

Now users can track the permanent links they have generated so that they don’t have to create a new one every time. This same feature will be used to display links on the public profile which will be released next week. Also, users will be able to edit and delete these as well soon so they can be reused.

Email preview

Now, you can preview the email that will be sent beside the link generator on the send page in real time. I will add support to the profile page to send emails directly from there to multiple comma separated emails.

Front page design

The home page has improved quite a lot from what we started, now we have more details about the features and a more clearer text explaining what upi.link aims to become.

upi.link - Progress Update 1

Sun, 17 Feb 2019 12:38:24 +0000

What am I building, in a sentence?

🔗upi.link: A programmable shortlink generator based on UPI (universal bank2bank payments service in 🇮🇳) sharable via social media & chat.

What did I complete till now?

Highlights

Deployed at upi.link

I used AWS Free tier to deploy this website to reduce cost for hosting this. It uses AWS Lambda Functions to generate shortlinks which are stored inside Redis with a TTL of 3 Days. Authentication is implemented using a postgres backend along with secure cookies for sessions. I will port the lambda functions I wrote for shortlink generation out of AWS Lambda to my custom backend written in Golang as authentication. This is because I have seen that the cold start for Lambdas is pretty slow for my use case and takes about 5-6 seconds which for the users seems like 10 seconds.

The frontend is written using Nuxt.js which is a Universal app framework on top of Vue and is similar to what Next.js is for React. This was a neccessity for this use case as I need the links shared on social media as well as chat with HTML OG (Open Graph) tags to be populated dynamically based on their content. The only other option was to write a static website served through templating either in Go (Go templates) or Python (Django/Flask using Jinja templates), but then I would not be able to utilize the rapid development offered by Vue/Nuxt. Also, I can now separate my API and frontend, maybe totally host the API proxied via Apex/Up and the frontend via S3 buckets to reduce the cost to host the site.

Created a roadmap

I shared this project with a few friends and groups and lots of feature requests as well ideas propped up. To keep a track of all these I created a public Trello Board. You can email to the board for adding more feature requests.

Features

Donation Campaigns

One of the major features this week, I added a campaign collection feature. The first campaign to be hosted is the #BharatKeVeer campaign which was launched to collect funds for contributing to the wives of the CRPF Soldiers who lost their lives during the Pulwama terrorist attack.

Temp Link Generator

I updated the temporary link generator on the home page. Now you don’t need to bother with the Long Link and you can directly see the page.

Buttons for sharing on Whatsapp and Telegram were added below the generated shortlink

UPI App compatible QR Code

QR code with the embeded Deeplinking Intent is now displayed on the redirect page for people who open the link on their laptops/desktops so that they can simply scan the link from their phone.

Emails via AWS SES

My SES request for sending emails was approved, I added the email button behind authentication.

📌Misc

Progress Blogs

Well, this blog, honestly, I wanted to have a blog where I could track the progress on this project but I did not want to have another setup on say Ghost/Wordpress. So again Nuxt came to my rescue and now I just have to add a page to my Nuxt folder and it handles everything else such as routing etc.

Recorded demo video 📹

You can view this video on youtube

Presentation about upi.link at Barcamp Bangalore - Techlash

Sun, 03 Feb 2019 12:38:24 +0000

I recently got a chance to talk about upi.linkduring Techlash at Barcamp Bangalore. I got to tell people about having started this small project. Here is the link to my presentation:

Presentation

A small summary of upi.link

Introducing [upi.link](https://upi.link), which is is a Payment Service Provider (PSP) neutral UPI Request Link generator built upon the UPI Deeplinking Spec, enabling SMEs and individual users to collect payments directly by sharing links on chat apps or embedding on their own websites, without having to rely on building their own mobile apps and calling intents, making manual UPI requests on PSP apps, tying up with service aggregators that charge fees, or even purchasing licenses to APIs when they are just getting started.

Setting SO_REUSEPORT and similar socket options in Go 1.11

Tue, 08 Jan 2019 12:38:24 +0000

Go recently introduced a heavily requested feature that allows programmers to set socket options before accepting and creating connections. You can find a mention of this in Go 1.11 Release Notes. Although, not many have written on this and implementing this is a bit confusing due to a change in the way one has to implement this. So I decided to share this with others who might be interested in using this feature.

By reading how support for this has been added, we can get an idea about how to go about implementing this. The release notes mention that a new ListenConfig type has been added. This type contains a Control function which needs to be implemented. This Control function should contain code to set the Socket Option.

Let us see how one would start a UDP reader that performs a callback on receiving a packet.

type UDPOptions struct {
Address string
MinPacketLength int
MaxPacketLength int
}
func StartUDPReader(opt UDPOptions, callback func([]byte)) {
addr, err := net.ResolveUDPAddr("udp", opt.Address)
if err != nil {
h.sysLog.Fatalf("address resoultion failed: %v", err)
}
conn, err := net.ListenUDP("udp", addr)
if err != nil {
h.sysLog.Fatalf("listen failed: %v", err)
}
packet := make([]byte, opt.MaxPacketLength)
for {
n, _, err := conn.ReadFromUDP(packet)
if err == nil && n >= opt.MinPacketLength && n &lt;= opt.MaxPacketLength {
callback(packet)
}
}
}

This is how the reader would look after adding SO_REUSEPORT using the new way.

func StartUDPReader(opt UDPOptions, callback func([]byte)) {
lc := net.ListenConfig{
Control: func(network, address string, c syscall.RawConn) error {
var opErr error
err := c.Control(func(fd uintptr) {
opErr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
})
if err != nil {
return err
}
return opErr
},
}
lp, err := lc.ListenPacket(context.Background(), "udp", opt.Address)
if err != nil {
h.sysLog.Fatalf("dial failed: %v", err)
}
conn := lp.(*net.UDPConn)
err = ipv4.NewPacketConn(conn).SetControlMessage(ipv4.FlagDst|ipv4.FlagInterface, true)
if err != nil {
h.sysLog.Fatalf("set control msg failed: %v", err)
}
packet := make([]byte, opt.MaxPacketLength)
for {
n, _, err := conn.ReadFromUDP(packet)
if err == nil && n >= opt.MinPacketLength && n &lt;= opt.MaxPacketLength {
callback(packet)
}
}
}

Using this approach we can reuse the port and have zero downtime, between restarts by starting the new reader before stopping the currently running reader.

Streaming audio from Linux to Android using PulseAudio over LAN

Mon, 19 Nov 2018 19:50:35 +0000

Suppose you are in a situation where you want to watch a movie on your TV or monitor but don’t want to use speakers. Maybe you are looking to listen to an audio book stored on your laptop but don’t want to transfer it to your phone. Or you just want to buy an audio jack splitter. Look no further, PulseAudio to the rescue.

PulseAudio provides streaming via SimpleProtocol on TCP via a simple command. All you need to do is find the source and start streaming it.

You can find the source by running this command:

pactl list | grep "Monitor Source"

After this, you can run:

pactl load-module module-simple-protocol-tcp rate=48000 format=s16le channels=2 source=<SOURCE> record=true port=<PORT (eg 8000)>

Next, you will need to download PulseDroid, the apk can be found in the Github repository or you can use the following command to download it using wget:

wget https://github.com/dront78/PulseDroid/raw/master/bin/PulseDroid.apk

Just enter the IP address of your machine (you can find it by running ifconfig) and the port you chose and press the Start button.

Rohan Verma

Using Minio in Github actions to mock s3

The Ethics of Invention

Does NPD Governance Framework give Government a backdoor into private companies

The x4ivygA51F Rabbit Hole

Generating Go docs using gomarkdoc and pandoc

Use of Facial Recognition Tech by Law Enforcement

Goodbye upi.link

Utopia for Realists

Building Go Plugins inside Docker

Creating archive pages grouped by year in Hugo

Containerized development workflow using remote gopls server

PaisaVasool - HackInOut 2019

What is PaisaVasool

The problem it solves

The hack

The app

Our hack allows you to consolidate your UPI requests for free!

Our hack allows payers to pay with any VPA.

Instant reconcilation

Challenges we ran into

Detailed technical flow

Future

Facebook Birthday List to ICS file

Using Docker containers for building Archaic Projects

A review of Siempo Launcher for Android

Three months of self inflicted digital pain and how it changed my habits

Highlights about Siempo

Nothing on the screen when you tap that power button

Everything is Monochrome and Apps have no Logos

The essentials which don’t belong to a category

The blacklisted ones (social media)

Reflections on this experiment

upi.link - Progress Update 2

Highlights

Features

Implemented User Profiles

Track generated links

Email preview

Front page design

upi.link - Progress Update 1

What am I building, in a sentence?

What did I complete till now?

Highlights

Deployed at upi.link

Created a roadmap

Features

Donation Campaigns

Temp Link Generator

Whatsapp/Telegram Share buttons

UPI App compatible QR Code

Emails via AWS SES

📌Misc

Progress Blogs

Recorded demo video 📹

Presentation about upi.link at Barcamp Bangalore - Techlash

Setting SO_REUSEPORT and similar socket options in Go 1.11

Streaming audio from Linux to Android using PulseAudio over LAN