Building Intelligent, Insightful, and Intuitive Software

Web to Mobile and Now Voice Apps

2019-08-07T05:00:00+00:00

Smart speakers are here, and they are not only changing how we interact with technology, but also how we get things done. As everyday users increasingly rely on them and businesses ramp up to use them, Voice Apps are now the new frontier for app design. Have you wondered what it would take to design an app that runs on them?

People use now using their voice and smart speakers for just about everything - from listening to music, catching up on the news, checking the weather, and setting daily reminders. By 2021, forecasts show that the use of smart speakers will top tablets.

Voice interactions go beyond just smart speakers – with most phones having built-in Voice capabilities, users are starting to use voice applications there as well.

Part of the reason behind this growth in voice usage – is that Voice is a ‘natural’ interface. In other words, we as humans know how to talk very soon after we are born. We see this in that we generally prefer using voice to interact with others, as opposed to text messages. Voice as an interface can be as intuitive as a typical conversation. Voice, however, does come with its challenges…

A Little Context

The first smart-phone apps were clunky, awkward, and hard to use. App builders had not yet figured out how to properly expose the capabilities of computers into such tiny devices.

When the iPhone launched, app builders learned how to take features and style it for the smart-phone screen. And end-users rewarded these well-designed apps.

Even though web design and mobile app design are different, they do have a lot of things in common - from buttons to text fields and dropdown menus to name a few. Voice apps have similar parallels, designing them is different, but they do have a lot in common with web and mobile apps.

A Unique Medium

It is worth starting with noticing that in many ways, Voice is unique as an interaction medium.

Voice Apps (and conversations in general) are temporal. When you speak - and therefore provide a voice-based request, you need to wait for the response to finish before moving on with your task. Users cannot speed through the apps. The temporal property means that users can quickly become bored or impatient.

Voice Apps are also hard to use privately. Since requests to apps need to be spoken and often the response is said aloud, interactions by default become public.

Lastly, voice is pretty ineffective with many types of information - from things like lists to showing relationships between items. Anything involving the use of visuals like graphs or tables can be challenging, communicating purely with voice.

Opportunities and Challenges

The Uniqueness of Voice does come with incredible opportunities. The most obvious one is that Voice Apps take advantage of an interaction paradigm that we, as humans, are very familiar with - even before being exposed to any technology. Voice interfaces being intuitive allows more people to use them easily and lowers training costs when getting them adopted. However, what gets often forgotten is that Voice Apps still need to be designed so that they can be intuitive, i.e., they need to support usage in many situations.

To put it simply, an intuitive Voice App for a Pizzeria will likely need to support many requests including “send me a pizza”, “which of your pizza’s are the healthiest”, and “how long would it take for a pizza to arrive here”. While it is not necessary to support every request, picking which requests would be supported and especially what will not be supported needs to be done carefully.

The temporal nature of voice means that people are generally comfortable receiving and providing short responses. These quick responses allow for increased usage but also means that smart speaker users are often multi-tasking when leveraging Voice App and therefore need functionality that does not get in the way of what they are doing. The need for efficient conversations also means that users often demand Voice Apps being able to grow with them so that they can be used quickly.

The inability of Voice Apps to be used privately does result in challenges but also calls for considering the usage of Voice Apps in a similar manner that private conversations happen today. In addition, the public nature of voice inspires looking into collaborative and multi-user scenarios when thinking about Voice Apps.

Similarities with all Apps

While there are a lot of opportunities and challenges when considering voice as a medium, it is, however, worth remembering that there are a lot of similarities.

To start with, creating a truly great app on any platform, be it smart-speakers or smart-phones, means having designers work closely with developers. With Voice Apps, designers often end up taking the lead on making the conversation feel great and balance those needs with the actions that the user needs to take. At the same time, developers take the lead in building out the logic for the App.

In addition, most fundamentals of app interaction remain the same with Voice. What we call events for mobile and web apps, are known as intents with Voice Apps. It is just how those interactions are done that have changed. Clicking and tapping are intuitive ways to interact with an App’s features, and this is similar to utterances spoken by people (i.e., intents) for Voice Apps.

Whether on the web, mobile, or voice based - all apps are simply designed to help a user complete an action of some sort. To do this, they gather, process, and provide data - often using what is referred to as a ‘form’ with each piece of value being accessed using a ‘control’ of some sort. This core mechanism of interacting with apps remains unchanged regardless of where users are pushing buttons or providing voice commands.

At the end of the day - all apps operate within the fundamentals of basic app design. They need to have intuitive nesting of these controls, which can take the forms of menus, sub-menus, tabs, and accordions. So while the organization of app’s functions remains the same with Voice Apps, what changes is how the user navigates through them.

From here

If the differences between a Voice App and other types of apps scares you, remember that the fundamentals are similar in many ways. With the growing usage of Voice Apps, it is a good idea to get started building them. Doing so will allow you to internalize the differences and the strengths of the medium.

There are multiple ways to start. If you are not technically inclined or new to app development, Alexa Skills Blueprints is an excellent place to start. If you are experienced technically, then using Violet, an open-source application-framework for building Voice Apps will suit you just fine.

The nature of Voice as a computing interface and the recent technological advances makes it an exciting opportunity.

What do you think? I would love to hear your thoughts.

Image Credit: Voice by Eucalyp from the Noun Project.
Updated: 5th Jan 2020 - General improvements.

Getting Inputs for your Voice App

2019-03-09T05:00:00+00:00

You have built a Voice App and it works well. You needed to use voice to expose data that you already had. It took a few iterations, but now users are happy with it. Your next challenge is needing to having your Voice App get data from the user.

If you are using the Violet framework then collecting data from users should not be hard. If you need an intro to Violet, see the article on Building a Voice App in 30 minutes. We will build on top of that article here.

Getting a Single Input from Users

The easiest way to get input from your users is in getting a single value during a conversation. Lets say that you wanted to allow:

User: What game nights have already been planned for Friday?
App: Let me see.
App: There are no game nights planned for Friday. Would you like to add one?
User: No.

To build this out, we will need to first tell the framework that we need to accept an input of type date. Providing a type allows the underlying system to be smart. In this case, expecting a date allows for inputs to not only accept days of the week, but also commonly used “today”, “tomorrow”, or even “25th November”.

violet.addInputTypes({
  "plannedDayQuery": "date"
});

Here, we are calling the parameter plannedDayQuery, and once we have declared it to the framework we can use it anywhere in our conversations.

Building out the above conversation examples gives us the following conversation flow:

<!-- -->
<choice>
  <expecting>What game nights have already been planned for
                [[plannedDayQuery]]?</expecting>
  <say>Let me see.</say>
  <resolve value="app.getGameNightsForDate(response)"/>
</choice>
<!-- -->

With the app logic looking something like:

var app = {
  // ...
  getGameNightsForDate: (response)=>{
    var dateToQuery = response.get('plannedDayQuery');
    // ,,,
  }
}

Custom Types

Often one of the predefined types do not work for us. In such situations we can define a custom type. For example, if we wanted to only accept a week day, we could do something like:

violet.addInputTypes({
  "plannedDayQuery": {
    type: "dayOfWeekType",
    values: ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"]
  },
  // ...
});

Thinking about Multiple Inputs

We are now able to get a single input from the user. The big challenge in collecting data however if that we often need multiple inputs from the user for a task - you know where we use Form’s in web and mobile apps.

We can definitely do this by having nested decision elements, with each having an ask and one or more expecting elements. However, this does get a little verbose.

In the Voice world, support for these Voice Forms, is provided by what is often referred to as ‘multi-turn conversations’ or ‘dialogs’.

You can do this in Violet by adding a dialog element consisting of multiple item elements. These items are values to be provided by the user. Items are unique in that like decisions they consist of an ask element to prompt the user, but like choices they consist of an expecting element that defines what the user can say. The dialog prompts users based on the items returned by the elicit attribute, and the provided dialog.nextReqdParam() implements requesting all the items that have been tagged as required.

Designing our Voice Form

So lets get back to adding data to our Game Night App. The interaction could go something like:

User: I’m looking to organize a game night for 3 hours on Thursday to play Chess over Dinner. App: Let me see what I can do. App: Great, you are all set.

More realistically, users have a hard time remembering all the details in shot and often give the information one-by-one. We therefore likely want to see something like:

User: I’m looking to organize a game night.
App: What day would you like it to be on?
User: I’d like it to be this Thursday.
App: How long would you like it to be?
User: 3 hours.
App: What would you like the main game to be?
User: Chess.
App: Do you want snacks, lunch, or dinner?
User: Dinner.
App: Let me see what I can do. App: Great, you are all set.

In the above the day, the duration, the game, and the food would be items. Creating a conversation flow from the above, we get:

<app>
  <!-- ... -->
  <dialog id="create" elicit="dialog.nextReqdParam()">
      <!-- allow the user to give us all the information -->
      <expecting>I'm looking to organize a game night for [[duration]]
                 hours on [[day]] to play [[game]] over [[food]]
      </expecting>
      <!-- setup up a more realistic multi-turn dialog -->
      <expecting>I'm looking to organize a game night</expecting>
      <item name="day" required>
        <ask>What day would you like it to be on?</ask>
        <expecting>I'd like it to be this [[day]]</expecting>
      </item>
      <item name="duration" required>
        <ask>How long would you like it to be?</ask>
        <expecting>[[duration]] hours</expecting>
      </item>
      <item name="game" required>
        <ask>What would you like the main game to be</ask>
        <expecting>[[game]]</expecting>
      </item>
      <item name="food" required>
        <ask>Do you want snacks, lunch or dinner?</ask>
        <expecting>[[food]]</expecting>
      </item>
      <say>Let me see what I can do.</say>
      <resolve value="app.createGameNight(response)">
        <say>Great, you are all set</say>
      </resolve>
    </dialog>
</app>

Now, all we need to do is implement createGameNight. A simple implementation looks like:

// ...
var app = {
  // ...
  createGameNight: (response)=>{
      var dt = calcDateInFuture(response.get('day'),
                                response.get('time'));

      model.futureGameNights.push({
        startTime: dt,
        duration: parseInt(response.get('duration')),
        game: response.get('game'),
        food: response.get('food')
      });

    }
}

Onwards

Hope you found this helpful. I will love to hear (@VineetSinha) what you build and how it goes. Feel free to reach out if you are stuck.

If you are looking for a version of this article for the Salesfore platform - there is trailhead Quick Start on Violet.

See the code for this project here: gameNight.js

Image Credit: Kaufdex from Pixabay.
Updated: 22nd Aug - minor details to align with the Violet 1.0 release.

Building a Voice App in 30 minutes

2019-02-02T05:00:00+00:00

You have mastered building web apps. Maybe even building mobile apps. But what about these smart speakers that are now sitting in people’s living rooms? What about our new found ability to talk to apps on our phones?

Did you find yourself wrestling with native APIs and small platform-specific details when you tried building a skill for Alexa or an action for Google Assistant? This doesn’t have to be the case. Building of Voice App can be simple and we can do it in 30 minutes.

To get started quickly - lets assume that we let the big guys - Amazon and/or Google - take care of the Speech Recognition. Doing this allows us to take advantage of a platform with error rates that are close to humans. It also allows us to deploy whatever we build to voice enabled devices by either company.

Lets also assume that we will use JavaScript for building the Voice App - it is becoming the go-to language for building most User Interfaces, and there is also a lot of tooling built using it to support Voice Apps. Continuing with that, we will build the App using the UI focused Voice Framework - Violet.

Set things up

The first step involved making sure that Node.js is installed. Once you are done with that, lets create a project. Lets say we want the app to help with planning Game Nights. So we create a folder for the project, and then tell the Node package manager, npm, to initialize a project and install the Violet dependencies:

mkdir gameNight
cd gameNight
npm init -y
npm install violet --save

Create the Skeleton for the App

Now, all you need to do is to create a file called server.js, load the violet module and initialize it. This looks something like:

// Load the Violet Module
const violet = require('violet').script();

// App Logic
var app = {
  //...
};


// The Conversation Flow (with a pointer to the app logic)
violet.addFlowScript(`<app>
  <!-- -->
</app>`, {app});

The app doesn’t do much yet, but it sets up a place to put your Logic separately from the View, i.e. the Conversation Flow. You can run this app by typing npm start.

The Conversation Flow is the part of your app that defines how the conversation will take place - what the user can say and how your app will respond in return. When your app needs to ask follow up questions to your user, you would ideally add these interactions here. The conversation flow is also the part where Voice Designers will focus on and this part is often loaded from a separate file. The App Framework manages loading the conversation and tracking the state so that you don’t have to.

Define the Conversation

We now have the basics working, so lets build out what kind of a conversation the app will have with its users. Lets say we want the app to help with planning Game Nights.

We could write out a script of how a user might want to start asking:

User: What can you do?
App: I can help you with planning Game Nights

We can implement the above easily, by telling the framework that the user is going to be making a choice, during which we will be expecting a prompt from the user, and the app should respond back by doing a say. The conversational flow ends up looking like:

<!-- flowScript -->
<choice id="launch">
  <expecting>What can you do?</expecting>
  <say>I can help you with planning Game Nights</say>
</choice>
<!-- flowScript -->

Adding an id allows us refer the element later, and in this case we use the reserved id: launch which means that the above script is also triggered when the user just launches your app, for example by saying: “Alexa, open Game Night”.

So, lets now plan a more central conversation with the App. It could look like:

User: What game nights have already been planned?
App: Let me see.
App: Would you like to hear of game nights that are upcoming or in the past?
User: In the past.
App: I had a game night scheduled on Thursday at 5pm where Chess was played.

The above is a little more involved, and supporting it can be done by adding a nested decision for the user to make when being we ask a question. Writing out the flow give us:

<!-- flowScript -->
<choice id="list">
  <expecting>What game nights have already been planned?</expecting>
  <say>Let me see.</say>
  <decision>
    <ask>Would you like to hear of game nights that are upcoming
            or in the past?</ask>
    <choice>
      <expecting>In the past.</expecting>
      <resolve value="app.getPastGameNights(response)"/>
    </choice>
    <choice>
      <expecting>Upcoming.</expecting>
      <resolve value="app.getUpcomingGameNights(response)"/>
    </choice>
  </decision>
</choice>
<!-- flowScript -->

In the above, the resolve tag tells the framework to call application logic that was provided earlier.

Test the App

We have a simple conversation built above. It helps to test things often - especially with Voice Apps. With that in mind, lets create dummy App Logic so that the code runs and we can test it.

// An Extremely Simple Model (to hold the data)
var model = {
    pastGameNights: [],
    futureGameNights: []
}

// The Controller
var app = {
  getPastGameNights: (response)=>{
    response.say('Dummy implementation');
  },
  getUpcomingGameNights: (response)=>{
    response.say('Dummy implementation');
  }
}

Now when you type npm start in the console you can run the app. Violet comes with a built in web tooling and should let you run test the code on your local machine.

Go to the url - likely http://localhost:8080/ - in your web browser. Feel free to pick an intent (think of it as a bit of speech that a user would make) and send a request.

The request will mock how an external service will talk to your Voice app that you are just running.

Now you can test the app - make sure the conversation flows as expected!

Building the Logic

Now that we have the basic conversation built, we can build the App Logic. From the earlier snippet, all we need are two methods getPastGameNights(...) and getUpcomingGameNights(...).

If we just assume that the data will be in memory, the logic becomes as simple as:

// An Extremely Simple Model
var model = {
    pastGameNights: [],
    futureGameNights: []
}

// The Controller
var app = {
  getPastGameNights: (response)=>{
    var results = model.pastGameNights;
    if (results.length == 0) {
      response.say('Sorry, I did not have anything scheduled');
    } else {
      var dt = results[0].startTime;
      response.say(`I had a game night scheduled
          on ${utils.getDayAndMonth(dt)}
          at ${utils.getTime(dt)}
          where ${results[0].game} was played`);
    }
  },
  getUpcomingGameNights: (response)=>{
    var results = model.futureGameNights;
    if (results.length == 0) {
      response.say('Sorry, I do not have anything scheduled');
    } else {
      var dt = results[0].startTime;
      response.say(`I have a game night scheduled
        on ${utils.getDayAndMonth(dt)}
        at ${utils.getTime(dt)} to play ${results[0].game}`);
    }
  }
}

We do need to think about how we would add data to the above model, and that can be done in many ways. You could build our your web/mobile interface for that, or you can access a database (Violet provides plugins Postgres and Salesforce). Also, feel free to see the next article on how to go about getting inputs for your Voice App.

Deploy the App

You should now be able to test the app again and make sure that it is working as you would expect.

The next step is in deploying the code to run on a server that is visible on the internet - for example, using a service like Heroku.

Once deployed, you will need to register your app with Amazon’s Alexa service or Google’s Dialogflow. To do this, open the web tooling of the deployed service, click on the ‘Register’ in the menu, select the appropriate service that you are interested in, and follow the instructions from there.

After registering your app, you should be able to test it using the testing features provided by Alexa and/or Dialogflow.

Alternatively, before deploying your app, you can also test it locally by connecting it to Alexa/Dialogflow by using a service like ngrok.

Polishing the UX

Now that we have the app deployed, we can start to use it. Often, this is where we notice that the app doesn’t feel right, and we have to make a number of small modifications to it. These changes usually include:

accepting more inputs from the user - often done by adding more expecting elements within one choice statement (see example);
changing how the app responds to make the conversation feel more natural;
realizing that the AI can get confused with words that sound similar (like ‘how’ and ‘wow’), so changing prompts to the user and accepting different inputs; and
adding a use case or two once we start using the app - people do use voice apps a little differently than their web and mobile counterparts; and
refining edge cases.

Where we go from here

Hope you found this helpful. In the next article we look into how we could get inputs for your Voice App.

But, before that try building a Voice App yourself and let me @VineetSinha know how it goes.

Code: See the code for this project here: gameNight.js.

Image Credit: Alexa Voice App with People by Brother UK.
Updated: 9th Mar - added link to follow up post.
Updated: 22nd Aug - minor details to align with the Violet 1.0 release.

One year of learning from others about their experiences

2018-10-01T05:00:00+00:00

The last year has been a fun ride. Since we open-sourced and started talking about the ideas behind Violet we have been able to learn lessons from the experiences of how some of the best Voice Apps have been built.

Not only have we built these into Violet, but we have also been talking about them at different places.

A Conversational Flow Language

We learned that the temporal nature of voice and users earlier familiarity with speech means that getting the User Experience right in the Voice world is far harder than in a Mobile or Web world.

Team end up needing to spend a fair bit of effort designing a good voice conversation. We have therefore pulled together a number of Conversation Design Resources.

Beyond just designing a good voice conversation, teams need to also iterate on the voice scripts once they have it implemented. This iteration can often be hard because of the large amount of state-management code in most Voice Apps.

We noticed that Voice Apps essentially consisted of one of a few ‘user-experience design patterns’. These design patterns resulted in us defining an HTML-inspired Conversational Flow Language (and added support for it into Violet).

Improved Developer Focus & support beyond Alexa

As we learned of the need to better support Conversation Design, we also realized that most Voice Apps need a significant amount of business logic. We also realized that the code behind a voice app could easily become intertwined and that it would be helpful to allow developers to separate the Conversation Design from the Business Logic.

We have updated Violet’s API’s to help make such separation of concerns easier. Also, as we received feedback from developers we have improved API documentation as well as now we have a tutorial for building Voice Apps using the Salesforce API.

As we received requests for conversations running on more devices, we added support for Google Dialogflow. This has allowed us to build actions for Google Home as well as provide voice conversations on mobile devices.

I shared a number of our learnings, in particular on the User Experience side of things at the VOICE Summit this year. You can find the slides as well as see a video of the presentation.

I also presented at Dreamforce again this year in a session focused on helping developers build voice skills on Violet. You can see a video of the presentation.

I am going to be sharing more of these lessons here in the upcoming months.

Thanks!

The learning this year for all of us would not have been possible without the collaboration of a number of people. In particular, I want to thank Mitchell Harris at the RAIN Agency, Steve Tingris at Dabble Lab, and Diana Deibel at Grand Studio for collaborating on these ideas.

Image Credit: People using messenger by Brother UK

A Voice Application Framework: Announcing Violet

2017-11-13T05:00:00+00:00

For the greater part of this last year, I have had the pleasure of building a lot of Voice prototypes. These have included Alexa skills, actions for Google Home, as well as building standalone devices (powered by AVS) with custom wake-words and voices.

When building such Voice Apps/Skills, I quickly noticed the need for programming with a fairly powerful yet low-level platform-specific API’s. It reminded me of the first Web apps that I built, where we needed to program to the different DOM API of Netscape Navigator and Internet Explorer.

The amount of technology behind smart speakers is incredible. On top of that, Amazon and Google have not only been able to simplify that power in building devices for our homes - Amazon Echo and Google Home respectively - but have also been able to build API for developers to build Skills that run on them.

These Skills, unfortunately, do require developers to track where a conversation is for every single user. Doing this makes it hard for developers to focus on the actual conversation use-case, the script design, and the business logic to support the skill.

Violet is an Open-Source Node.js based framework that focused on helping developers build Voice Apps. It started as a reusable-library for prototypes that I have been building and after over a dozen iterations it has become something that others have found helpful.

Violet is built on the alexa-app package. It makes it easy to build and test conversations locally. It has primitives to automate state-management of conversations using the goals backed by a per-user stack. It also includes a plugin framework with plugins to access various parts such as data access to PostgreSQL and Salesforce.

We were excited to talk about it at Dreamforce this week. And we were fortunate to get some really good feedback. We would love to hear about your thoughts. In particular, how can a framework help you build great Voice Apps? Are there common aspects that you have found when building a Voice App that you would like to see in an App Framework like Violet?

Architexa goes Open Source

2015-06-09T05:00:00+00:00

The dream behind Architexa started when I was working on my PhD. I realized that as programmers, we often have to look at a lot of code. And, that if we have tools to help experienced programmers get a better view of the code (and in particular its architecture), we could save them a lot of time.

The underlying stats are shocking – coders spend 5x as much time modifying code than writing new code and 3x as much time understanding code than modifying code (more – When Understanding means Rewriting). With that, the work started first as a tool (Relo) to create box-and-arrow diagrams (like class-diagrams) using the code as a guide. It then added another tool (Strata) to guesstimate a layered-architectural-diagram using code dependencies with heuristics for breaking cycles. And, finally with Architexa we added a tool to show run time information in code using sequence diagrams.

While we added the above, we got a lot of great feedback – and even got love letters! The Architexa team grew to over 10 people, and we eventually even got some investors. However, growing the business was hard for a number of reasons, and we realized that we would not be able to build Architexa as-is into something that every developer would want to use. We really had to choose between focusing on premium work that was often framework-specific or rebuilding from scratch to have something that might not have all the benefits of Architexa but would be something that a developer could benefit from quickly.

As part of that we decided to open source Architexa. You can see the best version that we could share at: https://github.com/i3labs/architexa

Over the next little bit, I will be polishing the repository and including licensing information. The code will be licensed primarily as AGPL, and we will also work to have a more appropriate license for anyone wanting it and support towards commercial applications. Beyond this, I will be phasing out the forums (feel free to ask questions on Stack Overflow) and will look forward to any Pull Requests (especially those that align with the Roadmap).

Beyond the administrative tasks, the short term roadmap for Architexa includes adding automated tests and making it compatible with Eclipse Luna. Longer term features will depend on those that are willing to support the contributed code, but it will be great to have a stand alone version of Architexa, integration into other code editors, increased support for JVM languages and frameworks, and finally support for non-Java languages.

Let me know what you think.

Another blog is born

2015-04-03T05:00:00+00:00

Over the past few years I have had the fortune of writing papers, blogging both publicly and internally for the companies I have worked at, teaching, and even presenting at conferences. This blog is an attempt at pulling all that into a single place.

Oh, and part of the agenda is discussing best practices with various emerging technologies, innovation, and some of the open sourcing projects that I care about.

I am looking forward to hearing what you think about it.

Any entires on this site that are earlier than this post will be those that I have brought in from one of the other places where I have written before.

Code on Different Types of Codebases

2013-01-28T17:36:11+00:00

A lot has been written about blogging and working on open source projects to improve your coding skills. They are great ideas. In addition, I like attending user groups.

But there is more. Becoming a great software developer also means making sure you are able to think as the situation demands, as opposed to being biased in the manner that your current project is developed. It means making sure you are comfortable building very different types of projects.

In the last few years, work has meant building Eclipse plugins, a flex based app, and a web service. However, I recently began to notice a lot of browser extensions. I wanted to see if building them had any lessons for me. Instead of trying to brainstorm random ideas, I heard of a need for our sales team. A few weekends later and a lot of studying Google’s extensions documentation, I built something. It wasn’t great, but it was helpful. Over the next few weeks, my colleagues got excited about it, and I could make improving the chrome extension to be part of my work.

I learned, and took advantage of, an interesting approach to security for Chome extensions. While I have used Sandboxes before, writing code on both sides of the sandbox was new to me.

Building low level coding skills are easy. If you have programmed in 2-3 languages, and know how to implement a couple of algorithms, then you should be able to implement most systems. However, you are likely not going to be making the best high level decisions unless you have experience building projects of very different architectures.

Now, I do have a day job. So, I can’t spend too much time on such side projects, but there is very little that can substitute feeling the difference between building something that only has JavaScript in a few places, versus building something that is primarily JavaScript. And this difference is getting to be really important these days – especially for organizations that really try to do a good job for their users.

Tips

There are 3 tips that have worked very well for me, when I have been working on such projects.

1. Think Small: It is hard to get your boss to approve working on a fun project. While working on these projects, I knew I had a limited amount of time. I also know of all the fun projects that I want to do that are just sitting on the side. So, when I select a project to do, I work very hard to somehow scope it down so that it can be finished in a weeks worth of time. Finishing a project within 40 hours of coding time is really hard, but at least this way the project gets finished before I have gotten busy and/or bored with the project.

Oh yeah, most such projects have taken 2-3x as long as I was planning. But, isn’t that the same as the industry?

2. Build Useful: Beyond trying to build the project in a week, I want something that it useful to at least one person. Thinking about making this person happy not only helps me get the job done and ignore other fun things, but it also makes me focus on the coding details that will actually make the user happy (as opposed to me focusing on the interesting technical challenges). And, yes, if it does include things that I can put on my resume, then that is only for the better.

3. Go Different: Beyond selecting a small project, I have typically tried to do one or two things very differently about each project. This includes moving from building a desktop app, to building a server side app, to building an algorithmically intensive app, to a rich interactive browser based app. And while I do the above, why not choose a different language to make the above in?

Building an app in a new language within 40 hours is not easy. But, try it. You will quickly realize that you need to sacrifice some things that you thought were obvious, and you will feel really good when you realize that you were able to accomplish the challenge.

Thoughts

I have learned a lot doing the above. I had built many things with Javascript, and while I am now much better at it, I still realize that I really like having a static strongly typed language for development. And not just because the compiler effectively creates some tests from me, but also because I consider myself as an engineer, and feel the responsibility in making sure whatever I build is able to be maintained by others. Here, Microsoft’s experiment with Typescript looks really interesting, and I am putting it on my list of things to experiment with.

Have you tried something similar to the above? What’s worked for you?

Image Credit: Allianz Arena by Dirk Loop

Breaking Out of Your Shell as a Coder

2013-01-15T18:47:36+00:00

Working on a software project after it has shipped for 1 or 2 years can be fun. But sometimes you feel like you need to do something a little different – to get different ideas, different perspectives, and different types of challenges. I feel like I am in the midst of such a journey, one that has begun to have some results that were unexpected when I started. The big lesson has really been that I need to *talk* to more people about coding.

I have to start by saying, it is awesome working on a successful project. Our teams’ vision is now being used by thousands of new users every month, and some are even sending us love letters 🙂 These just add a certain energy to us day-to-day. We have also ended up learning a lot of things: That small UI changes from increased prompts to reorganizing menus and dialog boxes can make users much happier. That paying attention to the right type of profiling can result in a few lines of code that improve the performance of our code by over 100x. That sometimes results can be large if we remove features that we are not happy with and spend time improving the rest of the software.

And to top it all, we have been fortunate to be very customer/user focused and have not had to deal with the bureaucracy that you find in some organizations. But even with the above, I have felt a a need to try new stuff. To make sure that I am not not stuck in my old ways, and that I know how to improve.

Failure: Online Forums

As an introvert, one thing that I have tried a fair bit is reading blog posts online. And that has been great. Discussions on aggregation sites (like Hacker News, Reddit, DZone, etc) have often been interesting. The thing though has been that I have always had a little bit of an uneasy feeling with them.

I have generally not been sure how much of the opinions that I read online are that of biased people, and on the other end – on aggregation/voting sites that the opinions were following the crowd mentality.

While blogs and online forums are good, I felt wanting to see the facial expressions of someone when they say something that is incorrect. There is so much we can learn from getting a breadth of smart people talking about a topic.

Failure: Going to User Groups

Even though I am generally a self-learner, one of the first things I started doing was making sure to attend user groups. Our main development is in Java these days, so going to the local Java User Group’s seemed like the best idea. And I did not really find it that helpful.

As a self-learner, I am mostly up-to-date about the latest frameworks, so I quickly got bored of attending any of the basic introductory talks on frameworks that I am familiar with. This just resulted in me coming only once or twice a year when the topic was interesting.

After about 2 years or so of not being very happy, I wanted to take a different approach. Hearing the perspectives of others at these User Groups sounded like they had most of what would be helpful, but I wanted a way to make sure I was not wasting my time. I wanted to see if I could get these perspectives and if I could take my ‘bored’ time at user groups to work on more advanced skills on the same topic.

Success: Coding and Interacting at User Groups

At a recent talk on Grail 2.0, I found it helpful to hear the latest stage of the roadmap of the project and see where things were going. I was able to get a sense of whether my fears of the project stagnating were true or not. And I found some interesting resources that I had not realized were available. And when the presenter gave an introduction to Grails – which I had read a couple of tutorials on before – I took the opportunity to actually build a full-fledged Grails app.

I know that some people might consider it bad etiquette to do something different while a topic is being presented, but I did want to hear the rest of the presentation and I hope the presenter appreciated me staying around for the talk.

Beyond the core presentation, some of the user group members got together immediately afterwards to get some drinks. During this time we talked about general development processes, challenges that teams are facing, and what the job trends are. In fact, I found this part of the meeting so helpful, that I have decided to actually spend more time volunteering and helping the organizers.

I also looked for smaller groups that have a lot more discussions among members. One of them has been focusing on Software Craftsmanship. I found it helpful sharing general development thoughts with people who care about the code that they are writing. And at the same time, as someone who has seen attempts at Test-Driven Development fail way too many times, it has been very helpful to hear how others have gotten tests to work effectively for themselves.

Takeaways…

So, yes, I am going to be attending user groups in the next few months. And I will make sure to be sharing more tips and learnings from these. Though, at the end of the day, everyone’s situation is different. If there is a big lesson here, it is actually to actively attend and participate in such meetups.

If you happen to live somewhere where there aren’t such meetups – try starting one. It is incredibly helpful to have people to talk to who don’t just think like your team/company. And if you live out in some place where there are few coders living in the area, make sure you attend multiple conferences!

Comments/Discussion on Hacker News.

Pains of Open Source

2012-09-12T15:27:05+00:00

I am a big fan of open source. I use it every day, and recommend it often. But I do want to have a balanced perspective on it. I recently posted about the pleasures of using open source. This is the other side of the coin, the pains of using it.

My hidden agenda about talking about such problems with open source usage is so that we as a community can come up with solutions to some of these challenges. But first, I really want to hear and write down the main challenges, so that we fix the right problems.

Too many sub-par projects

Perhaps it is that there is a lot of the not-invented-here syndrome that happens to us developers, or perhaps because most of the solutions to a particular problem are not good enough. But regardless, there are often way too many people putting effort into similar projects.

Now, I am not against having people try and build out different approaches — but there is a limit to that. For most frameworks that I have needed to try using as many as 3/4th of the projects are useless. Many projects were only developer to support a long deprecated api, and other projects are just not being maintained as the developers have found something more interesting (or perhaps something that might pay their bills).

Using the right framework often means trying out 2-3 solutions, going through its code to see how much it makes sense for your needs, and making sure that you are safe by being able to have your answer through determining the most active communities (in mailing lists, etc).

Bag Of Bolts

When 80% of the code in your project is an open source library, then most of your code is in figuring out how to hook up these projects together. Even though most libraries won’t force a way of using them, just integrating all their (open source) code bases is non-trivial.

By default, just bringing in such libraries will make your resulting web-app feel like a bag of bolts. Fortunately, project management and requirements usually take priority to such libraries, and therefore we as developers need to spend our time essentially becoming integration experts (to smooth out the kinks).

But here is the thing, the part that is unique to your application is in your code. So often, your 20% of the code is the hard part. While you might be using an open source search library like Lucene, it will be up to you to make sure that the indexes are not only created at the right time, with the right performance, but that they are stored in the right places, and have the right data in them. Some of these parts will be trivial as they are part of the default usage scenarios, but others might need the redesign of a key component of the library.

The minor problems: Who to Pay, Licensing, and Versioning

Using an open source libraries comes with an acknowledgement that we will get stuck, especially when we can’t figure out the code. Sites like StackOverflow are amazing, but some problems take more time to solve. I have often been in scenarios where we have a budget to give someone to solve the problem, but that we are often not sure as to who would be the best person/organization to give that to.

Sometimes, getting an open source library that gave us the freedom to do what we need is hard. Honestly, this used to be a big challenge, but recently most libraries are on a permissive license. Additionally, it can sometimes be a pain to update the versions of a library (tools like Maven and the entire ecosystem that Sonatype has fostered are incredibly helpful here).

Summary/Thoughts

I guess most of the above can be put simply put as ‘community’ and ‘code readability/documentation’. With perhaps documentation helping to build community. I guess, coders love to code. And one of the first things that most developers do, is write code. The means that we should expect some Open Source libraries have very limited documentation. Most do have have decent documentation to help you get started, but are often limited when it comes to anything beyond that. Perhaps we need to incentivize the community of users (such as StackOverflow) to gain karma by helping to document some of these projects.