From "Code and Pray" to "Crash and Fix"

August 30, 2010

Software industry is changing every hour and making progress every day. Software development methods and techniques are also getting better every year. Thus, we old folks have to stay updated and one way to achieve this is to learn from the apprentices and the interns.

This week I like to discuss different intern techniques: those in our days and those nowadays. In our intern days, one common technique was "code and pray." We would code something and run it by the complier and pray that it worked. Some practictioners also call this coincidental style of coding.

Recently I noticed that the interns at our company have developed this to a higher level. I noticed that they are now using "crash and fix" technique. It's a beauty in this particular case.

Crash and Fix GQL

As we all know, we need to create and provide an index.yaml file to Google App Engine so that it can search its data store properly.

Locally (that is on our development server), if we make a new kind of query, the Engine updates the index file. When we repeat that query some time later, the Engine is ready to do the search.

However, on the real server, if we make a new kind of query, the Engine just scans the index file. If the index necessary for the query is not there, the Engine throws exceptions.

There is no quota/limit on the local engine, but the real engine places some limit on the number of indexes.

Thus, the right thing to do, at least for us old folks, is:

1. Find typical usage patterns.


2. Make queries for just these patterns.


3. The resultant index.yaml file, developed on the local server, will be relatively small.


4. Use this small index.yaml file on the real server,

This would take time and insight. Modern interns have better ideas:

1. Deploy the app on the real server with an empty index.yaml file.


2. When a user makes a new type of query, the Engine throws excpetions. Log those exceptions somewhere.


3. Check those logs. The log tells us the specifications of the index it needs for a particular query..


4. Copy such specifications into a new index.yaml file.


5. Deploy the app on the real server again, and repeat.

This also took time but not insight. Do the modern interns have an edge over us old folks? Pay special notice to the comment "# goodpt3," "# goodpt2" etc. Those below 1 were added before those below 2 and so on. The interns will check only the topmost section, which is added the most recently. Time-saving and brain-saving, cool?

indexes:

# AUTOGENERATED

# This index.yaml is automatically updated whenever the dev_appserver
# detects that a new type of query is run.  If you want to manage the
# index.yaml file manually, remove the above marker line (the line
# saying "# AUTOGENERATED").  If you want to manage some indexes
# manually, move them above the marker line.  The index.yaml file is
# automatically uploaded to the admin console when you next deploy
# your application using appcfg.py.

...

- kind: Inventory
  properties:
  - name: expenses
  - name: locations
  - name: distance
  - name: skus
  - name: unit_space
    direction: desc
  - name: transports
...

- kind: Inventory
  properties:
  - name: gender
  - name: name
  - name: distance
  - name: skus
  - name: unit_space
    direction: desc
  - name: transports

# goodpt3
...
- kind: Inventory
  properties:
  - name: unit_space
  - name: gender
  - name: locations
  - name: phone
  - name: skus
  - name: transports
...


# goodpt2

...
 
- kind: Inventory
  properties:
  - name: unit_space
  - name: expenses
  - name: gender
  - name: locations
  - name: phone
  - name: distance
  - name: skus
  - name: transports
 
# goodpt1 - kind: Inventory
  properties:
  - name: education
  - name: transports
  - name: gender
  - name: locations
  - name: skus
  - name: unit_space
    direction: desc
...

- kind: Inventory
  properties:
  - name: transports
  - name: gender
  - name: locations
  - name: distance
  - name: skus
  - name: unit_space
    direction: desc
...

The file has almost 1200 lines and uses up 92% of the index quota. If we add new models/entities, we must make new queries and will need new indexes. Will the remaining 8% cover most of our future needs, or am I over-planning?