Architect Agile Way

Facebook and Google Plus Rants --- Profitable Software Business --- Part 3

Great skills can still lead to weak or nearly useless products.

How ?

When you don't understand your domain properly, or if you don't plan ahead for changes.

I don't make money teaching design. But developers, software business owners should listen to Hoare, David Parnas, Dijskstra (small head), Eric Evans, Peter Coad, Richard Feynman, Rasmus Lerdof, Linus Torvald, Charles Simonyi (MS Word, Intentional software). Scott Bain, Kent Beck etc.

If they don't know them or won't listen, they should not be in software business, right? Can anyone claim to be wiser( streetwise or worldly wise or whatever) than their collective wisdom?

On bitbucket, google code repository, GitHub, I find a lot of django/GoogleAppEnginePython codes for almost anything: social network, forum, message board, LMS(learning mgmt system), ERP, CMS(content mgmt system), wiki, quizz, collaborative writing, shared whiteboard, mobile app etc.

Shouldn't we extract some value from them?

As we all know, Python is not Perl/PHP or even Java. It's very good-natured language.

Log-in, log-out from App1, quiz from App2, friendship invite from App3 etc are there for us to mix and match as our kids play with Lego.

If some of us can find a business model to go with that Lego-style app, we may make some money for us.

Does anyone have some guide/advice for that kind of not-cut-and-paste but rip-off-and-rebuild development? HAPPY NEW YEAR!

I asked a Singaporean programmers group this:

"Ok. Find me something that looks like this. (1) It's free and online. (2) I type main question, then choices, including if necessary, images too. And the answer. (3) I can add as many question I like. (4) I can print it out, and it comes out nicely. (5) I can let my students practice there. (6) There can be tip/hints for students. (7) Answers are graded. (8) I can edit them. (9) I can download my question sets. They work offline too! (10) I can go on, but this should be enough. This is 1 of 1000 things we can do for education/training etc. So I have no need to protect this. ... Please don't mention SCORM, IMS, Moodle, Atutor, Coursera, hot potatoes, whatever. If a nice thing is there, why would I code or ask others to code."

They cannot find. Mine is world's first and very good. Still they don't want to work for me for 8 hrs * 15 days for free. They cannot see the benefits.

Facebook and Google Plus Rants --- Profitable Software Business --- Part 2

Recurring Revenue Streams

What are we going to do about education?

Every newborn fish can swim. But every newborn Myanmar doesn't read or like Nat Shin Naung/Nawaday right away. Every newborn Singaporean doesn't read or like Water Margins, Han Fei Tzu etc.

Education is an ongoing business for every new generation. A lot like tax-prep software business.

Every new year is a new business.

Little Knowledge and Sophisticated Goals

The title is a phrase from Peter Small's writings.

I could not and did not bother to pronounce English correctly. I ignored phonetics and phonology because I lacked ability. But that has not prevented me ( so far, at least)to do anything I want with English.

Now I want to get big in software business. I cannot do phonology-like coding.

Will that prevent me from expressing my self?

Do I need phonemes /a/ vs /o/ vs /u/ to express my romantic love in sonnets? my patriotic love in dramas? my sense of humor in Seinfeld-like, Keaton-like, Bean-like acts? my observations in Lamb-like, Thurber-like essays?

What You Admire You already Have

Lightning-fast swoops and cloud-splitting soars inspire young eagles ... because they get it, have it already, at least in small measures

Product Development: Sum is More Than the Parts

Are you thirsty?

I have a bottle of H and another bottle of O and a kit with which you can combine that H and that O?

Will you be quenched and satisfied?

Or would you like just a bottle of high quality water at the cheapest price in town from me?

Education is a bit more complex than H2O. Yet many pieces are there, almost free. Who will mix them into A drinkable, usable form? Why not us?

Magic in Software

I read lots of silly stuff and had/have strange role models. One favorite theme : a kid was surrounded by real or rival-witch-created snakes/or some other obstacles. The kid took out a piece of paper, tore it into pieces, threw them onto the ground, OR fanned some leaves on the ground with that paper, and they turned into fierce birds etc , snatched away all those snakes etc. .... Silly? ...

What about Gmail, Hotmail, Filckr, Facebook, Salesforce.com? ....

They all started on a piece of paper, they are still a roll of characters/words and lines. Modern abracadabra, right? ....

My heart sings with software. Software is my heart. I live or die with it. One day, you just whisper into a screen and my Abracadabra will come out to serve you like the Genie of Aladdin's Lamp.

Inciting Google Plus Communities

Hi Everyone,

I want to reinvent Moodle. I want to retain about one-fifth of its features and do it in Django.

The problem is I am weak in coding. Is there anyone who wants to do this as an open source project? I can promise only 2 things :

1. It will be better than Moodle 
2. It will be better than Khan Academy on Google App Engine! Cheers!

Facebook and Google Plus Rants --- Profitable Software Business

Recruitment on Facebook

When our team grows, of course, our receptionists and canteen staff will have PhD's and know Hadoop, Cassandra, MapReduce, R and lots of Big Data statistical packages.

But for the time being, others have been making good money with Excel!!! Yes, Excel!!! In Myanmar, in Singapore, and in New York (and in Wall Street ) !!!

From Hadoop to Excel, the space is a bit very wide. Our team hope to score some nice goals.

Any one, who has learned Excel or Hadoop or WHATEVER for 2 or 3 days, are encourage to join us! For the time being, we accept candidates without PhD, so hurry.

Adaptability, Flexibility or even Luck, but not the Ability to Predict

One of my friends gave university tuition while he was still a university student. Only very good students could do that. And I also did that. But then, since he has far better social skills than me, I sometimes asked him for advice. On one occasion, I said, "Some of my students do not like my best works/essays at all. What do I do?" He replied, "That happens to me too. What is good for me, they may not like it. Some shoddy works, they liked them a lot."

That lesson still stays with me. So, for my software business, I rely more on my adaptability, flexibility or even luck than on my ability to predict. So, though I have 20,000 features in mind, I intend to fight with 2 or 3 features first. ;-)

Software Development Business

Ancient China: population--> training to be soldiers --> forming into corps--> launching wars

Modern software follow similar steps.

1.  You create commodity software or use them for free. 
2. You turn them into building blocks(Lego-like pieces) of your application.
3. You weave. mesh, gel  the Lego pieces into useful apps. 
4. Then you sell these apps.

Easy Winning , as in TOEFL, IELTS

In 1995, I checked 7 TOEFL/IELTS brands, not books, and knew right away it is easy to do World Best classes there.

Last 3 days, we checked 5 world best online education sites' database schemas and became relieved and happy. Data models, domain models are heart and brains of web software. Our team is happy in this area.

How High-tech do you want to go?

I am starting a "very-very-very-hi tech" software company. And I am recruiting ordinary programmers (though I won't reject anyone as "over-qualified.") These 2 facts are not contradictory.

A CFO of Dell once asked, "How much techno do you need to own to compete? "

As students, you yourself might have asked, How many books do I need to read to compete well in the exams?

Even in high-tech business, you can compete well without relying on technology. Let's see.

Recruitment on Facebook

I predict 100 Instagram's from Myanmar and Singapore each. And when you predict something, you must also know when/how it can go wrong.    

Instagram came from another bigger app by Kevin Systrom, called Burbn, according to this month Fast Company magazine. Burbn is a clone of all popular app features at that time.

Singapore programmer Chang Sau Sheong, in "Cloning Internet Applications with Ruby" book, has done a Burbn like app already. There are many equally capable developers in Myanmar and Singapore.

But will they reduce their bigger clone app into a smaller, more useful app like Instagram? If they do, my predictions will be true. If not, well we will get 200 clone apps, including 2 of mine ;-(

Recruitment on Facebook

You can pay for 6-dollars-per-hour staff with 3 or 4 days of training. But to become a McDonald, you need a system.

Do I have a system to play in big league of software with ordinary programmers?

I think so. I'm recruiting. Let everyone know. No experience required. No commitment required. Only these requirements - you live in Singapore, you can read and type. You can work about 4 hours per week at home and 1 hour per week at our place. If you are over-qualified, that's better. We are not afraid of any talent

Recruitment on Facebook

I admire the source code of Django, Tornado, NLTK, boto, etc. As many of world greatest coders in history advise, their logic/code is "dumb" and therefore good. When I coded shopping cart, CRUD etc, my logic/code stayed "dumb" and good.

But when I moved even a bit farther, e.g. FSM with Pygame, an expert system etc, my logic/code became "clever," complex and unstable. I've thought about and studied this for years.

Our team is training to be a world leader in software -- both biz-wise and techno-wise.

Anyone interested in becoming better programmers are welcome. We are humble to learn and generous to share. Love and Thanks to all.

Some Unsued Assets

1. Good reading and note-taking, domain modeling, good and fast understanding of requirements of an industry, good and fast understanding of a user's job/task/process/activity 
2. Clean, small, flexible code/software.

If salesforec.com is not that combo, what is that?

If Flickr is not that combo, what is that? If Evernote is not that combo, what is that?

I have given you proofs, please join us. Even if with half a man, we will get to our goal. But, still we want you. Join us!

Writing Clearly - Crashing into Mars

I want to write a lot about software and software business. And I have to restrain myself to stick to my target audience, you : young students and  your parents. This post is about software but you will find it easy to understand and useful to your study.

Writing clearly is very important. How important? Not doing so can crash expensive space crafts, like the crash of Mars Climate Orbiter in September 1999. That just cost NASA US$ 125 million.

I got this example from Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making. It's a wonderful book, full of inspiring examples for young scientists, students and entrepreneurs.

What the Crash Had to do with Clear Writing?

Mars Climate Orbiter's software mixed up between English units and metric units, between feet and meters.

Kindergarten mistakes?

Not so fast.

The authors of that software borrowed some pieces from the software of a previous mission called Mars Global Surveyor. The part that caused the problem is called "the thruster equation", which does conversion between feet and meters. That must be a very simple, clear, short piece. What so special about conversion between feet and meters after all?

Pieces of software are just like paragraphs in a long story.

If you write clearly, they read like paragraphs in a good story. You know what each means, which parts of the story are where and so on.

If you are careless, undisciplined or irresponsible, any reader will trip, slip and fall among your lines and paragraphs.

And your software may down a plane, shut down a bank and a hospital, or even crash some space missions.

So, why not start a good habit now. Let's write a bit more clearly starting from today!

Mars Global Surveyor with s a bit messy software.

Mars Climate Orbiter that crashed into mars.

Life Being Stranger than Fiction: MVC in Joomla!

How many steps do you think you have to take to write a Hello World MVC in a web app? About 17, and about hundreds of code lines and configuration lines if you are doing ti in Joomla! I am not making this up. Check the links below.

1. http://docs.joomla.org/Developing_a_Model-View-Controller_Component/2.5/Introduction
2. http://docs.joomla.org/Developing_a_Model-View-Controller_Component/2.5/Adding_a_view_to_the_site_part
3. http://docs.joomla.org/Developing_a_Model-View-Controller_Component/2.5/Adding_a_menu_type_to_the_site_part
4. http://docs.joomla.org/Developing_a_Model-View-Controller_Component/2.5/Adding_a_model_to_the_site_part
5. http://docs.joomla.org/Developing_a_Model-View-Controller_Component/2.5/Adding_a_variable_request_in_the_menu_type
6. ...
7. ...
8. ...
9. ...
10. ...
11. ...
12. ...
13. ...
14. ..
15. ...
16. ...
17. http://docs.joomla.org/Developing_a_Model-View-Controller_Component/2.5/Example_of_menu_parameters_and_stylesheets

Simplicity Despite of Technical Knowledge

Old Style Javascript

A web page is a simple concept to understand, and that is why it caught on so fast in the first place.

And our desires to interact with it are also simple.

Say, there are a number of thingies ("a", "b", "c", "d", and "e") on the page and I want to take some simple actions on them e.g. ( "touch", "drag", "mouse-enter", "mouse-move-away", "click", "focus" and so on ).

Thus, if I click "a" on the web page, I expect some reaction.
Thus, if I drag "b" on the web page, I expect some reaction, maybe the same or something different.

Very simple and very reasonable expectations.
But what did old javascript offer?

Your action: Click a submit button of a form
Your expected reaction: Check the value of a field/blank in the form

Then, you must code/write something long, verbose and ugly. You can verify this by reading any old javascript book.

My Style Javascript

I refuse to touch such smelly, messy stuffs. So, I did something like:

onClick="clicker('id1964')"     

onMouseOver="mover('id1')" 

onMouseOut="mouter('id1964')"  

onClick="displayText('id11', 'Khine 1980');"


/* KYAW TEST June 25, 2007 */

function loader() { alert("on load on body tag"); }

function unloader() { alert("on unload on body tag"); }

function blure() { alert("on blur on body tag"); }

function focuse() { alert("on focus on body tag"); }

function clicker(elemId) {
    var elem=document.getElementById(elemId)
    elem.style.color="red"
    alert("this box follow the link anyway!!!");  
}
function mover(textAreaName) {
    document.getElementById(textAreaName).style.visibility="hidden";
    alert("blah");  
}
function mouter(textAreaName) {
    document.getElementById(textAreaName).style.visibility="visible";    
}

function displayText(textId, mess) {    
    document.getElementById(textId).firstChild.nodeValue=mess; 
}

The function "loader" acts when something is loaded.
The function "focuse" acts when something is focused.
The function "mouter" acts when the mouse is outted.
And so on.

Does it take brains to write simply like that?

Let jQuery Beat Me But ...

As you can see in the comment lines, these were done in June 2007. At that time, I wasn't keen on javascript and so hadn't heard of jQuery and John Resig and his wonderful inventions and re-inventions.  Now with his marvellous library, to get similar effects, I write just 1 or 2 lines.

Conclusion

Insist on simple, reasonable stuffs as a discerning buyer, on behalf of your own customers.
Refuse to use, reuse messy codes.


If you can find a John Resig, use his/her stuffs.

If you can't, at least make and keep things simple for yourself and, above all else, for your customers.

My Visions for Software

I'm referring to IT's Great Power to Spark Changes by Nayan Chanda. Because I have taken time, this post will be very short.

1. Collaboration is getting better. Soon, we will be able to teleport ourselves to any part of the world for a few cents
2. Distribution is getting better. You just write your content/app on your PC, phone, pad, watch, or shirt. Then, just press a button, and the world's population can see/use your stuff in a few seconds in their own devices.
3. Even content may be getting better despite what Rupert Murdoch said here “Devices and platforms are proliferating but this clever technology is merely an empty vessel without any great content.” Since NewCorps alone does not have monopoly on content creation genius, content costs will come down.

Our Opportunities

Imagine you can teleport your son/daughter to any library anywhere in the world, swiftly, safely, and cheaply.

Imagine your kid can use all the library services/apps there for free.

Will that make him/her any wiser, any more employable?

What else is needed?

Some Reasons for Learning to Write Software

on August 6, 2011

This is a kind of brain-storming with myself, and thus they were jotted down in no particular order.

1. to express myself in products, rather than in poetry, paint, dancing moves or tune etc.


2. to father many a virtual world, where schools really behave like schools, kids behave like kids, lessons behave like lessons etc.


3. to exploit market opportunities, technological opportunities, etc


4. to get ease and speed in self-expression


5. to join the conversations and movements of fellow programmers


6. to do some research


7. to own something hard to take away from me


8. to reuse codes written by others


9. to save kids from bad e-learning. Recalls what Bill Gates said in "Road Ahead": Bad processes, when computerized, do more damage faster!

Oh, even that lowly robot is helpful and lovely, isn't it?

Movie-Like Software and Our Chances

People and organisation seem to have an incurable amnesia. One day they say, "Software is king". The next day they say, "Software is commodity". And this pendulum swings go on and on. What is the truth behind this hype and reverse-hype?

Drucker's Predictions

Drucker has famously said that the IT Revolution has not even started yet. He described how printing business gave rise to publishing and other businesses and predicted that IT will give rise to many unforeseen industries.

Drucker also believes that IT will become like movie industry. By extension, it will be like publishing industry too.

Then, I want to ask:

1. Do we, small guys, still have a chance? 
2. Do we get access to technology, market and capital?

1. Can we mount a realistic challenge to Hollywood? e.g. Hong Kong, Bollywood, K Pop, J Pop, etc.
2. Can we pose a challenge to New York Times, Wall Street Journal? 
3. Or do we need something like Chinese Government's multi-billion media initiatives to make any inroads?

Since software is becoming more and more like writing novels and making a movie,

1. Can the next Harry Porter occur in Indonesia? 
2. Can the next Avatar come from Vietnam? 
3. Do we need a Silicon Valley or a Chinese Government to do well in the software market place?

I still believe we have a chance to become major players in software business.

1. Because change is a constant and it is our friend.
2. Or as our friend, Ko Moe Thu suggested, because we create changes that will stack the odds in our favour.

Polyglot Programming, Toolbox Programming

This is , in part, in response to the comments for our Ruby Tutor's Ruby Meta-programming series. Many commentators have suggested that we also try out their favorite programming languages such as Lisp, F#, Scala, Clojure etc.

Polyglot Programming

When you are in Munich, speak German.
When you are in Paris, speak French.
When you are in Osaka, speak Japanese.

All very fine. We have no qualms about switching languages provided we have fluency in those languages.

Does the parallel hold in software?

Write PL/SQL here, code in SQL-92 here, use hibernate here, employ iBatis here.
Use Java there, code in C here, write PHP there, write Ruby here.
Does the parallel really hold in software? I don't think so and here are my reasons.

1. Our attachments to our chosen programming language is far stronger than our attachments to our mother tongues. We switch our human languages depending on who we are talking to. We do so without second thoughts, without hesitation, without discomfort.
2. We are more rational about the human languages than about the programming languages.

Andy Hunt and Dave Thomas of Pragmatic Programmers have suggested a practical solution to this over-attachment: Learn a new programming language every month. My solution is an easier variation of that.

Toolbox Programming

In my method, there is no need to pick up a new language every month, year, or even every decade.
But when you pick up new ones, don't limit yourself to one, pick up as many new ones as you can.

When do you need to pick up new languages?

1. When your guy/gal is giving you trouble, when you start to feel pain. Example: the language is beautiful, the framework is beautiful, but you cannot deploy your app using them. Don't suffer the pain, switch.
2. When you decide that your guy/gal is perfect and that you cannot imagine your life without him/her. No one is perfect and s/he will forsake you someday and thus you'd better start finding some substitutes now.

What new languages do you pick up?
Since you are head over heel in love with this particular language, find as many languages similar to that one as you can.

You love it because it is a Functional language? Find 3 or 4 other Functional languages.

You love it because it enables you to write very small, compact programs. Find 3 or 4 other languages with such concise syntax and conventions.

You love it because it runs on JVM? Fine 3 or 4 other languages that can run on JVM as well.

Conclusion

Just after a few days each with Scala and Clojure, our Ruby Tutor has cured himself of his severe case of Ruby infatuation. Of course, he won't start using Scala or Clojure next week or month. He may not even use them at all. But he now knows he can hope for something Ruby-like if Ruby and its frameworks really prove nasty and stubborn. He won't grovel, kowtow, degrade himself, but switch, or wait, or both, or something else.

Fall in love, it's a fine noble sentiment. But not madly, not with the wrong types.

Once you find you are thinking it will be the end of the world to lose, or break up with that language,

Well, go ahead and pick up at least 3 new tools!

Programming - The Answer , My Friend, is ...

on August 11, 2011

My Own Failures

1. I wrote an application in a Ruby web framework 1.x.x. Everything worked, fine. Then the 2.x.x. version came along and I upgraded and my app broke in many places. I fixed them somehow.

Then the 3.x.x. version came along and I upgraded and my app broke in many places again.

That is not the framework's problem. The real problem is I had used too many plugins.

2. I wrote an application in a PHP web framework 1.6.x. Everything worked, fine. Then the 1.7.x. version came along and I upgraded and my app broke in many places.

This is not the framework's problem. The real problem is I had used too many PEAR's.

3. Before all this, I worked for a project that migrated from EJB1 to 2 to 3 etc.
And for another project, where the project managers took years to move from Struts 1 to 2.

Other Failures

1. Netscape's spaghetti code that cost the company its battle against Microsoft.
2. Friendster's social graph that slowed the site to a crawl.
3. MySpace's "technical debt" that ruined the site.
4. Ruby on Rails framework --- Its lack of hosting (Maybe their developers don't care or it is their blind spot). And its handling of data management; if they don't know this part well enough, they should leave it alone e.g. PHP has no ORM and has been doing just fine.
5. Microsoft alone has a dozen or more data access technologies from 1998 to 2008, according to a book by Joel Spolsky. I don't have time or experience to know such stuffs.

My Own Successes

1. Porting RedBlack tree, AA tree etc from Java to Ruby is easy.
2. Porting shortest path, quick sort etc from Ruby to Python is easy.
3. Porting Regular expressions, string/text handling etc from PHP and Perl to Ruby is easy.
4. Porting higher order functions amongst Perl, Python, Ruby is easy.
5. Porting any plain domain or business object from any language to any other language is very very easy. From Java to Pascal, from Perl to PHP, from Ruby to C++, whatever.

6. Modeling or making mental representations or doing objects, classes and APIs for Law domains, Insurance domains, Education domains are very very very easy and fast.

7. Dreaming, going into trance, seeing in my mind eyes some apps end to end from the users' point of views is extremely easy and fast. "Easy" doesn't mean I do these activities well because, often, I even forgot to do them at all!

8. Writing domain objects is very easy.

9. Combining those objects is very easy.

Other Successes

1. Real programmers code everything by themselves. A paradox because they do and they don't.
2. Real programmers don't trust others' codes. A paradox because they do and they don't.

3. Excel team wrote even their own compiler, and killed off all dependencies.
4. Algorithms are very infrequent. They come once or twice in a decade.

5. Facebook is using many programming languages. So is Twitter.
6. Google founders didn't learn HTML etc on purpose.

The answer

If you can't find Apple-style appealing packaging, sell your stuffs, your wares raw, plain, uncovered!

Have fun!

From "Code and Pray" to "Crash and Fix"

August 30, 2010

Software industry is changing every hour and making progress every day. Software development methods and techniques are also getting better every year. Thus, we old folks have to stay updated and one way to achieve this is to learn from the apprentices and the interns.

This week I like to discuss different intern techniques: those in our days and those nowadays. In our intern days, one common technique was "code and pray." We would code something and run it by the complier and pray that it worked. Some practictioners also call this coincidental style of coding.

Recently I noticed that the interns at our company have developed this to a higher level. I noticed that they are now using "crash and fix" technique. It's a beauty in this particular case.

Crash and Fix GQL

As we all know, we need to create and provide an index.yaml file to Google App Engine so that it can search its data store properly.

Locally (that is on our development server), if we make a new kind of query, the Engine updates the index file. When we repeat that query some time later, the Engine is ready to do the search.

However, on the real server, if we make a new kind of query, the Engine just scans the index file. If the index necessary for the query is not there, the Engine throws exceptions.

There is no quota/limit on the local engine, but the real engine places some limit on the number of indexes.

Thus, the right thing to do, at least for us old folks, is:

1. Find typical usage patterns.


2. Make queries for just these patterns.


3. The resultant index.yaml file, developed on the local server, will be relatively small.


4. Use this small index.yaml file on the real server,

This would take time and insight. Modern interns have better ideas:

1. Deploy the app on the real server with an empty index.yaml file.


2. When a user makes a new type of query, the Engine throws excpetions. Log those exceptions somewhere.


3. Check those logs. The log tells us the specifications of the index it needs for a particular query..


4. Copy such specifications into a new index.yaml file.


5. Deploy the app on the real server again, and repeat.

This also took time but not insight. Do the modern interns have an edge over us old folks? Pay special notice to the comment "# goodpt3," "# goodpt2" etc. Those below 1 were added before those below 2 and so on. The interns will check only the topmost section, which is added the most recently. Time-saving and brain-saving, cool?

indexes:

# AUTOGENERATED

# This index.yaml is automatically updated whenever the dev_appserver
# detects that a new type of query is run.  If you want to manage the
# index.yaml file manually, remove the above marker line (the line
# saying "# AUTOGENERATED").  If you want to manage some indexes
# manually, move them above the marker line.  The index.yaml file is
# automatically uploaded to the admin console when you next deploy
# your application using appcfg.py.

...

- kind: Inventory
  properties:
  - name: expenses
  - name: locations
  - name: distance
  - name: skus
  - name: unit_space
    direction: desc
  - name: transports
...

- kind: Inventory
  properties:
  - name: gender
  - name: name
  - name: distance
  - name: skus
  - name: unit_space
    direction: desc
  - name: transports

# goodpt3
...
- kind: Inventory
  properties:
  - name: unit_space
  - name: gender
  - name: locations
  - name: phone
  - name: skus
  - name: transports
...


# goodpt2

...
 
- kind: Inventory
  properties:
  - name: unit_space
  - name: expenses
  - name: gender
  - name: locations
  - name: phone
  - name: distance
  - name: skus
  - name: transports
 
# goodpt1 - kind: Inventory
  properties:
  - name: education
  - name: transports
  - name: gender
  - name: locations
  - name: skus
  - name: unit_space
    direction: desc
...

- kind: Inventory
  properties:
  - name: transports
  - name: gender
  - name: locations
  - name: distance
  - name: skus
  - name: unit_space
    direction: desc
...

The file has almost 1200 lines and uses up 92% of the index quota. If we add new models/entities, we must make new queries and will need new indexes. Will the remaining 8% cover most of our future needs, or am I over-planning?

An Online Programming Community and their Members

Here is a verbatim invitation to the  Singapore Python User Group on Facebook.

I admire the source code of Django, Tornado, NLTK, boto, etc. As many of the world greatest coders in history advise, their logic/code is "dumb" and therefore good.

When I coded shopping carts, CRUD websites etc, my logic/code stayed "dumb" and good. But when I moved even a bit farther, e.g. FSM with Pygame, an expert system etc, my logic/code became "clever," complex and unstable.

I've thought about and studied this for years.

Our team is training to be a world leader in software -- both biz-wise and techno-wise. Anyone interested in becoming better programmers are welcome.

We are humble to learn and generous to share.

Love and Thanks to all.

I'll let you see and hear how they respond in a few days' time.

Revolutions in Artificial Intelligence and Me ;-)

I am not interested in "hard" things. I don't want other people to do "hard" things either. So, when I read about previous AI revolutions and learned about their "hard" problems, I was not inspired, to say the least.

1. Tough to do even a tic tac toe problem.
2. Tough to overcome "knowledge bottleneck".
3. Tough to encode "commonsense", as demonstrated by CYC project.
4. Machines will learn on their own, but you have to train them first, and it's Tough to train them.

But when Google fans claimed what it is doing is a sort of AI, I became inspired a bit because this brand of AI seems accessible and easier to do. Even then, I took no action actually.

Now, AI seems to have entered a new epoch. Some AI researchers claimed in a New Scientist article that they've done "with relatively little programming" what it had been tough for older kinds of AI. They labelled their kind of AI as "morphological computing." Here I explain one of their most illuminating examples.

1 snail is boring.
10 snails are still boring.

1 octopus is interesting.
10 octopuses are still more interesting.

Rich behaviours, idiosyncratic behaviours, responsive actions.

Are octopuses more "intelligent" than snails?
Are octopus brains more "intelligent" than those of snails?

The researcher knew that this can't be because he also knew that both of them are "mollusc." According to him, the octopus also has no brain.
Another useful link.

What excites me most is that an entity can be made to look, sound, seem, feel intelligent while in reality it has no real intelligence.(An aside: this is far better than Searle's Chinese Room interpreter.)

What inspires me most is that I don't need to design/code/program/write a brain to make my software behave intelligent.

That's, for a programmer, a bit too much even to dream about and wish for!

I share. Please use it for your benefits.

on January 27, 2012

Data Prophet : Nate Silver

I'm summarizing an article Data Prophet in Wired October 2012. It's an interview with Nate Silver, a statistician, a data scientist.

He pointed out a number of problems with our typical predictive models.

Out of Sample Problems

Here we get no data for the phenomenon because we did not collect such data in the first place. e.g. If your model excludes real estate as an industry, whatever rise, fall, crisis happens in real estate, you will have no data for them. And your economic prediction will also miss problmes and effects arising from real estate

Over Fitting Problem

Here, you're mistaking mere coincidences or co-occurrences for patterns. e.g. You observed that whenever ice cream sales go up, murder cases go up too. Then, you assumed that there was a pattern.
 If you curb ice cream sales, will you prevent some murder cases then?

Nate advises that instead of looking for ideas, we should accept and see what is really there in the data.

Problem of Over-reacting to New Data

Maybe your model is too flexible or sensitive to new data or new types of data.

Here, I can give a personal experience. Recently I read in Newsweek about life-after-death experience of a neurosurgeon. He said that he got to a strange place and met strange beings, that there our senses were fused such that he could see a sound and hear a color at the same time etc. I believed every word of his. That is, a person like him who has lived like he has lived his life, can get to a place like that.

Nat Silver wisely warns that more is worse if we don't have a good framework, that we'd better stick to basic models if we are prone to over-reaction to new data. Wise poker players, stock investors, soldiers all now this.

I asked myself what if I got there myself. I would hear-see a sound-smell-sight. Then, I would see-hear it dissolve and another sense arise .... dissolve and arise .... dissolve and arise .... dissolve and arise ....I hope my basic, simplistic model will work there too.

The Need to Test in the Right Environment

An example of the problem situation may go like this: You created your model with data from New Mexico. You then tested the model in New Mexico again.

According  to Nate Silver, the right environment is" an environment where you don’t know what’s going to happen."

PayPal IPN, Google App Engine, Python Integration, an Example

References

1. PayPal IPN Python Code from Django Aware blog.
2. A Google Group discussion

3. Pro PayPal E-Commerce. I recommend this book.
4. Professional Web APIs with PHP. This book is good for non-PayPal stuffs too. A good book.

My Tutorial

(1) Make a sandbox/testing account with PayPal.
(2) Log in to that account. Get to IPN Simulator at the bottom of the page.
(3) In the form that shows up, make sure
1. notify url = http://your_app.appspot.com/path-to-your_ipn_handler
2. "receiver_email" = "ACCOUNT_EMAIL" that you use in your IPN handler code.
3. You should vary "payment_status" from "Completed" to "Failed" to "Pending" etc.
Send the form a few times.
(4) Paypal says it can't send.
Change " notify url" to www.google.com etc. You get the same failure result.
It is as expected: you haven't written the code to handle this POST request.

The code you need to write is:

import os
import sys
import logging
import wsgiref.handlers
from google.appengine.ext import webapp
from google.appengine.ext.webapp import template
from util.sessions import Session
from google.appengine.ext import db
sys.path.append(os.path.join(os.pardir,os.getcwd()))
from helper_rendering import doRender

import urllib, urllib2
from google.appengine.api import mail
from google.appengine.api import urlfetch
from google.appengine.ext.webapp.util import run_wsgi_app
PP_URL = "https://www.sandbox.paypal.com/cgi-bin/webscr"
ACCOUNT_EMAIL= "my_email_for_sandbox@nice.com"
#PP_URL = "https://www.paypal.com/cgi-bin/webscr"
#ACCOUNT_EMAIL = "my_email_for_real-paypal_acct@good.com"

class IPNHandler1(webapp.RequestHandler):
  def post(self):    
     parameters = None
     # Check payment is completed, not Pending or Failed.
     if self.request.get('payment_status') == 'Completed':
         if self.request.POST:
             parameters = self.request.POST.copy()
         if self.request.GET:
             parameters = self.request.GET.copy()
         logging.debug('IPN 1, 
                 or your own message for yourself to read in log.')
     else:
         self.response.out.write('Error, sorry. 
                The parameter payment_status was not Completed.')
     # Check the IPN POST request came from real PayPal, 
     # not from a fraudster.
     if parameters:
            parameters['cmd']='_notify-validate'
            params = urllib.urlencode(parameters)
            status = urlfetch.fetch(
                         url = PP_URL,
                         method = urlfetch.POST,  
                         payload = params,
                        ).content
            if not status == "VERIFIED":
                   self.response.out.write('Error. 
                    The request could not be verified, check for fraud.')
                   parameters['homemadeParameterValidity']=False 
                   # parameters = None
                   # You may log this data in your database, 
                   # for later investigation.
     # Check the money is really to go to your account, 
     # not to a fraudster's account.
     if parameters['receiver_email'] == ACCOUNT_EMAIL:  
         transaction_id = parameters['txn_id']
         # Check if this is a new, unique txn, 
         # not a fraudster re-using an old, verified txn.
         invoice_id = parameters['invoice']
         currency = parameters['mc_currency']
         amount = parameters['mc_gross']
         fee = parameters['mc_fee']
         # Check if they are the right product/item, right price, 
         # right currency, right amount, etc.
         email = parameters['payer_email']
         identifier = parameters['payer_id']
         # Email/notify/inform the user for whatever reason.

         parameters['your_parm']= "It is ok on 19 September, 2010." 
         logging.debug('IPN 100. All OK.')
         logging.debug(parameters['txn_id'])
         logging.debug(parameters['invoice'])
         logging.debug(parameters['payer_email'])
         # With this IPN testing, you can't see results on the browser.
         # See results on the log file maintained by Google AppEngine.
  
         #template_values = { 'params': parameters, }
         #doRender(self,'ipn_ok.htm', template_values )

if __name__ == '__main__':
  print "IPNHandler1 is being run as main()."

(1) Now send the IPN Simulator form again as explained in the beginning of this post.
Now Paypal says it has sent it successfully.
(2) Log in to your Google AppEngine developer account.
Get to the log. Set logging level to "debug"
Check the log entries. you will see something like:

 1. 09-29 01:54AM 28.745 /ipn_handle1 200 2141ms 408cpu_ms 4kb gzip(gfe)
      See details
      216.113.191.33 - - [29/Sep/2010:01:54:30 -0700] "POST /ipn_handle1 HTTP/1.1"
 200 4842 - "gzip(gfe)" 
"yourstruly.appspot.com" 
ms=1234 cpu_ms=123 api_cpu_ms=0 cpm_usd=0.123456 loading_request=1

   2.  D 09-29 01:54AM 29.395 IPN 1

   3.  D 09-29 01:54AM 30.682 IPN 100. All OK.

   4.   D 09-29 01:54AM 30.682 31929814

   5.   D 09-29 01:54AM 30.682 abc1234

   6.   D 09-29 01:54AM 30.682 buyer@alan.htet.com

   7. I 09-29 01:54AM 30.705 Creating session session-575150356408086346

   8. I 09-29 01:54AM 30.884 This request caused a new process to be started 
      for your application, and thus caused your application code to be loaded 
      for the first time. This request ... 

If you have any question, just ask by way of commenting below.

Google Query Language, General Patton and an Apprentice

on August 21, 2010

Topics: Google Query Language, General Patton and an apprentice at our software startup.

Google as you know allows its employees to take 20% of their time to do their own pet projects: 80% for the company and 20% for their own.

Our company? Yes, our ratio is almost the reverse, especially for the interns and the apprentices. They take 80% of their time to do their own pet projects and their own learning: 20% for the company and 80% for their own. So, they put in about 2 hours per day, out of 10 hours working time (10 hours include 2 hour lunch time and other breaks!)

Even then, I wonder, "Is this apprentice wasting our time?"

And General Patton? The famous American WW II tank commander? What does he have to do with software?

The answer: a lot.

If impatient, just go leap to the conclusion.

Example 1

from google.appengine.ext import db
from models.tutor import Residence
import inspect

query = db.GqlQuery("SELECT * FROM Residence WHERE " 
"dimensions IN :mydimensions "
"AND locations IN :mylocs "
"AND rent < :myrent "
"AND gender = :mygender"
,
mydimensions =["30x60", ],
mylocs = ['W', "W"],
myrent = 41,
mygender= "M"
)
print inspect.getmembers(query)

which gives the following result:

[('__class__', ), 
('__delattr__', ), 
('__dict__', {'_namespace': None, '_proto_query': , '_kwds': {'mylocs': ['W', 'W'], 'mydimensions': ['30x60'],
'myrent': 41, 'mygender': 'M'}, '_args': [], '_model_class': , 
'_keys_only': False, '_cursor': None, '_end_cursor': None, '_compile': True}),
('__doc__', 'A Query class that uses GQL query syntax instead of .filter() etc.'), 
('__getattribute__', ),
('__getitem__', bound method GqlQuery.__getitem__ of >), ('__hash__', ), 
('__init__', bound method GqlQuery.__init__ of ('__iter__', bound method GqlQuery.__iter__ of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c>>), ('__module__', 'google.appengine.ext.db'), 
('__new__', built-in method __new__ of type object at 0x8146a20), 
('__reduce__', built-in method __reduce__ of GqlQuery object at 0xaa7dd6c),
('__reduce_ex__', built-in method __reduce_ex__ of GqlQuery object at 0xaa7dd6c), 
('__repr__', method-wrapper '__repr__' of GqlQuery object at 0xaa7dd6c), 
('__setattr__', method-wrapper '__setattr__' of GqlQuery object at 0xaa7dd6c), 
('__str__', method-wrapper '__str__' of GqlQuery object at 0xaa7dd6c), ('__weakref__', None), ('_args', []), ('_compile', True), ('_cursor', None), ('_end_cursor', None), 
('_get_query', bound method GqlQuery._get_query of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c), 
('_keys_only', False), ('_kwds', {'mylocs': ['W', 'W'], 
'mydimensions': ['30x60'], 'myrent': 41, 'mygender': 'M'}), ('_model_class', class 'models.tutor.Residence'), 
('_namespace', None), 
('_proto_query', google.appengine.ext.gql.GQL object at 0xaa7d6ac),
('bind', bound method GqlQuery.bind of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c),
('count', bound method GqlQuery.count of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c),
('cursor', bound method GqlQuery.cursor of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c),
('fetch', bound method GqlQuery.fetch of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c), 
('get', bound method GqlQuery.get of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c),


('is_keys_only', bound method GqlQuery.is_keys_only of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c), ('run', bound method GqlQuery.run of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c), ('with_cursor', bound method GqlQuery.with_cursor of google.appengine.ext.db.GqlQuery object at 0xaa7dd6c)]

Example 2

from google.appengine.ext import db
from models.tutor import Residence
import inspect

query = db.GqlQuery("SELECT * FROM Residence WHERE " 
"dimensions IN :mydimensions "
"AND locations IN :mylocs "
"AND rent < :myrent "
"AND gender = :mygender"
,
mydimensions =["30x60", ],
mylocs = ['W', "W"],
myrent = 41,
mygender= "M"
)

print query.__dict__

which gives the following result:

{'_namespace': None, '_proto_query': , '_kwds': 
{'mylocs': ['W', 'W'], 'mydimensions': ['30x60'], 'myrent': 41, 'mygender': 'M'}, '_args': [], 
'_model_class': class 'models.tutor.Residence', '_keys_only': False, '_cursor': None, 
'_end_cursor': None, '_compile': True}

Example 3

from google.appengine.ext import db
from models.tutor import Residence
import inspect

query = db.GqlQuery("SELECT * FROM Residence WHERE " 
"dimensions IN :mydimensions "
"AND locations IN :mylocs "
"AND rent < :myrent "
"AND gender = :mygender"
,
mydimensions =["30x60", ],
mylocs = ['W', "W"],
myrent = 41,
mygender= "M"
)

obj1= query.__dict__['_proto_query']
print obj1

which gives the following result:

google.appengine.ext.gql.GQL object at 0xaa7814c 

Example 4

from google.appengine.ext import db
from models.tutor import Residence
import inspect


query = db.GqlQuery("SELECT * FROM Residence WHERE " 
"dimensions IN :mydimensions "
"AND locations IN :mylocs "
"AND rent < :myrent "
"AND gender = :mygender"
,
mydimensions =["30x60", ],
mylocs = ['W', "W"],
myrent = 41,
mygender= "M"
)

obj1= query.__dict__['_proto_query']
print obj1.__dict__['_GQL__symbols']

which gives the following result:

['SELECT', '*', 'FROM', 'Residence', 
'WHERE', 'dimensions', 'IN', ':mydimensions', 
'AND', 'locations', 'IN', ':mylocs', 
'AND', 'rent', '<', ':myrent', 
'AND', 'gender', '=', ':mygender']

which gradually leads to

['SELECT', '*', 'FROM', 'Residence', 
'WHERE', 'name', '=', ':myname', 
'AND', 'email', '=', ':myemail', 
'AND', 'phone', '=', ':myphone', 
'AND', 'rent', '=', ':myrent', 
'AND', 'neighborhood', '=', ':myneighborhood', 
'AND', 'transportation', '=', ':mytransportation', 
'AND', 'gender', '=', ':mygender', 
'AND', 'race', '=', ':myrace', 
'AND', 'dimensions', 'IN', ':mydimensions', 
'AND', 'locations', 'IN', ':mylocations', 
'ORDER', 'BY', 'email', 'DESC', ',', 'rent', 'ASC']

Conclusion

At one time in WW II, the rumors (which was actually true) went that the German tanks were far stronger than the American tanks. General Patton must have known this before anyone else.

Did he admit the truth? No.

Did he lie and refute the truth? No.

What did he say? He said in effect that he never intended to use his tanks that way. His tanks were to do something else, not ram/bump/jar into or out-punch/out-slug/out-shoot the opponent tanks. Read more about that here.

The younger software developer wants to look into the nuts and bolts of a tank (a deliberately mixed metaphor).
The older General just wants him to look into the logic of his query in pseudo-code or read the doc for GQL(mixed again).

Engineers and soldiers
programmers and suits
engineers and marketers

Anyway, we gave the apprentice a pat on the back. Whether he works his 2 hours as Engineer or Soldier, is fine with us.

Meta-programming in Ruby : Part 16 : instance_eval

There are altogether 16 posts on this topic of meta-programming in Ruby language. Our approach encourages hands-on experimentation. The exercises here are specially designed to facilitate self-study and other independent modes of learning.

Provided we can find time amidst our busy schedule(well, who is not busy these days?), we will also discuss meta-programming in other languages such as Lisp and python.

Code 1

instance_eval and self.

# instance_eval evaluates a block of code
# with self set to the object that's receiving the instance_eval call.

add_two = Proc.new { self + 2 }

puts 1.instance_eval(&add_two)   #=> 3
puts 2.instance_eval(&add_two)   #=> 4

"stellar coding staff".instance_eval{p self}
[1970, 1980, 1990, 2000].instance_eval{p self}   
{'a'=> 1970, 'water'=> 1990, 'gold'=>2010}.instance_eval{p self}  
puts "\n\n"


# Here's a second example using an implicit method receiver.
call_reverse = Proc.new { self.reverse } 

p "alan akhoe kid".instance_eval(&call_reverse)   
p ["x", "y", "z"].instance_eval(&call_reverse)               #=> ["z", "y", "x"]
p [1, 34, 117, 997].instance_eval(&call_reverse)             
# OK BAD # p 12121.instance_eval(&call_reverse)

Result 1

[root@localhost ruby_tutor]# ruby ie_self.rb
3
4
"stellar coding staff"
[1970, 1980, 1990, 2000]
{"a"=>1970, "water"=>1990, "gold"=>2010}


"dik eohka nala"
["z", "y", "x"]
[997, 117, 34, 1]

Code 2

We can display private attributes using instance_eval.

class C
   def initialize
     @a = 0
   end
   def next
     @a += 1
   end
 end
w = C.new
p w.next
p w.next; puts;puts

p w.instance_eval "@a"          #=> 2
p w.instance_eval { @a }        #=> 2
p C.new.instance_eval { @a }    #=> 0

Result 2

[root@localhost ruby_tutor]# ruby ie_and_private_vars.rb
1
2


2
2
0

The Crisis in Higher Education, Nicholas Carr, MIT Technology Review

The Crisis in Higher Education, Nicholas Carr, MIT Technology Review reviews many e-institutions such as Coursera, edX, Udacity, Open Learning Initiative, University of Phoenix etc.  Many professors, educators have valid concerns about MOOCs(massive open online courses).  The article shows no way out. I point out one way out here. Many more ways out may come in my upcoming posts.

The promoters of MOOCs have a "fairly naïve perception of what the analy­sis of large data sets allows," says Timothy Burke, a history professor at Swarthmore College. He contends that distance education has historically fallen short of expectations not for technical reasons but, rather, because of "deep philosophical problems" with the model. He grants that online education may provide efficient training in computer programming and other fields characterized by well-established procedures that can be codified in software. But he argues that the essence of a college education lies in the subtle interplay between students and teachers that cannot be simulated by machines, no matter how sophisticated the programming.
 

Alan Jacobs, a professor of English at Wheaton College in Illinois, raises similar concerns. In an e-mail to me, he observed that the work of college students "can be affected in dramatic ways by their reflection on the rhetorical situations they encounter in the classroom, in real-time synchronous encounters with other people." The full richness of such conversations can't be replicated in Internet forums, he argued, "unless the people writing online have a skilled novelist's ability to represent complex modes of thought and experience in prose."
 

In theory, this "flipped classroom" will allocate teaching time more rationally, enriching the experience of both professor and student.

Here, too, there are doubts. One cause for concern is the high dropout rate that has plagued the early MOOCs.
 

The greatest fear among the critics of MOOCs is that colleges will rush to incorporate online instruction into traditional classes without carefully evaluating the possible drawbacks.
 

The Way Out I Promised

A Myanmar saying goes : "With the fish's oil, fry the fish." The answer comes on page 92 of that same magazine. How Authors Write By Jason Pontin on October 24, 2012 :

The technologies of composition, not new media, inspire ­innovations in literary styles and forms.
 

The Atavist's authors have not been moved, as Baker was, by the creative possibilities of a new technology. Writers are excited to experimentation not by the media in which their works are published but, rather, by the technologies they use to compose the works.