Tuesday, June 30, 2009

Enumerators for Dependency Injection

A situation came up where I needed to select elements based on dependency injection. The elements were portions of a larger bitmap that were being used as tiles in a 2D graphics engine. Selection might be sequential, random, fixed, etc., but needed to be hidden from downstream processes. In particular, I needed to inject not the enumerators themselves, but enumerator factories.

At first I was going to use a custom interface. After some thought, however, I decided that the .NET IEnumerator<> interface fit the bill nicely. Technically an enumeration is some kind of ordered or unordered sequence from a collection. But a randomized sequence fits that bill, and it’s not too much of stretch to think of an infinitely looping sequence as an enumerator. (For example, think of the original collection as specifying the set of allowable items of an infinite sequence.)

So I came up the following helper functions:
static IEnumerator<TOutput> MakeEnumerator <TInput,TOutput,TState> (
TInput inputIn,
TState stateIn,
Func<TInput,TState,TState> funcMoveNext,
Func<TInput,TState,bool> funcMovedNextOk,
Func<TInput,TState,TOutput> funcCurrent)
{
while (true)
{
stateIn = funcMoveNext(inputIn,stateIn);

if (!funcMovedNextOk (inputIn,stateIn))
break;

yield return funcCurrent(inputIn,stateIn);
}
}


static Func<TInput,IEnumerator<TOutput>> MakeEnumeratorFunc <TInput,TOutput,TState> (
TState stateIn,
Func<TInput,TState,TState> funcMoveNext,
Func<TInput,TState,bool> funcMovedNextOk,
Func<TInput,TState,TOutput> funcCurrent)
{
return
(i)=> (
MakeEnumerator(
i,
stateIn,
funcMoveNext,
funcMovedNextOk,
funcCurrent));
}

The first helper is a generic enumerator builder that takes three functions and a state which control the enumeration. The functions transition from state to state, determine whether a transition has exhausted the enumerator, and select an enumerated item. (I separated the move next functionality and inverted its typical order because it makes things a lot more convenient.)

The second helper uses this first helper to build an enumerator factory function.

Below shows how this could be used to build an enumerator factory for canonical list enumeration. (You wouldn’t want to use these functions for this in a typical scenario; the traditional way would suffice there. They are mainly useful in dependency injection situations.)

Func<List<int>,IEnumerator<int>> xMakeEnumeratorFunc =
MakeEnumeratorFunc<List<int>,int,int>(
-1,
(i,s)=>(s+1),
(i,s)=>(s<i.Count),
(i,s)=>(i[s]));

List<int> xList = new List<int>() { 1, 3, 5, 7 };

IEnumerator<int> xEnumerator = xMakeEnumeratorFunc(xList);

// Inject xEnumerator here.

Fun, func, function!

(Tomorrow: I find a situation where a tradition boolean MoveNext is superior and post the overloaded functions.)

-Neil

Thursday, June 11, 2009

Enumerating Bits

OK, so the enumerating the set of subsets of non-zero bits is not a common task. But what about enumerating the set of non-zero bits in an integer? That’s something that tends to happen at least one or twice in any large project. Here’s how that can be done use a related function.

Given an integer (i), the least significant non-zero bit (b) can be found using this equation:

b = (i & -i)

(This is discussed as equation (37) in Donald Knuth’s The Art of Computer Programming, Volume 4, pre-fascicle 1A, page 8.)

It is easy to turn this into a loop which enumerates the bits in an integer:
while (xInteger>0)
{
int xBit = (xInteger & -xInteger);
xInteger &= ~xBit;
// Do something with xBit here.
}

(This is probably a case where one wouldn’t want to use a generic “yield return” enumerator, since if one is working at this level of detail, efficiency is probably the main concern.)

This can also be made to work in C# with unsigned integers by using various types of casting. But if efficiency is paramount, pay attention to the generated MSIL, since casting at some points generates more efficient code than does casting at other points.

Tests in release mode show that this construct is nearly four times faster in C# (.NET 3.5 version) than is shifting! But it looks arcane at first glance, so be sure to document the code well.

Tuesday, June 9, 2009

Power Set of a Bitmask

Here’s a cute trick for enumerating the power set of an integer bitmask. It’s based on equation (84) in Donald Knuth’s The Art of Computer Programming, Volume 4, pre-fascicle 1A, page 18, available in an "alpha test" version at:

http://www-cs-faculty.stanford.edu/~knuth/taocp.html

The idea is that, given a subset (x) of a mask (M) the next largest lexicographic subset (x') is:

x’ = (x – M) & M

Combined with the C# "yield return" statement, this allows a simple enumeration of the power set of a mask. You simply seed the subset with zero, and iterate until it is once again zero.

public static IEnumerable<int> PowerSet (
int maskIn)
{
int xSet = 0;

do
{
yield return xSet;
xSet = (xSet - maskIn) & maskIn;
}
while (xSet!=0);
}

For example, calling PowerSet(0x19) returns an enumerator which yields (in binary):

00000
00001
01000
01001
10000
10001
11000
11001


I can’t think of a specific use for it right off hand, but if a project uses enough bit manipulation, I’m sure something would arise.

Code Generator Oddity

The following is a exerpted example of my implementation of a well-known permutation algorithm. Note the code generation oddity flagged by the comment. This occurs in .NET 3.5 and Visual C# 2008. I leave the solution to the mystery to readers who like to disassemble and read MSIL, and think about what the JIT compiler may do to it.

protected static void Permute <TDatum> (
int beginIn,
int endIn,
TDatum[] dataIn,
Action<TDatum[]> dataActionIn)
where TDatum : IComparable
{
TDatum xDatumBegin;

if ((endIn-beginIn)<=1)
{
dataActionIn(dataIn);
}
else
{
Permute(
beginIn+1,
endIn,
dataIn,
dataActionIn);

for (int i=beginIn+1; i<endIn; i++)
{
xDatumBegin = dataIn[beginIn];

dataIn[beginIn] = dataIn[i];
dataIn[i] = xDatumBegin;

Permute(
beginIn+1,
endIn,
dataIn,
dataActionIn);

// Believe it or not, the compiler/loader
// will optimize the first block below
// to be almost 10% faster than the second.

#if thisIsFasterInDotNet
xDatumBegin = dataIn[beginIn];
dataIn[beginIn] = dataIn[i];
dataIn[i] = xDatumBegin;
#else
dataIn[i] = dataIn[beginIn];
dataIn[beginIn] = xDatumBegin;
#endif
}
}
}

Sunday, May 24, 2009

Zen Dream

Here’s something I once posted in a newsgroup.

Last night, I was practicing with some water-colors and I painted a series of concentric black and white circles, kind of like an archery target. Later, while I was asleep, I dreamt I was in a store. I saw that there were bows and arrows for sale, and I realized I had a consuming desire to practice archery. I picked out a bow, but I couldn’t find any arrows. Then I realized that they were haphazardly scattered about the floor, intermixed with long-handled paintbrushes. The arrows were missing feathers, and then other difficulties began to intervene, until I despaired of ever putting to together a decent archery set.

Then I woke up and realized I wasn’t interested in archery at all; I wanted to paint.

I had intended to end this story with the Zen saying "The skilled archer does not aim for the center of the target,” but I wanted to look it up in Yahoo to see if I could find the original reference. The first Yahoo hit I found was a newsgroup thread where I had quoted it earlier without reference. And if that's not some kind of koan, I don't know what is.

Monday, April 27, 2009

LINQ

It would not take even a Dr. Watson to notice that this blog has been idle for over a year. It might, however, take a Sherlock Holmes to discern the reason why. So I will sum it up with a word:

LINQ

Shortly after the last post, I began playing around with the new version of Microsoft Visual Studio, which included LINQ and all the new lambda programming goodies. I realized they made obsolete many of the issues I had discussed in the previous entry.

Concurrently, I decided to port a professional expert system engine I wrote and maintain to the new version of Visual Studio. It was written mainly in C++, with .NET interface code in C#. As I went along, I found myself thinking: “Perhaps I should just re-write the whole engine from scratch, using the newest C# technology.”

And so I did. And interspersed with other projects, etc., it took the better part of nine months to complete the engine, and another couple of months to work out the kinks and get it hooked up to the external components (UI, etc.). It was a lot of fun, but it did interfere with my blogging, to say the least.

In retrospect, it was a good decision. I learned the new lambda elements of C# in a way that would have been tough using a less whole-hearted approach. I also learned possibly more than I ever wanted to know about C# generics, lol. I want to share some of the things I discovered in subsequent posts in this blog, but for now I’ll just share what I consider the most important discovery:

Every programmer, it seems, hates exception handling, and I am no exception. So, when I started the re-write, I decided that I would, for once in my exception handling life, be good and pay attention to it from the ground up. So I started my re-write by crafting a set of classes to let me handle and pass exceptions and returns. They were not intended to replace the existing .NET exception handling, but rather to work hand in glove with it. Also, I began a discipline of handing errors and exceptions fully and correctly in the code at the exact moment I was writing the code. No more “I’ll save error handling for a rainy day.” It took a few weeks, but it stuck; it’s now a habit and I don’t fear error handling any longer.

Best programming discovery I ever made.

Thursday, March 20, 2008

Hierarchical vs. Relational

Some Thoughts About Hierarchical vs. Relational Representation
Part 1 – Rationale

As a rule of thumb, I like to design things so that each instance has three possible representations: in-memory object, hierarchical (e.g. XML), and relational. This tends to work out well because these are the three media of exchange in most computer architecture. Thus, in-memory for quick manipulation, hierarchical for static storage and communication via remoting or clipboard, and relational for dynamic storage.

This tends to work seamlessly except in one particular area: referenced objects (i.e. indexed objects in relational systems). Which is unfortunate, because referenced objects are a key paradigm in most computer languages and are the backbone of relational databases. This problem of coordinating referenced objects comes up in virtually every large program or code library, from e-commerce databases, to 3D graphics systems, to spreadsheets, etc.

I’ve tried several generic solutions to this problem. Most worked, but most became unwieldy as the code grew and special cases were considered. Because of this, I’ve decided to accept the fact that generic solutions here are difficult, and decided that instead, hope lies in the design patterns approach.

So, over the course of the next few blog entries, I will be presenting a series of design patterns aimed at tackling this problem. I don’t presume to think that any of these techniques are novel. The point is to arrange a series of rules-of-thumb and ad-hoc solutions into a more coherent set of design patterns.

-Neil

Saturday, January 12, 2008

Pine Board is Flawed

The pine board I was planning on using to make my first goban turned out to be irredeemably flawed on the bottom side. So I guess it’s back the attic floor with that one, and I’ll start with trying to make a goban from cherry! Maybe the value of the wood will drive me to do a good job, lol.

Wednesday, January 9, 2008

Opening Game Study


One thing I’ve noticed in my play against SmartGo is that my closed moyo tend to develop too early. So, although I securely capture some area, I allow SmartGo to capture large segments of the remaining board and so I get crushed.

My game is too “chunky” and lacks the “integrated” look one sees in most games.

At first I wondered whether this was because of the computer style of play vs. a human style of play; was I learning bad habits? However, I decided it must be because I was not fully developing the opening game before proceeding to closed moyo.

To that end (or beginning, lol), I bought “In the Beginning” by Ikuro Ishigure. I have to say that with just one day of reading, it has solved a lot of the problems I was having with the opening. The example here, though riddled with mistakes and lacking good joseki, does show more of an integrated pattern. And, though I got stomped in mid game, I did do significantly better through the opening game than I have been (based on computer analysis of the game).

-Neil

p.s. And I retrieved the pine board for goban number 1 from the attic flooring (yes, I did replace it with other boards, lol). I still need to mark it for size and decide on how to cut it for minimal warping.

Sunday, December 30, 2007

Goban R&D

Scrounging about the house and yard, I have located sufficient wood for three gobans:

One in 3/4" furniture-grade pine with nice grain and figurings. (I can possibly double it to 1.5", depending on knot holes.)

One in 3/4" high-grade cherry.

One in 2" high-grade cherry! (If I sacrifice some turning stock, which I'm loathe to do.) If I decide to use this, there will also be enough left over to make legs for it as a thin floor board.

I think the cherry boards, though darker than traditional boards, would look awesome with Yunzi jade-luster stones.

So I will start with the pine and work my way up, teaching myself the process as I go.

Pictures to follow, I hope.

-Neil

Saturday, December 29, 2007

My Best Game to Date Against SmartGo


This is my best game to date against SmartGo.

-Neil

Friday, December 28, 2007

Learning to Play Go

My first experience of Go was in high school in Hawaii. My social studies teacher was of Hawaiian/Japanese ancestry and she kept a Go set in the classroom. It was a really large, heavy floor goban and a set of stones that (if memory serves) were of some really nice intermediate grade material like marble. I seem to remember the goban as having incised lines, but I may be mistaken about that. Unfortunately, Go was not in the revival that it now experiences, so none of us took the opportunity to learn how to play.

Later in life, I decided I wanted to learn to play, but I kept putting it off. I don’t know exactly why I chose this time in my life to start playing Go, but I think it has a lot to do with my interest in the aesthetic of wabi-sabi. Of all the games I’ve encountered, Go seems to best embody the concept of wabi-sabi. This is certainly true of the equipment, but it is also true of the game itself.

About a six weeks ago, I decided to take the plunge. Being a computer programmer (and living in a very small town), it was natural that I should decide to use my computer to help me learn. So I made a promise to myself: “When I can learn to beat Igowin freeware, I’ll buy a nice software learning package.” It took about two weeks to beat Igowin, and – after looking around – I decided to try the 15-day trial of SmartGo. After 15 days with SmartGo, I decided that I liked it enough to pay and register it, so I did. It’s not perfect, but it seems to be a great learning tool and I am enjoying it.

My next goal is to beat SmartGo at least once. When I can do that, I’ve promised myself, I will buy a nice goban and some stones. In honor of my high school memory of Go, I have decided on a Japanese size goban and biconvex stones in the size 33 range.

Also, as part of my pursuit of wabi-sabi, I have decided to make my own goban someday. It may be a while, though, since I want to wait for just the right piece of lumber. I’m not sure running down to the local “wood-r-us” store and picking up lumber by the board-foot will give me the aesthetic I desire, lol. I want the goban to mean something to me as a work of art and a personal process. In art and craft, as in Go, the first and last moyos to be claimed lie within. As the Zen saying goes: "The skilled archer aims not at the target but at himself."

-Neil

Saturday, September 15, 2007

Jung in Boston

Let me state I am generally skeptical of paranormal anecdotes and research. I am trained in the proper use of statistics, and have used statistical testing and modeling in my professional career. I am aware that the most amazing coincidences are to be expected in large populations: no supernatural explanations are required. I have seen more than a few strange coincidences in my own life, none of which bothers me much, but the following experience continues to make me wonder.

I was staying in a hotel in Boston on business. It was my first visit to Boston, and I had been looking forward to doing some sight seeing. One thing I particularly wanted to see was the Old North Church. As a child, "Paul Revere's Ride" was one of my favorite poems. I had been captivated by Longfellow’s imagery of the church at night:

“A moment only he feels the spell,
Of the place and the hour, and the secret dread,
Of the lonely belfry and the dead;”

I talked about the possibility with friends until they were tired of hearing about it. Unfortunately, I did not have as much free time as I had hoped, and I had to cancel my plans for sight seeing.

I had a meeting scheduled with a business associate who lived in Cambridge. We had been planning on meeting for dinner at his house, but it was Halloween night and his wife and children were busy with "trick-or-treat," so he drove into Boston instead. He suggested a North Boston restaurant, and after dinner we decided to walk to a bistro on the next block for coffee and desert. Along the way, we came to a street corner where local youths -- dressed in black and with faces painted white -- were throwing eggs at passing cars (apparently a local Halloween tradition). Afraid that pedestrians might also prove tempting targets, we decided to backtrack and detour.

We made our way to a courtyard that seemed like it might be a shortcut through the middle of the block. It was a warm night for late October and there was a light fog in the air. We walked past a college-age couple who sat on a bench in the courtyard studying. The place had an other-worldly, "stage set" feeling, like a film-noir street scene. At the end of the courtyard was a short flight of steps that ended in an iron rail fence with an alleyway to each side. When we reached the top of the steps, I noticed a plaque on the fence. It was an historical marker that began: "The Old North Church…"

I had somehow, through a series of coincidences, and in a bizarre setting, obtained the object of my quest without trying. And not only that, but as in the poem: at a secret hour and seemingly under a spell. It's difficult to express, but the experience had a numinous, scripted feeling -- as if I were a character in a book or movie. It's this feeling of the whole thing being "set up," I think, that haunts me even more than the event itself.

To commemorate the experience, I bought an "Old North Church" souvenir coffee cup at the airport. It has yet to do anything strange.

-Neil

Friday, September 14, 2007

New Year's Eve, A Diary

From a diary, New Year's Eve 1991-1992

---------------------------------

December 31, 1990* – New Year’s Eve

3:00PM – I should have started this diary a year sooner. 1990* was an eventful year. It opened with the United States about to go to war in Iraq, and has closed with the dissolution of the Soviet Union. The world has changed radically during the last 3 years**, and the next three may prove as tumultuous. I’m not sure what we expected from the downfall of communism in Europe, but what we got was a range of effects, from economic turmoil to bloodshed. The patient was almost too far gone when the resuscitation was applied, and has come back choking spasmodically and still in serious condition. A rise in nationalism has led to fighting in some republics of the former USSR and Yugoslavia. Russia, which was fairly strong of old, and the center of the USSR, is doing surprisingly well. There is some apprehension concerning Yeltsin and whether he may yet prove to be a dictator, or conversely, whether he can control the military if the country revolts in the face of exploding prices.

10:00PM – It’s after midnight in Moscow and the Soviet Union is no more. Here in _____ it’s windy and rainy, after a cold and windy day.

11:15PM – I plan to take a photo of myself at midnight.

11:58PM – Bye bye 1991.

12:02AM – The first minutes of 1992. What will this year bring? I should do like _____ [a friend] and try to wonder where I’ll be this time next year. But for now I’m tired and I will go to bed.

---------------------------------

I did not continue the diary much past this point. I do still have the photo I took of myself.

-Neil

*These errors are in the original, both should read 1991.

**The length of time I had been living in the city in which I was writing this diary.

Tuesday, August 28, 2007

What the Maasai Can Teach Us About Software Engineering

An article on the Maasai started me thinking about why I like my teakettle better than the microwave. The teakettle is no easier to use than the microwave; it probably takes slightly more energy; and of course, boiled water is boiled water.

However, when the teakettle whistles when the water boils, that whistling is an integral act that stems from the nature of the process. Certainly, someone had to design the whistle and attach it to the kettle. But once this was done, the whistle noise occurs automatically and naturally arises naturally as a side-effect the primary process for which the kettle was designed (i.e. boiling water).

A microwave oven, by contrast, beeps because of a series of processes disconnected from function being performed and of an arbitrary character. Even in a very advanced microwave oven that can sense the water boiling, the chain of events resulting in the beep is still arbitrary.

One problem with software is that nearly all of it has this arbitrary nature. And things become more arbitrary and indirect as one moves up from the hardware, until the entire experience for the user is arbitrary and disconnected not just from the physical hardware, but even from the form and semantics of the software that is creating the user experience.

So how and where to recapture the lost integrity of software?

Unfortunately, I don’t know of any way of doing it at the user level. At the programming level, however, it is still quite possible. The trick is to stop seeing oneself as a designer and to see oneself as an architect in the true sense of the word. In computer science, we have mistaken the meaning of the word architect; it doesn’t mean “really good high level designer.” Architect is from the words “arch tech,” and is Greek for “master builder.” To be an architect, one must build things, not design them.

This is doubly confounded because we have also mistaken the roles of design vs. building. Nothing of value is ever designed, it must be built. Sometimes good things appear to be designed, but only because some builders get really good at doing the building in their heads or on paper.* But trying to design something without going through a building process nearly always results in something that is at best sub-optimal and usually creates more problems than it solves.

Our elevation of the design myth over building also causes us to mistake the meaning of the word building. Building is not a construction process. (Construction is simply one of the operations that enable building.) Building is a growth process, and growth processes are non-linear, non-monotonic (i.e. they involve deletion as well as addition**), and are to a certain extent unpredictable.


As Christopher Alexander said:

“It creates order, not by forcing it, nor by imposing it on the world (through plans or drawings or components): but because it is a process which draws order from its surroundings – it allows it to come together”

“But if course, by this means far more order can come into being, than could possibly come into being through an invented act.”

“It is vastly more complex than any other kind of order. It cannot be created by decision. It cannot be designed. It cannot be predicted by a plan.”

And this is exactly the kind of order that we have forgotten but that Maasai herder and a man living in the house made of trees he cut himself still remember. For them, things “just work.” Literally: “stuff just works,” that is, the stuff itself works by dint of its very nature, not because of arbitrary bits glued on during artificial processes. And that’s what we need to find again in the 21st century, because the Maasai and tree-man are not living in the past, they are also living in the 21st century, and it makes no sense to call them primitive or unsophisticated when it is we who have forgotten things.

-Neil

*In fact, the idea that things even can be designed is probably a myth that arose from people not understanding how skilled builders work.

** We forget that to put up a building, one often has to put up a scaffolding. We do this because we think of the building (a noun) is the result of building (a verb). But we forget that this semantic separation is artificial and is a trick of our minds. That dismantled scaffolding is as surely a part of the finished building as child’s experiences are part of the adult. Both were left behind in form, yet not in effect, since in both cases the finished product, in its very nature, bears the unmistakable marks of the construction process.

Monday, August 27, 2007

Book Review: A Bright and Shining Lie

A Bright and Shining Lie – John Paul Vann and America in Vietnam
by Neil Sheehan.

I begin this blog with my thoughts about a work by another Neil.*

Exciting and interesting on the whole, with a few tedious sections. More balanced than one would believe based on the title, and than most reviews indicate.

It analyzes the American mistakes in Vietnam unblinkingly, yet doesn't come across as overly cynical. Where the players involved – political or military – had good intentions, it is clear about separating the laudable aspects of those intentions from the situations which made them unrealistic and unattainable.

Sheehan’s views are slanted to the left, and there is a bit of “novelization” involved as he seeks to show how the war is mirrored in the life of one of its protagonists: John Paul Vann. But it’s not too bad, given the politically charged nature of the topic, any book on Vietnam is bound to step on a few toes.** He depicts many of the military and political people involved in the conflict as being seriously concerned for the Vietnamese people as a whole, yet in many cases still blinded by the kind of naïve, benignly-intentioned racism common in the post-colonial era.

The book also contains an excellent analysis of how WWII and the French/Vietnamese conflict transitioned into the American/Vietnamese conflict. This shows how the actual problems that made the American involvement in Vietnam an unfortunate quagmire are in many ways a result of things that happened before the war (as popularly defined) even began. By the time the conflict began, it was being carried in part by the momentum of past events and its own inertia rather than the needs of the day.

In a way, two overlapping wars were being fought by three protagonists: the South Vietnamese vs. the North Vietnamese as a continuation of the French/Vietnamese independence conflict***, and the Americans vs. the North Vietnamese as a proxy war between the West and Communism.

One thing many Americans failed to realize is that the Vietnamese peasantry identified with the independence conflict, and really didn’t care much about the Cold War. The Southern leaders were, in this view, seen as representing earlier Mandarins who became corrupted by colonialism, while the Northern leaders were seen as representing the earlier Mandarins who had stayed true to Vietnamese nationalism. In some cases, in fact, this was literally true.

Thus, though peasants would pragmatically support whoever was in their area at the time, they were always somewhat more sympathetic to the North Vietnamese. So, in the South, it became very much an urban vs. rural war – the boundary of every city and base was a war front.

The fact that both the U.S. and the South Vietnamese tended to conduct the war as a war of sortie from cities and bases into the countryside only intensified the urban/peasant polarization and made it continue to resemble the earlier war of independence. The main protagonist, John Paul Vann, for all his other flaws, realized this and saw that the war could not be won as long as this divide remained. He saw many of the ways in which the Vietnam conflict was different from earlier wars and even earlier guerrilla conflicts, and yet missed many other important things.

In summary: the book is not without its flaws and biases, but if studied in conjunction with other sources, it is a valuable contribution to the understanding of the American experience in Vietnam.

And that's what I learned from A Bright and Shining Lie.

-Neil

*Other than our first names, Neil Sheehan and I are unrelated.

**I think Sheehan is a bit overly critical of the role of the Catholic Church in the time leading up to the conflict. Not that there is no blame there, but I’m sure there are balancing, positive aspects that are ignored. Also, though he mentions the atrocities and illegalities committed by all sides, he seems to gloss over them somewhat more when it comes to discussing North Vietnamese actions.

***Because of the post-WWII history of Vietnam, North Vietnam was very much identified with independence side of that conflict, and South Vietnam with the French colonial. That this identification also split along West/Communist lines was largely the result of an historical accident involving a single individual: Ho Chi Minh.