Saturday, February 18, 2017

Get pip of python install to work behind proxy

OK, here is a simple task, use pip to install packages...
Other than if you are behind a proxy that uses https.

What happens is even though you issue the proper command line like below:
# pip install virtualenv --proxy http://user:pass@proxy:8080
you still get an error message like this:
.....
CertificateError: hostname 'proxy' doesn't match either of 'www.python.org', 'docs.python.org', 'bugs.python.org'.....

You tried to use HTTP_PROXY environment variable as well, still this Cert error??!!

Hmmmm.... what is going on?

It is because the ssl.py actually checks the hostname of the certificate subjectAltName, which would be python.org hostnames.  However, the 'hostname' it connected to is actually your proxy server, not the python servers.

OK, here is what you need to do - update the /usr/lib64/python2.7/ssl.py file
# vi /usr/lib64/pyton2.7/ssl.py

look for "if len(dnsnames) > 1:"

immediately before this line, you should insert below block:
value = 'proxy'
if _dnsname_match(value, hostname):
    return
dnsnames.append(value)

This is tested on python 2.7 behind a firewall.

Not all proxy servers will cause this problem, but if it does, here is the quick fix.

Friday, December 04, 2015

Common Sense solution to Monty Hall question - a "hotly debated" Math (Probability) topic

Descret statistics and experiment design.

Monty Hall problem keeps popping back up recently, most of them with a catchy headline “even geniuses/ Ph.D. got it wrong”.

Well since it was originated from one of the Berkeley profs, where I study now.   I might jump in to support the intellectual exercise. 

Some people made their names by sounding as if she is smarter than the smartest Mathematicians by claiming "they were all wrong" (i.e. here: The Time Everyone “Corrected” the World's Smartest Woman and here http://marilynvossavant.com/game-show-problem/)

As much as a catchy title, the real math is much more interesting... The truth is to claim to be "the" smartest, it needs a lot of assumptions.
Here is why:  (according to Wikipedia) 
"The behavior of the host is key to the 2/3 solution. Ambiguities in the "Parade" version do not explicitly define the protocol of the host. However, Marilyn vos Savant's solution (vos Savant 1990a) printed alongside Whitaker's question implies "

So she implied something in her solution that was not explicitly mentioned in the text.  While those implied assumptions are given support, her answer would be correct.  However, it is also fair for people to argue, if those assumptions does not hold, other answers might be correct.  

Ok, according to Wikipedia, Marilyn later corrected (clarified) her question in a followup post.  However, that also means "most people" who disagreed with her answer, weren't wrong, they were merely given a vague question that only got clarified after they had provided their solutions.  Her "implied" original posted question which is still available to see on her website: 
Suppose you’re on a game show, and you’re given the choice of three doors. Behind one door is a car, behind the others, goats. You pick a door, say #1, and the host, who knows what’s behind the doors, opens another door, say #3, which has a goat. He says to you, "Do you want to pick door #2?" Is it to your advantage to switch your choice of doors?
Craig F. Whitaker
Columbia, Maryland


For this particular question text, take it literally without assuming information out of the text, the answer should be "I don't know" like most Mathematicians PhD argued.  (see host behavior section of the Wikipedia page)

But, Marilyn's solution would be correct if she spells out "The HOST will NEVER open a door with the car"!!! 

(in other words, there WAS NO chance that the door the host opens first would be the car.  This is the key, you did NOT lose a 1/3 chance because the HOST will never give it to you, which means this had not been a 1 chose 3 game ever. 

What does that mean?  Why most Mathematicians got it "wrong"? 
Well, there is a thing called Mathematical model ... basically, that's the process of expressing a question in mathematical equations.  Here is what went wrong.  Marilyn built a mathematical model that "implied" the player KNOWS the host will always open a door with Goat.  The other PhDs, did not build this into their model. 

That's why Paul Erdös kept asking a "Common Sense" solution -- he was able to figure it out, but unfortunately for whatever reason Andrew Vazsonyi refused to understand it, instead he decided to stick with his decision tree approach which means he stuck with his mathematical model refusing to understand if his model correctly reflects the actual question. (https://web.archive.org/web/20140413131827/http://www.decisionsciences.org/DecisionLine/Vol30/30_1/vazs30_1.pdf)

So those "smarter than smartest" people have been implying information that was not explicitly provided in the original text, and they even built computer simulations with this assumption built in!

Since Paul Erdös' commonsense solution was lost by Andrew Vazsonyi, here is my common sense explanation and why I call Marilyn's posts intellectual scam. 

Here is the game: 
There are 3 doors, one of them has a car behind it.  you didn't know which one (BUT the host KNOWS), so you randomly chose one, you THINK you have 1/3 chance of winning the car (ACTUALLY YOU DON'T, keep reading!).

Now, the host will open a door, BUT he would *NEVER* open the door you chose!!!   (That's why you never had a 1/3 chance of winning, he will NOT doors randomly, instead, he is using his knowledge to play tricks on you, hence you now have a chance to play it back at him, but this was not explicitly spelled out to begin with).

There SHOULD be the following possibilities: 
1) you selected the door with the car: 1/3 of the chance (but since the HOST won't open it, even though you have 1/3 chance of choosing the door with the car, you DON"T have 1/3 chance winning)

Here, knowing you are not going to win right away (he will always pick a goat door), you LIMITED
the host's choice to 2 doors, (because he will NOT open door 1 whatsoever.)

So remember, if you were correct, the HOST has 2 options to play tricks on you.


2) you selected 1 of the 2 doors with goats, you have 1/3 of the chance to be here as well, however, because your choice was a goat, AND the host will NEVER open the door with the car, you LIMITED the host's choice to ONLY ONE.

Yes, remember, here, the host HAS to open the goat that you did not select!!!  He has NO other options, because he would NEVER open the car in the first round.

3) you selected the other door with goats, this is the same as 2), which is also 1/3 chance.

OK, you have 1/3 chance choosing a door with the car, and allow the host play tricks on you with 2 doors.
HOWEVER, you have 2/3 of the chance choosing a door with a goat, which FORCE the host to open the other goat door for you (remember, in this situation, the host actually has no options to play tricks other than open the OTHER door with goat).

Because in the cases (#2 & #3 you selected a goat door) you forced the host to open the other goat door, you know in this 2/3 of the chances, switching is a sure-fire of the car. 

It's a little bit complicated for #1, because if you already chose the door with the car if you switch, you would have lost the car, so in this 1/3 of the cases, you should stay. 
BUT since you don't know which you chose the first round, you really could not make the adjustment like we did here, BUT you know in 2/3 of the cases (where your chose was a goat), switch is a sure-fire of the car, but in the 1/3 of the cases (your selection was the car), staying would be better...   Since you don't know, you might just switch, because it has higher chance 2/3 .vs. 1/3


OK, the interesting thing comes here: 
According to this text, the host opens another door shows you a goat -- was that by chance?  The text didn't say!  It did mention the host is aware of where the car was, however, it did not specify that the host WILL NEVER open the door with the car!!! 
IF he opened the door to a goat by chance, then you just missed your 1/3 chance of winning, and were given a second chance. 

Then why Marilyn could prove she was correct? Well, she assumed the host will never open the car first round.   Under that assumption, her solution (switch) is the best solution for the math model she built  (here in the explanation of her solution she added "...and the host always opens a loser. ")

This is the key to her "scam", this "always" was not given in the original question (mathematically this means she is adding additional conditions and restrictions and changed the actual model).



OK, now here is the commonsense description of a properly construct the question that Marilyn solved: 
a) a car is randomly placed behind 1 of 3 doors, the other two hide a goat each;
b) you have zero information to begin with other than a);
c) you can choose one of the 3 doors as round 1;
d) the host will then have to open a door with a goat (he is forced by rule to do so, and he is not allowed to open a door with a car behind it, he cannot choose to skip opening a door either, and you KNOW this), this is round 2;
e) you are then asked "switch or stay", this is round 3.


And here is the Commonsense Solution: 
1) for round 1 above, do you agree you have 1/3 chance choosing right, 2/3 wrong? (of course, no one denies that, right?)
2) now it comes the interesting part, because of "host always open a loser", IF your first selection was a loser, then the host is FORCED to open the other loser (so by now both you and the host are holding on to a loser door, but there are only 2 losers, so the leftover must be the winner)  And it does not matter which loser you first selected, because by the rule, the host will have to eliminate the other loser anyways, so either one of the 2 losers you pick, the host had to remove the other loser and leave you with the winner. 
3) so now if I ask you "how likely was your first round choice wrong?" the answer, of course, is 2/3, but if you were wrong, then the only door left has not been chosen by you nor the host would be the car, so in this 2/3 of the cases, you should switch. 
4) because of the host has the option to eliminate one of two losers if you chose correctly during round 1, but this is only 1/3 of the chances which you should stay. 


You ask why? I still don't get it?  like mentioned before, the host was forced by rule to eliminate one of the 3 possibilities (the 1/3 chance where he opens the door showing the car), and if you know this rule, then here are the "possibilities"  (let's say you select door #1)
1) car is behind door #1, the host can freely open either #2 or #3, he is not helping you - you have 1/3 chance
2) car is behind door #2, the host is FORCED by rule to open #3, so if you switch, you get the #2, winner - you have 1/3 chance;
3) car is behind door #3, the host is FORCED by rule to open #2, so if you switch, you get #3, winner - you have 1/3 chance;

See, if you stay, your only hope is it turns out to be 1), 1/3 chance; however, if it was either 2) or 3) above, the host is forced to point you to the winner by open the other loser.   So you have two times of these 1/3 chance winning if you switch. 

This ONLY works like Marilyn solved IF the host is forced by rule to always open a loser, and you KNOW this rule.  Her mathematical model assumed this very specifically, so was her computer simulations -- because the computer model was programmed to "always open a loser", the simulation result supported her claim.

Why it's a scam?  Because the original question did NOT specify this "host has to open a door with goat" which in essence change the round 3 to a different question, and most Mathematicians and PhD actually used common sense to think about it like this (without assuming the player knows a rule forces host to always open a goat):  If you do NOT KNOW the host opened goat door was the rule of the game (instead, you think you were lucky that he did not open a door with car prove you were wrong at round 2), then there is no reason to switch in round 3"

 There have been comprehensive discussion of host behavior and why he opens a door with goat i.e. some variation says host only open a door to goat if he knew you were correct, otherwise he would skip round 2, and ask you if you want to switch without opening a door in which case switch is a sure losing strategy.  Of course Marilyn's math model was not built this way.


Thursday, June 10, 2010

The missing "I" in IT (1)

Various articles have pointed out the diminished importance of CIO in organizations. It should not have been a surprise. 


Since the tech bubble in 2000, people should have foreseen this coming, why?
I call it the missing "I" in IT - or as I sometimes say - "IT is about I, before T", or in more avid language, the dog should wag the tail, not the other way around.

Apparently, most CIO focused too much on "T", instead of "I", hence deprecated their own value.  (The tail waged the dog)

OK, let's do a quick quiz, in less than 10 words, can you please explain what is "IT"?   "Computer and related stuff" is the most common answer, next ? "Computer and related stuff used in doing business" - at least business is mentioned.

However, both answers are in the T before I camp, not the I camp.  Computers and related stuff are the technology part of IT that "can be used" in processing information.  However, computers can be used to fry an egg in addition to process data.  (for those of you who are too young to know, when Intel first released its 80486 CPU, CPU fans were not around yet, and the CPU generated so much hit, there was a video posted people fried an egg on it.)

IT is about information.   One of the oldest and still widely used IT innovation is pen and paper.  Seriously, it was high tech at the time it was invented, and still very important tool we use this very day - although we don't call it high-tech anymore.

Now, what a low tech pen/paper has anything to do with our discussion of IT?  It was an invention to record information.  Think about it, what pens do to paper is not to make the paper more valuable in terms of weight, or any other physical measure --  instead, it is making marks that is now called information.

I always advocate people think about computers as pen/paper, because
IT should be about using technology to collect and process information.  
Not about how to sharpen your pencil more reliably.

If a CIO's focus is all "I make sure my computers run!" the CIO adds barely any value to a company.  Think about it, a company has all the pencils that are sharp all the time, but barely any one literate, when can these pencils do to help the business?  Unfortunately sharpening pencils that barely used happens all too often, and it depreciate the value of CIO in the business world.

Running the most reliable gadget that does little to enhance information processing within an Organization is a big trap for many CIOs who lost their prestiges status - they are just a glorified pencil sharpener.


IT should be about efficiency, not technology.   Management has always been about information, collect information regarding how things work, process information to understand how things work, analyze information to facilitate decision making, send out information to execute, present information to manage public relationship, etc.

Information is control - the better, more comprehensive information flows within an organization, the better control the leadership has.  Of course technology helps here.  That is actually the core value of IT - helps information flow with in an organization more effectively, efficiently with all the available technologies.

A classic example is inventory level.  Let's say a warehouse retailer, carries 15 days worth of inventory because it does inventory count weekly, it takes 3 days for the vendor to ship the products, and they need 5 days buffer - because they are not sure how fast each item sells.

Now with an inventory management solution, they should no longer need to count inventory every week - each in and out are traced, so they can save 6 days out of 7 days counting cycle.  So this inventory system can save 6 out of 15 days inventory for this retailer.  If this retailer is large, cutting 40% inventory can mean tens of millions.
Now, how about the POS system that tracks all items sold?  That will make 5 day buffer probably no more than 3 days (the order and shipping time).
A forecast system that forecast sales, and inventory level that enables pre-order of short of stock merchant will cut another 5 days off the cycle -- because no buffer needed any more, and orders can be placed early to accommodate shipping time.

So it can maintain 1 day inventory - now this is what now call JIT.  It was revolutionary at the time, and was enabled/enhanced by information technologies.  The companies invented this architecture knows what is IT.  How many CIOs have done anything remotely similar to such innovation?

Next we will talk about
  • out sourcing