Mish's Global Economic Trend Analysis

---

• China Manufacturing PMI Plunges to 32-Month Low of 47.7; Reflections on Stocks Rallying on "Bad News"
• China to Protect Iran Even if Result Starts World War III; What's the Best Way to Deal with Iran?
• Maximum Intervention Moves Into Overdrive; Foreign Banks can Fund themselves Cheaper in US Dollars than US Banks; Discount Rate Cut Coming Up?
• China Cuts Bank Reserve Ratios by .5 Percentage Points; Central Banks Cut Rates on Dollar Swap Lines; German 1-Year Bond Yield Negative First Time Ever; Futures Soar
• German Finance Minister says "Big Bazooka" Not Ready, Would Not Stem Crisis, Even IF it Was; Plans Too “Intricate and Complex” for Investors to Understand.

China Manufacturing PMI Plunges to 32-Month Low of 47.7; Reflections on Stocks Rallying on "Bad News" Posted: 30 Nov 2011 10:56 PM PST Equity markets soared on central bank manipulations and various rumors the past few days. However, neither rumors nor trivial actions (which is all that happened) can save the global economy. Yesterday stocks rallied on news China Cuts Bank Reserve Ratios by .5 Percentage Points and Central Banks Cut Rates on Dollar Swap Lines. However, the reason Chinese central bank reacted is hugely deteriorating conditions in China. The reason the Fed reacted is hugely deteriorating conditions in Europe. Equities have rallied on reported "good news". However the first irony is the global economic picture outside the US is horrendous. The second irony is bottoms are formed on bad news (and tops on good news), but central banks intervention is really bad news widely recognized as good news. With that in mind, please consider the HSBC China Manufacturing PMI for November 2011. November data showed Chinese manufacturing sector operating conditions deteriorating at the sharpest rate since March 2009. Behind the renewed contraction of the sector were marked reductions in both production and incoming new business. The latest survey findings also showed a marked easing in price pressures, with average input costs falling for the first time in 16 months. In response, manufacturers reduced their output charges at a marked rate. After adjusting for seasonal variation, the HSBC Purchasing Managers' Index™ (PMI™) – a composite indicator designed to give a single-figure snapshot of operating conditions in the manufacturing economy – dropped from 51.0 to a 32-month low of 47.7 in November, signalling a solid deterioration in manufacturing sector performance. Additionally, the month-on-month decline in the index was the largest in three years. Manufacturing production in China fell for the first time in four months during November, with the rate of decline the fastest since March 2009. Panelists generally attributed reduced output to falling new business. The latest decline in new orders was marked, and the steepest in 32 months. Moreover, the month-on-month decline in the respective index was among the greatest since data collection began in April 2004. China Manufacturing PMI Stocks ignoring bad news is normally a very good sign. Stocks rallying on government intervention as bad news is presented as good is a different story indeed. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

China to Protect Iran Even if Result Starts World War III; What's the Best Way to Deal with Iran? Posted: 30 Nov 2011 06:31 PM PST Does the US have the right to defend itself? If so why doesn't any nation have the right to defend itself? What is the best way for the US to deal with Iran? Here is a video in Chinese, with English subtitles, in which China says it will defend Iran. Link if video does not play: "China will not hesitate to protect Iran even with a third World War" Here are a few panels about 2 minutes 15 seconds into the video in which China states an intent to protect Iran, if Iran is attacked, even if it means World War III. I support the position (a few moments later and shown in the screen shot below) that suggests the Iranian people have little trust in their leaders and the best way to deal with Iran is to let the people rise up against the government as happened in Egypt and Libya. With the US threatening Iran at every turn, and with the needless war in which the US destroyed Iraq killing or ruining the lives of hundreds-of-thousands of Iraqis, it is no wonder Iran wants to protect itself. Any country would want to do the same. The US has no business instigating another war, yet that is exactly what economic sanctions are. The downright scary policies of Mitt Romney and Newt Gingrich go even further, and would have the US marching off to World War III before we know it. The best way for the US to deal with Iran is to support the Iranian people (not the leaders). Nearly all the private citizens of Iran would have no grudge against the US if we would simply stop our policies of aggression in the region. We do not need another war and certainly cannot afford one. Ron Paul offers the best hope of stopping yet another disastrous, and needless march to war. In case you missed it, please consider President Obama and Mitt Romney are Nearly One and the Same! Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

Maximum Intervention Moves Into Overdrive; Foreign Banks can Fund themselves Cheaper in US Dollars than US Banks; Discount Rate Cut Coming Up? Posted: 30 Nov 2011 08:54 AM PST Steen Jakobsen, chief economist for Saxo Bank offers his take on the liquidity moves by global central bankers. Please consider Steen's Chronicle, Maximum Intervention Moves Into Overdrive Our theme for Q4 was 'Maximum Intervention' and today was a new high for this exact concept. The day after the European Union Finance Ministers (ECO-FIN) meeting (which once again failed to produce any progress on the EU debt crisis) the Chinese cut the RRR-ratio - the minimum reserves each commercial bank must hold of customer deposits and notes - from 21.5 percent to 21.0 percent. (The RRR started the year in 18.5 percent and this is the first cut since 2008. Back in 2006 the RRR ratio was just below 8.0 percent for a number of years.) This is an indication that China's help to the growing outlook of a 'Perfect Storm' will be monetary easing despite relatively stubborn inflation numbers. Coup-de-grace Then in coup-de-grace style the Federal Reserve and five other major central banks cut the dollar funding rate for overnight swaps by 50 basis points, down from 100 basis points, and at the same time made this programme run through to February 2013. This immediately raises the hope for further cuts in policy rates in the US and Europe. Right now, foreign banks can fund themselves cheaper in US Dollars than US banks. This will almost certainly mean the discount rate will be cut by 25 bps and before the weekend. Mounting pressure on Monti The market loves liquidity and this action shows the true determination of policymakers to address the growing funding crisis, but its ultimate success will depend on progress in the EU debt crisis, and whether this will again merely be a stand-alone action of throwing liquidity at a problem which remains one of solvency. In other words, the lack of structural changes in Europe – are the same both before and after this coordinated intervention. Alas, technocrat Monti remains more important to the future of this risk-on move than the move itself. However, it should not be ignored that the market is looking for excuses to take the S&P 500 index higher, and there is no denial that the underlying economic data from the US has continuously surprised to the upside over the past month. Fundamentals are improving in the US, and the ADM report this morning gave indications that Non-Farm Pay-Rolls data on Friday could yet be another positive news story. 'Monster Santa' rally Earlier this week I described this week as likely to be one of consolidation and potential for a rest at 1215/1220 – we have now clearly overshot this on the upside and the target 1240/50 might well be in full view. The market also likes the "seasonal play" of buying into year-end, and I am getting plenty of ammunition from tech-based analysts that the price action reminds them of September/October 2010 – the post Jackson Hole rally - making this the start of a 'monster Santa' rally. I will remain sidelined with a negative bias, if only because, my generic models are short, and from a tactical point of view, I feel that to get a real solution to solvency, unlike liquidity, we need to see stock markets in mini-crash mode before politicians and policymakers truly understand the necessity to make a long-term commitment to austerity and growth. The signal today was: We are prepared to buy more time, and feel confident enough to float the market with cheap liquidity and an indication of further easing to come. The market will love that, but investors should also remember to stay sober when drinking from the cool-aid name: cheap money. Go Santa, Safe travels! Discount Rate Discussion The Federal Reserve website has this discussion of the Discount Rate. The discount rate is the interest rate charged to commercial banks and other depository institutions on loans they receive from their regional Federal Reserve Bank's lending facility--the discount window. The Federal Reserve Banks offer three discount window programs to depository institutions: primary credit, secondary credit, and seasonal credit, each with its own interest rate. All discount window loans are fully secured. Under the primary credit program, loans are extended for a very short term (usually overnight) to depository institutions in generally sound financial condition. Depository institutions that are not eligible for primary credit may apply for secondary credit to meet short-term liquidity needs or to resolve severe financial difficulties. Seasonal credit is extended to relatively small depository institutions that have recurring intra-year fluctuations in funding needs, such as banks in agricultural or seasonal resort communities. The discount rate charged for primary credit (the primary credit rate) is set above the usual level of short-term market interest rates. (Because primary credit is the Federal Reserve's main discount window program, the Federal Reserve at times uses the term "discount rate" to mean the primary credit rate.) The discount rate on secondary credit is above the rate on primary credit. The discount rate for seasonal credit is an average of selected market rates. Discount rates are established by each Reserve Bank's board of directors, subject to the review and determination of the Board of Governors of the Federal Reserve System. The discount rates for the three lending programs are the same across all Reserve Banks except on days around a change in the rate. Further information on the discount window, including interest rates, is available from the Federal Reserve System's discount window web site. Discount Window Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

China Cuts Bank Reserve Ratios by .5 Percentage Points; Central Banks Cut Rates on Dollar Swap Lines; German 1-Year Bond Yield Negative First Time Ever; Futures Soar Posted: 30 Nov 2011 07:09 AM PST Equity futures sharply reversed an overnight pullback on a pair of central bank actions, one in China, the other an agreement between the US and Europe. China Cuts Bank Reserve Ratios by .5 Percentage Points The Wall Street Journal reports China Cuts Reserve-Requirement Ratio The People's Bank of China, China's central bank, said Wednesday it will cut the reserve-requirement ratio for banks by half of a percentage point, the first such cut since December 2008. The cut essentially frees up banks to lend additional money. The cut late Wednesday in Beijing cheered European markets, with the benchmark Stoxx Europe 600 index up 0.8% midday, while London's FTSE was up 0.8%. "The data for the last few weeks has been bad," said Mark Williams, China economist at Capital Economics. "There's zero growth in property starts, electricity output growth has slowed, the export numbers for November will be awful and they may have had a sneak preview of that. All of these things could have triggered a shift in policy." Wednesday's move will take the reserve-requirement rate to 21% for major banks. It will free up around 390 billion yuan (about $61 billion) in funds for the banks to lend, according to calculations by The Wall Street Journal based on data for bank deposits in October. The cut in reserve ratio "is a clear signal that Beijing has decided that the balance of risks now lies with growth, rather than inflation," said Stephen Green, regional head of research in Greater China for Standard Chartered, in a note following the PBOC's move. Mr. Green predicts that China will reduce the reserve ratio again in January due to a potential liquidity crunch coming up before Chinese New Year. The PBOC has raised the reserve requirement ratio six times so far this year, and has raised benchmark lending and deposit rates five times since October last year to combat stubbornly high inflation. The previous reserve ratio increase took effect June 20, and the last interest rate hike was effective July 7. There will likely be more such reserve ratio cuts, with one more cut of 0.5 percentage point coming as soon as the beginning of next year, said Yao Wei, China economist with Société Générale, adding that she doesn't expect any interest rate cut in the next six months. Central Banks Cut Rates on Dollar Swap Lines Bloomberg reports European Stocks Rally After Central Banks Cut Rates on Dollar Swap Lines European stocks rallied for their longest stretch of gains in seven weeks as the Federal Reserve and five other central banks lowered the cost of dollar funding and China cut its reserve ratio for banks. The Fed, Bank of Canada, Bank of England, Bank of Japan, European Central Bank and Swiss National Bank agreed to reduce the interest rate on dollar liquidity swap lines by 50 basis points and extend their authorization through Feb. 1, 2013. Finance ministers of the 27-nation European Union are meeting in Brussels today to seek agreement on how to temporarily guarantee banks' bond issuance in order to improve funding conditions for lending. EU leaders agreed last month to provide the guarantees to restore investor confidence in banks. German 1-Year Bond Yield Negative First Time Ever Investment Week reports German 1-year bunds move to negative yield for first time ever The yield on 1-year German bunds turned negative today for the first time ever, according to Bloomberg data, as the European Central Bank looks set to ramp up measures to fight the debt crisis. The yield on the 1-year note fell 13 basis points to -0.05% by midday. This is the first time it has seen a negative yield since Bloomberg began compiling data on the asset class in 1995. Yields on the 6-month bunds, known as Bubills, turned negative last week, dropping to -0.05% on Friday. It was the first time 6-month bunds have offered a negative yield since the creation of the euro. S&P Equity Futures are up another 3 Percent, Bond Market Yawns Global equities are sharply higher with this global coordinated action. S&P 500 futures are up another 3 percent and will gap higher. Meanwhile Spanish 10-year bonds rallied (yields fell) a mere 7 basis points to 6.32%, Spanish 2-year bonds rallied a mere 8 basis points to 5.51%, Italian 10-year bonds rallied 10 basis points to 7.13%, and Italian 10-year bonds rallied 9 basis points to 7.00%. Whatever the equity markets see, the bond market doesn't. A flight to safety of German bonds is back on, that China needs to cut reserve requirements is a huge sign of weakness (and no it will not stop a hard Chinese landing). Also bear in mind that on September 15, there was coordinated swap-line action that did nothing. Bloomberg reports ECB Coordinated Policy Action Is 'Big Deal,' Blanchflower Says September 15, 2011 11:35 AM EDT The Frankfurt-based ECB said today that it will coordinate with the Federal Reserve and other central banks to conduct three dollar liquidity-providing operations with a maturity of approximately three months. The loans are in addition to the bank's regular seven-day dollar offerings and will be conducted as fixed-rate tenders with full allotment, the bank said. It will offer the loans on Oct. 12, Nov. 9 and Dec. 7. "The dollar funding situation has caused headaches for some banks," said David Schnautz, a fixed-income strategist at Commerzbank AG based in London. "The ECB's measures help ease those problems. It will be interesting to see if there is more to come." Basis swaps allow banks to borrow in one currency, while simultaneously lending in another. The ECB's measure is a "really big deal," according to Dartmouth College Professor David Blanchflower. "The fact that these central banks have acted together and said we'll backstop banks is really big news," Blanchflower, a former member of the Bank of England's Monetary Policy Committee, said today at the Bloomberg Markets 50 Summit in New York. Here is an interesting chart on ZeroHedge that shows what happened the last time there was coordinated swap-line action. What's Changed? Nothing much that I can see. China cut the reserve-requirement rate to 21% from 21.5% and the Fed and ECB renewed swap lines at a slightly lower rate. Yields on Italian bonds are still at or above 7%, and nothing has been done to solve any long-term structural issues. Nonetheless it's party time for equities, crude, and metals, particularly gold and copper. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

German Finance Minister says "Big Bazooka" Not Ready, Would Not Stem Crisis, Even IF it Was; Plans Too “Intricate and Complex” for Investors to Understand. Posted: 30 Nov 2011 12:39 AM PST In a huge non-surprise to the bond markets (but not to bullish equity buffoons), Wolfgang Schauble admits euro bail-out fund won't halt crisis Europe's "big bazooka" bail-out fund is not ready and won't stem the debt crisis that on Tuesday pounded Italy and the European Central Bank (ECB), admitted Wolfgang Schauble, Germany's finance minister. Mr Schauble said eurozone finance ministers, who are meeting in Brussels, could not agree on the terms of the European Financial Stability Facility (EFSF). He told Germany's Handelsblatt that although Europe needed a fund "capable of action", plans for the EFSF were too "intricate and complex" for investors to understand. The finance ministers, who were meeting ahead of a full Ecofin summit today, acknowledged the €440bn (£376bn) fund would not win support to leverage it up to €1 trillion. Its capacity would be between €500bn and €700bn instead – a total that is unlikely to be big enough to rescue Spain and Italy. However, the ministers concurred that the €8bn of international aid to Greece should be disbursed before Athens runs out of cash in two weeks. Evangelos Venizelos, Greece's finance minister, said: "In Greece we have all the necessary conditions in order to go ahead with the next disbursement." Necessary Conditions Met?! The only way "necessary conditions" can possibly have been met is if "necessary conditions" have changed. Germany, the Netherlands, and the IMF have all insisted that all Greek coalition leaders sign off on agreement to IMF and EU demands. However, the leader of the Greek New Democracy party still refuses to sign as noted on November 22 in Showdown in Greece; EU Gives Deadline on Signatures; Samaras Won't Sign, Sends Letter Instead, Seeks Policy Changes. Either the EU has blinked or I missed a "signing party".  Regardless, Greece is going to default anyway, signing party or not. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

You are subscribed to email updates from Mish's Global Economic Trend Analysis To stop receiving these emails, you may unsubscribe now.	Email delivery powered by Google
Google Inc., 20 West Kinzie, Chicago IL USA 60610

Fast Blog Finder v3.0 Free Edition registration

Hi and thank you for registering your copy
of Fast Blog Finder v3.0 Free Edition.

Please, keep this mail safe because you might
need to check the information later on

Here's your personal key [we suggest to keep it private!]

1322759347-7548-10510

Copy-paste it on the program activation screen and click Activate.

Use your personal key to unlock Fast Blog Finder v3.0 Free Edition
on two computers for your personal use.

In the next days we'll be sending you quick-tips
to help you get the most out of Fast Blog Finder.
Stay tuned...

Julia Gulevich
G-Lock Software
http://www.glocksoft.com

Damn Cool Pics

---

• A Social Media Revolution - China's Answer to Social Networking [Infographic]
• Mall Santa Musical by Improv Everywhere
• UFO Treehouse Hotel In Sweden

A Social Media Revolution - China's Answer to Social Networking [Infographic] Posted: 30 Nov 2011 11:05 AM PST Almost 500 million Chinese citizens are online and a quarter of all social network users in the world are Chinese. However, because government policies in the country block many western social networking sites such as Facebook and Twitter, a vibrant domestic ecosystem of similar online platforms has emerged. Here we take a look at the different players in the space and background. Click on Image to Enlarge. Source: gplus

Mall Santa Musical by Improv Everywhere Posted: 29 Nov 2011 09:20 PM PST Mall Santa Musical, the latest mission by Improv Everywhere, features a musical that breaks out in a New Jersey shopping mall when "a man is suddenly compelled to express his desire to sit on Santa's lap via song". This is IE's fifth installment in their series of Spontaneous Musical projects. Via: laughingsquid

UFO Treehouse Hotel In Sweden Posted: 29 Nov 2011 04:29 PM PST Located near the Swedish village of Harads, this UFO Treehotel is one of a kind. So far there are five unique tree rooms available and twenty-four planned to be built. The five tree rooms that are currently available for booking are the Cabin, the Mirrorcube, the Bird's Nest, the Blue Cone and the UFO. It ain't cheap at $600 a night, but isolation, a luxury treehouse, and UFOs should never come cheap.

You are subscribed to email updates from Damn Cool Pictures To stop receiving these emails, you may unsubscribe now.	Email delivery powered by Google
Google Inc., 20 West Kinzie, Chicago IL USA 60610

Just How Smart Are Search Robots?

---

Just How Smart Are Search Robots? Posted: 29 Nov 2011 12:51 PM PST Posted by iPullRank Matt Cutts announced at Pubcon that Googlebot is “getting smarter.” He also announced that Googlebot can crawl AJAX to retrieve Facebook comments coincidentally only hours after I unveiled Joshua Giardino's research that suggested Googlebot is actually a headless browser based off the Chromium codebase at SearchLove New York. I'm going to challenge Matt Cutts' statements, Googlebot hasn't just recently gotten smarter, it actually hasn’t been a text-based crawler for some time now; nor has BingBot or Slurp for that matter. There is evidence that Search Robots are headless web browsers and the Search Engines have had this capability since 2004. Disclaimer: I do not work for any Search Engine. These ideas are speculative based on patent research done by Joshua Giardino, myself, some direction from Bill Slawski and what can be observed on Search Engine Results Pages. A headless browser is simply a full-featured web browser with no visual interface. Similar to the TSR (Terminate Stay Resident) programs that live on your system tray in Windows they run without you seeing anything on your screen but other programs may interact with them. With a headless browser you can interface with it via a command-line or scripting language and therefore load a webpage and programmatically examine the same output a user would see in Firefox, Chrome or (gasp) Internet Explorer. Vanessa Fox alluded that Google may be using these to crawl AJAX in January of 2010. However Search Engines would have us believe that their crawlers are still similar to Unix’s Lynx browser and can only see and understand text and its associated markup. Basically they have trained us to believe that Googlebot, Slurp and Bingbot are a lot like Pacman in that you point it in a direction and it gobbles up everything it can without being able to see where it’s going or what it’s looking at. Think of the dashes that Pacman eats as webpages. Every once in a while it hits a wall and is forced in another direction. Think of SEOs as the power pills. Think of ghosts as technical SEO issues that might trip up Pacman and cause him to not complete the level that is your page. When an SEO gets involved with a site it helps a search engine spider eat the ghost; when they don’t Pacman dies and starts another life on another site.  That’s what they have been selling us for years the only problem is it’s simply not true anymore and hasn’t been for some time. To be fair though Google normally only lies by omission so it’s our fault for taking so long to figure it out. I encourage you to read Josh’s paper in full but some highlights that indicate this are: A patent filed in 2004 entitled “Document Segmentation Based on Visual Gaps” discusses methods Google uses to render pages visually and traversing the Document Object Model (DOM) to better understand the content and structure of a page. A key excerpt from that patent says “Other techniques for generating appropriate weights may also be used, such as based on examination of the behavior or source code of Web browser software or using a labeled corpus of hand-segmented web pages to automatically set weights through a machine learning process.”   The wily Mr. Cutts suggested at Pubcon that GoogleBot will soon be taking into account what is happening above the fold as an indication user experience quality as though it were a new feature. That’s curious because according to the “Ranking Documents Based on User Behavior and/or Feature Data” patent from June 17, 2004 they have been able to do this for the past seven years. A key excerpt from that patent describes “Examples of features associated with a link might include the font size of the anchor text associated with the link; the position of the link (measured, for example, in a HTML list, in running text, above or below the first screenful viewed on an 800.times.600 browser display, side (top, bottom, left, right) of document, in a footer, in a sidebar, etc.); if the link is in a list, the position of the link in the list; font color and/or attributes of the link (e.g., italics, gray, same color as background, etc.);” This is evidence that Google has visually considered the fold for some time. I also would say that this is live right now as there are instant previews that show a cut-off at the point which Google is considering the fold.   It is no secret that Google has been executing JavaScript to a degree for some time now but “Searching Through Content Which is Accessible Through Web-based Forms” shows an indication that Google is using a headless browser to perform the transformations necessary to dynamically input forms. “Many web sites often use JavaScript to modify the method invocation string before form submission. This is done to prevent each crawling of their web forms. These web forms cannot be automatically invoked easily. In various embodiments, to get around this impediment, a JavaScript emulation engine is used. In one implementation, a simple browser client is invoked, which in turn invokes a JavaScript engine.” Hmmm…interesting. Google also owns a considerable amount of IBM patents as of June and August of 2011 and with that comes a lot of their awesome research into remote systems, parallel computing and headless machines for example the “Simultaneous network configuration of multiple headless machines” patent. Though Google has clearly done extensive research of their own in these areas. Not to be left out there’s a Microsoft patent entitled “High Performance Script Behavior Detection Through Browser Shimming” where there is not much room for interpretation; in so many words it says Bingbot is a browser. "A method for analyzing one or more scripts contained within a document to determine if the scripts perform one or more predefined functions, the method comprising the steps of: identifying, from the one or more scripts, one or more scripts relevant to the one or more predefined functions; interpreting the one or more relevant scripts; intercepting an external function call from the one or more relevant scripts while the one or more relevant scripts are being interpreted, the external function call directed to a document object model of the document; providing a generic response, independent of the document object model, to the external function call; requesting a browser to construct the document object model if the generic response did not enable further operation of the relevant scripts; and providing a specific response, obtained with reference to the constructed document object model, to the external function call if the browser was requested to construct the document object model."(emphasis mine) Curious, indeed.  Furthermore, Yahoo filed a patent on Feb 22, 2005 entitled  "Techniques for crawling dynamic web content" which says "The software system architecture in which embodiments of the invention are implemented may vary. FIG 1 is one example of an architecture in which plug-in modules are integrated with a conventional web crawler and a browser engine which, in one implementation, functions like a conventional web browser without a user interface (also referred to as a "headless browser")." Ladies and gentlemen I believe they call that a "smoking gun." The patent then goes on to discuss automatic and custom form filling and methods for handling JavaScript. Search Engine crawlers are indeed like Pacman but not the floating mouth without a face that my parents jerked across the screen of arcades and bars in the mid-80’s. Googlebot and Bingbot are actually more like the ray-traced Pacman with eyes, nose and appendages that we’ve continued to ignore on console systems since the 90’s. This Pacman can punch, kick, jump and navigate the web with lightning speed in 4 dimensions (the 4th is time – see the freshness update). That is to say Search Engine crawlers can render the page as we see them in our own web browsers and have achieved such a high level of programmatic understanding that allows them to emulate a user. Have you ever read the EULA for Chrome? Yeah me neither, but as with most Google products they ask you to opt-in to a program in which your usage data is sent back to Google. I would surmise that this usage data is not just used to inform the ranking algorithm (slightly) but that it is also used as a means to train Googlebot’s machine learning algorithms in order to teach it to input certain fields in forms. For example Google can use user form inputs to figure out what type of data goes into which field and then programmatically fill forms with generic data of that type. If 500 users put in an age in a form field named “age” it has a valid data set that tells it to input an age. Therefore Pacman no longer runs into doors and walls, he has keys and can scale the face of buildings.  Instant Previews - This is why you’re seeing annotated screenshots in Instant Previews of the SERPs. The instant previews are in fact an impressive feat in that they not only take a screenshot of a page but they also visually highlight and extract text pertinent to your search query. This simply cannot be accomplished with a text-based crawler.    Flash Screenshots - You may have also noticed in Google Webmaster Tools screenshots of Flash sites. Wait I thought Google couldn’t see Flash?   AJAX POST Requests Confirmed - Matt Cutts also confirmed that GoogleBot can in fact handle AJAX POST requests coincidentally a matter of hours after the “Googlebot Is Chrome” article was tweeted by Rand, it made its way to the front of HackerNews and brought my site down. By definition AJAX is content loaded by JavaScript when an action takes place after a page is loaded. Therefore it cannot be crawled with a text-based crawler because a text-based crawler does not execute JavaScript it only pulls down existing code as it is rendered at the initial load.   Google Crawling Flash - Mat Clayton also showed me some server logs where GoogleBot has been accessing URLs that are only accessible via embedded in Flash modules on Mixcloud.com:  66.249.71.130 "13/Nov/2011:11:55:41 +0000" "GET /config/?w=300&h=300&js=1&embed_type=widget_standard&feed=http%3A//www.mixcloud.com/chrisreadsubstance/bbe-mixtape-competition-2010.json&tk=TlVMTA HTTP/1.1" 200 695 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.71.116 "13/Nov/2011:11:51:14 +0000" "GET /config/?w=300&h=300&js=1&feed=http%3A//www.mixcloud.com/ZiMoN/electro-house-mix-16.json&embed_type=widget_standard&tk=TlVMTA HTTP/1.1" 200 694 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  Granted this is not new, but another post from 2008 explains that Google "explores Flash files in the same way that a person would, by clicking buttons, entering input, and so on." Oh, you mean like a person would with a browser?   Site Speed – Although Google could potentially get site load times from toolbars and usage data from Chrome it’s far more dependable for them to get it by crawling the web themselves. Without actually executing all the code on a page it’s not realistic that the calculation of page load time would be accurate.   So far this might sound like Googlebot is only a few steps from SkyNet and due to years of SEOs and Google telling us their search crawler is text-based it might sound like science fiction to you. I assure you that it’s not and that a lot of the things I’m talking about can be easily accomplished by programmers far short of the elite engineering team at Google. PhantomJS is a headless Webkit browser that can be controlled via a JavaScript API. With a little bit of script automation a browser can easily be turned into a web crawler. Ironically the logo is a ghost similar to the ones in Pacman and the concept is quite simple really; PhantomJS is used to load a webpage as a user sees it in Firefox, Chrome or Safari, extract features and follow the links. PhantomJS has infinite applications for scraping and otherwise analyzing sites and I encourage the SEO community to embrace it as we move forward. Josh has used PhantomJS to prepare some proof of concepts that I shared at SearchLove. I had mentioned before when I released GoFish that I’d had trouble scraping the breakout terms from Google Insights using a text-based crawler due to the fact that it’s rendered using AJAX. Richard Baxter suggested that it was easily scrapable using an XPath string which leads me to believe that the ImportXML crawling architecture in Google Docs is based on a headless browser as well. In any event here Josh pulls the breakout terms from the page using PhantomJS: Creating screenshots with a text-based crawler is impossible but with a headless webkit browser it’s a piece of cake. Here’s an example that Josh has prepared to show screenshots being created programmatically using PhantomJS. Chromium is Google’s open source fork of the Webkit browser and I seriously doubt that Google’s motives for building a browser were altruistic. The aforementioned research would suggest that GoogleBot is a multi-threaded headless browser based on that same code. Well actually they do but they say the "instant preview crawler" is a completely separate entity. Think of the Instant Crawler as Ms. Pacman. A poster on Webmaster Central complained that they were seeing "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.597 Safari/534.14" rather than "Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13" as the Google Web Preview user agent in their logs. John Mu reveals "We use the Chrome-type user-agent for the Instant Previews testing tool, so that we're able to compare what a browser (using that user-agent) would see with what we see through Googlebot accesses for the cached preview image." While the headless browser and Googlebot as we know it may be separate in semantic explanation I believe that they always crawl in parallel and inform indexation and ultimately rankings. In other words it's like a 2-player simultaneous version of Pacman with a 3D Ms. Pacman and a regular Pacman playing the same levels at the same time. After all it wouldn't make sense for the crawlers to crawl the whole web twice independently. So why aren't they more transparent about these capabilities as they pertain to rankings? Two Words: Search Quality. As long as Search Engines can hide behind the deficiencies of a text-based crawler they can continue to use it as a scapegoat for their inability to serve up the best results. They can continue to move towards things such as the speculated AuthorRank and lean on SEOs to literally optimize their Search Engines. They can continue to say vague things like “don’t chase the algorithm”, “improve your user experience” and “we’re weighing things above the fold” that force SEOs to scramble and make Google’s job easier. Google’s primary product (and only product if you’re talking to Eric Schmidt in court) is Search and if it is publicly revealed that their capabilities are far beyond what they advertise they would then be responsible for a higher level of search quality if not indexation of impossible rich media like Flash. In short they don’t tell us because with great power comes great responsibility. A lot of people have asked me as Josh and I've led up to unveiling this research “what is the actionable insight?” and “how does it change what I do as far as SEO?” There are really three things as far as I’m concerned: You're not Hiding Anything with Javascript - Any content you thought you were hiding with post-load JavaScript -- stop it. Bait and switching is now 100% ineffective. Pacman sees all.   User Experience is Incredibly Important - Google can literally see your site now! As Matt Cutts said they are looking at what's above the fold and therefore they can consider how many ads are rendered on the page in determining rankings. Google can leverage usage data in concert with the design of the site as a proxy to determine out how useful a site is to people. That's both exciting and terrifying but it also means every SEO needs to pick up a copy of "Don't Make Me Think" if they haven't already.   SEO Tools Must Get Smarter - Most SEO tools are built on text-based scrapers and while many are quite sophisticated (SEOmoz clearly is leading the pack right now) they are still very much the 80’s Pacman. If we are to understand what Google is truly considering when ranking pages we must include more aspects in our own analyses. When discussing things such as Page Authority and the likelihood of spam we should be visually examining pages programmatically rather than just limiting ourselves to the metrics like keyword density and the link graph. In other words we need a UX Quality Score that is influenced by visual analysis and potential spammy transformations.   We should be comparing how much the rendered page differs from what would otherwise be expected of the code. We could call this a Delta Score.   When measuring the distribution of link equity from a page the dynamic transformations must also be taken into account as Search Engines are able to understand how many links are truly on a page. This could also be included within the Delta Score.   On another note Natural Language Processing should also be included in our analyses as it is presumably a large part of what makes Google’s algo tick. This is not so much for scoring but for identifying the key concepts that a machine will associate with a given piece of content and truly understanding what a link is worth in context of what you are trying to rank for. In other words we need contextual analysis of the link graph. There are two things that I will agree with Matt Cutts on. The only constant is change and we must stop chasing the algorithm. However we must also realize that Google will continue to feed us misinformation about their capabilities or dangle enough to make us jump to conclusions and hold on to them. Therefore we must also hold them accountable for their technology. Simply put if they can definitively prove they are not doing any of this stuff – then at this point they should be; after all these are some of the most talented engineers in the universe. Google continues to make Search Marketing more challenging and revoke the data that allows us to build better user experiences but the simple fact is that our relationship is symbiotic. Search Engines need SEOs and Webmasters to make the web faster, easier for them to understand and we need Search Engines to react to and reward quality content by making it visible. The issue is that Google holds all the cards and I’m happy to have done my part to pull one. Your move Matt. Do you like this post? Yes No

2nd November Index Update: Our Broadest Index Yet, and New PA/DA Scores are Live Posted: 29 Nov 2011 12:36 AM PST Posted by randfish Hey gang - it's that magical time again when Linkscape's web index has updated with brand new data (for the second time this month). Open Site Explorer, the Mozbar and the PRO Web App all have new links and scores to check out. This index also features the updated Page Authority and Domain Authority models covered by Matt last week on the blog. Here's the current index's metrics: 38,295,116,929 (38 billion) URLs 466,742,600 (466 million) Subdomains 125,007,049 (125 million) Root Domains 387,379,700,299 (387 billion) Links Followed vs. Nofollowed 2.03% of all links found were nofollowed 55.57% of nofollowed links are internal, 44.43% are external Rel Canonical - 10.34% of all pages now employ a rel=canonical tag The average page has 70.61 links on it (down 6.67 from last index; we're likely biasing to a different set of webpages with the broader vs. deeper focus of this release) 59.02 internal links on average 11.59 external links on average As you can see, we're crawling a LOT more root domains - we expect to have data for an extremely high percentage of all the domains that you might find active on the web. However, because of this broader crawl, we're not reaching as deeply into some large domains (some of that is us weeding out crap, including many more millions of binary files, error-producing webpages and other web "junk"). You can see below a chart of the root domains we've crawled in the last 6 months vs. the total URLs in each index. We work toward a few key metrics to judge our progress on the index: Correlations with Google rankings (not only of PA/DA, but of link counts, linking root domains, mozRank, etc) Percent of successful API requests (meaning a request for link data on a URL from any source that we had link data for) Raw size and freshness (total # of root domains and URLs in the index, though, as Danny Sullivan has pointed out, this may not be a great metric on which to judge a web corpus) We've gotten better with most of these recently - PA/DA have better correlations, more of your requests (via Open Site Explorer, the Mozbar or any third-party application) now have link data, and we're slowly improving freshness (this index was actually completed last week, but didn't launch due to the Thanksgiving holiday). However, we are not improving as much on raw index size (root domains, yes, which we've seen correlate with other metrics, but raw URL count, no). This will continue to be a focus for us in the months to come, and we're still targeting 100 billion+ URLs as a goal (though we're not willing to sacrifice quality, accuracy or freshness to get there). As always, if you've got feedback on the new scores, on the link data or anything related to the index, please do let us know. We love to hear from you! Do you like this post? Yes No

You are subscribed to email updates from SEOmoz Daily SEO Blog To stop receiving these emails, you may unsubscribe now.	Email delivery powered by Google
Google Inc., 20 West Kinzie, Chicago IL USA 60610